syzbot
BUG: workqueue lockup in update_super_work
Status:
moderation: reported on 2026/11/21 17:40
Subsystems:
ext4
Labels:
prio:low
[Documentation on labels]
Reported-by: syzbot+9827e26568b1b4955681@syzkaller.appspotmail.com
First crash: 1h33m, last: 1h33m
β¨ AI-generated root cause summary:
The workqueue watchdog detected a 35-second lockup on CPU 0's per-CPU worker pool (pool 2), where all four busy workers (update_super_work, smc_ib_port_event_work, srcu_invoke_callbacks, and wg_packet_tx_worker) are in TASK_RUNNING (state:R) and switched out at __switch_to, unable to be scheduled onto CPU 0 while CPU 1 (pool 6) continues running normally. Because srcu_invoke_callbacks and smc_ib_port_event_work (which blocked on rtnl_lock) cannot resume on CPU 0, the stall cascades to fsnotify_mark_destroy_workfn (synchronize_srcu), cfg80211_wiphy_work, reg_todo, and batadv_tt_purge. The exact code currently monopolizing CPU 0 cannot be conclusively determined from this report because show_cpu_pool_busy_workers() checks task_is_running() rather than whether a worker is actively executing on the CPU, and therefore skipped trigger_single_cpu_backtrace(0).
▶
▼
β¨ Full AI root cause analysis
Verified Crash Mechanism
The workqueue watchdog (wq_watchdog_timer_fn() in kernel/workqueue.c) reported a 35-second stall on pool 2 (cpus=0 node=0 flags=0x0 nice=0), indicating that pool 2 had pending items on pool->worklist while neither pool->last_progress_ts nor per_cpu(wq_watchdog_touched_cpu, 0) advanced for 35 seconds. Meanwhile, CPU 1's per-CPU pool (pool 6, cpus=1) remained healthy (hung=0s).
Inspecting the four busy workers in pool 2 shows that all four are pinned to CPU 0 (cpus=0), are in TASK_RUNNING (state:R) state, and are switched out at __switch_to():
kworker/0:3 (pid:1166, update_super_work -> ext4_commit_super() -> wait_on_buffer() -> io_schedule()): slept in TASK_UNINTERRUPTIBLE (calling wq_worker_sleeping()), was subsequently woken up into TASK_RUNNING, and has been waiting on CPU 0's runqueue without resuming past schedule()/wq_worker_running().
kworker/0:2 (pid:1102, smc_ib_port_event_work -> ib_query_port() -> rxe_query_port() -> ib_get_eth_speed() -> rtnl_lock()): similarly slept in __mutex_lock_common(), was woken into TASK_RUNNING (potentially receiving rtnl_lock ownership via __mutex_handoff() in __mutex_unlock_slowpath()), and is waiting on CPU 0's runqueue.
kworker/0:6 (pid:4854, wg_packet_tx_worker -> wg_packet_create_data_done() -> wg_timers_any_authenticated_packet_traversal() -> mod_peer_timer() -> rcu_read_unlock_bh() -> local_bh_enable()): was preempted via __local_bh_enable_ip() -> preempt_check_resched() -> preempt_schedule() -> __schedule().
kworker/0:1 (pid:10, srcu_invoke_callbacks() -> local_bh_enable()): was likewise preempted at __local_bh_enable_ip() -> preempt_check_resched() -> preempt_schedule() -> __schedule().
Because pid:4854 and pid:10 were preempted while task_is_running(tsk) was true, schedule() (kernel/sched/core.c:7472) did not call sched_submit_work() / wq_worker_sleeping(). Consequently, pool 2->nr_running remained non-zero, so need_more_worker(pool) (kernel/workqueue.c:989) returned false and pool 2's three idle workers (9, 4823, 4816) were not wokenβthough even if woken, they are also SCHED_NORMAL (nice=0) threads bound to CPU 0.
Cascading Stalls
Because CPU 0 has not scheduled any of these TASK_RUNNING workers since t - 35s:
pid:10 (srcu_invoke_callbacks, in-flight 35s) cannot finish on CPU 0, stalling fsnotify_mark_destroy_workfn (fs/notify/mark.c:1187, synchronize_srcu(&fsnotify_mark_srcu), in-flight 35s on pool 8) and its flush_work waiter BAR(5280).
pid:1102 (smc_ib_port_event_work in rtnl_lock()) cannot resume on CPU 0, stalling rtnl_lock-dependent work in pool 8 and pool 6 (cfg80211_wiphy_work, reg_todo, and batadv_tt_purge, which triggered MAYDAY on bat_events).
What Remains Inconclusive
The four backtraced workers are victims sitting on CPU 0's runqueue rather than the code currently executing on CPU 0. Under CONFIG_PREEMPT=y, four runnable SCHED_NORMAL kworkers cannot be starved for 35+ seconds unless CPU 0 is stuck in non-preemptible kernel context (such as a softirq/hardirq loop, spinlock deadlock, or local_bh_disable()/preempt_disable() loop), monopolized by a higher scheduling class (SCHED_FIFO/SCHED_RR/SCHED_DEADLINE), or stalled externally.
The exact task or interrupt/softirq context executing on CPU 0 could not be determined from this report because show_cpu_pool_busy_workers() (kernel/workqueue.c:8054) checks task_is_running(worker->task) (true for any TASK_RUNNING task on the runqueue, not only the task actively running on the CPU) to set found_running = true, and therefore skipped calling trigger_single_cpu_backtrace(0) at kernel/workqueue.c:8101. Notice also that CONFIG_CMDLINE configures watchdog_thresh=55 (110s softlockup threshold) and CONFIG_RCU_CPU_STALL_TIMEOUT=100 (100s RCU stall threshold), so the 30s default wq_watchdog_thresh fired at 35s before a softlockup or RCU CPU stall dump could capture CPU 0's active stack.
Sample crash report:
BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 35s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=8 refcnt=9
in-flight: 1166:update_super_work for 44s ,1102:smc_ib_port_event_work for 41s
pending: 2*nsim_dev_hwstats_traffic_work, free_obj_work, vmstat_shepherd, 2*psi_avgs_work
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=4 refcnt=5
in-flight: 4836:reg_todo for 10s
pending: 3*nsim_dev_hwstats_traffic_work
workqueue events_unbound: flags=0x2
pwq 8: cpus=0-1 flags=0x6 nice=0 active=6 refcnt=8
in-flight: 42:cfg80211_wiphy_work for 35s cfg80211_wiphy_work ,40:fsnotify_mark_destroy_workfn for 35s BAR(5280)
pending: 2*macvlan_process_broadcast, cfg80211_wiphy_work
pwq 8: cpus=0-1 flags=0x6 nice=0 active=24 refcnt=25
in-flight: 13:macvlan_process_broadcast for 4s ,12:cfg80211_wiphy_work for 46s cfg80211_wiphy_work ,1437:cfg80211_wiphy_work for 29s ,937:cfg80211_wiphy_work for 42s cfg80211_wiphy_work ,1396:cfg80211_wiphy_work for 44s cfg80211_wiphy_work
pending: 5*macvlan_process_broadcast, 2*cfg80211_wiphy_work, crng_reseed, nsim_dev_trap_report_work, cfg80211_wiphy_work, nsim_dev_trap_report_work, toggle_allocation_gate, nsim_dev_trap_report_work, flush_memcg_stats_dwork, 2*nsim_dev_trap_report_work
workqueue events_power_efficient: flags=0x82
pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5
pending: do_cache_clean, 3*neigh_periodic_work
pwq 8: cpus=0-1 flags=0x6 nice=0 active=9 refcnt=10
pending: 3*neigh_periodic_work, gc_worker, 2*neigh_periodic_work, fb_flashcursor, wg_ratelimiter_gc_entries, crda_timeout_work
workqueue events_dfl_long: flags=0x2
pwq 8: cpus=0-1 flags=0x6 nice=0 active=6 refcnt=7
pending: 6*defense_work_handler
workqueue rcu_gp: flags=0x108
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 10:srcu_invoke_callbacks for 35s
pending: srcu_invoke_callbacks
workqueue mm_percpu_wq: flags=0x108
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: vmstat_update
workqueue writeback: flags=0x4a
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
in-flight: 96:wb_workfn for 43s
workqueue bat_events: flags=0x6000a
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=45 MAYDAY
in-flight: 2871(RESCUER):batadv_tt_purge for 18s
pending: mayday_cursor_func
inactive: 3*batadv_tt_purge, 4*batadv_mcast_mla_update, 5*batadv_iv_send_outstanding_bat_ogm_packet, batadv_mcast_mla_update, 3*batadv_purge_orig, 10*batadv_iv_send_outstanding_bat_ogm_packet, 2*batadv_purge_orig, batadv_bla_periodic_work, batadv_dat_purge, 2*batadv_bla_periodic_work, 2*batadv_dat_purge, 2*batadv_bla_periodic_work, 2*batadv_dat_purge, batadv_tt_purge
workqueue wg-kex-wg0: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg1: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
in-flight: 26:wg_packet_tx_worker for 3s
workqueue wg-kex-wg2: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg0: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-crypt-wg1: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg2: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg0: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg1: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg2: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg0: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5
in-flight: 4854:wg_packet_tx_worker for 37s
pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg1: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
pending: wg_packet_decrypt_worker, wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue loop1: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: loop_rootcg_workfn
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=35s workers=7 idle: 9 4823 4816
pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=0s workers=8 idle: 4808 24 4908 4310 4809 55
pool 8: cpus=0-1 flags=0x6 nice=0 hung=4s workers=9 manager: 5306
Showing backtraces of busy workers in stalled worker pools:
pool 2:
task:kworker/0:3 state:R running task stack:0 pid:1166 tgid:1166 ppid:2 task_flags:0x4208060 flags:0x00000010
Workqueue: events update_super_work
Call trace:
__switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
context_switch kernel/sched/core.c:5583 [inline]
__schedule+0x1368/0x2b50 kernel/sched/core.c:7382
__schedule_loop kernel/sched/core.c:7459 [inline]
schedule+0x13c/0x20c kernel/sched/core.c:7474
io_schedule+0x1c/0xe0 kernel/sched/core.c:8146
bit_wait_io+0x1c/0xe8 kernel/sched/wait_bit.c:258
__wait_on_bit+0xb8/0x178 kernel/sched/wait_bit.c:52
out_of_line_wait_on_bit+0x130/0x184 kernel/sched/wait_bit.c:67
wait_on_bit_io include/linux/wait_bit.h:106 [inline]
__wait_on_buffer+0x6c/0x7c fs/buffer.c:118
wait_on_buffer include/linux/buffer_head.h:476 [inline]
ext4_commit_super+0x2b4/0x458 fs/ext4/super.c:6417
update_super_work+0x2dc/0x384 fs/ext4/super.c:817
process_one_work kernel/workqueue.c:3495 [inline]
process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
kthread+0x304/0x3d4 kernel/kthread.c:436
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
pool 2:
task:kworker/0:2 state:R running task stack:0 pid:1102 tgid:1102 ppid:2 task_flags:0x4208060 flags:0x00000010
Workqueue: events smc_ib_port_event_work
Call trace:
__switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
context_switch kernel/sched/core.c:5583 [inline]
__schedule+0x1368/0x2b50 kernel/sched/core.c:7382
__schedule_loop kernel/sched/core.c:7459 [inline]
schedule+0x13c/0x20c kernel/sched/core.c:7474
schedule_preempt_disabled+0x18/0x2c kernel/sched/core.c:7531
__mutex_lock_common kernel/locking/mutex.c:726 [inline]
__mutex_lock+0x5d8/0x10b8 kernel/locking/mutex.c:821
mutex_lock_nested+0x24/0x30 kernel/locking/mutex.c:874
rtnl_lock+0x20/0x2c net/core/rtnetlink.c:80
ib_get_eth_speed+0x148/0x770 drivers/infiniband/core/verbs.c:2060
rxe_query_port+0x98/0x37c drivers/infiniband/sw/rxe/rxe_verbs.c:56
__ib_query_port drivers/infiniband/core/device.c:2304 [inline]
ib_query_port+0x16c/0x76c drivers/infiniband/core/device.c:2336
smc_ib_remember_port_attr net/smc/smc_ib.c:370 [inline]
smc_ib_port_event_work+0xfc/0x894 net/smc/smc_ib.c:394
process_one_work kernel/workqueue.c:3495 [inline]
process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
kthread+0x304/0x3d4 kernel/kthread.c:436
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
pool 2:
task:kworker/0:1 state:R running task stack:0 pid:10 tgid:10 ppid:2 task_flags:0x4208060 flags:0x00000010
Workqueue: rcu_gp srcu_invoke_callbacks
Call trace:
__switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
context_switch kernel/sched/core.c:5583 [inline]
__schedule+0x1368/0x2b50 kernel/sched/core.c:7382
preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7561
preempt_schedule+0x60/0x78 kernel/sched/core.c:7585
__local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480
local_bh_enable include/linux/bottom_half.h:33 [inline]
srcu_invoke_callbacks+0x290/0x568 kernel/rcu/srcutree.c:2274
process_one_work kernel/workqueue.c:3495 [inline]
process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
kthread+0x304/0x3d4 kernel/kthread.c:436
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
pool 2:
task:kworker/0:6 state:R running task stack:0 pid:4854 tgid:4854 ppid:2 task_flags:0x4208060 flags:0x00000010
Workqueue: wg-crypt-wg0 wg_packet_tx_worker
Call trace:
__switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
context_switch kernel/sched/core.c:5583 [inline]
__schedule+0x1368/0x2b50 kernel/sched/core.c:7382
preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7561
preempt_schedule+0x60/0x78 kernel/sched/core.c:7585
__local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480
local_bh_enable include/linux/bottom_half.h:33 [inline]
rcu_read_unlock_bh include/linux/rcupdate.h:923 [inline]
mod_peer_timer+0x21c/0x25c drivers/net/wireguard/timers.c:38
wg_timers_any_authenticated_packet_traversal+0x98/0xc0 drivers/net/wireguard/timers.c:218
wg_packet_create_data_done drivers/net/wireguard/send.c:247 [inline]
wg_packet_tx_worker+0x130/0x75c drivers/net/wireguard/send.c:276
process_one_work kernel/workqueue.c:3495 [inline]
process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
kthread+0x304/0x3d4 kernel/kthread.c:436
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
* Struck through repros no longer work on HEAD.