BUG: workqueue lockup in update_super_work

Status: moderation: reported on 2026/11/21 17:40
Subsystems: ext4
Labels: prio:low
[Documentation on labels]
Reported-by: syzbot+9827e26568b1b4955681@syzkaller.appspotmail.com
First crash: 1h33m, last: 1h33m
✨ AI-generated root cause summary:

The workqueue watchdog detected a 35-second lockup on CPU 0's per-CPU worker pool (pool 2), where all four busy workers (update_super_work, smc_ib_port_event_work, srcu_invoke_callbacks, and wg_packet_tx_worker) are in TASK_RUNNING (state:R) and switched out at __switch_to, unable to be scheduled onto CPU 0 while CPU 1 (pool 6) continues running normally. Because srcu_invoke_callbacks and smc_ib_port_event_work (which blocked on rtnl_lock) cannot resume on CPU 0, the stall cascades to fsnotify_mark_destroy_workfn (synchronize_srcu), cfg80211_wiphy_work, reg_todo, and batadv_tt_purge. The exact code currently monopolizing CPU 0 cannot be conclusively determined from this report because show_cpu_pool_busy_workers() checks task_is_running() rather than whether a worker is actively executing on the CPU, and therefore skipped trigger_single_cpu_backtrace(0).

▶ ▼ ✨ Full AI root cause analysis
Based on kernel commit a041bdbe1f3a:

Verified Crash Mechanism

The workqueue watchdog (wq_watchdog_timer_fn() in kernel/workqueue.c) reported a 35-second stall on pool 2 (cpus=0 node=0 flags=0x0 nice=0), indicating that pool 2 had pending items on pool->worklist while neither pool->last_progress_ts nor per_cpu(wq_watchdog_touched_cpu, 0) advanced for 35 seconds. Meanwhile, CPU 1's per-CPU pool (pool 6, cpus=1) remained healthy (hung=0s).

Inspecting the four busy workers in pool 2 shows that all four are pinned to CPU 0 (cpus=0), are in TASK_RUNNING (state:R) state, and are switched out at __switch_to():

  1. kworker/0:3 (pid:1166, update_super_work -> ext4_commit_super() -> wait_on_buffer() -> io_schedule()): slept in TASK_UNINTERRUPTIBLE (calling wq_worker_sleeping()), was subsequently woken up into TASK_RUNNING, and has been waiting on CPU 0's runqueue without resuming past schedule()/wq_worker_running().
  2. kworker/0:2 (pid:1102, smc_ib_port_event_work -> ib_query_port() -> rxe_query_port() -> ib_get_eth_speed() -> rtnl_lock()): similarly slept in __mutex_lock_common(), was woken into TASK_RUNNING (potentially receiving rtnl_lock ownership via __mutex_handoff() in __mutex_unlock_slowpath()), and is waiting on CPU 0's runqueue.
  3. kworker/0:6 (pid:4854, wg_packet_tx_worker -> wg_packet_create_data_done() -> wg_timers_any_authenticated_packet_traversal() -> mod_peer_timer() -> rcu_read_unlock_bh() -> local_bh_enable()): was preempted via __local_bh_enable_ip() -> preempt_check_resched() -> preempt_schedule() -> __schedule().
  4. kworker/0:1 (pid:10, srcu_invoke_callbacks() -> local_bh_enable()): was likewise preempted at __local_bh_enable_ip() -> preempt_check_resched() -> preempt_schedule() -> __schedule().

Because pid:4854 and pid:10 were preempted while task_is_running(tsk) was true, schedule() (kernel/sched/core.c:7472) did not call sched_submit_work() / wq_worker_sleeping(). Consequently, pool 2->nr_running remained non-zero, so need_more_worker(pool) (kernel/workqueue.c:989) returned false and pool 2's three idle workers (9, 4823, 4816) were not wokenβ€”though even if woken, they are also SCHED_NORMAL (nice=0) threads bound to CPU 0.

Cascading Stalls

Because CPU 0 has not scheduled any of these TASK_RUNNING workers since t - 35s:

  • pid:10 (srcu_invoke_callbacks, in-flight 35s) cannot finish on CPU 0, stalling fsnotify_mark_destroy_workfn (fs/notify/mark.c:1187, synchronize_srcu(&fsnotify_mark_srcu), in-flight 35s on pool 8) and its flush_work waiter BAR(5280).
  • pid:1102 (smc_ib_port_event_work in rtnl_lock()) cannot resume on CPU 0, stalling rtnl_lock-dependent work in pool 8 and pool 6 (cfg80211_wiphy_work, reg_todo, and batadv_tt_purge, which triggered MAYDAY on bat_events).

What Remains Inconclusive

The four backtraced workers are victims sitting on CPU 0's runqueue rather than the code currently executing on CPU 0. Under CONFIG_PREEMPT=y, four runnable SCHED_NORMAL kworkers cannot be starved for 35+ seconds unless CPU 0 is stuck in non-preemptible kernel context (such as a softirq/hardirq loop, spinlock deadlock, or local_bh_disable()/preempt_disable() loop), monopolized by a higher scheduling class (SCHED_FIFO/SCHED_RR/SCHED_DEADLINE), or stalled externally.

The exact task or interrupt/softirq context executing on CPU 0 could not be determined from this report because show_cpu_pool_busy_workers() (kernel/workqueue.c:8054) checks task_is_running(worker->task) (true for any TASK_RUNNING task on the runqueue, not only the task actively running on the CPU) to set found_running = true, and therefore skipped calling trigger_single_cpu_backtrace(0) at kernel/workqueue.c:8101. Notice also that CONFIG_CMDLINE configures watchdog_thresh=55 (110s softlockup threshold) and CONFIG_RCU_CPU_STALL_TIMEOUT=100 (100s RCU stall threshold), so the 30s default wq_watchdog_thresh fired at 35s before a softlockup or RCU CPU stall dump could capture CPU 0's active stack.

▶ ▼ ✨ AI Jobs (2)
ID Workflow Result Correct Bug Created Started Finished Revision
a95daf5d bisect-cause πŸƒ BUG: workqueue lockup in update_super_work 2026/11/21 16:48 2026/11/21 16:48 b2c69459
d2abe826 assessment-security DenialOfService: βœ… Exploitable: ❌ FilesystemTrigger: ❌ FuzzingArtifact: ❌ MemoryCorruption: ❌ NetworkNeedsSetup: ❌ NetworkTrigger: ❌ PeripheralTrigger: ❌ RemoteTrigger: ❌ Unprivileged: ❌ UserNamespace: ❌ VMGuestTrigger: ❌ VMHostTrigger: ❌ ❓ BUG: workqueue lockup in update_super_work 2026/11/21 16:44 2026/11/21 16:44 2026/11/21 17:40 b2c69459

Sample crash report:
BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 35s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=8 refcnt=9
    in-flight: 1166:update_super_work for 44s ,1102:smc_ib_port_event_work for 41s
    pending: 2*nsim_dev_hwstats_traffic_work, free_obj_work, vmstat_shepherd, 2*psi_avgs_work
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=4 refcnt=5
    in-flight: 4836:reg_todo for 10s
    pending: 3*nsim_dev_hwstats_traffic_work
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=6 refcnt=8
    in-flight: 42:cfg80211_wiphy_work for 35s cfg80211_wiphy_work ,40:fsnotify_mark_destroy_workfn for 35s BAR(5280)
    pending: 2*macvlan_process_broadcast, cfg80211_wiphy_work
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=24 refcnt=25
    in-flight: 13:macvlan_process_broadcast for 4s ,12:cfg80211_wiphy_work for 46s cfg80211_wiphy_work ,1437:cfg80211_wiphy_work for 29s ,937:cfg80211_wiphy_work for 42s cfg80211_wiphy_work ,1396:cfg80211_wiphy_work for 44s cfg80211_wiphy_work
    pending: 5*macvlan_process_broadcast, 2*cfg80211_wiphy_work, crng_reseed, nsim_dev_trap_report_work, cfg80211_wiphy_work, nsim_dev_trap_report_work, toggle_allocation_gate, nsim_dev_trap_report_work, flush_memcg_stats_dwork, 2*nsim_dev_trap_report_work
workqueue events_power_efficient: flags=0x82
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5
    pending: do_cache_clean, 3*neigh_periodic_work
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=9 refcnt=10
    pending: 3*neigh_periodic_work, gc_worker, 2*neigh_periodic_work, fb_flashcursor, wg_ratelimiter_gc_entries, crda_timeout_work
workqueue events_dfl_long: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=6 refcnt=7
    pending: 6*defense_work_handler
workqueue rcu_gp: flags=0x108
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    in-flight: 10:srcu_invoke_callbacks for 35s
    pending: srcu_invoke_callbacks
workqueue mm_percpu_wq: flags=0x108
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
workqueue writeback: flags=0x4a
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 96:wb_workfn for 43s
workqueue bat_events: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=45 MAYDAY
    in-flight: 2871(RESCUER):batadv_tt_purge for 18s
    pending: mayday_cursor_func
    inactive: 3*batadv_tt_purge, 4*batadv_mcast_mla_update, 5*batadv_iv_send_outstanding_bat_ogm_packet, batadv_mcast_mla_update, 3*batadv_purge_orig, 10*batadv_iv_send_outstanding_bat_ogm_packet, 2*batadv_purge_orig, batadv_bla_periodic_work, batadv_dat_purge, 2*batadv_bla_periodic_work, 2*batadv_dat_purge, 2*batadv_bla_periodic_work, 2*batadv_dat_purge, batadv_tt_purge
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    in-flight: 26:wg_packet_tx_worker for 3s
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5
    in-flight: 4854:wg_packet_tx_worker for 37s
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_decrypt_worker, wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue loop1: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: loop_rootcg_workfn
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=35s workers=7 idle: 9 4823 4816
pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=0s workers=8 idle: 4808 24 4908 4310 4809 55
pool 8: cpus=0-1 flags=0x6 nice=0 hung=4s workers=9 manager: 5306
Showing backtraces of busy workers in stalled worker pools:
pool 2:
task:kworker/0:3     state:R  running task     stack:0     pid:1166  tgid:1166  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: events update_super_work
Call trace:
 __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
 context_switch kernel/sched/core.c:5583 [inline]
 __schedule+0x1368/0x2b50 kernel/sched/core.c:7382
 __schedule_loop kernel/sched/core.c:7459 [inline]
 schedule+0x13c/0x20c kernel/sched/core.c:7474
 io_schedule+0x1c/0xe0 kernel/sched/core.c:8146
 bit_wait_io+0x1c/0xe8 kernel/sched/wait_bit.c:258
 __wait_on_bit+0xb8/0x178 kernel/sched/wait_bit.c:52
 out_of_line_wait_on_bit+0x130/0x184 kernel/sched/wait_bit.c:67
 wait_on_bit_io include/linux/wait_bit.h:106 [inline]
 __wait_on_buffer+0x6c/0x7c fs/buffer.c:118
 wait_on_buffer include/linux/buffer_head.h:476 [inline]
 ext4_commit_super+0x2b4/0x458 fs/ext4/super.c:6417
 update_super_work+0x2dc/0x384 fs/ext4/super.c:817
 process_one_work kernel/workqueue.c:3495 [inline]
 process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
 worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
 kthread+0x304/0x3d4 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
pool 2:
task:kworker/0:2     state:R  running task     stack:0     pid:1102  tgid:1102  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: events smc_ib_port_event_work
Call trace:
 __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
 context_switch kernel/sched/core.c:5583 [inline]
 __schedule+0x1368/0x2b50 kernel/sched/core.c:7382
 __schedule_loop kernel/sched/core.c:7459 [inline]
 schedule+0x13c/0x20c kernel/sched/core.c:7474
 schedule_preempt_disabled+0x18/0x2c kernel/sched/core.c:7531
 __mutex_lock_common kernel/locking/mutex.c:726 [inline]
 __mutex_lock+0x5d8/0x10b8 kernel/locking/mutex.c:821
 mutex_lock_nested+0x24/0x30 kernel/locking/mutex.c:874
 rtnl_lock+0x20/0x2c net/core/rtnetlink.c:80
 ib_get_eth_speed+0x148/0x770 drivers/infiniband/core/verbs.c:2060
 rxe_query_port+0x98/0x37c drivers/infiniband/sw/rxe/rxe_verbs.c:56
 __ib_query_port drivers/infiniband/core/device.c:2304 [inline]
 ib_query_port+0x16c/0x76c drivers/infiniband/core/device.c:2336
 smc_ib_remember_port_attr net/smc/smc_ib.c:370 [inline]
 smc_ib_port_event_work+0xfc/0x894 net/smc/smc_ib.c:394
 process_one_work kernel/workqueue.c:3495 [inline]
 process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
 worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
 kthread+0x304/0x3d4 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
pool 2:
task:kworker/0:1     state:R  running task     stack:0     pid:10    tgid:10    ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: rcu_gp srcu_invoke_callbacks
Call trace:
 __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
 context_switch kernel/sched/core.c:5583 [inline]
 __schedule+0x1368/0x2b50 kernel/sched/core.c:7382
 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7561
 preempt_schedule+0x60/0x78 kernel/sched/core.c:7585
 __local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480
 local_bh_enable include/linux/bottom_half.h:33 [inline]
 srcu_invoke_callbacks+0x290/0x568 kernel/rcu/srcutree.c:2274
 process_one_work kernel/workqueue.c:3495 [inline]
 process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
 worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
 kthread+0x304/0x3d4 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
pool 2:
task:kworker/0:6     state:R  running task     stack:0     pid:4854  tgid:4854  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: wg-crypt-wg0 wg_packet_tx_worker
Call trace:
 __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
 context_switch kernel/sched/core.c:5583 [inline]
 __schedule+0x1368/0x2b50 kernel/sched/core.c:7382
 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7561
 preempt_schedule+0x60/0x78 kernel/sched/core.c:7585
 __local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480
 local_bh_enable include/linux/bottom_half.h:33 [inline]
 rcu_read_unlock_bh include/linux/rcupdate.h:923 [inline]
 mod_peer_timer+0x21c/0x25c drivers/net/wireguard/timers.c:38
 wg_timers_any_authenticated_packet_traversal+0x98/0xc0 drivers/net/wireguard/timers.c:218
 wg_packet_create_data_done drivers/net/wireguard/send.c:247 [inline]
 wg_packet_tx_worker+0x130/0x75c drivers/net/wireguard/send.c:276
 process_one_work kernel/workqueue.c:3495 [inline]
 process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
 worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
 kthread+0x304/0x3d4 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2026/11/21 16:22 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a041bdbe1f3a b2c69459 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 BUG: workqueue lockup in update_super_work
* Struck through repros no longer work on HEAD.