BUG: soft lockup in wg_ratelimiter_gc_entries

Status: moderation: reported on 2026/11/22 04:33
Subsystems: wireguard
Labels: prio:low
[Documentation on labels]
Reported-by: syzbot+e696759843d287a23861@syzkaller.appspotmail.com
First crash: 3h56m, last: 3h56m
✨ AI-generated root cause summary:

CPU 0 reported a 24s soft lockup inside wg_ratelimiter_gc_entries() at cond_resched() -> __might_resched() -> lock_is_held_type(), while CPU 1 was in arena_map_free() -> remove_vm_area() -> debug_check_no_obj_freed() scanning a 4GB+64KB vm_area. Inspection of drivers/net/wireguard/ratelimiter.c and lockdep irqtrace events shows wg_ratelimiter_gc_entries() has bounded iteration (table_size <= 8192), drops table_lock per bucket, and keeps IRQs and preemption enabled, with zero interrupts occurring between entry (ktime_get_coarse_with_offset, stamp 1142073) and the watchdog IRQ (stamp 1142074). Thus, no kernel infinite loop or IRQ/preemption leak is present in wg_ratelimiter_gc_entries(); the 24s elapsed time without timer interrupts indicates an inconclusive external vCPU/hypervisor stall (arm64 lacks kvm_check_and_clear_guest_paused() support), potentially exacerbated by heavy debug/sanitizer CPU load such as CPU 1's ~1M-chunk debug_check_no_obj_freed() loop under CONFIG_DEBUG_OBJECTS_FREE.

▶ ▼ ✨ Full AI root cause analysis
Based on kernel commit c44104f3642d:

Technical Analysis

1. Crash Site & Execution State on CPU 0

The soft lockup watchdog fired on CPU 0 while kworker/u8:6 (PID 1180) was executing wg_ratelimiter_gc_entries() (drivers/net/wireguard/ratelimiter.c:55-82) on events_power_efficient (system_power_efficient_wq):

2. Invariants Checked in drivers/net/wireguard/ratelimiter.c (Ruled Out as Root Cause)

  • Bounded loop and preemption: wg_ratelimiter_init() (drivers/net/wireguard/ratelimiter.c:173-177) caps table_size at 8192 and max_entries = table_size * 8 (<= 65536). Each iteration of for (i = 0; i < table_size; ++i) acquires table_lock (non-IRQ-disabling spin_lock), scans table_v4[i] and table_v6[i], unlocks table_lock, and invokes cond_resched().
  • Object lifecycle and locking: Both wg_ratelimiter_allow() (hlist_add_head_rcu, line 149) and wg_ratelimiter_gc_entries() (entry_uninit() $\rightarrow$ hlist_del_rcu() + call_rcu(..., entry_free), lines 48-52) serialize hash-bucket modifications under table_lock, while wg_ratelimiter_uninit() calls cancel_delayed_work_sync(&gc_work) before clearing the tables and destroying entry_cache. Even if a bucket list were somehow corrupted, an infinite loop in hlist_for_each_entry_safe() would stall inside lines 64-75 with table_lock held, not at cond_resched() on line 78.
  • Impossibility of a 24s CONFIG_PREEMPT=y CPU-bound lockup here: In kernel/watchdog.c:798-826, watchdog_timer_fn() queues softlockup_fn() via stop_one_cpu_nowait() on migration/0 (SCHED_FIFO, priority 99) every sample_period. On CONFIG_PREEMPT=y, any hardware interrupt returning to wg_ratelimiter_gc_entries() at line 78 (preempt_count == 0, IRQs enabled) would immediately preempt kworker/u8:6 via arm64_exit_to_kernel_mode_preempt() (arch/arm64/kernel/entry-common.c:105-110) and run softlockup_fn(), resetting watchdog_touch_ts.

3. Evidence from irqtrace and Concurrent CPU 1 Activity (Inconclusive Trigger)

  • print_irqtrace_events(current) (kernel/locking/lockdep.c:4244-4265) records current->irqtrace on kworker/u8:6:
    • 1142062 / 1142066: neigh_periodic_work() (net/core/neighbour.c:987, 1054), which also runs on system_power_efficient_wq.
    • 1142073: ktime_get_coarse_with_offset() (kernel/time/timekeeping.c:1071 $\rightarrow$ seqcount_lockdep_reader_access(), include/linux/seqlock.h:75), called once at the start of wg_ratelimiter_gc_entries() on line 57 (ktime_get_coarse_boottime_ns()) before entering the loop.
    • 1142074: __el1_irq (arch/arm64/kernel/entry-common.c:566), the timer interrupt in which watchdog_timer_fn() reported the 24s soft lockup.
  • Because every el1_interrupt() updates irq_events via irqentry_enter_from_kernel_mode() / irqentry_exit_to_kernel_mode_preempt() and call_rcu() also toggles IRQs, the single-step increment (1142073 $\rightarrow$ 1142074) proves zero hardware interrupts were delivered to CPU 0 (and entry_uninit() was not even called) between line 57 and line 78 of wg_ratelimiter_gc_entries().
  • The 24s elapsed running_clock() delta without intermediate timer interrupts therefore cannot be caused by wg_ratelimiter_gc_entries(). It indicates an external vCPU/hypervisor pause or host CPU starvation (Hardware name: linux,dummy-virt), where arm64's kvm_check_and_clear_guest_paused() (include/asm-generic/kvm_para.h:12) is a no-op stub returning false in watchdog_timer_fn() (kernel/watchdog.c:841), making the exact external stall trigger inconclusive from the kernel report alone.
  • Concurrently, the NMI backtrace shows CPU 1 (kworker/u8:1, PID 5013) inside bpf_map_free_deferred() $\rightarrow$ arena_map_free() (kernel/bpf/arena.c:401) $\rightarrow$ free_vm_area() $\rightarrow$ remove_vm_area() (mm/vmalloc.c:3385) $\rightarrow$ __debug_check_no_obj_freed() (lib/debugobjects.c:1141-1180). Because arena_map_alloc() reserves KERN_VM_SZ = SZ_4G + GUARD_SZ (kernel/bpf/arena.c:48, 302) and CONFIG_ARM64_4K_PAGES=y sets ODEBUG_CHUNK_SIZE = 4096 (lib/debugobjects.c:34-35), CONFIG_DEBUG_OBJECTS_FREE=y forces CPU 1 to iterate over 1,048,592 chunks (x26 = 0x7b4cc = 505,036 remaining), acquiring db->lock and scanning obj_hash on each chunk. While this does not block CPU 0 via any shared kernel lock or interrupt mask, it represents substantial debug-configuration CPU work occurring alongside the QEMU/host stall.
▶ ▼ ✨ AI Jobs (2)
ID Workflow Result Correct Bug Created Started Finished Revision
9443f304 bisect-cause 🏃 BUG: soft lockup in wg_ratelimiter_gc_entries 2026/11/22 03:56 2026/11/22 03:56 b2c69459
256acfcf assessment-security DenialOfService: ❌ Exploitable: ❌ FilesystemTrigger: ❌ FuzzingArtifact: ❌ MemoryCorruption: ❌ NetworkNeedsSetup: ❌ NetworkTrigger: ❌ PeripheralTrigger: ❌ RemoteTrigger: ❌ Unprivileged: ❌ UserNamespace: ❌ VMGuestTrigger: ❌ VMHostTrigger: ❌ ❓ BUG: soft lockup in wg_ratelimiter_gc_entries 2026/11/22 03:55 2026/11/22 03:55 2026/11/22 04:33 b2c69459
▶ ▼ Similar bugs (10)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in wg_ratelimiter_gc_entries (6) prio:normal wireguard 1 6 140d 238d 0/29 auto-obsoleted due to no activity on 2026/10/02 11:31
upstream INFO: rcu detected stall in wg_ratelimiter_gc_entries (5) wireguard 1 1 363d 363d 0/29 auto-obsoleted due to no activity on 2026/02/21 09:54
linux-6.1 INFO: rcu detected stall in wg_ratelimiter_gc_entries 1 1 1311d 1311d 0/3 auto-obsoleted due to no activity on 2023/08/18 11:48
upstream INFO: rcu detected stall in wg_ratelimiter_gc_entries (4) wireguard 1 2 568d 572d 0/29 auto-obsoleted due to no activity on 2025/08/01 02:14
upstream INFO: rcu detected stall in wg_ratelimiter_gc_entries wireguard 1 1 1288d 1288d 0/29 auto-obsoleted due to no activity on 2023/08/11 08:26
linux-6.1 INFO: rcu detected stall in wg_ratelimiter_gc_entries (2) 1 1 841d 841d 0/3 auto-obsoleted due to no activity on 2024/11/10 07:30
upstream INFO: rcu detected stall in wg_ratelimiter_gc_entries (2) wireguard 1 syz, C done done 3 1194d 1192d 0/29 auto-obsoleted due to no activity on 2023/11/23 21:08
upstream INFO: rcu detected stall in wg_ratelimiter_gc_entries (3) wireguard 1 1 703d 703d 0/29 auto-obsoleted due to no activity on 2025/03/18 13:29
linux-5.15 INFO: rcu detected stall in wg_ratelimiter_gc_entries (2) 1 1 388d 388d 0/3 auto-obsoleted due to no activity on 2026/02/06 13:45
linux-5.15 INFO: rcu detected stall in wg_ratelimiter_gc_entries 1 1 847d 847d 0/3 auto-obsoleted due to no activity on 2024/11/04 15:33

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/u8:6:1180]
Modules linked in:
irq event stamp: 1142074
hardirqs last  enabled at (1142073): [<ffff8000804df8e0>] seqcount_lockdep_reader_access include/linux/seqlock.h:75 [inline]
hardirqs last  enabled at (1142073): [<ffff8000804df8e0>] ktime_get_coarse_with_offset+0x160/0x1ac kernel/time/timekeeping.c:1071
hardirqs last disabled at (1142074): [<ffff800085a87210>] __el1_irq arch/arm64/kernel/entry-common.c:566 [inline]
hardirqs last disabled at (1142074): [<ffff800085a87210>] el1_interrupt+0x28/0x60 arch/arm64/kernel/entry-common.c:582
softirqs last  enabled at (1142066): [<ffff800084b1facc>] spin_unlock_bh include/linux/spinlock.h:407 [inline]
softirqs last  enabled at (1142066): [<ffff800084b1facc>] neigh_periodic_work+0x60c/0x8f0 net/core/neighbour.c:1054
softirqs last disabled at (1142062): [<ffff800084b1f594>] spin_lock_bh include/linux/spinlock.h:353 [inline]
softirqs last disabled at (1142062): [<ffff800084b1f594>] neigh_periodic_work+0xd4/0x8f0 net/core/neighbour.c:987
CPU: 0 UID: 0 PID: 1180 Comm: kworker/u8:6 Tainted: G             L      syzkaller #0 PREEMPT 
Tainted: [L]=SOFTLOCKUP
Hardware name: linux,dummy-virt (DT)
Workqueue: events_power_efficient wg_ratelimiter_gc_entries
pstate: 10000005 (nzcV daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : lock_is_held_type+0x128/0x1c0 kernel/locking/lockdep.c:6021
lr : lockdep_recursion_finish kernel/locking/lockdep.c:470 [inline]
lr : lock_is_held_type+0x108/0x1c0 kernel/locking/lockdep.c:6017
sp : ffff8000a03279c0
x29: ffff8000a03279c0 x28: 0000000000004650 x27: dfff800000000000
x26: 00000000ffffffff x25: 0000000000000000 x24: ffff000018318000
x23: ffff8000883a5348 x22: ffff8000886176a0 x21: ffff000018318ba0
x20: 0000000000000002 x19: 0000000000000000 x18: 1fffe00003327c22
x17: 0000000000000000 x16: 00000000f1f1f1f1 x15: 00000000f3f3f3f3
x14: ffff700014064f16 x13: ffff80008ab36980 x12: ffff700011223ac5
x11: 1ffff00011223ac4 x10: ffff700011223ac4 x9 : dfff800000000000
x8 : 0000000041b58ab3 x7 : ffff700014064f4c x6 : ffff8000a0327a60
x5 : dfff800000000000 x4 : 00000000f1f1f1f1 x3 : 1fffe00003063001
x2 : 0000000000000000 x1 : ffff7fffe1a13000 x0 : 00000000000000c0
Call trace:
 __daif_local_irq_restore arch/arm64/include/asm/irqflags.h:175 [inline] (P)
 arch_local_irq_restore arch/arm64/include/asm/irqflags.h:195 [inline] (P)
 lock_is_held_type+0x128/0x1c0 kernel/locking/lockdep.c:6018 (P)
 lock_is_held include/linux/lockdep.h:249 [inline]
 __might_resched+0x49c/0x588 kernel/sched/core.c:9161
 wg_ratelimiter_gc_entries+0x28c/0x3e4 drivers/net/wireguard/ratelimiter.c:78
 process_one_work+0x820/0x1940 kernel/workqueue.c:3495
 process_scheduled_works kernel/workqueue.c:3578 [inline]
 worker_thread+0x42c/0xc90 kernel/workqueue.c:3659
 kthread+0x2ec/0x384 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 5013 Comm: kworker/u8:1 Tainted: G             L      syzkaller #0 PREEMPT 
Tainted: [L]=SOFTLOCKUP
Hardware name: linux,dummy-virt (DT)
Workqueue: events_unbound bpf_map_free_deferred
pstate: 10000005 (nzcV daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:211 [inline]
pc : _raw_spin_unlock_irqrestore+0x40/0xb4 kernel/locking/spinlock.c:221
lr : __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:210 [inline]
lr : _raw_spin_unlock_irqrestore+0x80/0xb4 kernel/locking/spinlock.c:221
sp : ffff8000a1427890
x29: ffff8000a1427890 x28: dfff800000000000 x27: 0000000000000003
x26: 000000000007b4cc x25: 000000000000b139 x24: 0000000000000000
x23: 00000000000013b1 x22: dead000000000122 x21: dead000000000100
x20: ffff80008e652990 x19: 0000000000000000 x18: ffff000069de23a8
x17: ffff7fffe1a39000 x16: ffff000069de1fd0 x15: 0098968000000000
x14: 7fffffffffffffff x13: ffff80008ab3f490 x12: ffff700011cca533
x11: 1ffff00011cca532 x10: ffff700011cca532 x9 : dfff800000000000
x8 : ffff8000a1427730 x7 : ffff8000a14278a0 x6 : ffff8000a1427850
x5 : ffff8000a1427790 x4 : 0000000000000002 x3 : 1fffe00003174751
x2 : 0000000000000000 x1 : 0000000000176e5f x0 : 00000000000000c0
Call trace:
 __daif_local_irq_restore arch/arm64/include/asm/irqflags.h:175 [inline] (P)
 arch_local_irq_restore arch/arm64/include/asm/irqflags.h:195 [inline] (P)
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:210 [inline] (P)
 _raw_spin_unlock_irqrestore+0x40/0xb4 kernel/locking/spinlock.c:221 (P)
 __debug_check_no_obj_freed lib/debugobjects.c:1180 [inline]
 debug_check_no_obj_freed+0x280/0x558 lib/debugobjects.c:1201
 remove_vm_area+0x130/0x2a0 mm/vmalloc.c:3385
 free_vm_area+0x34/0x5c mm/vmalloc.c:4936
 arena_map_free+0xcc/0x160 kernel/bpf/arena.c:401
 bpf_map_free kernel/bpf/syscall.c:906 [inline]
 bpf_map_free_deferred+0x194/0x640 kernel/bpf/syscall.c:933
 process_one_work+0x820/0x1940 kernel/workqueue.c:3495
 process_scheduled_works kernel/workqueue.c:3578 [inline]
 worker_thread+0x42c/0xc90 kernel/workqueue.c:3659
 kthread+0x2ec/0x384 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2026/11/22 00:45 https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux master c44104f3642d b2c69459 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu2-arm64 BUG: soft lockup in wg_ratelimiter_gc_entries
* Struck through repros no longer work on HEAD.