syzbot
BUG: soft lockup in mas_find
Status:
moderation: reported on 2026/11/21 00:02
Subsystems:
mm
Labels:
prio:low
[Documentation on labels]
Reported-by: syzbot+d7b9230ce97cb3fa544a@syzkaller.appspotmail.com
First crash: 7h03m, last: 7h03m
✨ AI-generated root cause summary:
CPU 0 triggered a 132s soft lockup warning inside modprobe's mmap path (mtree_range_walk -> mt_slot -> debug_lockdep_rcu_enabled), while CPU 1 was spinning in csd_lock_wait() inside smp_call_function_many_cond() during exit_mmap -> free_ldt_pgtables(). Inspection of the irqtrace stamps and call paths shows that hardirqs were enabled on CPU 0 (IF=1, EFLAGS=0x246) and that zero interrupts occurred on CPU 0 between syscall entry (stamp 1567 in do_syscall_64) and the soft lockup timer interrupt 132s later (stamp 1568 in sysvec_apic_timer_interrupt). This rules out an active infinite loop in mtree_range_walk or a kernel IRQ-disabled hang, and instead indicates an external vCPU execution stall or lost/delayed interrupt delivery (such as a hypervisor/host vCPU pause not masked by kvm_check_and_clear_guest_paused()) that also prevented CPU 0 from servicing an earlier IPI waited on by CPU 1.
▶
▼
✨ Full AI root cause analysis
Summary of Findings
This report shows a BUG: soft lockup - CPU#0 stuck for 132s! detected on CPU 0 (modprobe) in watchdog_timer_fn() (kernel/watchdog.c:798), alongside an NMI backtrace from CPU 1 (syz-executor) spinning in csd_lock_wait() (kernel/smp.c:389). After inspecting the call stacks, irqtrace metadata, and kernel configuration, the evidence indicates that CPU 0 did not spin in a kernel infinite loop, but rather experienced a 132-second stall in execution / interrupt delivery (consistent with a host/hypervisor vCPU pause in the GCE VM that was not reported via PVCLOCK_GUEST_STOPPED), which also left CPU 1 waiting on CPU 0 to process a previously queued CSD IPI. Without a reproducer or host-side hypervisor trace, the exact external trigger of the vCPU stall remains inconclusive.
Detailed Analysis of Both CPUs
1. CPU 0 (modprobe:7944) — mmap / Maple Tree Walk
- Call path:
ksys_mmap_pgoff (mm/mmap.c:619) → vm_mmap_pgoff (mm/util.c:581) → do_mmap (mm/mmap.c:573) → mmap_region (mm/vma.c:3173) → __mmap_region (mm/vma.c:3071) → __mmap_setup (mm/vma.c:2580) → vma_find (include/linux/mm.h:1811) → mas_find (lib/maple_tree.c:5447) → mas_find_setup (lib/maple_tree.c:5406) → mas_walk (lib/maple_tree.c:4421) → mas_state_walk (lib/maple_tree.c:2976) → mtree_range_walk (lib/maple_tree.c:2081) → mt_slot (lib/maple_tree.c:676) → debug_lockdep_rcu_enabled (kernel/rcu/update.c:321).
- Invariants checked:
mmap_region() (mm/vma.c:3154) holds current->mm->mmap_lock for write (mmap_assert_write_locked(current->mm)), and modprobe is performing a standard mmap(NULL, 0x9088, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, ...) syscall.
- None of the functions from
do_syscall_64 down to mtree_range_walk / debug_lockdep_rcu_enabled disable local IRQs or preemption.
EFLAGS on CPU 0 is 0x00000246 (IF=1), confirming hardirqs were enabled when sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1062) fired.
- Ruled-out hypothesis (active infinite loop in
mtree_range_walk / mas_walk):
- The lockdep irqtrace fields on
modprobe show:
- With
CONFIG_HZ_100=y, CONFIG_PREEMPT=y (PREEMPT(full)), and watchdog_thresh=55 (sample_period = 22s, softlockup threshold 2 * 55 = 110s), if CPU 0 had been actively looping in mtree_range_walk() for 132 seconds with IF=1, the 100 Hz timer tick and watchdog_timer_fn() would have fired repeatedly, advancing irq event stamp by thousands and triggering watchdog/0 preemption (preempt_count == 0) or reporting at 110s.
- Instead, zero interrupts and zero context switches (
__schedule() would update hardirqs last enabled/disabled) occurred on CPU 0 between entering do_syscall_64 (1567) and the single sysvec_apic_timer_interrupt (1568) 132 seconds later.
2. CPU 1 (syz-executor:1055) — free_ldt_pgtables / csd_lock_wait
- Call path:
do_exit → exit_mm → __mmput → exit_mmap → arch_exit_mmap → free_ldt_pgtables (arch/x86/kernel/ldt.c:411) → tlb_finish_mmu (mm/mmu_gather.c:529) → tlb_flush_mmu (mm/mmu_gather.c:403) → flush_tlb_mm_range (arch/x86/mm/tlb.c:1441) → flush_tlb_multi (arch/x86/mm/tlb.c:1367) → smp_call_function_many_cond (kernel/smp.c:970) → csd_lock (kernel/smp.c:422) → csd_lock_wait (kernel/smp.c:389).
- Mechanism:
- In
smp_call_function_many_cond(), line 970 acquires csd_lock(csd) on per_cpu_ptr(cfd->csd, cpu) before llist_add() (line 985) and before sending the TLB flush IPI (line 998).
- CPU 1 is spinning on
!(csd->node.u_flags & CSD_FLAG_LOCK) (kernel/smp.c:389) because an earlier smp_call_function* request using CPU 1's cfd->csd entry for CPU 0 had not yet been executed and unlocked by CPU 0.
- Note that
CONFIG_CSD_LOCK_WAIT_DEBUG=y is compiled in, but # CONFIG_CSD_LOCK_WAIT_DEBUG_DEFAULT is not set and csdlock_debug is not enabled on CONFIG_CMDLINE, so static_branch_unlikely(&csdlock_debug_enabled) at kernel/smp.c:384 is false and csd_lock_wait_toolong() is not active.
- When CPU 0 resumed and took
sysvec_apic_timer_interrupt (1568), watchdog_timer_fn() (kernel/watchdog.c:798) checked kvm_check_and_clear_guest_paused() (arch/x86/kernel/kvmclock.c:157), which only touches the watchdogs if the hypervisor set PVCLOCK_GUEST_STOPPED in pvti.flags.
- Because host vCPU preemption/scheduling delays on GCE do not necessarily set
PVCLOCK_GUEST_STOPPED, is_softlockup() (kernel/watchdog.c:744) observed now - touch_ts == 132s > 110s and emitted the soft lockup report (followed by panic due to CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=1).
What Remains Inconclusive
- Without the prior syzkaller execution log or host/hypervisor traces, it cannot be definitively proven whether the 132s interrupt/execution gap on CPU 0 was caused purely by host-level GCE vCPU pausing/overcommit or by an unrecorded guest operation prior to
modprobe's mmap() that delayed virtual APIC timer and IPI delivery. However, no defect was found in mm/vma.c, lib/maple_tree.c, arch/x86/kernel/ldt.c, or kernel/smp.c on the reported stacks.
Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 132s! [modprobe:7944]
Modules linked in:
irq event stamp: 1568
hardirqs last enabled at (1567): [<ffffffff8c0e4178>] do_syscall_64+0xc8/0x530 arch/x86/entry/syscall_64.c:86
hardirqs last disabled at (1568): [<ffffffff8c0ebcce>] sysvec_apic_timer_interrupt+0xe/0xc0 arch/x86/kernel/apic/apic.c:1062
softirqs last enabled at (168): [<ffffffff816dd5be>] local_bh_disable include/linux/bottom_half.h:20 [inline]
softirqs last enabled at (168): [<ffffffff816dd5be>] fpregs_lock arch/x86/include/asm/fpu/api.h:69 [inline]
softirqs last enabled at (168): [<ffffffff816dd5be>] fpu_reset_fpstate_regs arch/x86/kernel/fpu/core.c:824 [inline]
softirqs last enabled at (168): [<ffffffff816dd5be>] fpu_flush_thread+0x2be/0x500 arch/x86/kernel/fpu/core.c:883
softirqs last disabled at (166): [<ffffffff816dd5be>] local_bh_disable include/linux/bottom_half.h:20 [inline]
softirqs last disabled at (166): [<ffffffff816dd5be>] fpregs_lock arch/x86/include/asm/fpu/api.h:69 [inline]
softirqs last disabled at (166): [<ffffffff816dd5be>] fpu_reset_fpstate_regs arch/x86/kernel/fpu/core.c:824 [inline]
softirqs last disabled at (166): [<ffffffff816dd5be>] fpu_flush_thread+0x2be/0x500 arch/x86/kernel/fpu/core.c:883
CPU: 0 UID: 0 PID: 7944 Comm: modprobe Not tainted syzkaller #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/06/2026
RIP: 0010:debug_lockdep_rcu_enabled+0x29/0x40 kernel/rcu/update.c:321
Code: 90 f3 0f 1e fa 31 c0 83 3d 9b 54 af 05 00 74 1d 83 3d 8a 82 af 05 00 74 14 65 48 8b 0d 98 93 d4 08 31 c0 83 b9 a4 0b 00 00 00 <0f> 94 c0 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
RSP: 0018:ffffc900118cf310 EFLAGS: 00000246
RAX: 0000000000000000 RBX: dffffc0000000000 RCX: ffff888054eaddc0
RDX: 0000000000000000 RSI: ffffffff91a3ba40 RDI: ffffc900118cf700
RBP: 0000000000000000 R08: ffff888054eaddc0 R09: 0000000000000005
R10: 0000000000000004 R11: 0000000000000000 R12: ffff888078ff0040
R13: 00007f11eb0bcfff R14: ffff88807f6394a8 R15: 0000000000000005
FS: 0000000000000000(0000) GS:ffff8881233ea000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f11eb0bd000 CR3: 00000000776c0000 CR4: 00000000003526f0
DR0: 0000200000000300 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
<TASK>
mt_slot lib/maple_tree.c:676 [inline]
mtree_range_walk+0x4d4/0x8b0 lib/maple_tree.c:2081
mas_state_walk lib/maple_tree.c:2976 [inline]
mas_walk+0x216/0x490 lib/maple_tree.c:4421
mas_find_setup lib/maple_tree.c:5406 [inline]
mas_find+0x635/0xcb0 lib/maple_tree.c:5447
vma_find include/linux/mm.h:1811 [inline]
__mmap_setup mm/vma.c:2580 [inline]
__mmap_region mm/vma.c:3071 [inline]
mmap_region+0x70f/0x2800 mm/vma.c:3173
do_mmap+0xe0a/0x1300 mm/mmap.c:573
vm_mmap_pgoff+0x27a/0x520 mm/util.c:581
ksys_mmap_pgoff+0x4dc/0x760 mm/mmap.c:619
do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
do_syscall_64+0x16a/0x530 arch/x86/entry/syscall_64.c:89
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f11eb3e6242
Code: 08 00 04 00 00 eb e2 90 41 f7 c1 ff 0f 00 00 75 27 55 89 cd 53 48 89 fb 48 85 ff 74 33 41 89 ea 48 89 df b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5e 5b 5d c3 0f 1f 00 c7 05 46 40 01 00 16 00
RSP: 002b:00007ffca5caf3f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f11eb3e6242
RDX: 0000000000000001 RSI: 0000000000009088 RDI: 0000000000000000
RBP: 0000000000000802 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000802 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f11eb0bd0c0 R14: 00007ffca5caf830 R15: 00000fff94b95e82
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 1055 Comm: syz-executor Not tainted syzkaller #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/06/2026
RIP: 0010:csd_lock_wait kernel/smp.c:389 [inline]
RIP: 0010:csd_lock kernel/smp.c:422 [inline]
RIP: 0010:smp_call_function_many_cond+0x61e/0x1500 kernel/smp.c:970
Code: b6 04 04 84 c0 0f 85 d5 03 00 00 44 8b 3b 44 89 fe 83 e6 01 31 ff e8 d1 79 0c 00 41 83 e7 01 75 07 e8 b6 74 0c 00 eb 3f f3 90 <48> b8 00 00 00 00 00 fc ff df 41 0f b6 04 04 84 c0 75 0f f7 03 01
RSP: 0018:ffffc90003de7320 EFLAGS: 00000293
RAX: ffffffff81bca53e RBX: ffff8880b8244328 RCX: ffff8880338a5dc0
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90003de7470 R08: ffff8880338a735f R09: 1ffff11006714e6b
R10: dffffc0000000000 R11: 0000000000000000 R12: 1ffff11017048865
R13: 0000000000000000 R14: ffff8880b8244320 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8881234ea000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffd5c1fc2b0 CR3: 0000000036a00000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
<TASK>
__flush_tlb_multi arch/x86/include/asm/paravirt.h:46 [inline]
flush_tlb_multi arch/x86/mm/tlb.c:1367 [inline]
flush_tlb_mm_range+0x910/0x10b0 arch/x86/mm/tlb.c:1441
tlb_flush arch/x86/include/asm/tlb.h:23 [inline]
tlb_flush_mmu_tlbonly include/asm-generic/tlb.h:460 [inline]
tlb_flush_mmu+0x1af/0xa30 mm/mmu_gather.c:403
tlb_finish_mmu+0xf4/0x220 mm/mmu_gather.c:529
free_ldt_pgtables+0x192/0x360 arch/x86/kernel/ldt.c:411
arch_exit_mmap arch/x86/include/asm/mmu_context.h:229 [inline]
exit_mmap+0x1b7/0xa40 mm/mmap.c:1300
__mmput+0x118/0x420 kernel/fork.c:1192
exit_mm+0x18e/0x240 kernel/exit.c:586
do_exit+0x6a8/0x2360 kernel/exit.c:967
do_group_exit+0x22d/0x2f0 kernel/exit.c:1123
get_signal+0x136a/0x1430 kernel/signal.c:3113
arch_do_signal_or_restart+0xbb/0x860 arch/x86/kernel/signal.c:337
__exit_to_user_mode_loop kernel/entry/common.c:66 [inline]
exit_to_user_mode_loop+0x10e/0x770 kernel/entry/common.c:101
__exit_to_user_mode_prepare include/linux/irq-entry-common.h:207 [inline]
syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:230 [inline]
syscall_exit_to_user_mode include/linux/entry-common.h:343 [inline]
do_syscall_64+0x32c/0x530 arch/x86/entry/syscall_64.c:107
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f3e021646d7
Code: Unable to access opcode bytes at 0x7f3e021646ad.
RSP: 002b:00007fffcc89ee80 EFLAGS: 00000202 ORIG_RAX: 000000000000003d
RAX: fffffffffffffe00 RBX: 0000555564733500 RCX: 00007f3e021646d7
RDX: 0000000040000000 RSI: 00007fffcc89eedc RDI: ffffffffffffffff
RBP: 00007fffcc89eedc R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000008
R13: 0000000000000003 R14: 00007fffcc89f138 R15: 0000000000000000
</TASK>
* Struck through repros no longer work on HEAD.