BUG: soft lockup in __run_timers

Status: auto-obsoleted due to no activity on 2024/12/16 23:50
Reported-by: syzbot+be7849202482501d1735@syzkaller.appspotmail.com
First crash: 694d, last: 694d
▶ ▼ Similar bugs (13)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
android-5-10 BUG: soft lockup in __run_timers 1 2 722d 791d 0/2 auto-obsoleted due to no activity on 2024/11/18 04:44
android-54 BUG: soft lockup in __run_timers 1 1 776d 776d 0/2 auto-obsoleted due to no activity on 2024/09/26 00:22
upstream BUG: soft lockup in __run_timers kernel 1 1 1284d 1280d 0/29 auto-obsoleted due to no activity on 2023/05/06 12:08
android-54 BUG: soft lockup in __run_timers (2) 1 2 495d 557d 0/2 auto-obsoleted due to no activity on 2025/07/03 11:27
upstream INFO: rcu detected stall in __run_timers (5) kernel 1 syz, C error error 63 1654d 2167d 0/29 closed as invalid on 2022/02/08 10:39
upstream INFO: rcu detected stall in __run_timers (8) prio:normal kernel 1 1 103d 103d 0/29 auto-obsoleted due to no activity on 2026/07/31 00:54
linux-5.15 INFO: rcu detected stall in __run_timers 1 1 1213d 1213d 0/3 auto-obsoleted due to no activity on 2023/08/15 19:18
linux-6.1 INFO: rcu detected stall in __run_timers (2) origin:upstream 1 syz error 3 19d 304d 0/3 upstream: reported syz repro on 2025/10/12 10:55
upstream INFO: rcu detected stall in __run_timers (7) kernfs 1 25 702d 807d 0/29 auto-obsoleted due to no activity on 2024/11/08 17:44
linux-6.1 INFO: rcu detected stall in __run_timers 1 1 661d 661d 0/3 auto-obsoleted due to no activity on 2025/01/28 16:20
upstream INFO: rcu detected stall in __run_timers (6) kernel 1 2 1622d 1638d 0/29 auto-closed as invalid on 2022/06/02 21:46
linux-5.15 INFO: rcu detected stall in __run_timers (2) 1 9 13d 343d 0/3 upstream: reported on 2025/09/03 11:07
linux-6.6 INFO: rcu detected stall in __run_timers 1 2 24d 63d 0/2 upstream: reported on 2026/06/10 07:15

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 143s! [syz.1.325:1962]
Modules linked in:
CPU: 0 PID: 1962 Comm: syz.1.325 Tainted: G        W         5.15.161-syzkaller-00425-gb92c0d35d015 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
RIP: 0010:expire_timers kernel/time/timer.c:1502 [inline]
RIP: 0010:__run_timers+0x71d/0xa10 kernel/time/timer.c:1773
Code: 00 20 00 21 de 31 ff e8 a1 c0 0f 00 b8 00 00 20 00 21 c3 75 2c e8 e3 bc 0f 00 48 8b 5c 24 20 48 89 df e8 d6 5d 85 03 4c 89 ef <4c> 89 f6 48 8b 54 24 58 e8 e6 02 00 00 48 89 df e8 6e 59 85 03 eb
RSP: 0018:ffffc90000007da0 EFLAGS: 00000202
RAX: 0000000000000101 RBX: ffff8881f7027900 RCX: 0000000000000102
RDX: 0000000000000102 RSI: 0000000000000000 RDI: ffff8881065f84e0
RBP: ffffc90000007f08 R08: ffffffff8160a30f R09: 0000000000000000
R10: 0000000000000000 R11: dffffc0000000001 R12: dffffc0000000000
R13: ffff8881065f84e0 R14: ffffffff8455f410 R15: ffff8881f7027908
FS:  00007ffac86996c0(0000) GS:ffff8881f7000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffac8656f98 CR3: 0000000126566000 CR4: 00000000003506b0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
 <IRQ>
 run_timer_softirq+0x69/0xf0 kernel/time/timer.c:1786
 handle_softirqs+0x25e/0x5c0 kernel/softirq.c:565
 __do_softirq kernel/softirq.c:603 [inline]
 invoke_softirq kernel/softirq.c:425 [inline]
 __irq_exit_rcu+0x52/0xf0 kernel/softirq.c:652
 irq_exit_rcu+0x9/0x10 kernel/softirq.c:664
 sysvec_apic_timer_interrupt+0x9a/0xc0 arch/x86/kernel/apic/apic.c:1096
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1b/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:finish_task_switch+0x16f/0x7b0 kernel/sched/core.c:5024
Code: 74 08 4c 89 ff e8 91 70 65 00 49 8b 1f 48 85 db 4c 8b 6d c0 0f 85 ce 00 00 00 4c 89 e7 e8 89 c9 98 03 fb 49 8d 9d 38 0b 00 00 <48> 89 d8 48 c1 e8 03 49 be 00 00 00 00 00 fc ff df 42 0f b6 04 30
RSP: 0018:ffffc90001027680 EFLAGS: 00000282
RAX: 0000000080000001 RBX: ffff888116a79ef8 RCX: 0000000000000002
RDX: 0000000040000002 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffffc900010276d0 R08: ffffffff8191fd19 R09: fffffbfff0e52e93
R10: 0000000000000000 R11: dffffc0000000001 R12: ffff8881f70378c0
R13: ffff888116a793c0 R14: 1ffff1103ee07081 R15: ffff8881f7038408
 context_switch kernel/sched/core.c:5155 [inline]
 __schedule+0xcd4/0x1590 kernel/sched/core.c:6512
 preempt_schedule_irq+0xc7/0x140 kernel/sched/core.c:6916
 irqentry_exit_cond_resched+0x2a/0x30 kernel/entry/common.c:398
 irqentry_exit+0x30/0x40 kernel/entry/common.c:432
 sysvec_apic_timer_interrupt+0x55/0xc0 arch/x86/kernel/apic/apic.c:1096
 asm_sysvec_apic_timer_interrupt+0x1b/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:preempt_schedule_thunk+0x0/0x18 arch/x86/entry/thunk_64.S:34
Code: 80 c1 03 38 c1 7c c3 4c 89 f7 e8 37 1f b2 00 eb b9 0f 1f 44 00 00 55 48 89 e5 e8 47 d8 6f 00 48 c7 c0 80 5c 40 85 5d c3 cc cc <55> 48 89 e5 57 56 52 51 50 41 50 41 51 41 52 41 53 e8 96 b0 e3 03
RSP: 0018:ffffc90001027978 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff888128bf90b0 RCX: 0000000000000001
RDX: ffffc900027a5000 RSI: 000000000000002f RDI: 0000000000000001
RBP: ffffc90001027988 R08: dffffc0000000000 R09: 0000000000000003
R10: fffff52000204f24 R11: dffffc0000000001 R12: ffff888128bf90c0
R13: 0000000000000000 R14: 0000000000000000 R15: dffffc0000000000
 spin_unlock include/linux/spinlock.h:403 [inline]
 fasync_insert_entry+0x223/0x2e0 fs/fcntl.c:954
 fasync_add_entry fs/fcntl.c:977 [inline]
 fasync_helper+0x52/0xa0 fs/fcntl.c:995
 perf_fasync+0x84/0xa0 kernel/events/core.c:6524
 ioctl_fioasync fs/ioctl.c:383 [inline]
 do_vfs_ioctl+0x1272/0x2a80 fs/ioctl.c:795
 __do_sys_ioctl fs/ioctl.c:872 [inline]
 __se_sys_ioctl+0x99/0x190 fs/ioctl.c:860
 __x64_sys_ioctl+0x7b/0x90 fs/ioctl.c:860
 x64_sys_call+0x98/0x9a0 arch/x86/include/generated/asm/syscalls_64.h:17
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7ffac9a1fef9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffac8699038 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007ffac9bd7f80 RCX: 00007ffac9a1fef9
RDX: 00000000200006c0 RSI: 0000000000005452 RDI: 000000000000000d
RBP: 00007ffac9a92b76 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007ffac9bd7f80 R15: 00007ffec4461878
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 1287 Comm: syz.0.216 Tainted: G        W         5.15.161-syzkaller-00425-gb92c0d35d015 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
RIP: 0010:__sanitizer_cov_trace_pc+0x5d/0x60 kernel/kcov.c:207
Code: 0b 00 00 83 fa 02 75 21 48 8b 91 40 0b 00 00 48 8b 32 48 8d 7e 01 8b 89 3c 0b 00 00 48 39 cf 73 08 48 89 44 f2 08 48 89 3a 5d <c3> 66 90 55 48 89 e5 4c 8b 45 08 65 48 8b 15 10 0d 92 7e 65 8b 05
RSP: 0018:ffffc900001cfa68 EFLAGS: 00000046
RAX: ffffffff81924400 RBX: 0000000000000000 RCX: ffff88810bc80000
RDX: 0000000000010500 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc900001cfab0 R08: ffffffff819243f2 R09: ffffc900001cfc00
R10: 0000000000000000 R11: dffffc0000000001 R12: ffffc900001cfdc0
R13: ffff88812d469240 R14: 0000000000000000 R15: ffffc900001cfc66
FS:  0000000000000000(0000) GS:ffff8881f7100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd57b4f8f98 CR3: 0000000006a0f000 CR4: 00000000003506a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 perf_prepare_sample+0x168/0x1a80 kernel/events/core.c:7455
 __perf_event_output kernel/events/core.c:7633 [inline]
 perf_event_output_forward+0xdb/0x1b0 kernel/events/core.c:7653
 __perf_event_overflow+0x20b/0x390 kernel/events/core.c:9323
 perf_swevent_hrtimer+0x3fd/0x560 kernel/events/core.c:10719
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x41a/0xad0 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x40c/0xaa0 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1085 [inline]
 __sysvec_apic_timer_interrupt+0xfd/0x3c0 arch/x86/kernel/apic/apic.c:1102
 sysvec_apic_timer_interrupt+0x44/0xc0 arch/x86/kernel/apic/apic.c:1096
 asm_sysvec_apic_timer_interrupt+0x1b/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:harmonize_features net/core/dev.c:3516 [inline]
RIP: 0010:netif_skb_features+0x777/0xb70 net/core/dev.c:3608
Code: 81 fb 88 a8 00 00 0f 94 c0 44 08 f0 49 83 c5 69 4d 21 e5 a8 01 4d 0f 44 ec 48 8b 5d b8 4c 89 ff 31 f6 e8 2c e5 ff ff 41 89 c6 <48> b8 00 00 00 00 00 fc ff df 48 8b 4d b0 0f b6 04 01 84 c0 0f 85
RSP: 0018:ffffc900001d02e0 EFLAGS: 00000246
RAX: 000000000000dd86 RBX: ffff88811f3b9800 RCX: ffff88810bc80000
RDX: dffffc0000000000 RSI: 0000000000000048 RDI: 0000000000000000
RBP: ffffc900001d0330 R08: ffffffff83e69624 R09: ffffffff83e692ec
R10: 0000000000000002 R11: ffff88810bc80000 R12: 0000000000004000
R13: 0000000000004000 R14: 000000000000dd86 R15: ffff88811f3b9780
 validate_xmit_skb+0x7f/0xce0 net/core/dev.c:3693
 __dev_queue_xmit+0x12d2/0x2e70 net/core/dev.c:4246
 dev_queue_xmit+0x17/0x20 net/core/dev.c:4288
 neigh_connected_output+0x417/0x450 net/core/neighbour.c:1533
 neigh_output include/net/neighbour.h:517 [inline]
 ip6_finish_output2+0xf95/0x16e0 net/ipv6/ip6_output.c:126
 __ip6_finish_output+0x60f/0x7c0 net/ipv6/ip6_output.c:197
 ip6_finish_output+0x31/0x210 net/ipv6/ip6_output.c:207
 NF_HOOK_COND include/linux/netfilter.h:294 [inline]
 ip6_output+0x1f7/0x4d0 net/ipv6/ip6_output.c:230
 dst_output include/net/dst.h:444 [inline]
 NF_HOOK include/linux/netfilter.h:305 [inline]
 ndisc_send_skb+0x73e/0xc90 net/ipv6/ndisc.c:509
 ndisc_send_rs+0x532/0x6a0 net/ipv6/ndisc.c:703
 addrconf_rs_timer+0x2d1/0x600 net/ipv6/addrconf.c:3978
 call_timer_fn+0x3b/0x2d0 kernel/time/timer.c:1457
 expire_timers kernel/time/timer.c:1502 [inline]
 __run_timers+0x72a/0xa10 kernel/time/timer.c:1773
 run_timer_softirq+0x69/0xf0 kernel/time/timer.c:1786
 handle_softirqs+0x25e/0x5c0 kernel/softirq.c:565
 __do_softirq kernel/softirq.c:603 [inline]
 invoke_softirq kernel/softirq.c:425 [inline]
 __irq_exit_rcu+0x52/0xf0 kernel/softirq.c:652
 irq_exit_rcu+0x9/0x10 kernel/softirq.c:664
 sysvec_apic_timer_interrupt+0x9a/0xc0 arch/x86/kernel/apic/apic.c:1096
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1b/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:update_stack_state+0x3bf/0x460 arch/x86/kernel/unwind_frame.c:242
Code: 67 a2 7d 00 65 48 8b 05 df 68 cd 7e 49 39 06 74 0d 48 89 df e8 42 fc ff ff 49 89 c6 eb 19 48 89 d8 48 c1 e8 03 42 80 3c 38 00 <74> 08 48 89 df e8 37 a2 7d 00 4c 8b 33 48 8b 5d d0 48 83 c3 48 48
RSP: 0018:ffffc90000eb7210 EFLAGS: 00000246
RAX: 1ffff920001d6fd7 RBX: ffffc90000eb7eb8 RCX: 1ffff920001d6e77
RDX: 1ffff920001d6e6e RSI: ffffc90000eb7eb0 RDI: ffffc90000eb73b8
RBP: ffffc90000eb72c8 R08: ffffc90000eb7448 R09: 000000000000000c
R10: ffffc90000eb7450 R11: dffffc0000000001 R12: 1ffff920001d6e74
R13: 0000000000000000 R14: ffffc90000eb7388 R15: dffffc0000000000
 unwind_next_frame+0x3cb/0x700 arch/x86/kernel/unwind_frame.c:305
 arch_stack_walk+0x10d/0x140 arch/x86/kernel/stacktrace.c:25
 stack_trace_save+0x113/0x1c0 kernel/stacktrace.c:122
 save_stack+0xf6/0x1e0 mm/page_owner.c:139
 __reset_page_owner+0x36/0x190 mm/page_owner.c:156
 reset_page_owner include/linux/page_owner.h:26 [inline]
 free_pages_prepare mm/page_alloc.c:1472 [inline]
 free_pcp_prepare mm/page_alloc.c:1544 [inline]
 free_unref_page_prepare+0x7c8/0x7d0 mm/page_alloc.c:3534
 free_unref_page+0xe8/0x750 mm/page_alloc.c:3616
 free_the_page mm/page_alloc.c:805 [inline]
 __free_pages+0x61/0xf0 mm/page_alloc.c:5855
 __vunmap+0x7bc/0x8f0 mm/vmalloc.c:2652
 __vfree mm/vmalloc.c:2700 [inline]
 vfree+0x7f/0xb0 mm/vmalloc.c:2731
 kcov_put kernel/kcov.c:408 [inline]
 kcov_close+0x2b/0x50 kernel/kcov.c:510
 __fput+0x3fe/0x910 fs/file_table.c:280
 ____fput+0x15/0x20 fs/file_table.c:308
 task_work_run+0x129/0x190 kernel/task_work.c:164
 exit_task_work include/linux/task_work.h:32 [inline]
 do_exit+0xc48/0x2ca0 kernel/exit.c:878
 do_group_exit+0x141/0x310 kernel/exit.c:1000
 get_signal+0x7a3/0x1630 kernel/signal.c:2896
 arch_do_signal_or_restart+0xbd/0x1680 arch/x86/kernel/signal.c:867
 handle_signal_work kernel/entry/common.c:154 [inline]
 exit_to_user_mode_loop+0xa0/0xe0 kernel/entry/common.c:178
 exit_to_user_mode_prepare+0x5a/0xa0 kernel/entry/common.c:214
 __syscall_exit_to_user_mode_work kernel/entry/common.c:296 [inline]
 syscall_exit_to_user_mode+0x26/0x160 kernel/entry/common.c:307
 do_syscall_64+0x47/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x66/0xd0
RIP: 0033:0x7fb1b89ceef9
Code: Unable to access opcode bytes at RIP 0x7fb1b89ceecf.
RSP: 002b:00007fb1b76060e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: 0000000000000001 RBX: 00007fb1b8b87138 RCX: 00007fb1b89ceef9
RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007fb1b8b8713c
RBP: 00007fb1b8b87130 R08: 00007ffea5ca50b0 R09: 0000000000000000
R10: 0000000000000008 R11: 0000000000000246 R12: 00007fb1b8b8713c
R13: 0000000000000000 R14: 00007ffea5c5ad20 R15: 00007ffea5c5ae08
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/09/17 23:45 android13-5.15-lts b92c0d35d015 c673ca06 .config console log report info [disk image] [vmlinux] [kernel image] ci2-android-5-15-perf BUG: soft lockup in __run_timers
* Struck through repros no longer work on HEAD.