INFO: rcu detected stall in io_handle_tw_list

Status: auto-obsoleted due to no activity on 2025/01/28 05:01
Subsystems: io-uring
[Documentation on labels]
First crash: 673d, last: 651d
▶ ▼ Similar bugs (1)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in io_handle_tw_list (2) mm 1 14 232d 550d 0/29 auto-obsoleted due to no activity on 2026/03/24 04:28

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	1-...!: (1 GPs behind) idle=931c/1/0x4000000000000000 softirq=29846/29852 fqs=12
rcu: 	(detected by 0, t=10502 jiffies, g=25285, q=758 ncpus=2)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 7879 Comm: syz.5.466 Not tainted 6.12.0-rc5-syzkaller-00044-gc1e939a21eb1 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:get_current arch/x86/include/asm/current.h:49 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x8/0x70 kernel/kcov.c:216
Code: 8b 3d 34 28 9d 0c 48 89 de 5b e9 43 d6 5d 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 8b 04 24 <65> 48 8b 0c 25 c0 d7 03 00 65 8b 15 90 fd 6e 7e 81 e2 00 01 ff 00
RSP: 0018:ffffc90000a18d20 EFLAGS: 00000002
RAX: ffffffff8bbff8f3 RBX: 0000000000000001 RCX: ffff888034df9e00
RDX: ffff888034df9e00 RSI: ffff88807b10d340 RDI: ffff8880b872c9d0
RBP: dffffc0000000000 R08: ffffffff81823402 R09: 1ffffffff203a095
R10: dffffc0000000000 R11: fffffbfff203a096 R12: 1ffff110170e593b
R13: ffff88807b10d340 R14: ffff8880b872c9d0 R15: ffff8880b872c880
FS:  00007ff453f616c0(0000) GS:ffff8880b8700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020003c80 CR3: 000000005d1bc000 CR4: 0000000000350ef0
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 timerqueue_del+0x23/0x100 lib/timerqueue.c:54
 __remove_hrtimer kernel/time/hrtimer.c:1118 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1671 [inline]
 __hrtimer_run_queues+0x3d0/0xd50 kernel/time/hrtimer.c:1755
 hrtimer_interrupt+0x396/0x990 kernel/time/hrtimer.c:1817
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1038 [inline]
 __sysvec_apic_timer_interrupt+0x112/0x420 arch/x86/kernel/apic/apic.c:1055
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
 sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1049
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0xd8/0x140 kernel/locking/spinlock.c:194
Code: 9c 8f 44 24 20 42 80 3c 23 00 74 08 4c 89 f7 e8 6e e3 2d f6 f6 44 24 21 02 75 52 41 f7 c7 00 02 00 00 74 01 fb bf 01 00 00 00 <e8> 33 7a 92 f5 65 8b 05 64 22 33 74 85 c0 74 43 48 c7 04 24 0e 36
RSP: 0018:ffffc9000311f660 EFLAGS: 00000206
RAX: 83a0897b82304e00 RBX: 1ffff92000623ed0 RCX: ffffffff8170bfba
RDX: dffffc0000000000 RSI: ffffffff8c0acac0 RDI: 0000000000000001
RBP: ffffc9000311f700 R08: ffffffff942cc907 R09: 1ffffffff2859920
R10: dffffc0000000000 R11: fffffbfff2859921 R12: dffffc0000000000
R13: 1ffff92000623ecc R14: ffffc9000311f680 R15: 0000000000000246
 spin_unlock_irqrestore include/linux/spinlock.h:406 [inline]
 get_partial_node+0x248/0x280 mm/slub.c:2860
 get_partial mm/slub.c:2940 [inline]
 ___slab_alloc+0xc17/0x14b0 mm/slub.c:3798
 __slab_alloc+0x58/0xa0 mm/slub.c:3908
 __slab_alloc_node mm/slub.c:3961 [inline]
 slab_alloc_node mm/slub.c:4122 [inline]
 __do_kmalloc_node mm/slub.c:4263 [inline]
 __kmalloc_noprof+0x25a/0x400 mm/slub.c:4276
 kmalloc_noprof include/linux/slab.h:882 [inline]
 io_cqring_event_overflow+0xd3/0x660 io_uring/io_uring.c:737
 io_req_cqe_overflow+0xf2/0x150 io_uring/io_uring.c:767
 __io_submit_flush_completions+0x2c4/0xcf0 io_uring/io_uring.c:1452
 io_submit_flush_completions io_uring/io_uring.h:150 [inline]
 ctx_flush_and_put io_uring/io_uring.c:1035 [inline]
 io_handle_tw_list+0x473/0x500 io_uring/io_uring.c:1075
 tctx_task_work_run+0x9a/0x370 io_uring/io_uring.c:1135
 tctx_task_work+0x9a/0x100 io_uring/io_uring.c:1153
 task_work_run+0x251/0x310 kernel/task_work.c:239
 get_signal+0x15e8/0x1740 kernel/signal.c:2690
 arch_do_signal_or_restart+0x96/0x860 arch/x86/kernel/signal.c:337
 exit_to_user_mode_loop kernel/entry/common.c:111 [inline]
 exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
 __syscall_exit_to_user_mode_work kernel/entry/common.c:207 [inline]
 syscall_exit_to_user_mode+0xc9/0x370 kernel/entry/common.c:218
 do_syscall_64+0x100/0x230 arch/x86/entry/common.c:89
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7ff45317e719
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ff453f61038 EFLAGS: 00000246 ORIG_RAX: 00000000000000e3
RAX: 0000000000000000 RBX: 00007ff453335f80 RCX: 00007ff45317e719
RDX: 0000000000000000 RSI: 0000000020003c80 RDI: 0000000000000000
RBP: 00007ff4531f132e R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007ff453335f80 R15: 00007ffe36ec6bc8
 </TASK>
rcu: rcu_preempt kthread starved for 10478 jiffies! g25285 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:25920 pid:17    tgid:17    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5328 [inline]
 __schedule+0x18af/0x4bd0 kernel/sched/core.c:6690
 __schedule_loop kernel/sched/core.c:6767 [inline]
 schedule+0x14b/0x320 kernel/sched/core.c:6782
 schedule_timeout+0x1be/0x310 kernel/time/timer.c:2615
 rcu_gp_fqs_loop+0x2df/0x1330 kernel/rcu/tree.c:2045
 rcu_gp_kthread+0xa7/0x3b0 kernel/rcu/tree.c:2247
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 0 UID: 0 PID: 7882 Comm: syz.4.467 Not tainted 6.12.0-rc5-syzkaller-00044-gc1e939a21eb1 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:csd_lock_wait kernel/smp.c:340 [inline]
RIP: 0010:smp_call_function_many_cond+0x19f3/0x2ca0 kernel/smp.c:884
Code: 45 8b 65 00 44 89 e6 83 e6 01 31 ff e8 86 f3 0b 00 41 83 e4 01 49 bc 00 00 00 00 00 fc ff df 75 07 e8 31 ef 0b 00 eb 38 f3 90 <42> 0f b6 04 23 84 c0 75 11 41 f7 45 00 01 00 00 00 74 1e e8 15 ef
RSP: 0018:ffffc9000312f3e0 EFLAGS: 00000246
RAX: ffffffff8188eb0b RBX: 1ffff110170e8919 RCX: 0000000000040000
RDX: ffffc90005172000 RSI: 000000000003ffff RDI: 0000000000040000
RBP: ffffc9000312f5e0 R08: ffffffff8188eada R09: 1ffffffff2859900
R10: dffffc0000000000 R11: fffffbfff2859901 R12: dffffc0000000000
R13: ffff8880b87448c8 R14: ffff8880b863fc80 R15: 0000000000000001
FS:  00007fa8ab57b6c0(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000110c43a0bc CR3: 00000000598c6000 CR4: 0000000000350ef0
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1051
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:2085 [inline]
 text_poke_bp_batch+0x352/0xb30 arch/x86/kernel/alternative.c:2295
 text_poke_bp+0xb0/0x100 arch/x86/kernel/alternative.c:2522
 __static_call_transform+0x51a/0x810 arch/x86/kernel/static_call.c:111
 arch_static_call_transform+0x141/0x380 arch/x86/kernel/static_call.c:163
 __static_call_update+0xd8/0x5e0 kernel/static_call_inline.c:147
 tracepoint_update_call kernel/tracepoint.c:317 [inline]
 tracepoint_add_func+0x918/0x9e0 kernel/tracepoint.c:358
 tracepoint_probe_register_prio_may_exist+0x122/0x190 kernel/tracepoint.c:482
 bpf_raw_tp_link_attach+0x48b/0x6e0 kernel/bpf/syscall.c:3849
 bpf_raw_tracepoint_open+0x177/0x1f0 kernel/bpf/syscall.c:3880
 __sys_bpf+0x3c0/0x810 kernel/bpf/syscall.c:5695
 __do_sys_bpf kernel/bpf/syscall.c:5760 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:5758 [inline]
 __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5758
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fa8aa77e719
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fa8ab57b038 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007fa8aa935f80 RCX: 00007fa8aa77e719
RDX: 0000000000000010 RSI: 0000000020000600 RDI: 0000000000000011
RBP: 00007fa8aa7f132e R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fa8aa935f80 R15: 00007ffc720215f8
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/10/30 04:55 upstream c1e939a21eb1 66aeb999 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in io_handle_tw_list
2024/10/08 05:16 upstream 87d6aab2389e 402f1df0 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: rcu detected stall in io_handle_tw_list
* Struck through repros no longer work on HEAD.