INFO: rcu detected stall in trace_contention_end

Status: auto-obsoleted due to no activity on 2025/08/04 17:46
Subsystems: kernel
[Documentation on labels]
First crash: 466d, last: 466d

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 GPs behind) idle=fed4/1/0x4000000000000000 softirq=39047/39048 fqs=1
rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-1): P8717/3:b..l
rcu: 	(detected by 1, t=10502 jiffies, g=24821, q=2016 ncpus=2)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 970 Comm: kworker/0:2 Not tainted 6.15.0-rc5-syzkaller-00022-g01f95500a162 #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/29/2025
Workqueue: rcu_gp process_srcu
RIP: 0010:rcu_is_watching_curr_cpu include/linux/context_tracking.h:128 [inline]
RIP: 0010:rcu_is_watching+0xd/0xc0 kernel/rcu/tree.c:736
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 53 65 ff 05 13 35 03 12 <e8> 6e 01 cc 09 48 c7 c3 e8 3d a5 93 89 c5 83 f8 07 0f 87 82 00 00
RSP: 0018:ffffc90000007e28 EFLAGS: 00000082
RAX: 0000000080010001 RBX: ffff8880b8427940 RCX: ffffffff81a8606e
RDX: ffff888026e20000 RSI: ffffffff81a86370 RDI: 0000000000000001
RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff8880b8427840
R13: 0000000000000001 R14: ffff88802832f340 R15: ffffffff89817860
FS:  0000000000000000(0000) GS:ffff8881249df000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055557ff14808 CR3: 0000000061c1d000 CR4: 00000000003526f0
DR0: 0000000000000001 DR1: 0000000000020001 DR2: 0000000000000006
DR3: 0000000000000001 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 trace_hrtimer_expire_entry include/trace/events/timer.h:259 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1758 [inline]
 __hrtimer_run_queues+0x895/0xad0 kernel/time/hrtimer.c:1825
 hrtimer_interrupt+0x397/0x8e0 kernel/time/hrtimer.c:1887
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1038 [inline]
 __sysvec_apic_timer_interrupt+0x108/0x3f0 arch/x86/kernel/apic/apic.c:1055
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
 sysvec_apic_timer_interrupt+0x9f/0xc0 arch/x86/kernel/apic/apic.c:1049
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:trace_contention_end+0xdd/0x130 include/trace/events/lock.h:122
Code: 8b 7c 24 08 89 da 48 89 ee e8 3f df ff ff 65 ff 0d 48 03 0d 12 0f 85 38 ff ff ff e8 fd 17 95 ff e9 2e ff ff ff e8 13 ce 09 00 <84> c0 0f 85 5a ff ff ff 80 3d aa 5a d6 0e 00 0f 85 4d ff ff ff c6
RSP: 0018:ffffc90003937a10 EFLAGS: 00000282
RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff81970c51
RDX: 0000000000000000 RSI: ffffffff8bf48260 RDI: ffffffff8dcecd00
RBP: ffffc90003937b70 R08: 0000000000000000 R09: fffffbfff210a0a2
R10: ffffffff90850517 R11: 0000000000000000 R12: dffffc0000000000
R13: ffffc90003937ab0 R14: 0000000000000000 R15: 1ffff92000726f50
 __mutex_lock_common kernel/locking/mutex.c:610 [inline]
 __mutex_lock+0x1c0/0xb90 kernel/locking/mutex.c:746
 srcu_advance_state kernel/rcu/srcutree.c:1701 [inline]
 process_srcu+0x73/0x1920 kernel/rcu/srcutree.c:1861
 process_one_work+0x9cc/0x1b70 kernel/workqueue.c:3238
 process_scheduled_works kernel/workqueue.c:3319 [inline]
 worker_thread+0x6c8/0xf10 kernel/workqueue.c:3400
 kthread+0x3c2/0x780 kernel/kthread.c:464
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:153
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>
task:syz.6.567       state:R  running task     stack:26056 pid:8717  tgid:8716  ppid:6243   task_flags:0x400140 flags:0x00004002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5382 [inline]
 __schedule+0x116f/0x5de0 kernel/sched/core.c:6767
 preempt_schedule_common+0x44/0xc0 kernel/sched/core.c:6947
 preempt_schedule_thunk+0x16/0x30 arch/x86/entry/thunk.S:12
 __raw_spin_unlock include/linux/spinlock_api_smp.h:143 [inline]
 _raw_spin_unlock+0x3e/0x50 kernel/locking/spinlock.c:186
 spin_unlock include/linux/spinlock.h:391 [inline]
 filemap_map_pages+0xe2b/0x1680 mm/filemap.c:3748
 do_fault_around mm/memory.c:5476 [inline]
 do_read_fault mm/memory.c:5509 [inline]
 do_fault mm/memory.c:5652 [inline]
 do_pte_missing+0xf1a/0x3fb0 mm/memory.c:4160
 handle_pte_fault mm/memory.c:5997 [inline]
 __handle_mm_fault+0x103d/0x2a40 mm/memory.c:6140
 handle_mm_fault+0x3fe/0xad0 mm/memory.c:6309
 faultin_page mm/gup.c:1193 [inline]
 __get_user_pages+0x771/0x36f0 mm/gup.c:1491
 populate_vma_page_range+0x278/0x3a0 mm/gup.c:1929
 __mm_populate+0x1d8/0x380 mm/gup.c:2032
 mm_populate include/linux/mm.h:3487 [inline]
 vm_mmap_pgoff+0x362/0x450 mm/util.c:584
 ksys_mmap_pgoff+0x7d/0x5c0 mm/mmap.c:607
 __do_sys_mmap arch/x86/kernel/sys_x86_64.c:89 [inline]
 __se_sys_mmap arch/x86/kernel/sys_x86_64.c:82 [inline]
 __x64_sys_mmap+0x125/0x190 arch/x86/kernel/sys_x86_64.c:82
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xcd/0x260 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f835a18e969
RSP: 002b:00007f835af11038 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
RAX: ffffffffffffffda RBX: 00007f835a3b5fa0 RCX: 00007f835a18e969
RDX: b635773f06ebbeee RSI: 0000000000b36000 RDI: 0000200000000000
RBP: 00007f835a210ab1 R08: ffffffffffffffff R09: 0000000000000000
R10: 0000000000008031 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f835a3b5fa0 R15: 00007ffdcfb2cfa8
 </TASK>
rcu: rcu_preempt kthread starved for 10500 jiffies! g24821 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:28856 pid:16    tgid:16    ppid:2      task_flags:0x208040 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5382 [inline]
 __schedule+0x116f/0x5de0 kernel/sched/core.c:6767
 __schedule_loop kernel/sched/core.c:6845 [inline]
 schedule+0xe7/0x3a0 kernel/sched/core.c:6860
 schedule_timeout+0x123/0x290 kernel/time/sleep_timeout.c:99
 rcu_gp_fqs_loop+0x1ea/0xb00 kernel/rcu/tree.c:2046
 rcu_gp_kthread+0x270/0x380 kernel/rcu/tree.c:2248
 kthread+0x3c2/0x780 kernel/kthread.c:464
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:153
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 1 UID: 0 PID: 81 Comm: kworker/u8:4 Not tainted 6.15.0-rc5-syzkaller-00022-g01f95500a162 #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/29/2025
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:340 [inline]
RIP: 0010:smp_call_function_many_cond+0x4a5/0x1290 kernel/smp.c:885
Code: 89 ee e8 be 12 0c 00 85 ed 74 48 48 8b 44 24 20 49 89 c4 83 e0 07 49 c1 ec 03 48 89 c5 4d 01 f4 83 c5 03 e8 5d 17 0c 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 de 0b 00 00 8b 43 08 31
RSP: 0018:ffffc9000215f938 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b8441720 RCX: ffffffff81af1bb9
RDX: ffff88801db02440 RSI: ffffffff81af1b93 RDI: 0000000000000005
RBP: 0000000000000003 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffffed10170882e5
R13: 0000000000000001 R14: dffffc0000000000 R15: ffff8880b853b040
FS:  0000000000000000(0000) GS:ffff888124adf000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdc20ee1000 CR3: 000000000e180000 CR4: 00000000003526f0
Call Trace:
 <TASK>
 on_each_cpu_cond_mask+0x40/0x90 kernel/smp.c:1052
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:2455 [inline]
 text_poke_bp_batch+0x220/0x760 arch/x86/kernel/alternative.c:2665
 text_poke_flush arch/x86/kernel/alternative.c:2856 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:2853 [inline]
 text_poke_finish+0x30/0x40 arch/x86/kernel/alternative.c:2863
 arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x376/0x550 kernel/jump_label.c:919
 static_key_enable_cpuslocked+0x1b7/0x270 kernel/jump_label.c:210
 static_key_enable+0x1a/0x20 kernel/jump_label.c:223
 toggle_allocation_gate mm/kfence/core.c:850 [inline]
 toggle_allocation_gate+0xfa/0x280 mm/kfence/core.c:842
 process_one_work+0x9cc/0x1b70 kernel/workqueue.c:3238
 process_scheduled_works kernel/workqueue.c:3319 [inline]
 worker_thread+0x6c8/0xf10 kernel/workqueue.c:3400
 kthread+0x3c2/0x780 kernel/kthread.c:464
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:153
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/05/06 17:39 upstream 01f95500a162 ae98e6b9 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: rcu detected stall in trace_contention_end
* Struck through repros no longer work on HEAD.