INFO: rcu detected stall in advance

ID	Workflow	Result	Correct	Bug	Created	Started	Finished	Revision	Error
a6793476-f33a-4942-9e72-80a7696c9105	repro		💥	INFO: rcu detected stall in advance_sched	2026/03/07 04:31	2026/03/07 04:31	2026/03/07 04:47	31e9c887f7dc24e04b3ca70d0d54fc34141844b0	Error 504, Message: Cancelled while waiting for stream data; Failed to close the streaming context; status = CANCELLED: Cancelled while waiting for stream data === Source Location Trace: === learning/brain/research/stairways/orchestrator/public/stream.h:459 learning/serving/servables/wiz/orch_wiz_servable.cc:1037 ; Failed to run inference for model: go/debugproto name: "prod-common-global__/aistudio/gemini-v4m-rev22-riftrunner-sc__main__/aistudio/gemini-v4m-rev22-riftrunner-sc__2026020501__prefill__variant__d9ef83c4-247c-43cf-a2bf-d65477f9516e" version { value: 1 } signature_name: "serving_default" , Status: DEADLINE_EXCEEDED, Details: [map[@type:type.googleapis.com/google.rpc.DebugInfo detail:[ORIGINAL ERROR] generic::deadline_exceeded: [original: beyond::dependency::DEADLINE_EXCEEDED] Deadline exceeded. Check error_details_ext for the original status returned by handler.; [model=/aistudio/gemini-v4m-rev22-riftrunner-sc,target=PredictionService]; for endpoint: [google.rpc.error_details_ext] { code: 1 message: "Cancelled while waiting for stream data; Failed to close the streaming context; status = CANCELLED: Cancelled while waiting for stream data\n=== Source Location Trace: === \nlearning/brain/research/stairways/orchestrator/public/stream.h:459\nlearning/serving/servables/wiz/orch_wiz_servable.cc:1037\n; Failed to run inference for model: go/debugproto \nname: \"prod-common-global__/aistudio/gemini-v4m-rev22-riftrunner-sc__main__/aistudio/gemini-v4m-rev22-riftrunner-sc__2026020501__prefill__variant__d9ef83c4-247c-43cf-a2bf-d65477f9516e\"\nversion {\n value: 1\n}\nsignature_name: \"serving_default\"\n" details { type_url: "type.googleapis.com/language_labs.genai.debug.GeminiApiDebugInfo" value: "\222\001\323\001\n\263\001Deadline exceeded. Check error_details_ext for the original status returned by handler.; [model=/aistudio/gemini-v4m-rev22-riftrunner-sc,target=PredictionService]; for endpoint: \022\033net/rpc/rpc-status.cc:562:0" } } 525002922 { 3 { 1: "/aistudio/gemini-v4m-rev22-riftrunner-sc" } 6 { 5 { 1 { 1: "/aistudio/gemini-v4m-rev22-riftrunner-sc" 3: "serving_default" } 2: 131072 3: 1 } 6 { 2: 1 3: 0 4: "Streaming didn\'t start, relying on Uniserve retries: Deadline exceeded. Check error_details_ext for the original status returned by handler.; [model=/aistudio/gemini-v4m-rev22-riftrunner-sc,target=PredictionService]; for endpoint: " 5 { 1 { 1: 1772858276 2: 890517344 } 2 { 1: 597 2: 959072966 } } 6: "\000" } } }]]

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: rcu: Tasks blocked on level-0 rcu_node (CPUs 0-1): P29 rcu: (detected by 1, t=10502 jiffies, g=84453, q=3632 ncpus=2) task:ktimers/1 state:R running task stack:22880 pid:29 tgid:29 ppid:2 task_flags:0x4208040 flags:0x00080000 Call Trace: <IRQ> sched_show_task+0x49d/0x630 kernel/sched/core.c:7819 rcu_print_detail_task_stall_rnp kernel/rcu/tree_stall.h:292 [inline] print_other_cpu_stall+0xf78/0x1340 kernel/rcu/tree_stall.h:681 check_cpu_stall kernel/rcu/tree_stall.h:856 [inline] rcu_pending kernel/rcu/tree.c:3667 [inline] rcu_sched_clock_irq+0x9a9/0x1140 kernel/rcu/tree.c:2704 update_process_times+0x235/0x2d0 kernel/time/timer.c:2474 tick_sched_handle kernel/time/tick-sched.c:298 [inline] tick_nohz_handler+0x3e9/0x710 kernel/time/tick-sched.c:319 __run_hrtimer kernel/time/hrtimer.c:1777 [inline] __hrtimer_run_queues+0x4f6/0xd00 kernel/time/hrtimer.c:1841 hrtimer_interrupt+0x45d/0xa90 kernel/time/hrtimer.c:1903 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1045 [inline] __sysvec_apic_timer_interrupt+0x102/0x3e0 arch/x86/kernel/apic/apic.c:1062 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1056 [inline] sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1056 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697 RIP: 0010:lock_acquire+0x222/0x340 kernel/locking/lockdep.c:5872 Code: ff ff ff e8 d0 80 35 09 f7 44 24 08 00 02 00 00 0f 84 3a ff ff ff 65 48 8b 05 aa bf 16 10 48 3b 44 24 58 75 33 fb 48 83 c4 60 <5b> 41 5c 41 5d 41 5e 41 5f 5d e9 4f 3f 39 09 cc 48 8d 3d a7 d6 41 RSP: 0018:ffffc90000a3f8e0 EFLAGS: 00000282 RAX: 44c50bd0e83b0c00 RBX: 0000000000000000 RCX: 0000000000000046 RDX: 00000000ec1a1bfd RSI: ffffffff8cfe08a2 RDI: ffffffff8b3f5be0 RBP: ffffffff8ad45021 R08: ffffffff8ad45021 R09: ffffffff8d5ae940 R10: dffffc0000000000 R11: fffffbfff1db686f R12: 0000000000000002 R13: ffffffff8d5ae940 R14: 0000000000000000 R15: 0000000000000246 rcu_lock_acquire include/linux/rcupdate.h:331 [inline] rcu_read_lock include/linux/rcupdate.h:867 [inline] __rt_spin_lock kernel/locking/spinlock_rt.c:50 [inline] rt_spin_lock+0x1dd/0x3e0 kernel/locking/spinlock_rt.c:57 spin_lock include/linux/spinlock_rt.h:44 [inline] advance_sched+0xca/0xc90 net/sched/sch_taprio.c:930 __run_hrtimer kernel/time/hrtimer.c:1777 [inline] __hrtimer_run_queues+0x542/0xd00 kernel/time/hrtimer.c:1841 hrtimer_run_softirq+0x1a3/0x2e0 kernel/time/hrtimer.c:1858 handle_softirqs+0x1df/0x650 kernel/softirq.c:622 __do_softirq kernel/softirq.c:656 [inline] run_ktimerd+0x69/0x100 kernel/softirq.c:1138 smpboot_thread_fn+0x542/0xa60 kernel/smpboot.c:160 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x510/0xa50 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK>

Crashes (14):
Time	Kernel	Commit	Syzkaller	Config	Log	Report	VM info	Assets (help?)	Manager	Title
2026/01/16 10:57	upstream	603c05a1639f	d6526ea3	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2026/01/11 01:53	upstream	97313d6113ab	d6526ea3	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2026/01/02 04:00	upstream	b69053dd3ffb	d6526ea3	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/12/05 14:12	upstream	2061f18ad76e	d6526ea3	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/28 22:41	upstream	6cf62f0174de	d6526ea3	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/28 09:11	upstream	e538109ac71d	d6526ea3	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/26 12:36	upstream	30f09200cc4a	d6526ea3	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/25 18:27	upstream	8a2bcda5e139	64219f15	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/18 03:08	upstream	e7c375b18160	ef766cd7	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/17 19:13	upstream	6a23ae0a96a6	ef766cd7	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/17 10:57	upstream	6a23ae0a96a6	ef766cd7	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/11/17 07:45	upstream	6a23ae0a96a6	f7988ea4	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/10/31 09:55	upstream	d127176862a9	2c50b6a9	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched
2025/10/30 14:28	upstream	e53642b87a4f	fd2207e7	.config	console log	report	info	[disk image] [vmlinux] [kernel image]	ci-upstream-kasan-gce-smack-root	INFO: rcu detected stall in advance_sched

Crashes (14):

Assets (help?)