INFO: rcu detected stall in discover_timer

Status: upstream: reported C repro on 2020/01/05 20:03
Reported-by: syzbot+c9f96b046ae341acce55@syzkaller.appspotmail.com
First crash: 2412d, last: 2200d
Fix bisection: failed (error log, bisect log)
  
▶ ▼ Similar bugs (3)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in discover_timer block 1 19 2710d 2887d 0/29 closed as dup on 2019/01/02 16:30
upstream INFO: rcu detected stall in discover_timer (2) block 1 7 2445d 2508d 0/29 auto-closed as invalid on 2020/03/03 10:41
linux-4.19 INFO: rcu detected stall in discover_timer 1 1 2524d 2524d 0/1 auto-closed as invalid on 2020/01/14 13:25
▶ ▼ Last patch testing requests (2)
Created Duration User Patch Repo Result
2023/01/29 20:32 14m retest repro linux-4.14.y report log
2022/09/10 00:27 9m retest repro linux-4.14.y report log

Sample crash report:
INFO: rcu_preempt self-detected stall on CPU
	1-...: (1 GPs behind) idle=df2/140000000000002/0 softirq=18648/18649 fqs=1 
	 (t=10500 jiffies g=1586 c=1585 q=2596)
rcu_preempt kthread starved for 10498 jiffies! g1586 c1585 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
rcu_preempt     R  running task    29776     8      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2808 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3384
 schedule+0x92/0x1c0 kernel/sched/core.c:3428
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1746
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
NMI backtrace for cpu 1
CPU: 1 PID: 7698 Comm: syz-executor487 Not tainted 4.14.169-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x142/0x197 lib/dump_stack.c:58
 nmi_cpu_backtrace.cold+0x57/0x94 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x141/0x189 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:158 [inline]
 rcu_dump_cpu_stacks+0x186/0x1d2 kernel/rcu/tree.c:1396
 print_cpu_stall kernel/rcu/tree.c:1542 [inline]
 check_cpu_stall kernel/rcu/tree.c:1610 [inline]
 __rcu_pending kernel/rcu/tree.c:3390 [inline]
 rcu_pending kernel/rcu/tree.c:3452 [inline]
 rcu_check_callbacks.cold+0x43d/0xd0a kernel/rcu/tree.c:2792
 update_process_times+0x31/0x70 kernel/time/timer.c:1590
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:165
 tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1223
 __run_hrtimer kernel/time/hrtimer.c:1223 [inline]
 __hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1287
 hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1321
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
 smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
RIP: 0010:__phys_addr+0x1/0xe0 arch/x86/mm/physaddr.c:15
RSP: 0018:ffff8880aed07b68 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff10
RAX: ffff8880a95e00c0 RBX: 1ffff11015da0f74 RCX: 0000000000000007
RDX: 0000000000000100 RSI: ffff888095301948 RDI: ffff8880a8458d80
RBP: ffff8880aed07b88 R08: 000000002ac14993 R09: 0000000000000003
R10: 0000000000000000 R11: ffff8880a95e00c0 R12: ffff8880a8458d80
R13: 00000000ffffffff R14: ffff88808b8e62c0 R15: ffff8880a9e19900
 __alloc_skb+0xe8/0x500 net/core/skbuff.c:212
 alloc_skb include/linux/skbuff.h:980 [inline]
 new_skb+0x28/0x1d0 drivers/block/aoe/aoecmd.c:67
 aoecmd_cfg_pkts drivers/block/aoe/aoecmd.c:427 [inline]
 aoecmd_cfg+0x180/0x5c0 drivers/block/aoe/aoecmd.c:1392
 discover_timer drivers/block/aoe/aoemain.c:44 [inline]
 discover_timer+0xcd/0x170 drivers/block/aoe/aoemain.c:21
 call_timer_fn+0x161/0x670 kernel/time/timer.c:1279
 expire_timers kernel/time/timer.c:1318 [inline]
 __run_timers kernel/time/timer.c:1636 [inline]
 __run_timers kernel/time/timer.c:1604 [inline]
 run_timer_softirq+0x5b7/0x1520 kernel/time/timer.c:1649
 __do_softirq+0x244/0x9a0 kernel/softirq.c:288
 invoke_softirq kernel/softirq.c:368 [inline]
 irq_exit+0x160/0x1b0 kernel/softirq.c:409
 exiting_irq arch/x86/include/asm/apic.h:648 [inline]
 smp_apic_timer_interrupt+0x146/0x5e0 arch/x86/kernel/apic/apic.c:1102
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
 </IRQ>
RIP: 0010:setfl fs/fcntl.c:80 [inline]
RIP: 0010:do_fcntl+0x943/0xe10 fs/fcntl.c:347
RSP: 0018:ffff88809c87fdf0 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff10
RAX: ffffffff819382d3 RBX: ffffffff816b3630 RCX: 000000007eab9e99
RDX: 1ffff1101537b9a0 RSI: ffff8880a95e0940 RDI: ffff8880a9bdccf4
RBP: ffff88809c87fea8 R08: 0000000000002057 R09: ffffffff8955fb38
R10: ffff8880a95e0940 R11: ffff8880a95e00c0 R12: 1ffff1101390ffc0
R13: ffff8880a9bdccc0 R14: 0000000000040000 R15: ffff8880a9bdccf0
 SYSC_fcntl fs/fcntl.c:463 [inline]
 SyS_fcntl+0xd5/0x110 fs/fcntl.c:448
 do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x4486e9
RSP: 002b:00007ff6c0268db8 EFLAGS: 00000246 ORIG_RAX: 0000000000000048
RAX: ffffffffffffffda RBX: 00000000006ddc28 RCX: 00000000004486e9
RDX: 0000000000042000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: 00000000006ddc20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006ddc2c
R13: 00007fff3f44bb4f R14: 00007ff6c02699c0 R15: 00000000006ddc2c
INFO: rcu_sched detected stalls on CPUs/tasks:
	1-...: (1 GPs behind) idle=df2/140000000000002/0 softirq=18648/18649 fqs=2 
	(detected by 0, t=10551 jiffies, g=1081, c=1080, q=67)
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 7698 Comm: syz-executor487 Not tainted 4.14.169-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
task: ffff8880a95e00c0 task.stack: ffff88809c878000
RIP: 0010:llist_add_batch+0x5e/0xb0 lib/llist.c:45
RSP: 0018:ffff8880aed07558 EFLAGS: 00000046
RAX: 0000000000000000 RBX: ffff888088ba6838 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff888088ba6838 RDI: ffff888088ba6838
RBP: ffff8880aed07588 R08: ffff8880a95e00c0 R09: 0000000000000003
R10: 0000000000000000 R11: ffff8880a95e00c0 R12: ffff888088ba6838
R13: ffffed1011174d07 R14: ffffed1015da4f19 R15: ffff8880aed278c8
FS:  00007ff6c0269700(0000) GS:ffff8880aed00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000002000056c CR3: 000000008b9b5000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 llist_add include/linux/llist.h:221 [inline]
 irq_work_queue kernel/irq_work.c:102 [inline]
 irq_work_queue+0x194/0x1f0 kernel/irq_work.c:87
 __perf_event_overflow+0x2a8/0x330 kernel/events/core.c:7522
 perf_swevent_hrtimer+0x220/0x350 kernel/events/core.c:8728
 __run_hrtimer kernel/time/hrtimer.c:1223 [inline]
 __hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1287
 hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1321
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
 smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
RIP: 0010:arch_local_save_flags arch/x86/include/asm/paravirt.h:774 [inline]
RIP: 0010:arch_local_irq_save arch/x86/include/asm/paravirt.h:796 [inline]
RIP: 0010:slab_alloc_node mm/slab.c:3302 [inline]
RIP: 0010:kmem_cache_alloc_node_trace+0x90/0x770 mm/slab.c:3659
RSP: 0018:ffff8880aed07aa0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff10
RAX: 0000000000000286 RBX: 0000000001090220 RCX: 0000000000000240
RDX: 0000000000000100 RSI: 0000000001090220 RDI: ffff8880aa800ac0
RBP: ffff8880aed07b18 R08: 0000000000000000 R09: 0000000000000003
R10: 0000000000000000 R11: ffff8880a95e00c0 R12: ffff8880aa800ac0
R13: ffff8880aa800ac0 R14: 0000000000000000 R15: 0000000001090220
 __do_kmalloc_node mm/slab.c:3681 [inline]
 __kmalloc_node_track_caller+0x3d/0x80 mm/slab.c:3696
 __kmalloc_reserve.isra.0+0x40/0xe0 net/core/skbuff.c:137
 __alloc_skb+0xcf/0x500 net/core/skbuff.c:205
 alloc_skb include/linux/skbuff.h:980 [inline]
 new_skb+0x28/0x1d0 drivers/block/aoe/aoecmd.c:67
 aoecmd_cfg_pkts drivers/block/aoe/aoecmd.c:427 [inline]
 aoecmd_cfg+0x180/0x5c0 drivers/block/aoe/aoecmd.c:1392
 discover_timer drivers/block/aoe/aoemain.c:44 [inline]
 discover_timer+0xcd/0x170 drivers/block/aoe/aoemain.c:21
 call_timer_fn+0x161/0x670 kernel/time/timer.c:1279
 expire_timers kernel/time/timer.c:1318 [inline]
 __run_timers kernel/time/timer.c:1636 [inline]
 __run_timers kernel/time/timer.c:1604 [inline]
 run_timer_softirq+0x5b7/0x1520 kernel/time/timer.c:1649
 __do_softirq+0x244/0x9a0 kernel/softirq.c:288
 invoke_softirq kernel/softirq.c:368 [inline]
 irq_exit+0x160/0x1b0 kernel/softirq.c:409
 exiting_irq arch/x86/include/asm/apic.h:648 [inline]
 smp_apic_timer_interrupt+0x146/0x5e0 arch/x86/kernel/apic/apic.c:1102
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
 </IRQ>
RIP: 0010:setfl fs/fcntl.c:80 [inline]
RIP: 0010:do_fcntl+0x943/0xe10 fs/fcntl.c:347
RSP: 0018:ffff88809c87fdf0 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff10
RAX: ffffffff819382d3 RBX: ffffffff816b3630 RCX: 000000007eab9e99
RDX: 1ffff1101537b9a0 RSI: ffff8880a95e0940 RDI: ffff8880a9bdccf4
RBP: ffff88809c87fea8 R08: 0000000000002057 R09: ffffffff8955fb38
R10: ffff8880a95e0940 R11: ffff8880a95e00c0 R12: 1ffff1101390ffc0
R13: ffff8880a9bdccc0 R14: 0000000000040000 R15: ffff8880a9bdccf0
 SYSC_fcntl fs/fcntl.c:463 [inline]
 SyS_fcntl+0xd5/0x110 fs/fcntl.c:448
 do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x4486e9
RSP: 002b:00007ff6c0268db8 EFLAGS: 00000246 ORIG_RAX: 0000000000000048
RAX: ffffffffffffffda RBX: 00000000006ddc28 RCX: 00000000004486e9
RDX: 0000000000042000 RSI: 0000000000000004 RDI: 0000000000000003
RBP: 00000000006ddc20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006ddc2c
R13: 00007fff3f44bb4f R14: 00007ff6c02699c0 R15: 00000000006ddc2c
Code: 00 fc ff df 49 01 c6 49 01 c5 e8 7e 80 50 fe 41 80 3e 00 75 3d 41 80 7d 00 00 49 8b 17 75 3d 49 89 14 24 48 89 d0 f0 49 0f b1 1f <48> 39 c2 75 da 48 89 55 d0 e8 54 80 50 fe 48 8b 55 d0 48 85 d2

Crashes (16):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/02/05 07:32 linux-4.14.y 9fa690a2a016 93e5e335 .config console log report syz C ci2-linux-4-14
2020/08/04 21:51 linux-4.14.y 7f2c5eb458b8 02034dac .config console log report ci2-linux-4-14
2020/05/06 13:57 linux-4.14.y d71f695ce745 4618eb2d .config console log report ci2-linux-4-14
2020/04/27 18:41 linux-4.14.y 050272a0423e 0ce7569e .config console log report ci2-linux-4-14
2020/04/03 23:36 linux-4.14.y 4520f06b03ae ef26b610 .config console log report ci2-linux-4-14
2020/03/14 15:42 linux-4.14.y 12cd844a39ed 749688d2 .config console log report ci2-linux-4-14
2020/03/14 15:40 linux-4.14.y 12cd844a39ed 749688d2 .config console log report ci2-linux-4-14
2020/03/14 14:50 linux-4.14.y 12cd844a39ed 749688d2 .config console log report ci2-linux-4-14
2020/03/03 08:32 linux-4.14.y 78d697fc93f9 350a7a26 .config console log report ci2-linux-4-14
2020/03/02 13:33 linux-4.14.y 78d697fc93f9 4a4e0509 .config console log report ci2-linux-4-14
2020/02/29 10:00 linux-4.14.y 78d697fc93f9 c88c7b75 .config console log report ci2-linux-4-14
2020/02/23 21:11 linux-4.14.y 98db2bf27b9e d801cb02 .config console log report ci2-linux-4-14
2020/02/13 13:37 linux-4.14.y e0f8b8a65a47 e6247653 .config console log report ci2-linux-4-14
2020/01/07 07:31 linux-4.14.y 84f5ad468100 53430d97 .config console log report ci2-linux-4-14
2020/01/07 06:49 linux-4.14.y 84f5ad468100 53430d97 .config console log report ci2-linux-4-14
2020/01/05 20:02 linux-4.14.y 84f5ad468100 d646e21f .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.