INFO: rcu detected stall in run_timer_softirq (3)

Status: auto-closed as invalid on 2020/08/30 01:43
Subsystems: netfilter
[Documentation on labels]
Reported-by: syzbot+8878cb4b6ee2945a51a6@syzkaller.appspotmail.com
First crash: 2531d, last: 2297d
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
INFO: rcu detected stall in run_timer_softirq (3) 0 (1) 2020/07/06 15:12
▶ ▼ Similar bugs (4)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in run_timer_softirq net 1 1 2898d 2897d 0/29 closed as invalid on 2018/09/11 09:35
linux-4.14 INFO: rcu detected stall in run_timer_softirq 1 3 2389d 2412d 0/1 auto-closed as invalid on 2020/05/30 20:42
linux-4.19 INFO: rcu detected stall in run_timer_softirq 1 9 2298d 2403d 0/1 auto-closed as invalid on 2020/08/29 05:43
upstream INFO: rcu detected stall in run_timer_softirq (2) net 1 27 2813d 2893d 0/29 closed as dup on 2019/01/02 16:38

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	0-...!: (1 GPs behind) idle=18e/1/0x4000000000000004 softirq=258758/258759 fqs=0 
	(t=10501 jiffies g=344461 q=14)
rcu: rcu_preempt kthread starved for 10502 jiffies! g344461 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: RCU grace-period kthread stack dump:
rcu_preempt     R  running task    29264    10      2 0x80004000
Call Trace:
 context_switch kernel/sched/core.c:3386 [inline]
 __schedule+0x934/0x1f90 kernel/sched/core.c:4082
 schedule+0xdc/0x2b0 kernel/sched/core.c:4156
 schedule_timeout+0x486/0xc50 kernel/time/timer.c:1895
 rcu_gp_fqs_loop kernel/rcu/tree.c:1658 [inline]
 rcu_gp_kthread+0xa10/0x1940 kernel/rcu/tree.c:1818
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
NMI backtrace for cpu 0
CPU: 0 PID: 14248 Comm: syz-executor.4 Not tainted 5.6.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x197/0x210 lib/dump_stack.c:118
 nmi_cpu_backtrace.cold+0x70/0xb2 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x23b/0x28b lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x183/0x1cf kernel/rcu/tree_stall.h:254
 print_cpu_stall kernel/rcu/tree_stall.h:475 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:549 [inline]
 rcu_pending kernel/rcu/tree.c:3030 [inline]
 rcu_sched_clock_irq.cold+0x51a/0xc37 kernel/rcu/tree.c:2276
 update_process_times+0x2d/0x70 kernel/time/timer.c:1726
 tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:171
 tick_sched_timer+0x53/0x140 kernel/time/tick-sched.c:1314
 __run_hrtimer kernel/time/hrtimer.c:1517 [inline]
 __hrtimer_run_queues+0x364/0xe40 kernel/time/hrtimer.c:1579
 hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1641
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1119 [inline]
 smp_apic_timer_interrupt+0x160/0x610 arch/x86/kernel/apic/apic.c:1144
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829
RIP: 0010:expire_timers kernel/time/timer.c:1450 [inline]
RIP: 0010:__run_timers kernel/time/timer.c:1773 [inline]
RIP: 0010:__run_timers kernel/time/timer.c:1740 [inline]
RIP: 0010:run_timer_softirq+0x6cd/0x1790 kernel/time/timer.c:1786
Code: ff e8 57 49 10 00 4c 89 f7 e8 7f 8d 89 06 48 8b 95 10 ff ff ff 48 89 de 4c 89 e7 e8 fd c1 ff ff 48 8b 85 20 ff ff ff 80 38 00 <0f> 85 3e 0e 00 00 49 c7 46 38 00 00 00 00 4c 89 f7 e8 1d 8f 89 06
RSP: 0018:ffffc90000007de8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: ffffed1015d04e87 RBX: ffffffff86511890 RCX: ffffffff81650902
RDX: 0000000000000100 RSI: ffffffff81650960 RDI: 0000000000000004
RBP: ffffc90000007f08 R08: ffff8880326f60c0 R09: ffff8880326f6950
R10: fffffbfff154b438 R11: ffffffff8aa5a1c7 R12: ffff8880a7a00c10
R13: ffff8880a7a00c18 R14: ffff8880ae827400 R15: dffffc0000000000
 __do_softirq+0x262/0x98c kernel/softirq.c:292
 invoke_softirq kernel/softirq.c:373 [inline]
 irq_exit+0x19b/0x1e0 kernel/softirq.c:413
 exiting_irq arch/x86/include/asm/apic.h:546 [inline]
 smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1146
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829
 </IRQ>
RIP: 0010:preempt_count arch/x86/include/asm/preempt.h:26 [inline]
RIP: 0010:check_kcov_mode kernel/kcov.c:153 [inline]
RIP: 0010:write_comp_data+0x9/0x70 kernel/kcov.c:208
Code: 13 00 00 8b 80 84 13 00 00 48 8b 11 48 83 c2 01 48 39 d0 76 07 48 89 34 d1 48 89 11 5d c3 0f 1f 00 65 4c 8b 04 25 c0 1e 02 00 <65> 8b 05 28 90 8c 7e a9 00 01 1f 00 75 51 41 8b 80 80 13 00 00 83
RSP: 0018:ffffc9000614f180 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000002 RBX: 0000000000000002 RCX: ffffffff86707f51
RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000005
RBP: ffffc9000614f1a0 R08: ffff8880326f60c0 R09: fffffbfff16a3382
R10: fffffbfff16a3381 R11: ffffffff8b519c0f R12: ffffffff88fdcfc0
R13: 0000000000000003 R14: ffff88808941bdc0 R15: dffffc0000000000
 nf_hook_slow+0xd1/0x1e0 net/netfilter/core.c:513
 nf_hook include/linux/netfilter.h:262 [inline]
 __ip_local_out+0x403/0x870 net/ipv4/ip_output.c:114
 ip_local_out+0x2d/0x1b0 net/ipv4/ip_output.c:123
 __ip_queue_xmit+0x878/0x1c20 net/ipv4/ip_output.c:530
 sctp_v4_xmit+0x1a8/0x200 net/sctp/protocol.c:981
 sctp_packet_transmit+0x1ba6/0x3740 net/sctp/output.c:629
 sctp_outq_flush_transports net/sctp/outqueue.c:1147 [inline]
 sctp_outq_flush+0x2b8/0x2780 net/sctp/outqueue.c:1195
 sctp_outq_uncork+0x6c/0x80 net/sctp/outqueue.c:758
 sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1793 [inline]
 sctp_side_effects net/sctp/sm_sideeffect.c:1185 [inline]
 sctp_do_sm+0x50d/0x5330 net/sctp/sm_sideeffect.c:1156
 sctp_primitive_REQUESTHEARTBEAT+0xa0/0xd0 net/sctp/primitive.c:185
 sctp_apply_peer_addr_params+0x114/0x1f20 net/sctp/socket.c:2430
 sctp_setsockopt_peer_addr_params+0x4c7/0x590 net/sctp/socket.c:2684
 sctp_setsockopt net/sctp/socket.c:4716 [inline]
 sctp_setsockopt+0x2321/0x7350 net/sctp/socket.c:4655
 sock_common_setsockopt+0x94/0xd0 net/core/sock.c:3149
 __sys_setsockopt+0x261/0x4c0 net/socket.c:2130
 __do_sys_setsockopt net/socket.c:2146 [inline]
 __se_sys_setsockopt net/socket.c:2143 [inline]
 __x64_sys_setsockopt+0xbe/0x150 net/socket.c:2143
 do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45b3b9
Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f6fef7b7c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
RAX: ffffffffffffffda RBX: 00007f6fef7b86d4 RCX: 000000000045b3b9
RDX: 0000000000000009 RSI: 0000000000000084 RDI: 0000000000000003
RBP: 000000000075bfc8 R08: 000000000000009c R09: 0000000000000000
R10: 0000000020000300 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000000000000a9f R14: 00000000004d5208 R15: 000000000075bfd4

Crashes (32):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/02/11 09:05 upstream 0a679e13ea30 084454ae .config console log report ci-upstream-kasan-gce
2020/02/10 02:20 upstream d1ea35f4cdd4 35f5e45e .config console log report ci-upstream-kasan-gce
2020/01/27 05:15 upstream a45ea48e2bcd dd56146d .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/16 01:10 upstream 51d69817519f f9b69507 .config console log report ci-upstream-kasan-gce
2020/01/12 20:58 upstream 6327edceb62b 31290a45 .config console log report ci-upstream-kasan-gce
2020/01/09 11:06 upstream b07f636fca1c ddc3e859 .config console log report ci-upstream-kasan-gce
2019/11/29 12:24 upstream 81b6b96475ac d29b9e84 .config console log report ci-qemu-upstream
2019/11/29 12:19 upstream 81b6b96475ac d29b9e84 .config console log report ci-qemu-upstream
2019/11/29 12:16 upstream 81b6b96475ac d29b9e84 .config console log report ci-qemu-upstream
2019/11/29 12:06 upstream 81b6b96475ac d29b9e84 .config console log report ci-qemu-upstream
2019/11/29 10:59 upstream 81b6b96475ac 4f7e1d0f .config console log report ci-qemu-upstream
2019/11/29 10:12 upstream 81b6b96475ac 4f7e1d0f .config console log report ci-qemu-upstream
2019/11/29 10:09 upstream 81b6b96475ac 4f7e1d0f .config console log report ci-qemu-upstream
2019/10/16 08:23 upstream 3b1f00aceb7a d4ea592f .config console log report ci-upstream-kasan-gce-root
2019/10/09 14:21 upstream e3280b54afed 312c6a5a .config console log report ci-upstream-kasan-gce
2019/09/30 13:48 upstream 97f9a3c4eee5 c7a4fb99 .config console log report ci-upstream-kasan-gce-root
2019/09/21 21:04 upstream 227c3e9eb5cf d96e88f3 .config console log report ci-upstream-kasan-gce
2019/09/21 09:53 upstream f97c81dc6ca5 d96e88f3 .config console log report ci-upstream-kasan-gce-selinux-root
2019/09/20 01:06 upstream 3c2edc36a774 4d3ae0b7 .config console log report ci-upstream-kasan-gce-selinux-root
2019/09/19 17:39 upstream b41dae061bbd eb940044 .config console log report ci-upstream-kasan-gce-root
2019/09/19 03:26 upstream d013cc800a2a 46c0be24 .config console log report ci-upstream-kasan-gce-selinux-root
2019/09/18 17:12 upstream 35f7a9526615 c2dcd700 .config console log report ci-upstream-kasan-gce-root
2019/09/17 04:22 upstream cef7298262e9 51ca0454 .config console log report ci-upstream-kasan-gce-root
2019/09/15 02:31 upstream 1609d7604b84 32d59357 .config console log report ci-upstream-kasan-gce
2019/09/12 07:04 upstream 3120b9a6a3f7 f4e53c10 .config console log report ci-upstream-kasan-gce
2019/09/11 05:07 upstream 3120b9a6a3f7 a60cb4cd .config console log report ci-upstream-kasan-gce
2020/05/02 01:43 upstream 052c467cb587 bc734e7a .config console log report ci-upstream-kasan-gce-386
2020/02/14 01:57 upstream 0bf999f9c5e7 c5ed587f .config console log report ci-upstream-kasan-gce-386
2020/02/13 06:17 upstream f2850dd5ee01 84f4fc8a .config console log report ci-upstream-kasan-gce-386
2020/01/16 01:21 upstream 51d69817519f f9b69507 .config console log report ci-upstream-kasan-gce-386
2019/12/04 10:15 upstream 63de37476ebd 0ecb9746 .config console log report ci-qemu-upstream-386
2019/10/17 20:23 linux-next 3ef845da3c3b 8c88c9c1 .config console log report ci-upstream-linux-next-kasan-gce-root
* Struck through repros no longer work on HEAD.