INFO: rcu detected stall in inet_sendmsg

Status: auto-closed as invalid on 2020/01/06 14:34
Reported-by: syzbot+a8033fbb1f13b4f6bb8c@syzkaller.appspotmail.com
First crash: 2674d, last: 2567d
▶ ▼ Similar bugs (7)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in inet_sendmsg sctp 1 35 2517d 2821d 0/29 auto-closed as invalid on 2020/01/26 18:56
linux-4.14 INFO: rcu detected stall in inet_sendmsg (2) 1 1 2318d 2318d 0/1 auto-closed as invalid on 2020/09/11 11:53
linux-4.19 INFO: rcu detected stall in inet_sendmsg 1 3 2567d 2569d 0/1 auto-closed as invalid on 2020/01/06 19:02
upstream INFO: rcu detected stall in inet_sendmsg (4) net 1 1 1437d 1437d 0/29 auto-obsoleted due to no activity on 2023/01/16 15:35
linux-4.19 INFO: rcu detected stall in inet_sendmsg (2) 1 4 2168d 2336d 0/1 auto-closed as invalid on 2021/02/08 07:43
upstream INFO: rcu detected stall in inet_sendmsg (2) perf 1 13 2255d 2339d 15/29 fixed on 2020/07/17 17:58
upstream INFO: rcu detected stall in inet_sendmsg (3) perf 1 syz error error 68 1607d 2245d 0/29 auto-obsoleted due to no activity on 2022/10/08 22:05

Sample crash report:
INFO: rcu_preempt self-detected stall on CPU
	0-...: (10499 ticks this GP) idle=60a/140000000000001/0 softirq=16723/16723 fqs=76 
	 (t=10500 jiffies g=3050 c=3049 q=358)
rcu_preempt kthread starved for 10339 jiffies! g3050 c3049 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1
rcu_preempt     I29824     8      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2807 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
 schedule+0x92/0x1c0 kernel/sched/core.c:3427
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
NMI backtrace for cpu 0
CPU: 0 PID: 8976 Comm: syz-executor.1 Not tainted 4.14.142 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x138/0x197 lib/dump_stack.c:53
 nmi_cpu_backtrace.cold+0x57/0x94 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x141/0x189 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:158 [inline]
 rcu_dump_cpu_stacks+0x186/0x1d2 kernel/rcu/tree.c:1396
 print_cpu_stall kernel/rcu/tree.c:1542 [inline]
 check_cpu_stall kernel/rcu/tree.c:1610 [inline]
 __rcu_pending kernel/rcu/tree.c:3390 [inline]
 rcu_pending kernel/rcu/tree.c:3452 [inline]
 rcu_check_callbacks.cold+0x43d/0xd0a kernel/rcu/tree.c:2792
 update_process_times+0x31/0x70 kernel/time/timer.c:1588
 tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
 tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
 __run_hrtimer kernel/time/hrtimer.c:1220 [inline]
 __hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1284
 hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1318
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
 smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
 apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
 </IRQ>
RIP: 0010:__sanitizer_cov_trace_pc+0x41/0x60 kernel/kcov.c:87
RSP: 0018:ffff88805ab9eaa0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: 0000000000040000 RBX: ffff88808bed4238 RCX: ffffc90009458000
RDX: 000000000003ffff RSI: ffffffff84efa21b RDI: ffff88808bed42d8
RBP: ffff88805ab9eaa0 R08: 0000000000000000 R09: ffff88805fe3ad38
R10: ffff88805fe3ad18 R11: ffff88805fe3a380 R12: dffffc0000000000
R13: ffff88808bed4040 R14: ffff88808bed42d0 R15: ffff88808bed42c0
 list_add_tail include/linux/list.h:93 [inline]
 list_move_tail include/linux/list.h:183 [inline]
 hhf_dequeue+0x5eb/0xa60 net/sched/sch_hhf.c:438
 dequeue_skb net/sched/sch_generic.c:148 [inline]
 qdisc_restart net/sched/sch_generic.c:241 [inline]
 __qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
 __dev_xmit_skb net/core/dev.c:3235 [inline]
 __dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 br_dev_queue_push_xmit+0x367/0x530 net/bridge/br_forward.c:55
 NF_HOOK include/linux/netfilter.h:250 [inline]
 NF_HOOK include/linux/netfilter.h:244 [inline]
 br_forward_finish+0xbc/0x320 net/bridge/br_forward.c:67
 NF_HOOK include/linux/netfilter.h:250 [inline]
 NF_HOOK include/linux/netfilter.h:244 [inline]
 __br_forward+0x560/0x9c0 net/bridge/br_forward.c:111
 deliver_clone+0x61/0xc0 net/bridge/br_forward.c:127
 maybe_deliver net/bridge/br_forward.c:168 [inline]
 maybe_deliver net/bridge/br_forward.c:156 [inline]
 br_flood+0x3c8/0x530 net/bridge/br_forward.c:210
 br_dev_xmit+0x9a4/0xd40 net/bridge/br_device.c:83
 __netdev_start_xmit include/linux/netdevice.h:4033 [inline]
 netdev_start_xmit include/linux/netdevice.h:4042 [inline]
 xmit_one net/core/dev.c:3009 [inline]
 dev_hard_start_xmit+0x18c/0x8b0 net/core/dev.c:3025
 __dev_queue_xmit+0x1d95/0x25e0 net/core/dev.c:3525
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 neigh_resolve_output net/core/neighbour.c:1364 [inline]
 neigh_resolve_output+0x4d8/0x870 net/core/neighbour.c:1344
 neigh_output include/net/neighbour.h:500 [inline]
 ip_finish_output2+0x766/0x14a0 net/ipv4/ip_output.c:229
 ip_do_fragment+0x1999/0x2170 net/ipv4/ip_output.c:678
 ip_fragment.constprop.0+0x148/0x200 net/ipv4/ip_output.c:549
 ip_finish_output+0x7bd/0xc60 net/ipv4/ip_output.c:315
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x24a/0xd40 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:462 [inline]
 ip_local_out+0x97/0x170 net/ipv4/ip_output.c:124
 ip_send_skb+0x3e/0xc0 net/ipv4/ip_output.c:1418
 ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1438
 raw_sendmsg+0x1930/0x2450 net/ipv4/raw.c:669
 inet_sendmsg+0x122/0x500 net/ipv4/af_inet.c:762
 sock_sendmsg_nosec net/socket.c:646 [inline]
 sock_sendmsg+0xce/0x110 net/socket.c:656
 kernel_sendmsg+0x44/0x50 net/socket.c:664
 sock_no_sendpage+0x107/0x130 net/core/sock.c:2564
 inet_sendpage+0x3b8/0x580 net/ipv4/af_inet.c:780
 kernel_sendpage+0x92/0xf0 net/socket.c:3406
 sock_sendpage+0x8b/0xc0 net/socket.c:871
 pipe_to_sendpage+0x242/0x340 fs/splice.c:451
 splice_from_pipe_feed fs/splice.c:502 [inline]
 __splice_from_pipe+0x348/0x780 fs/splice.c:626
 splice_from_pipe+0xf0/0x150 fs/splice.c:661
 generic_splice_sendpage+0x3c/0x50 fs/splice.c:832
 do_splice_from fs/splice.c:851 [inline]
 do_splice fs/splice.c:1147 [inline]
 SYSC_splice fs/splice.c:1402 [inline]
 SyS_splice+0xd92/0x1430 fs/splice.c:1382
 do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x4598e9
RSP: 002b:00007fbfc9a97c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000113
RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00000000004598e9
RDX: 0000000000000005 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 000000000075bfc8 R08: 0000000000010005 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fbfc9a986d4
R13: 00000000004c90ca R14: 00000000004df570 R15: 00000000ffffffff
INFO: rcu_sched detected stalls on CPUs/tasks:
	0-...: (1 GPs behind) idle=60a/140000000000000/0 softirq=16720/16723 fqs=71 
	(detected by 1, t=10562 jiffies, g=2179, c=2178, q=15)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 8976 Comm: syz-executor.1 Not tainted 4.14.142 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
task: ffff88805fe3a380 task.stack: ffff88805ab98000
RIP: 0010:__sanitizer_cov_trace_pc+0x23/0x60 kernel/kcov.c:68
RSP: 0018:ffff88805ab9eaa8 EFLAGS: 00000246
RAX: ffff88805fe3a380 RBX: ffff88808bed4238 RCX: ffff88808bed4200
RDX: 0000000000000001 RSI: ffffffff84efa11c RDI: ffff88808bed4248
RBP: ffff88805ab9eb00 R08: 0000000000000000 R09: ffff88805fe3ad38
R10: ffff88805fe3ad18 R11: ffff88805fe3a380 R12: dffffc0000000000
R13: ffff88808bed4040 R14: 0000000000000001 R15: ffff88808bed42c0
FS:  00007fbfc9a98700(0000) GS:ffff8880aee00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2cc23000 CR3: 00000000a1be9000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 dequeue_skb net/sched/sch_generic.c:148 [inline]
 qdisc_restart net/sched/sch_generic.c:241 [inline]
 __qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
 __dev_xmit_skb net/core/dev.c:3235 [inline]
 __dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 br_dev_queue_push_xmit+0x367/0x530 net/bridge/br_forward.c:55
 NF_HOOK include/linux/netfilter.h:250 [inline]
 NF_HOOK include/linux/netfilter.h:244 [inline]
 br_forward_finish+0xbc/0x320 net/bridge/br_forward.c:67
 NF_HOOK include/linux/netfilter.h:250 [inline]
 NF_HOOK include/linux/netfilter.h:244 [inline]
 __br_forward+0x560/0x9c0 net/bridge/br_forward.c:111
 deliver_clone+0x61/0xc0 net/bridge/br_forward.c:127
 maybe_deliver net/bridge/br_forward.c:168 [inline]
 maybe_deliver net/bridge/br_forward.c:156 [inline]
 br_flood+0x3c8/0x530 net/bridge/br_forward.c:210
 br_dev_xmit+0x9a4/0xd40 net/bridge/br_device.c:83
 __netdev_start_xmit include/linux/netdevice.h:4033 [inline]
 netdev_start_xmit include/linux/netdevice.h:4042 [inline]
 xmit_one net/core/dev.c:3009 [inline]
 dev_hard_start_xmit+0x18c/0x8b0 net/core/dev.c:3025
 __dev_queue_xmit+0x1d95/0x25e0 net/core/dev.c:3525
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
 neigh_resolve_output net/core/neighbour.c:1364 [inline]
 neigh_resolve_output+0x4d8/0x870 net/core/neighbour.c:1344
 neigh_output include/net/neighbour.h:500 [inline]
 ip_finish_output2+0x766/0x14a0 net/ipv4/ip_output.c:229
 ip_do_fragment+0x1999/0x2170 net/ipv4/ip_output.c:678
 ip_fragment.constprop.0+0x148/0x200 net/ipv4/ip_output.c:549
 ip_finish_output+0x7bd/0xc60 net/ipv4/ip_output.c:315
 NF_HOOK_COND include/linux/netfilter.h:239 [inline]
 ip_mc_output+0x24a/0xd40 net/ipv4/ip_output.c:390
 dst_output include/net/dst.h:462 [inline]
 ip_local_out+0x97/0x170 net/ipv4/ip_output.c:124
 ip_send_skb+0x3e/0xc0 net/ipv4/ip_output.c:1418
 ip_push_pending_frames+0x64/0x80 net/ipv4/ip_output.c:1438
 raw_sendmsg+0x1930/0x2450 net/ipv4/raw.c:669
 inet_sendmsg+0x122/0x500 net/ipv4/af_inet.c:762
 sock_sendmsg_nosec net/socket.c:646 [inline]
 sock_sendmsg+0xce/0x110 net/socket.c:656
 kernel_sendmsg+0x44/0x50 net/socket.c:664
 sock_no_sendpage+0x107/0x130 net/core/sock.c:2564
 inet_sendpage+0x3b8/0x580 net/ipv4/af_inet.c:780
 kernel_sendpage+0x92/0xf0 net/socket.c:3406
 sock_sendpage+0x8b/0xc0 net/socket.c:871
 pipe_to_sendpage+0x242/0x340 fs/splice.c:451
 splice_from_pipe_feed fs/splice.c:502 [inline]
 __splice_from_pipe+0x348/0x780 fs/splice.c:626
 splice_from_pipe+0xf0/0x150 fs/splice.c:661
 generic_splice_sendpage+0x3c/0x50 fs/splice.c:832
 do_splice_from fs/splice.c:851 [inline]
 do_splice fs/splice.c:1147 [inline]
 SYSC_splice fs/splice.c:1402 [inline]
 SyS_splice+0xd92/0x1430 fs/splice.c:1382
 do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x4598e9
RSP: 002b:00007fbfc9a97c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000113
RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00000000004598e9
RDX: 0000000000000005 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 000000000075bfc8 R08: 0000000000010005 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fbfc9a986d4
R13: 00000000004c90ca R14: 00000000004df570 R15: 00000000ffffffff
Code: 00 00 00 e9 2f ec ff ff 65 48 8b 04 25 40 ee 01 00 48 85 c0 74 1a 65 8b 15 4b fb a4 7e 81 e2 00 01 1f 00 75 0b 8b 90 c8 12 00 00 <83> fa 01 74 01 c3 55 48 89 e5 48 8b 75 08 48 8b 88 d0 12 00 00 
rcu_sched kthread starved for 10412 jiffies! g2179 c2178 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1
rcu_sched       I29824     9      2 0x80000000
Call Trace:
 context_switch kernel/sched/core.c:2807 [inline]
 __schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
 schedule+0x92/0x1c0 kernel/sched/core.c:3427
 schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
 rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

Crashes (3):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/09/08 14:33 linux-4.14.y 414510bc00a5 a60cb4cd .config console log report ci2-linux-4-14
2019/09/06 13:41 linux-4.14.y 414510bc00a5 c16be727 .config console log report ci2-linux-4-14
2019/05/24 22:14 linux-4.14.y bbcb3c09eae4 85c57315 .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.