INFO: rcu detected stall in wg_packet_tx_worker (2)

Status: auto-closed as invalid on 2021/02/08 20:10
Subsystems: net
[Documentation on labels]
First crash: 2195d, last: 2099d
▶ ▼ Similar bugs (4)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in wg_packet_tx_worker net 1 syz, C done 24 2244d 2297d 15/29 fixed on 2020/07/17 17:58
linux-5.15 INFO: rcu detected stall in wg_packet_tx_worker 1 1 796d 796d 0/3 auto-obsoleted due to no activity on 2024/09/14 07:48
upstream INFO: rcu detected stall in wg_packet_tx_worker (4) net 1 1 1719d 1719d 0/29 closed as invalid on 2022/02/08 10:10
upstream INFO: rcu detected stall in wg_packet_tx_worker (3) net 1 syz, C inconclusive 5 1821d 1803d 20/29 fixed on 2021/11/10 00:50

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (1 GPs behind) idle=6d2/1/0x4000000000000000 softirq=45052/45054 fqs=58 
	(t=10501 jiffies g=75361 q=542)
rcu: rcu_preempt kthread starved for 10385 jiffies! g75361 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
rcu_preempt     R  running task    29080    10      2 0x00004000
Call Trace:
 context_switch kernel/sched/core.c:3778 [inline]
 __schedule+0x8e5/0x21e0 kernel/sched/core.c:4527
 schedule+0xd0/0x2a0 kernel/sched/core.c:4602
 schedule_timeout+0x148/0x250 kernel/time/timer.c:1879
 rcu_gp_fqs_loop kernel/rcu/tree.c:1887 [inline]
 rcu_gp_kthread+0xae5/0x1b50 kernel/rcu/tree.c:2057
 kthread+0x3b5/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
NMI backtrace for cpu 1
CPU: 1 PID: 9642 Comm: kworker/1:6 Not tainted 5.8.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: wg-crypt-wg0 wg_packet_tx_worker
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x18f/0x20d lib/dump_stack.c:118
 nmi_cpu_backtrace.cold+0x70/0xb1 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x1b3/0x223 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x194/0x1cf kernel/rcu/tree_stall.h:318
 print_cpu_stall kernel/rcu/tree_stall.h:551 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:625 [inline]
 rcu_pending kernel/rcu/tree.c:3635 [inline]
 rcu_sched_clock_irq.cold+0x5b3/0xccd kernel/rcu/tree.c:2518
 update_process_times+0x25/0xa0 kernel/time/timer.c:1710
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:176
 tick_sched_timer+0x1d1/0x2a0 kernel/time/tick-sched.c:1328
 __run_hrtimer kernel/time/hrtimer.c:1520 [inline]
 __hrtimer_run_queues+0x1d5/0xfc0 kernel/time/hrtimer.c:1584
 hrtimer_interrupt+0x32a/0x930 kernel/time/hrtimer.c:1646
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1080 [inline]
 __sysvec_apic_timer_interrupt+0x142/0x5e0 arch/x86/kernel/apic/apic.c:1097
 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline]
 sysvec_apic_timer_interrupt+0xb2/0xf0 arch/x86/kernel/apic/apic.c:1091
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:581
RIP: 0010:__list_add_valid+0x77/0xa0 lib/list_debug.c:35
Code: 48 8b 16 4c 39 e2 0f 85 5e 01 00 00 48 39 f5 0f 84 3e 01 00 00 4c 39 e5 0f 84 35 01 00 00 48 83 c4 08 b8 01 00 00 00 5d 41 5c <c3> 48 89 34 24 e8 9f dd 17 fe 48 8b 34 24 eb a1 48 89 f7 48 89 34
RSP: 0018:ffffc90004de7568 EFLAGS: 00000282
RAX: 0000000000000001 RBX: ffff888040166338 RCX: ffffffff8664d556
RDX: ffff888040166338 RSI: ffff888040166338 RDI: ffff888040166340
RBP: ffff888040166340 R08: 0000000000000001 R09: 0000000000000003
R10: 0000000000000000 R11: ffffffff8c61bae0 R12: ffff888040166000
R13: dffffc0000000000 R14: ffff8880402c4d50 R15: ffff888040166338
 __list_add include/linux/list.h:67 [inline]
 list_add_tail include/linux/list.h:100 [inline]
 list_move_tail include/linux/list.h:228 [inline]
 fq_codel_dequeue+0x864/0x1f50 net/sched/sch_fq_codel.c:299
 dequeue_skb net/sched/sch_generic.c:263 [inline]
 qdisc_restart net/sched/sch_generic.c:366 [inline]
 __qdisc_run+0x1ac/0x1620 net/sched/sch_generic.c:384
 __dev_xmit_skb net/core/dev.c:3788 [inline]
 __dev_queue_xmit+0x261f/0x2d60 net/core/dev.c:4105
 neigh_hh_output include/net/neighbour.h:498 [inline]
 neigh_output include/net/neighbour.h:507 [inline]
 ip_finish_output2+0x1523/0x21f0 net/ipv4/ip_output.c:228
 __ip_finish_output net/ipv4/ip_output.c:306 [inline]
 __ip_finish_output+0x779/0xcb0 net/ipv4/ip_output.c:288
 ip_finish_output+0x34/0x1f0 net/ipv4/ip_output.c:316
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip_output+0x196/0x310 net/ipv4/ip_output.c:430
 dst_output include/net/dst.h:443 [inline]
 ip_local_out+0xaf/0x1a0 net/ipv4/ip_output.c:125
 iptunnel_xmit+0x5a0/0x9b0 net/ipv4/ip_tunnel_core.c:82
 send4+0x4c6/0xd70 drivers/net/wireguard/socket.c:85
 wg_socket_send_skb_to_peer+0x192/0x220 drivers/net/wireguard/socket.c:174
 wg_packet_create_data_done drivers/net/wireguard/send.c:252 [inline]
 wg_packet_tx_worker+0x2f9/0x980 drivers/net/wireguard/send.c:280
 process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
 kthread+0x3b5/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
NMI backtrace for cpu 1
CPU: 1 PID: 9642 Comm: kworker/1:6 Not tainted 5.8.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: wg-crypt-wg0 wg_packet_tx_worker
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x60 kernel/kcov.c:196
Code: 48 89 ef 5d e9 21 b4 3f 00 5d be 03 00 00 00 e9 76 7b 24 02 66 0f 1f 44 00 00 48 8b be b0 01 00 00 e8 b4 ff ff ff 31 c0 c3 90 <65> 48 8b 14 25 00 ff 01 00 65 8b 05 d0 e1 8d 7e a9 00 01 ff 00 48
RSP: 0018:ffffc90004de7568 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff8664d556
RDX: 0000000000000001 RSI: ffff88804c1be200 RDI: 0000000000000001
RBP: ffff888040166340 R08: 0000000000000001 R09: 0000000000000003
R10: 0000000000000000 R11: ffffffff8c61bae0 R12: ffff888040166000
R13: dffffc0000000000 R14: ffff8880402c4d50 R15: ffff888040166338
FS:  0000000000000000(0000) GS:ffff8880ae700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000029f3a98 CR3: 000000009cdc3000 CR4: 00000000001506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 __list_del_entry include/linux/list.h:132 [inline]
 list_move_tail include/linux/list.h:227 [inline]
 fq_codel_dequeue+0x7cf/0x1f50 net/sched/sch_fq_codel.c:299
 dequeue_skb net/sched/sch_generic.c:263 [inline]
 qdisc_restart net/sched/sch_generic.c:366 [inline]
 __qdisc_run+0x1ac/0x1620 net/sched/sch_generic.c:384
 __dev_xmit_skb net/core/dev.c:3788 [inline]
 __dev_queue_xmit+0x261f/0x2d60 net/core/dev.c:4105
 neigh_hh_output include/net/neighbour.h:498 [inline]
 neigh_output include/net/neighbour.h:507 [inline]
 ip_finish_output2+0x1523/0x21f0 net/ipv4/ip_output.c:228
 __ip_finish_output net/ipv4/ip_output.c:306 [inline]
 __ip_finish_output+0x779/0xcb0 net/ipv4/ip_output.c:288
 ip_finish_output+0x34/0x1f0 net/ipv4/ip_output.c:316
 NF_HOOK_COND include/linux/netfilter.h:290 [inline]
 ip_output+0x196/0x310 net/ipv4/ip_output.c:430
 dst_output include/net/dst.h:443 [inline]
 ip_local_out+0xaf/0x1a0 net/ipv4/ip_output.c:125
 iptunnel_xmit+0x5a0/0x9b0 net/ipv4/ip_tunnel_core.c:82
 send4+0x4c6/0xd70 drivers/net/wireguard/socket.c:85
 wg_socket_send_skb_to_peer+0x192/0x220 drivers/net/wireguard/socket.c:174
 wg_packet_create_data_done drivers/net/wireguard/send.c:252 [inline]
 wg_packet_tx_worker+0x2f9/0x980 drivers/net/wireguard/send.c:280
 process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
 kthread+0x3b5/0x4a0 kernel/kthread.c:292
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/08/07 06:14 upstream 47ec5303d73e 1f122f88 .config console log report ci-upstream-kasan-gce-selinux-root
2020/11/10 20:07 net-next-old 8be33ecfc1ff cca87986 .config console log report info ci-upstream-net-kasan-gce
2020/11/09 19:20 net-next-old bff6f1db91e3 64069d48 .config console log report info ci-upstream-net-kasan-gce
2020/09/23 09:21 net-next-old 748d1c8a425e 3e8f6c27 .config console log report info ci-upstream-net-kasan-gce
* Struck through repros no longer work on HEAD.