BUG: workqueue lockup - pool cpus=1 node=0 flags=0x2 nice=0 stuck for 476s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=3 refcnt=4
    pending: nsim_dev_hwstats_traffic_work, vmstat_shepherd, nsim_dev_hwstats_traffic_work
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=3 refcnt=4
    pending: 3*psi_avgs_work
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
    pending: flush_memcg_stats_dwork, toggle_allocation_gate
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
    in-flight: 7577:nsim_dev_trap_report_work
    pending: crng_reseed
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: nsim_dev_trap_report_work
workqueue events_power_efficient: flags=0x180
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=5 refcnt=6
    pending: gc_worker, neigh_managed_work, check_lifetime, neigh_periodic_work, check_lifetime
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=4 refcnt=5
    pending: wg_ratelimiter_gc_entries, neigh_periodic_work, neigh_managed_work, check_lifetime
workqueue mm_percpu_wq: flags=0x8
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=3 MAYDAY
    pending: vmstat_update
workqueue writeback: flags=0x4a
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 7784:wb_workfn
workqueue wg-kex-wg0: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 6213:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 6930:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
    in-flight: 7353:wg_queued_expired_zero_key_material
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg1: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 7607:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 7795:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
    pending: 2*wg_packet_handshake_send_worker
workqueue wg-kex-wg2: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg0: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg1: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 6430:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 6793:wg_packet_handshake_send_worker
workqueue wg-kex-wg2: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 7132:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 6675:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
    in-flight: 7439:wg_packet_handshake_send_worker ,4980:wg_packet_handshake_send_worker
pool 2: cpus=0 node=0 flags=0x2 nice=0 hung=3s workers=4 manager: 7138
pool 6: cpus=1 node=0 flags=0x2 nice=0 hung=476s workers=5 manager: 7432
pool 8: cpus=0-1 flags=0x6 nice=0 hung=3s workers=7 manager: 5098
Showing backtraces of running workers in stalled CPU-bound worker pools:
pool 6:
task:kworker/1:6     state:R  running task     stack:0     pid:6930  tgid:6930  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker
Call Trace:
[<ffffffff8639c964>] context_switch kernel/sched/core.c:5260 [inline]
[<ffffffff8639c964>] __schedule+0xf10/0x4098 kernel/sched/core.c:6867
pool 6:
task:kworker/1:4     state:R  running task     stack:0     pid:6430  tgid:6430  ppid:2      task_flags:0x4208060 flags:0x00000000
Workqueue: wg-kex-wg1 wg_packet_handshake_receive_worker
Call Trace:
[<ffffffff8639c964>] context_switch kernel/sched/core.c:5260 [inline]
[<ffffffff8639c964>] __schedule+0xf10/0x4098 kernel/sched/core.c:6867
[<ffffffff863a052c>] preempt_schedule_irq+0x70/0x1c8 kernel/sched/core.c:7194
[<ffffffff86393306>] raw_irqentry_exit_cond_resched kernel/entry/common.c:173 [inline]
[<ffffffff86393306>] raw_irqentry_exit_cond_resched kernel/entry/common.c:165 [inline]
[<ffffffff86393306>] irqentry_exit+0x37a/0xb04 kernel/entry/common.c:216
[<ffffffff86392168>] do_irq+0x34/0x4c arch/riscv/kernel/traps.c:458
[<ffffffff863bb5fc>] handle_exception+0x13c/0x16a arch/riscv/kernel/entry.S:221
[<ffffffff80a9f822>] arch_local_irq_restore arch/riscv/include/asm/irqflags.h:51 [inline]
[<ffffffff80a9f822>] slab_update_freelist mm/slub.c:838 [inline]
[<ffffffff80a9f822>] __slab_free+0x106/0x2bc mm/slub.c:5941
[<ffffffff80aaac76>] do_slab_free mm/slub.c:6622 [inline]
[<ffffffff80aaac76>] ___cache_free+0x176/0x1ac mm/slub.c:6741
[<ffffffff80b52ec0>] qlink_free mm/kasan/quarantine.c:163 [inline]
[<ffffffff80b52ec0>] qlist_free_all+0x68/0x13c mm/kasan/quarantine.c:179
[<ffffffff80b53516>] kasan_quarantine_reduce+0x156/0x1bc mm/kasan/quarantine.c:286
[<ffffffff80b4ea64>] __kasan_kmalloc+0x80/0xa8 mm/kasan/common.c:406
[<ffffffff80a9d8f8>] kasan_kmalloc include/linux/kasan.h:263 [inline]
[<ffffffff80a9d8f8>] __kmalloc_cache_noprof+0x230/0x878 mm/slub.c:5780
[<ffffffff83289aa2>] kmalloc_noprof include/linux/slab.h:957 [inline]
[<ffffffff83289aa2>] kzalloc_noprof include/linux/slab.h:1094 [inline]
[<ffffffff83289aa2>] keypair_create drivers/net/wireguard/noise.c:100 [inline]
[<ffffffff83289aa2>] wg_noise_handshake_begin_session+0xee/0xd88 drivers/net/wireguard/noise.c:827
[<ffffffff8329371c>] wg_packet_send_handshake_response+0x218/0x2cc drivers/net/wireguard/send.c:96
[<ffffffff8329700a>] wg_receive_handshake_packet+0x6ba/0xa70 drivers/net/wireguard/receive.c:154
[<ffffffff83298b7e>] wg_packet_handshake_receive_worker+0x116/0x30c drivers/net/wireguard/receive.c:213
[<ffffffff801c25d4>] process_one_work+0x930/0x1e14 kernel/workqueue.c:3257
[<ffffffff801c3ff8>] process_scheduled_works kernel/workqueue.c:3340 [inline]
[<ffffffff801c3ff8>] worker_thread+0x540/0xcbc kernel/workqueue.c:3421
[<ffffffff801e2fcc>] kthread+0x37c/0x778 kernel/kthread.c:463
[<ffffffff80069946>] ret_from_fork_kernel+0x2a/0xbbc arch/riscv/kernel/process.c:228
[<ffffffff863bb74e>] ret_from_fork_kernel_asm+0x16/0x18 arch/riscv/kernel/entry.S:363
pool 6:
task:kworker/1:0     state:R  running task     stack:0     pid:6675  tgid:6675  ppid:2      task_flags:0x4208060 flags:0x00000000
Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker
Call Trace:
[<ffffffff8639c964>] context_switch kernel/sched/core.c:5260 [inline]
[<ffffffff8639c964>] __schedule+0xf10/0x4098 kernel/sched/core.c:6867
[<ffffffff863a052c>] preempt_schedule_irq+0x70/0x1c8 kernel/sched/core.c:7194
[<ffffffff86393306>] raw_irqentry_exit_cond_resched kernel/entry/common.c:173 [inline]
[<ffffffff86393306>] raw_irqentry_exit_cond_resched kernel/entry/common.c:165 [inline]
[<ffffffff86393306>] irqentry_exit+0x37a/0xb04 kernel/entry/common.c:216
[<ffffffff86392168>] do_irq+0x34/0x4c arch/riscv/kernel/traps.c:458
[<ffffffff863bb5fc>] handle_exception+0x13c/0x16a arch/riscv/kernel/entry.S:221
[<ffffffff80a9f822>] arch_local_irq_restore arch/riscv/include/asm/irqflags.h:51 [inline]
[<ffffffff80a9f822>] slab_update_freelist mm/slub.c:838 [inline]
[<ffffffff80a9f822>] __slab_free+0x106/0x2bc mm/slub.c:5941
[<ffffffff80aaac76>] do_slab_free mm/slub.c:6622 [inline]
[<ffffffff80aaac76>] ___cache_free+0x176/0x1ac mm/slub.c:6741
[<ffffffff80b52ec0>] qlink_free mm/kasan/quarantine.c:163 [inline]
[<ffffffff80b52ec0>] qlist_free_all+0x68/0x13c mm/kasan/quarantine.c:179
[<ffffffff80b53516>] kasan_quarantine_reduce+0x156/0x1bc mm/kasan/quarantine.c:286
[<ffffffff80b4ea64>] __kasan_kmalloc+0x80/0xa8 mm/kasan/common.c:406
[<ffffffff80a9d8f8>] kasan_kmalloc include/linux/kasan.h:263 [inline]
[<ffffffff80a9d8f8>] __kmalloc_cache_noprof+0x230/0x878 mm/slub.c:5780
[<ffffffff83289aa2>] kmalloc_noprof include/linux/slab.h:957 [inline]
[<ffffffff83289aa2>] kzalloc_noprof include/linux/slab.h:1094 [inline]
[<ffffffff83289aa2>] keypair_create drivers/net/wireguard/noise.c:100 [inline]
[<ffffffff83289aa2>] wg_noise_handshake_begin_session+0xee/0xd88 drivers/net/wireguard/noise.c:827
[<ffffffff8329371c>] wg_packet_send_handshake_response+0x218/0x2cc drivers/net/wireguard/send.c:96
[<ffffffff8329700a>] wg_receive_handshake_packet+0x6ba/0xa70 drivers/net/wireguard/receive.c:154
[<ffffffff83298b7e>] wg_packet_handshake_receive_worker+0x116/0x30c drivers/net/wireguard/receive.c:213
[<ffffffff801c25d4>] process_one_work+0x930/0x1e14 kernel/workqueue.c:3257
[<ffffffff801c3ff8>] process_scheduled_works kernel/workqueue.c:3340 [inline]
[<ffffffff801c3ff8>] worker_thread+0x540/0xcbc kernel/workqueue.c:3421
[<ffffffff801e2fcc>] kthread+0x37c/0x778 kernel/kthread.c:463
[<ffffffff80069946>] ret_from_fork_kernel+0x2a/0xbbc arch/riscv/kernel/process.c:228
[<ffffffff863bb74e>] ret_from_fork_kernel_asm+0x16/0x18 arch/riscv/kernel/entry.S:363