BUG: soft lockup in task_numa_work

Status: auto-closed as invalid on 2022/04/01 16:00
Reported-by: syzbot+6a0539e45a2b77ff934c@syzkaller.appspotmail.com
First crash: 1715d, last: 1715d
▶ ▼ Similar bugs (3)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in task_numa_work mm 1 2 1079d 1080d 0/29 auto-obsoleted due to no activity on 2023/11/28 14:56
upstream INFO: rcu detected stall in task_numa_work (2) mm bpf 1 syz, C error error 28 22d 587d 0/29 upstream: reported C repro on 2025/01/04 04:47
linux-6.1 INFO: rcu detected stall in task_numa_work 1 1 800d 800d 0/3 auto-obsoleted due to no activity on 2024/09/12 14:33

Sample crash report:
kvm: vcpu 0: requested 128 ns lapic timer period limited to 200000 ns
watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [syz-executor.2:2511]
Modules linked in:
irq event stamp: 8250
hardirqs last  enabled at (8249): [<ffffffff8129070b>] kvm_wait arch/x86/kernel/kvm.c:799 [inline]
hardirqs last  enabled at (8249): [<ffffffff8129070b>] kvm_wait+0x14b/0x240 arch/x86/kernel/kvm.c:779
hardirqs last disabled at (8250): [<ffffffff81003d00>] trace_hardirqs_off_thunk+0x1a/0x1c
softirqs last  enabled at (8242): [<ffffffff88400678>] __do_softirq+0x678/0x980 kernel/softirq.c:318
softirqs last disabled at (7469): [<ffffffff813927d5>] invoke_softirq kernel/softirq.c:372 [inline]
softirqs last disabled at (7469): [<ffffffff813927d5>] irq_exit+0x215/0x260 kernel/softirq.c:412
CPU: 1 PID: 2511 Comm: syz-executor.2 Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:61
Code: 48 89 df e8 f4 20 7f f9 e9 2e ff ff ff 48 89 df e8 e7 20 7f f9 eb 82 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 14 43 4e 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 04 43 4e 00 f4 c3 90 90 41 56 41 55
RSP: 0018:ffff88823874fab8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
RAX: 1ffffffff13e3054 RBX: ffff88809ff84f20 RCX: 1ffff110152bedca
RDX: dffffc0000000000 RSI: ffff8880a95f6e30 RDI: ffff8880a95f6e04
RBP: 0000000000000003 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000286
R13: ffffed1013ff09e4 R14: 0000000000000001 R15: ffff8880ba12be00
FS:  00007fc5fdabb700(0000) GS:ffff8880ba100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fff3fb119c0 CR3: 000000023b375000 CR4: 00000000003426e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 arch_safe_halt arch/x86/include/asm/paravirt.h:94 [inline]
 kvm_wait arch/x86/kernel/kvm.c:799 [inline]
 kvm_wait+0x179/0x240 arch/x86/kernel/kvm.c:779
 pv_wait arch/x86/include/asm/paravirt.h:689 [inline]
 pv_wait_head_or_lock kernel/locking/qspinlock_paravirt.h:471 [inline]
 __pv_queued_spin_lock_slowpath+0x86a/0xae0 kernel/locking/qspinlock.c:474
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:679 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:53 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:88 [inline]
 do_raw_spin_lock+0x189/0x220 kernel/locking/spinlock_debug.c:113
 spin_lock include/linux/spinlock.h:329 [inline]
 change_pte_range mm/mprotect.c:62 [inline]
 change_pmd_range mm/mprotect.c:244 [inline]
 change_pud_range mm/mprotect.c:272 [inline]
 change_p4d_range mm/mprotect.c:292 [inline]
 change_protection_range+0xb5f/0x1fd0 mm/mprotect.c:317
 change_protection+0xa9/0xc0 mm/mprotect.c:338
 change_prot_numa+0x2f/0x80 mm/mempolicy.c:599
 task_numa_work+0x51c/0xac0 kernel/sched/fair.c:2642
 task_work_run+0x148/0x1c0 kernel/task_work.c:113
 tracehook_notify_resume include/linux/tracehook.h:193 [inline]
 exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
 prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
 syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
 do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7fc600545ae9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fc5fdabb188 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: 0000000000000000 RBX: 00007fc600658f60 RCX: 00007fc600545ae9
RDX: 0000000020000400 RSI: 000000004400ae8f RDI: 0000000000000007
RBP: 00007fc60059ff6d R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007fff3fa7ccaf R14: 00007fc5fdabb300 R15: 0000000000022000
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 10231 Comm: kworker/u4:9 Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: writeback wb_workfn (flush-8:0)
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:161 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x5c/0xe0 kernel/locking/spinlock.c:184
Code: 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 75 72 48 83 3d cd 31 d8 01 00 74 64 48 89 df 57 9d 0f 1f 44 00 00 e8 94 5c 4e f9 <bf> 01 00 00 00 e8 fa 1b 28 f9 65 8b 05 73 8e e8 77 85 c0 74 39 5b
RSP: 0018:ffff8880ba007d00 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000000
RDX: 1ffff11015685512 RSI: 0000000000000000 RDI: ffff8880ab42a890
RBP: ffffffff8d3a7488 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000005 R11: ffffffff8c66501b R12: dffffc0000000000
R13: 1ffff11017400fa5 R14: ffff88809ec66bd0 R15: ffffffff8d3a7488
FS:  0000000000000000(0000) GS:ffff8880ba000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1724770018 CR3: 0000000009e6d000 CR4: 00000000003426f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 debug_object_deactivate lib/debugobjects.c:568 [inline]
 debug_object_deactivate+0x1f9/0x2e0 lib/debugobjects.c:529
 debug_hrtimer_deactivate kernel/time/hrtimer.c:421 [inline]
 debug_deactivate kernel/time/hrtimer.c:471 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1435 [inline]
 __hrtimer_run_queues+0x1bc/0xe60 kernel/time/hrtimer.c:1527
 hrtimer_interrupt+0x326/0x9e0 kernel/time/hrtimer.c:1585
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1071 [inline]
 smp_apic_timer_interrupt+0x10c/0x550 arch/x86/kernel/apic/apic.c:1096
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:894
 </IRQ>
RIP: 0010:csd_lock_wait kernel/smp.c:108 [inline]
RIP: 0010:smp_call_function_single+0x1ee/0x420 kernel/smp.c:302
Code: 24 40 8b 7c 24 1c e8 a1 f9 ff ff 41 89 c5 eb 07 e8 e7 03 0a 00 f3 90 44 8b 64 24 58 31 ff 41 83 e4 01 44 89 e6 e8 42 05 0a 00 <45> 85 e4 75 e1 e8 c8 03 0a 00 e8 c3 03 0a 00 bf 01 00 00 00 e8 19
RSP: 0018:ffff88804e57ece0 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000000 RBX: 1ffff11009cafda0 RCX: ffffffff8158819e
RDX: 0000000000000001 RSI: ffff8880ab42a040 RDI: 0000000000000005
RBP: ffff88804e57eda8 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000005 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000002
 smp_call_function_many+0x743/0x8d0 kernel/smp.c:434
 flush_tlb_others arch/x86/include/asm/paravirt.h:309 [inline]
 flush_tlb_mm_range+0x179/0x320 arch/x86/mm/tlb.c:728
 flush_tlb_page arch/x86/include/asm/tlbflush.h:576 [inline]
 ptep_clear_flush+0x123/0x160 mm/pgtable-generic.c:87
 page_mkclean_one+0x425/0x860 mm/rmap.c:912
 rmap_walk_file+0x539/0xb10 mm/rmap.c:1897
 rmap_walk+0x105/0x190 mm/rmap.c:1915
 page_mkclean+0x20f/0x2b0 mm/rmap.c:981
 clear_page_dirty_for_io+0x305/0xee0 mm/page-writeback.c:2687
 mpage_submit_page+0x80/0x250 fs/ext4/inode.c:2215
 mpage_process_page_bufs+0x534/0x630 fs/ext4/inode.c:2345
 mpage_prepare_extent_to_map+0x9a2/0xf10 fs/ext4/inode.c:2707
 ext4_writepages+0x111d/0x37f0 fs/ext4/inode.c:2835
 do_writepages+0xe5/0x290 mm/page-writeback.c:2344
 __writeback_single_inode+0x10c/0x11d0 fs/fs-writeback.c:1385
 writeback_sb_inodes+0x537/0xef0 fs/fs-writeback.c:1647
 __writeback_inodes_wb+0xc6/0x280 fs/fs-writeback.c:1716
 wb_writeback+0x841/0xcc0 fs/fs-writeback.c:1822
 wb_check_old_data_flush fs/fs-writeback.c:1924 [inline]
 wb_do_writeback fs/fs-writeback.c:1977 [inline]
 wb_workfn+0x8ba/0x1250 fs/fs-writeback.c:2006
 process_one_work+0x864/0x1570 kernel/workqueue.c:2153
 worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
 kthread+0x33f/0x460 kernel/kthread.c:259
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
----------------
Code disassembly (best guess):
   0:	48 89 df             	mov    %rbx,%rdi
   3:	e8 f4 20 7f f9       	callq  0xf97f20fc
   8:	e9 2e ff ff ff       	jmpq   0xffffff3b
   d:	48 89 df             	mov    %rbx,%rdi
  10:	e8 e7 20 7f f9       	callq  0xf97f20fc
  15:	eb 82                	jmp    0xffffff99
  17:	90                   	nop
  18:	90                   	nop
  19:	90                   	nop
  1a:	90                   	nop
  1b:	90                   	nop
  1c:	e9 07 00 00 00       	jmpq   0x28
  21:	0f 00 2d 14 43 4e 00 	verw   0x4e4314(%rip)        # 0x4e433c
  28:	fb                   	sti
  29:	f4                   	hlt
* 2a:	c3                   	retq <-- trapping instruction
  2b:	90                   	nop
  2c:	e9 07 00 00 00       	jmpq   0x38
  31:	0f 00 2d 04 43 4e 00 	verw   0x4e4304(%rip)        # 0x4e433c
  38:	f4                   	hlt
  39:	c3                   	retq
  3a:	90                   	nop
  3b:	90                   	nop
  3c:	41 56                	push   %r14
  3e:	41 55                	push   %r13

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2021/12/02 16:00 linux-4.19.y 3f8a27f9e27b 61f86278 .config console log report info ci2-linux-4-19 BUG: soft lockup in task_numa_work
* Struck through repros no longer work on HEAD.