INFO: rcu detected stall in ext4_file_read_iter (2)

Status: auto-obsoleted due to no activity on 2022/11/29 18:58
Subsystems: mm kasan
[Documentation on labels]
First crash: 1440d, last: 1440d
▶ ▼ Similar bugs (2)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in ext4_file_read_iter cgroups mm 1 2 2366d 2446d 0/29 auto-closed as invalid on 2020/05/17 22:01
linux-4.19 INFO: rcu detected stall in ext4_file_read_iter 1 syz, C done error 1 2101d 2474d 0/1 upstream: reported C repro on 2019/11/01 11:29

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (10499 ticks this GP) idle=0694/1/0x4000000000000000 softirq=82863/82863 fqs=0
	(t=10500 jiffies g=126913 q=33 ncpus=2)
rcu: rcu_preempt kthread timer wakeup didn't happen for 10499 jiffies! g126913 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: 	Possible timer handling issue on cpu=0 timer-softirq=160765
rcu: rcu_preempt kthread starved for 10500 jiffies! g126913 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:I stack:28712 pid:   16 ppid:     2 flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5182 [inline]
 __schedule+0xadf/0x52b0 kernel/sched/core.c:6494
 schedule+0xda/0x1b0 kernel/sched/core.c:6570
 schedule_timeout+0x14a/0x2a0 kernel/time/timer.c:1935
 rcu_gp_fqs_loop+0x190/0x910 kernel/rcu/tree.c:1657
 rcu_gp_kthread+0x236/0x360 kernel/rcu/tree.c:1854
 kthread+0x2e4/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 29851 Comm: syz-executor.3 Not tainted 6.0.0-rc3-syzkaller-00031-gc5e4d5e99162 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022
RIP: 0010:preempt_count arch/x86/include/asm/preempt.h:27 [inline]
RIP: 0010:check_kcov_mode kernel/kcov.c:166 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x60 kernel/kcov.c:200
Code: 48 89 ef 5d e9 d1 c8 4b 00 5d be 03 00 00 00 e9 96 44 81 02 66 0f 1f 44 00 00 48 8b be a8 01 00 00 e8 b4 ff ff ff 31 c0 c3 90 <65> 8b 05 09 65 86 7e 89 c1 48 8b 34 24 81 e1 00 01 00 00 65 48 8b
RSP: 0018:ffffc90000007dc8 EFLAGS: 00000046
RAX: 0000000080000001 RBX: ffff888021195340 RCX: ffff8880596a322c
RDX: 0000000000000000 RSI: ffffffff878c12ff RDI: ffff88807f4885b0
RBP: ffff88807f488580 R08: 0000000000000006 R09: 1802b33030056660
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8880596a3200
R13: 1802b330b0056661 R14: 0000000000000000 R15: 1802b330b0056661
FS:  00007fc47ee78700(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2d825000 CR3: 00000000759c4000 CR4: 0000000000350ef0
Call Trace:
 <IRQ>
 advance_sched+0x32c/0x9a0 net/sched/sch_taprio.c:760
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x5fa/0xe40 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1095 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1112
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1106
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x38/0x70 kernel/locking/spinlock.c:194
Code: 74 24 10 e8 fa d2 dc f7 48 89 ef e8 82 54 dd f7 81 e3 00 02 00 00 75 25 9c 58 f6 c4 02 75 2d 48 85 db 74 01 fb bf 01 00 00 00 <e8> 33 16 d0 f7 65 8b 05 ec cb 7f 76 85 c0 74 0a 5b 5d c3 e8 00 de
RSP: 0018:ffffc90017707630 EFLAGS: 00000206
RAX: 0000000000000006 RBX: 0000000000000200 RCX: 1ffffffff21238b6
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001
RBP: ffffffff8c0eab80 R08: 0000000000000001 R09: ffffffff908de9e7
R10: 0000000000000001 R11: 1ffffffff201aca6 R12: 0000000000000292
R13: ffffffff912194e8 R14: 0000000000000001 R15: ffffc900177076ec
 kfence_guarded_alloc mm/kfence/core.c:373 [inline]
 __kfence_alloc+0x8c7/0x1580 mm/kfence/core.c:1042
 kfence_alloc include/linux/kfence.h:128 [inline]
 slab_alloc_node mm/slub.c:3161 [inline]
 slab_alloc mm/slub.c:3251 [inline]
 __kmem_cache_alloc_lru mm/slub.c:3258 [inline]
 kmem_cache_alloc+0x1dd/0x3b0 mm/slub.c:3268
 kmem_cache_zalloc include/linux/slab.h:723 [inline]
 jbd2_alloc_handle include/linux/jbd2.h:1596 [inline]
 new_handle fs/jbd2/transaction.c:476 [inline]
 jbd2__journal_start+0x18c/0x910 fs/jbd2/transaction.c:503
 __ext4_journal_start_sb+0x3a3/0x490 fs/ext4/ext4_jbd2.c:105
 __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
 ext4_dirty_inode+0x9d/0x110 fs/ext4/inode.c:5963
 __mark_inode_dirty+0x48b/0x1040 fs/fs-writeback.c:2381
 generic_update_time fs/inode.c:1860 [inline]
 inode_update_time fs/inode.c:1873 [inline]
 touch_atime+0x63d/0x700 fs/inode.c:1945
 file_accessed include/linux/fs.h:2516 [inline]
 filemap_read+0xc0c/0xe10 mm/filemap.c:2728
 generic_file_read_iter+0x3b0/0x5a0 mm/filemap.c:2806
 ext4_file_read_iter+0x1d5/0x600 fs/ext4/file.c:132
 call_read_iter include/linux/fs.h:2181 [inline]
 new_sync_read fs/read_write.c:389 [inline]
 vfs_read+0x67d/0x930 fs/read_write.c:470
 ksys_read+0x127/0x250 fs/read_write.c:607
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7fc47dc89279
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fc47ee78168 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 00007fc47dd9bf80 RCX: 00007fc47dc89279
RDX: 0000000000002020 RSI: 00000000200021c0 RDI: 0000000000000003
RBP: 00007fc47dce32e9 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffce4d81bbf R14: 00007fc47ee78300 R15: 0000000000022000
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 29851 Comm: syz-executor.3 Not tainted 6.0.0-rc3-syzkaller-00031-gc5e4d5e99162 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022
RIP: 0010:hlock_class kernel/locking/lockdep.c:222 [inline]
RIP: 0010:check_wait_context kernel/locking/lockdep.c:4752 [inline]
RIP: 0010:__lock_acquire+0x6be/0x56d0 kernel/locking/lockdep.c:5003
Code: 00 00 45 3b a5 70 0a 00 00 0f 8d 99 00 00 00 49 63 c4 48 8d 04 80 49 8d 1c c6 48 8d 7b 20 48 89 fa 48 c1 ea 03 42 0f b6 14 3a <84> d2 74 09 80 fa 03 0f 8e af 27 00 00 0f b7 53 20 66 81 e2 ff 1f
RSP: 0018:ffffc90000007ae8 EFLAGS: 00000016
RAX: 000000000000000a RBX: ffff88802239c5c8 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88802239c5e8
RBP: 0000000000000002 R08: 0000000000000000 R09: dffffc0000000000
R10: fffffbfff211bd28 R11: 0000000000000000 R12: 0000000000000002
R13: ffff88802239bb00 R14: ffff88802239c578 R15: dffffc0000000000
FS:  00007fc47ee78700(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b2d825000 CR3: 00000000759c4000 CR4: 0000000000350ef0
Call Trace:
 <IRQ>
 lock_acquire kernel/locking/lockdep.c:5666 [inline]
 lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5631
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:162
 debug_object_deactivate lib/debugobjects.c:732 [inline]
 debug_object_deactivate+0x101/0x300 lib/debugobjects.c:720
 debug_hrtimer_deactivate kernel/time/hrtimer.c:425 [inline]
 debug_deactivate kernel/time/hrtimer.c:481 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1653 [inline]
 __hrtimer_run_queues+0x3f3/0xe40 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1095 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1112
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1106
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
RIP: 0010:_raw_spin_unlock_irqrestore+0x38/0x70 kernel/locking/spinlock.c:194
Code: 74 24 10 e8 fa d2 dc f7 48 89 ef e8 82 54 dd f7 81 e3 00 02 00 00 75 25 9c 58 f6 c4 02 75 2d 48 85 db 74 01 fb bf 01 00 00 00 <e8> 33 16 d0 f7 65 8b 05 ec cb 7f 76 85 c0 74 0a 5b 5d c3 e8 00 de
RSP: 0018:ffffc90017707630 EFLAGS: 00000206
RAX: 0000000000000006 RBX: 0000000000000200 RCX: 1ffffffff21238b6
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001
RBP: ffffffff8c0eab80 R08: 0000000000000001 R09: ffffffff908de9e7
R10: 0000000000000001 R11: 1ffffffff201aca6 R12: 0000000000000292
R13: ffffffff912194e8 R14: 0000000000000001 R15: ffffc900177076ec
 kfence_guarded_alloc mm/kfence/core.c:373 [inline]
 __kfence_alloc+0x8c7/0x1580 mm/kfence/core.c:1042
 kfence_alloc include/linux/kfence.h:128 [inline]
 slab_alloc_node mm/slub.c:3161 [inline]
 slab_alloc mm/slub.c:3251 [inline]
 __kmem_cache_alloc_lru mm/slub.c:3258 [inline]
 kmem_cache_alloc+0x1dd/0x3b0 mm/slub.c:3268
 kmem_cache_zalloc include/linux/slab.h:723 [inline]
 jbd2_alloc_handle include/linux/jbd2.h:1596 [inline]
 new_handle fs/jbd2/transaction.c:476 [inline]
 jbd2__journal_start+0x18c/0x910 fs/jbd2/transaction.c:503
 __ext4_journal_start_sb+0x3a3/0x490 fs/ext4/ext4_jbd2.c:105
 __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
 ext4_dirty_inode+0x9d/0x110 fs/ext4/inode.c:5963
 __mark_inode_dirty+0x48b/0x1040 fs/fs-writeback.c:2381
 generic_update_time fs/inode.c:1860 [inline]
 inode_update_time fs/inode.c:1873 [inline]
 touch_atime+0x63d/0x700 fs/inode.c:1945
 file_accessed include/linux/fs.h:2516 [inline]
 filemap_read+0xc0c/0xe10 mm/filemap.c:2728
 generic_file_read_iter+0x3b0/0x5a0 mm/filemap.c:2806
 ext4_file_read_iter+0x1d5/0x600 fs/ext4/file.c:132
 call_read_iter include/linux/fs.h:2181 [inline]
 new_sync_read fs/read_write.c:389 [inline]
 vfs_read+0x67d/0x930 fs/read_write.c:470
 ksys_read+0x127/0x250 fs/read_write.c:607
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7fc47dc89279
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fc47ee78168 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 00007fc47dd9bf80 RCX: 00007fc47dc89279
RDX: 0000000000002020 RSI: 00000000200021c0 RDI: 0000000000000003
RBP: 00007fc47dce32e9 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffce4d81bbf R14: 00007fc47ee78300 R15: 0000000000022000
 </TASK>
NMI backtrace for cpu 1
CPU: 1 PID: 11 Comm: kworker/u4:1 Not tainted 6.0.0-rc3-syzkaller-00031-gc5e4d5e99162 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022
Workqueue: events_unbound toggle_allocation_gate
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x46/0x14f lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x206/0x250 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:166 [inline]
 rcu_dump_cpu_stacks+0x2c7/0x4d0 kernel/rcu/tree_stall.h:371
 print_cpu_stall kernel/rcu/tree_stall.h:660 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:744 [inline]
 rcu_pending kernel/rcu/tree.c:3660 [inline]
 rcu_sched_clock_irq.cold+0x9f/0x79c kernel/rcu/tree.c:2342
 update_process_times+0x11a/0x1a0 kernel/time/timer.c:1839
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:243
 tick_sched_timer+0xee/0x120 kernel/time/tick-sched.c:1480
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe40 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1095 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1112
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1106
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:csd_lock_wait kernel/smp.c:414 [inline]
RIP: 0010:smp_call_function_many_cond+0x5c3/0x1430 kernel/smp.c:988
Code: 89 ee e8 30 ad 0a 00 85 ed 74 48 48 8b 44 24 08 49 89 c4 83 e0 07 49 c1 ec 03 48 89 c5 4d 01 f4 83 c5 03 e8 4f b0 0a 00 f3 90 <41> 0f b6 04 24 40 38 c5 7c 08 84 c0 0f 85 b5 0b 00 00 8b 43 08 31
RSP: 0018:ffffc90000107948 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b9a425a0 RCX: 0000000000000000
RDX: ffff888011a6bb00 RSI: ffffffff817159e1 RDI: 0000000000000005
RBP: 0000000000000003 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffffed10173484b5
R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000000001
 on_each_cpu_cond_mask+0x56/0xa0 kernel/smp.c:1154
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1302 [inline]
 text_poke_bp_batch+0x233/0x6c0 arch/x86/kernel/alternative.c:1490
 text_poke_flush arch/x86/kernel/alternative.c:1660 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:1657 [inline]
 text_poke_finish+0x16/0x30 arch/x86/kernel/alternative.c:1667
 arch_jump_label_transform_apply+0x13/0x20 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x32f/0x410 kernel/jump_label.c:801
 static_key_disable_cpuslocked+0x152/0x1b0 kernel/jump_label.c:207
 static_key_disable+0x16/0x20 kernel/jump_label.c:215
 toggle_allocation_gate mm/kfence/core.c:825 [inline]
 toggle_allocation_gate+0x183/0x390 mm/kfence/core.c:803
 process_one_work+0x991/0x1610 kernel/workqueue.c:2289
 worker_thread+0x665/0x1080 kernel/workqueue.c:2436
 kthread+0x2e4/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/08/31 18:56 upstream c5e4d5e99162 51e54e30 .config console log report info ci-upstream-kasan-gce-root INFO: rcu detected stall in ext4_file_read_iter
* Struck through repros no longer work on HEAD.