possible deadlock in static_key_slow_dec

Status: auto-obsoleted due to no activity on 2023/01/11 02:59
Subsystems: cgroups
[Documentation on labels]
Reported-by: syzbot+b011e55d1b4c015100d2@syzkaller.appspotmail.com
First crash: 2845d, last: 1436d
Cause bisection: introduced by (bisect log) [merge commit]:
commit 7cd18c11799720bf077309be1b84b8395af1e2ff
Author: Stephen Rothwell <sfr@canb.auug.org.au>
Date: Wed Aug 14 04:46:44 2019 +0000

  Merge remote-tracking branch 'tip/auto-latest'

Crash: possible deadlock in static_key_slow_dec (log)
Repro: C syz .config
  
Fix bisection: failed (error log, bisect log)
  
▶ ▼ Duplicate bugs (1)
Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
possible deadlock in __static_key_slow_dec cgroups 4 syz, C done error 2 2358d 2358d 0/29 closed as dup on 2021/01/20 05:22
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
possible deadlock in static_key_slow_dec 0 (2) 2019/08/16 07:07
▶ ▼ Similar bugs (3)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 possible deadlock in static_key_slow_dec (3) 4 3 1731d 1780d 0/1 auto-closed as invalid on 2022/03/22 05:24
linux-4.19 possible deadlock in static_key_slow_dec 4 1 2544d 2544d 0/1 auto-closed as invalid on 2019/12/30 04:27
linux-4.19 possible deadlock in static_key_slow_dec (2) 4 2 2049d 2132d 0/1 auto-closed as invalid on 2021/05/08 21:09
▶ ▼ Last patch testing requests (6)
Created Duration User Patch Repo Result
2022/12/24 08:31 22m retest repro linux-next no crash log
2022/12/22 11:31 21m retest repro upstream no crash log
2022/12/22 03:31 21m retest repro upstream no crash log
2022/09/13 00:27 10m retest repro upstream report log
2022/09/12 07:27 18m retest repro linux-next no crash log
2022/09/12 04:27 16m retest repro linux-next error

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.6.0-rc5-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor754/10070 is trying to acquire lock:
ffffffff89768e70 (cpu_hotplug_lock.rw_sem){++++}, at: __static_key_slow_dec kernel/jump_label.c:254 [inline]
ffffffff89768e70 (cpu_hotplug_lock.rw_sem){++++}, at: static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:270

but task is already holding lock:
ffff88809011c7d8 (&mm->mmap_sem#2){++++}, at: vm_mmap_pgoff+0x152/0x200 mm/util.c:504

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&mm->mmap_sem#2){++++}:
       down_write+0x8d/0x150 kernel/locking/rwsem.c:1534
       mpol_rebind_mm+0x20/0xc0 mm/mempolicy.c:382
       cpuset_attach+0x214/0x400 kernel/cgroup/cpuset.c:2203
       cgroup_migrate_execute+0xc34/0x1210 kernel/cgroup/cgroup.c:2445
       cgroup_attach_task+0x57c/0x8e0 kernel/cgroup/cgroup.c:2738
       __cgroup1_procs_write.constprop.0+0x3a7/0x490 kernel/cgroup/cgroup-v1.c:522
       cgroup_file_write+0x20d/0x710 kernel/cgroup/cgroup.c:3695
       kernfs_fop_write+0x268/0x490 fs/kernfs/file.c:315
       __vfs_write+0x76/0x100 fs/read_write.c:494
       vfs_write+0x262/0x5c0 fs/read_write.c:558
       ksys_write+0x127/0x250 fs/read_write.c:611
       do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&cpuset_rwsem){++++}:
       percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
       cpuset_read_lock+0x3b/0x130 kernel/cgroup/cpuset.c:340
       __sched_setscheduler.constprop.0+0x251/0x2000 kernel/sched/core.c:4869
       _sched_setscheduler+0xee/0x180 kernel/sched/core.c:5041
       __kthread_create_on_node+0x2ea/0x410 kernel/kthread.c:349
       kthread_create_on_node+0xbb/0xf0 kernel/kthread.c:388
       create_worker+0x23b/0x530 kernel/workqueue.c:1926
       workqueue_prepare_cpu+0x9c/0xf0 kernel/workqueue.c:5026
       cpuhp_invoke_callback+0x22c/0x1d20 kernel/cpu.c:172
       cpuhp_up_callbacks kernel/cpu.c:599 [inline]
       _cpu_up+0x27d/0x540 kernel/cpu.c:1165
       do_cpu_up kernel/cpu.c:1200 [inline]
       do_cpu_up+0xfe/0x1a0 kernel/cpu.c:1172
       smp_init+0x239/0x24e kernel/smp.c:604
       kernel_init_freeable+0x32e/0x5ae init/main.c:1438
       kernel_init+0xd/0x1bb init/main.c:1352
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

-> #0 (cpu_hotplug_lock.rw_sem){++++}:
       check_prev_add kernel/locking/lockdep.c:2475 [inline]
       check_prevs_add kernel/locking/lockdep.c:2580 [inline]
       validate_chain kernel/locking/lockdep.c:2970 [inline]
       __lock_acquire+0x201b/0x3ca0 kernel/locking/lockdep.c:3954
       lock_acquire+0x197/0x420 kernel/locking/lockdep.c:4484
       percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
       cpus_read_lock+0x3b/0x130 kernel/cpu.c:292
       __static_key_slow_dec kernel/jump_label.c:254 [inline]
       static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:270
       sw_perf_event_destroy+0x81/0x130 kernel/events/core.c:8840
       _free_event+0x33b/0x12b0 kernel/events/core.c:4616
       put_event+0x40/0x50 kernel/events/core.c:4710
       perf_mmap_close+0x4f7/0xcb0 kernel/events/core.c:5754
       remove_vma+0xa9/0x170 mm/mmap.c:177
       remove_vma_list mm/mmap.c:2568 [inline]
       __do_munmap+0x729/0x10e0 mm/mmap.c:2812
       do_munmap mm/mmap.c:2820 [inline]
       mmap_region+0x1ef/0x1540 mm/mmap.c:1713
       do_mmap+0x843/0x1140 mm/mmap.c:1543
       do_mmap_pgoff include/linux/mm.h:2334 [inline]
       vm_mmap_pgoff+0x197/0x200 mm/util.c:506
       ksys_mmap_pgoff+0x457/0x5b0 mm/mmap.c:1593
       do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
  cpu_hotplug_lock.rw_sem --> &cpuset_rwsem --> &mm->mmap_sem#2

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&mm->mmap_sem#2);
                               lock(&cpuset_rwsem);
                               lock(&mm->mmap_sem#2);
  lock(cpu_hotplug_lock.rw_sem);

 *** DEADLOCK ***

1 lock held by syz-executor754/10070:
 #0: ffff88809011c7d8 (&mm->mmap_sem#2){++++}, at: vm_mmap_pgoff+0x152/0x200 mm/util.c:504

stack backtrace:
CPU: 1 PID: 10070 Comm: syz-executor754 Not tainted 5.6.0-rc5-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x188/0x20d lib/dump_stack.c:118
 check_noncircular+0x32e/0x3e0 kernel/locking/lockdep.c:1808
 check_prev_add kernel/locking/lockdep.c:2475 [inline]
 check_prevs_add kernel/locking/lockdep.c:2580 [inline]
 validate_chain kernel/locking/lockdep.c:2970 [inline]
 __lock_acquire+0x201b/0x3ca0 kernel/locking/lockdep.c:3954
 lock_acquire+0x197/0x420 kernel/locking/lockdep.c:4484
 percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
 cpus_read_lock+0x3b/0x130 kernel/cpu.c:292
 __static_key_slow_dec kernel/jump_label.c:254 [inline]
 static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:270
 sw_perf_event_destroy+0x81/0x130 kernel/events/core.c:8840
 _free_event+0x33b/0x12b0 kernel/events/core.c:4616
 put_event+0x40/0x50 kernel/events/core.c:4710
 perf_mmap_close+0x4f7/0xcb0 kernel/events/core.c:5754
 remove_vma+0xa9/0x170 mm/mmap.c:177
 remove_vma_list mm/mmap.c:2568 [inline]
 __do_munmap+0x729/0x10e0 mm/mmap.c:2812
 do_munmap mm/mmap.c:2820 [inline]
 mmap_region+0x1ef/0x1540 mm/mmap.c:1713
 do_mmap+0x843/0x1140 mm/mmap.c:1543
 do_mmap_pgoff include/linux/mm.h:2334 [inline]
 vm_mmap_pgoff+0x197/0x200 mm/util.c:506
 ksys_mmap_pgoff+0x457/0x5b0 mm/mmap.c:1593
 do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4472e9
Code: e8 2c bb 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 5b 07 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f8789a02da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
RAX: ffffffffffffffda RBX: 00000000006dcc38 RCX: 00000000004472e9
RDX: 0000000000000000 RSI: 0000000000003000 RDI: 0000000020ffd000
RBP: 00000000006dcc30 R08: 0000000000000004 R09: 0000000000000000
R10: 0000000000000011 R11: 0000000000000246 R12: 00000000006dcc3c
R13: 00007ffef74e260f R14: 00007f8789a039c0 R15: 0000000000000000

Crashes (35):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/03/13 06:06 upstream 3cc6e2c599cd d850e9d0 .config console log report syz C ci-upstream-kasan-gce-selinux-root
2020/03/10 14:59 upstream 30bb5572ce7a 35f53e45 .config console log report syz C ci-upstream-kasan-gce-root
2020/03/23 04:57 linux-next 770fbb32d34e 78267cec .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2019/08/16 07:06 linux-next 17da61ae48ec 8fd428a1 .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2020/03/19 13:44 upstream 5076190daded 2c31c529 .config console log report ci-upstream-kasan-gce-root
2020/03/06 21:41 upstream 63623fd44972 c88c7b75 .config console log report ci-upstream-kasan-gce-selinux-root
2020/03/04 21:26 upstream 63623fd44972 c88c7b75 .config console log report ci-upstream-kasan-gce
2020/02/27 12:58 upstream f8788d86ab28 59b57593 .config console log report ci-upstream-kasan-gce
2020/02/16 15:51 upstream db70e26e33ee cf914200 .config console log report ci-upstream-kasan-gce-selinux-root
2020/01/30 01:56 upstream b3a608222336 5ed23f9a .config console log report ci-upstream-kasan-gce-selinux-root
2019/12/20 15:33 upstream 7e0165b2f1a9 e30cbdae .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/21 20:20 upstream 7d194c2100ad b24d2b8a .config console log report ci-upstream-kasan-gce-selinux-root
2019/10/17 21:26 upstream 283ea345934d 8c88c9c1 .config console log report ci-upstream-kasan-gce
2019/09/24 15:34 upstream 4c07e2ddab5b 0942eab8 .config console log report ci-upstream-kasan-gce-selinux-root
2019/09/21 13:53 upstream f97c81dc6ca5 d96e88f3 .config console log report ci-upstream-kasan-gce
2019/03/26 07:15 upstream a3ac7917b730 55684ce1 .config console log report ci-upstream-kasan-gce-selinux-root
2019/03/25 19:48 upstream 8c2ffd917477 2c86e0a5 .config console log report ci-upstream-kasan-gce-selinux-root
2019/03/06 07:32 upstream 63bdf4284c38 16559f86 .config console log report ci-upstream-kasan-gce-smack-root
2019/01/11 18:43 upstream de6629eb262e c3f3344c .config console log report ci-upstream-kasan-gce-smack-root
2019/01/06 11:46 upstream b5aef86e089a 94f8adb5 .config console log report ci-upstream-kasan-gce-selinux-root
2018/11/28 05:34 upstream ef78e5ec9214 4b6d14f2 .config console log report ci-upstream-kasan-gce-smack-root
2018/11/27 13:40 upstream ef78e5ec9214 4b6d14f2 .config console log report ci-upstream-kasan-gce-root
2018/11/25 17:56 upstream e195ca6cb6f2 3d3ec907 .config console log report ci-upstream-kasan-gce-selinux-root
2018/11/04 05:07 upstream 83650fd58a93 8bd6bd63 .config console log report ci-upstream-kasan-gce-root
2020/01/24 13:39 upstream 4703d9119972 2e95ab33 .config console log report ci-upstream-kasan-gce-386
2020/03/13 17:52 linux-next 770fbb32d34e d850e9d0 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/02/09 01:14 linux-next 6dff1565d69c 06150bf1 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/01/12 14:42 linux-next 6c09d7dbb7d3 31290a45 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/11/24 12:19 linux-next b9d3d0140506 598ca6c8 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/11/05 14:16 linux-next 51309b9d73f5 0f3ec414 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/10/05 11:40 linux-next 311ef88adfa3 f3f7d9c8 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/10/04 03:56 linux-next 2521ffab5375 fc17ba49 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/08/16 06:36 linux-next 17da61ae48ec 8fd428a1 .config console log report ci-upstream-linux-next-kasan-gce-root
2019/02/02 03:08 linux-next dc4c89997735 564f9a4f .config console log report ci-upstream-linux-next-kasan-gce-root
2018/11/04 23:08 linux-next 25e9471b6a27 8bd6bd63 .config console log report ci-upstream-linux-next-kasan-gce-root
* Struck through repros no longer work on HEAD.