possible deadlock in perf_trace_init

Status: upstream: reported C repro on 2019/04/24 18:48
Reported-by: syzbot+b0ccd80ce35eb840e66d@syzkaller.appspotmail.com
First crash: 2680d, last: 2277d
Fix bisection the fix commit could be any of (bisect log):
  01fd1694b93c Linux 4.14.141
  4139fb08c05f Linux 4.14.187
   
▶ ▼ Similar bugs (2)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in perf_trace_init 4 syz 7 3264d 3269d 0/29 closed as invalid on 2017/10/27 06:57
upstream possible deadlock in perf_trace_init (2) trace 4 2 3182d 3187d 0/29 closed as dup on 2018/02/12 15:04
▶ ▼ Last patch testing requests (8)
Created Duration User Patch Repo Result
2023/01/18 21:32 10m retest repro linux-4.14.y report log
2023/01/18 12:32 10m retest repro linux-4.14.y report log
2023/01/18 02:32 9m retest repro linux-4.14.y report log
2023/01/17 17:32 11m retest repro linux-4.14.y report log
2022/09/03 22:27 9m retest repro linux-4.14.y report log
2022/09/03 21:27 10m retest repro linux-4.14.y report log
2022/09/03 20:27 10m retest repro linux-4.14.y report log
2022/09/03 19:27 10m retest repro linux-4.14.y report log
▶ ▼ Fix bisection attempts (7)
Created Duration User Patch Repo Result
2020/06/30 11:04 29m (2) bisect fix linux-4.14.y OK (2) job log
2020/05/31 10:18 22m bisect fix linux-4.14.y OK (0) job log log
2020/05/01 09:54 24m bisect fix linux-4.14.y OK (0) job log log
2020/04/01 05:39 22m bisect fix linux-4.14.y OK (0) job log log
2020/03/02 05:15 23m bisect fix linux-4.14.y OK (0) job log log
2020/02/01 04:51 24m bisect fix linux-4.14.y OK (0) job log log
2020/01/02 04:24 23m bisect fix linux-4.14.y OK (0) job log log

Sample crash report:
audit: type=1400 audit(1567658808.245:36): avc:  denied  { map } for  pid=7076 comm="syz-executor922" path="/root/syz-executor922931172" dev="sda1" ino=16483 scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1
======================================================
WARNING: possible circular locking dependency detected
4.14.141 #37 Not tainted
------------------------------------------------------
syz-executor922/7076 is trying to acquire lock:
 (event_mutex){+.+.}, at: [<ffffffff8162b8e8>] perf_trace_init+0x58/0xaa0 kernel/trace/trace_event_perf.c:216

but task is already holding lock:
 (&cpuctx_mutex/1){+.+.}, at: [<ffffffff816bbe90>] perf_event_ctx_lock_nested+0x150/0x2c0 kernel/events/core.c:1235

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #5 (&cpuctx_mutex/1){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3991
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       mutex_lock_double kernel/events/core.c:9904 [inline]
       __perf_event_ctx_lock_double kernel/events/core.c:9963 [inline]
       SYSC_perf_event_open+0x134c/0x2690 kernel/events/core.c:10225
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9983
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #4 (&cpuctx_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3991
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_event_init_cpu+0xc2/0x170 kernel/events/core.c:11223
       perf_event_init+0x2d8/0x31a kernel/events/core.c:11270
       start_kernel+0x3b6/0x6fd init/main.c:621
       x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:381
       x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:362
       secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240

-> #3 (pmus_lock){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3991
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_event_init_cpu+0x2f/0x170 kernel/events/core.c:11217
       cpuhp_invoke_callback+0x1ea/0x1ab0 kernel/cpu.c:184
       cpuhp_up_callbacks kernel/cpu.c:572 [inline]
       _cpu_up+0x228/0x530 kernel/cpu.c:1134
       do_cpu_up+0x121/0x150 kernel/cpu.c:1169
       cpu_up+0x1b/0x20 kernel/cpu.c:1177
       smp_init+0x157/0x170 kernel/smp.c:578
       kernel_init_freeable+0x30b/0x532 init/main.c:1067
       kernel_init+0x12/0x162 init/main.c:999
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

-> #2 (cpu_hotplug_lock.rw_sem){++++}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3991
       percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
       percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
       cpus_read_lock+0x3d/0xc0 kernel/cpu.c:295
       static_key_slow_inc+0x13/0x30 kernel/jump_label.c:123
       tracepoint_add_func kernel/tracepoint.c:223 [inline]
       tracepoint_probe_register_prio+0x4d6/0x6d0 kernel/tracepoint.c:283
       tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
       trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8106
       perf_try_init_event+0x164/0x200 kernel/events/core.c:9342
       perf_init_event kernel/events/core.c:9380 [inline]
       perf_event_alloc.part.0+0xd90/0x25b0 kernel/events/core.c:9640
       perf_event_alloc kernel/events/core.c:9993 [inline]
       SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10097
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9983
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #1 (tracepoints_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3991
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       tracepoint_probe_register_prio+0x36/0x6d0 kernel/tracepoint.c:279
       tracepoint_probe_register+0x2b/0x40 kernel/tracepoint.c:304
       trace_event_reg+0x277/0x330 kernel/trace/trace_events.c:305
       perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
       perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
       perf_trace_init+0x449/0xaa0 kernel/trace/trace_event_perf.c:221
       perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8106
       perf_try_init_event+0x164/0x200 kernel/events/core.c:9342
       perf_init_event kernel/events/core.c:9380 [inline]
       perf_event_alloc.part.0+0xd90/0x25b0 kernel/events/core.c:9640
       perf_event_alloc kernel/events/core.c:9993 [inline]
       SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10097
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9983
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #0 (event_mutex){+.+.}:
       check_prev_add kernel/locking/lockdep.c:1901 [inline]
       check_prevs_add kernel/locking/lockdep.c:2018 [inline]
       validate_chain kernel/locking/lockdep.c:2460 [inline]
       __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3991
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       perf_trace_init+0x58/0xaa0 kernel/trace/trace_event_perf.c:216
       perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8106
       perf_try_init_event+0xe6/0x200 kernel/events/core.c:9342
       perf_init_event kernel/events/core.c:9380 [inline]
       perf_event_alloc.part.0+0xd90/0x25b0 kernel/events/core.c:9640
       perf_event_alloc kernel/events/core.c:9993 [inline]
       SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10097
       SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9983
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

other info that might help us debug this:

Chain exists of:
  event_mutex --> &cpuctx_mutex --> &cpuctx_mutex/1

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&cpuctx_mutex/1);
                               lock(&cpuctx_mutex);
                               lock(&cpuctx_mutex/1);
  lock(event_mutex);

 *** DEADLOCK ***

2 locks held by syz-executor922/7076:
 #0:  (&pmus_srcu){....}, at: [<ffffffff816bf8aa>] perf_event_alloc.part.0+0xbaa/0x25b0 kernel/events/core.c:9636
 #1:  (&cpuctx_mutex/1){+.+.}, at: [<ffffffff816bbe90>] perf_event_ctx_lock_nested+0x150/0x2c0 kernel/events/core.c:1235

stack backtrace:
CPU: 1 PID: 7076 Comm: syz-executor922 Not tainted 4.14.141 #37
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x138/0x197 lib/dump_stack.c:53
 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1901 [inline]
 check_prevs_add kernel/locking/lockdep.c:2018 [inline]
 validate_chain kernel/locking/lockdep.c:2460 [inline]
 __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
 lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3991
 __mutex_lock_common kernel/locking/mutex.c:756 [inline]
 __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 perf_trace_init+0x58/0xaa0 kernel/trace/trace_event_perf.c:216
 perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:8106
 perf_try_init_event+0xe6/0x200 kernel/events/core.c:9342
 perf_init_event kernel/events/core.c:9380 [inline]
 perf_event_alloc.part.0+0xd90/0x25b0 kernel/events/core.c:9640
 perf_event_alloc kernel/events/core.c:9993 [inline]
 SYSC_perf_event_open+0xad1/0x2690 kernel/events/core.c:10097
 SyS_perf_event_open+0x34/0x40 kernel/events/core.c:9983
 do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x440449
RSP: 002b:00007ffd52629378 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 0000000000440449
RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000020000000
RBP: 00000000006ca018 R08: 0000000000000000 R09: 00000000004002c8
R10: 0000000000000003

Crashes (27):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/09/05 04:50 linux-4.14.y 01fd1694b93c 040fda58 .config console log report syz C ci2-linux-4-14
2019/09/05 04:31 linux-4.14.y 01fd1694b93c 040fda58 .config console log report syz C ci2-linux-4-14
2019/08/25 23:16 linux-4.14.y b5260801526c d21c5d9d .config console log report syz C ci2-linux-4-14
2019/08/20 20:19 linux-4.14.y 45f092f9e9cb 6b8391d0 .config console log report syz ci2-linux-4-14
2019/12/03 04:22 linux-4.14.y fbc5fe7a54d0 ab342da3 .config console log report ci2-linux-4-14
2019/12/02 09:29 linux-4.14.y fbc5fe7a54d0 f879db37 .config console log report ci2-linux-4-14
2019/11/28 21:21 linux-4.14.y 43598c571e7e 97264cb1 .config console log report ci2-linux-4-14
2019/11/22 03:29 linux-4.14.y f56f3d0e65ad 8098ea0f .config console log report ci2-linux-4-14
2019/11/20 13:06 linux-4.14.y 775d01b65b5d 432c7650 .config console log report ci2-linux-4-14
2019/11/12 00:22 linux-4.14.y 4762bcd451a9 377d77fa .config console log report ci2-linux-4-14
2019/11/11 05:33 linux-4.14.y c9fda4f22428 dc438b91 .config console log report ci2-linux-4-14
2019/11/04 22:59 linux-4.14.y ddef1e8e3f6e 18e12644 .config console log report ci2-linux-4-14
2019/11/01 10:53 linux-4.14.y ddef1e8e3f6e a41ca8fa .config console log report ci2-linux-4-14
2019/11/01 09:18 linux-4.14.y ddef1e8e3f6e a41ca8fa .config console log report ci2-linux-4-14
2019/11/01 09:15 linux-4.14.y ddef1e8e3f6e a41ca8fa .config console log report ci2-linux-4-14
2019/11/01 09:12 linux-4.14.y ddef1e8e3f6e a41ca8fa .config console log report ci2-linux-4-14
2019/10/30 12:30 linux-4.14.y ddef1e8e3f6e 5ea87a66 .config console log report ci2-linux-4-14
2019/10/18 01:49 linux-4.14.y b98aebd29824 8c88c9c1 .config console log report ci2-linux-4-14
2019/10/18 01:47 linux-4.14.y b98aebd29824 8c88c9c1 .config console log report ci2-linux-4-14
2019/09/24 19:00 linux-4.14.y f6e27dbb1afa f8368f99 .config console log report ci2-linux-4-14
2019/08/25 22:41 linux-4.14.y b5260801526c d21c5d9d .config console log report ci2-linux-4-14
2019/08/23 09:54 linux-4.14.y 45f092f9e9cb ca6f3cfa .config console log report ci2-linux-4-14
2019/08/20 19:42 linux-4.14.y 45f092f9e9cb 6b8391d0 .config console log report ci2-linux-4-14
2019/08/18 16:28 linux-4.14.y 45f092f9e9cb 55bf8926 .config console log report ci2-linux-4-14
2019/06/24 11:08 linux-4.14.y a5758c531177 472f0082 .config console log report ci2-linux-4-14
2019/04/30 17:58 linux-4.14.y fa5941f45d7e 20f16bef .config console log report ci2-linux-4-14
2019/04/24 17:47 linux-4.14.y 68d7a45eec10 8e3c52b1 .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.