Technical Root Cause Analysis
1. Components Involved
- Source File:
drivers/block/nbd.c
- Functions:
nbd_genl_reconnect_sock_cb()nbd_add_socket()nbd_reclassify_socket()nbd_mark_nsock_dead()recv_work()
- Structs & Fields:
struct nbd_sock: nsock->sock, nsock->tx_lockstruct socket / struct sock: sk->sk_lock
2. Background and Invariants
In the Linux kernel, standard userspace network sockets (such as TCP/IPv4 sockets belonging to the sk_lock-AF_INET lockdep class) establish an ordering dependency:
$$\text{sk_lock-AF_INET} \longrightarrow \text{&mm->mmap_lock}$$
This occurs because data-copying operations like tcp_sendmsg_locked() hold lock_sock(sk) and invoke copy_from_iter() on user-provided buffers; if those buffers are memory-mapped, page faults trigger __might_fault(), which acquires &mm->mmap_lock.
In turn, operations holding &mm->mmap_lock (e.g., mmap() or page allocation) can trigger direct memory reclaim (fs_reclaim). Under fs_reclaim, dirty pages may be written back to swap or filesystems backed by block devices, establishing:
$$\text{fs_reclaim} \longrightarrow \text{&q->q_usage_counter(io)}$$
When a block device request queue is managed by blk-mq:
- Freezing or updating queue scheduler configuration (
blk_mq_update_nr_hw_queues(), elevator_change()) takes &q->elevator_lock under q->q_usage_counter.
- Quiescing queues (
elevator_switch()) calls synchronize_srcu(&set->srcu) to wait for inflight request dispatch.
- Request dispatch invokes the block driver's
.queue_rq handler (nbd_queue_rq()), which locks &cmd->lock and calls nbd_handle_cmd(), which acquires &nsock->tx_lock.
Because NBD transmits and manages network sockets under &nsock->tx_lock inside the block I/O path, acquiring a socket lock (lock_sock()) while holding &nsock->tx_lock would bridge:
$$\text{&nsock->tx_lock} \longrightarrow \text{sk_lock-AF_INET}$$
thereby closing a circular locking dependency:
$$\text{fs_reclaim} \longrightarrow \text{&q->q_usage_counter(io)} \longrightarrow \text{&q->elevator_lock} \longrightarrow \text{set->srcu} \longrightarrow \text{&cmd->lock} \longrightarrow \text{&nsock->tx_lock} \longrightarrow \text{sk_lock-AF_INET} \longrightarrow \text{&mm->mmap_lock} \longrightarrow \text{fs_reclaim}$$
To prevent this collision, NBD sockets must not use the generic sk_lock-AF_INET lockdep class. The driver provides nbd_reclassify_socket() (defined at lines 1245–1272 of drivers/block/nbd.c), which initializes sk->sk_lock with dedicated lock classes (sk_lock-AF_INET-NBD and slock-AF_INET-NBD).
3. Defect and Failure Mechanism
While initial socket addition in nbd_add_socket() correctly calls nbd_reclassify_socket(sock) after nbd_get_socket():
static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg, bool netlink)
{
...
sock = nbd_get_socket(nbd, arg, &err);
if (!sock)
return err;
nbd_reclassify_socket(sock);
...
}
The netlink socket reconnection callback, nbd_genl_reconnect_sock_cb() (invoked via nbd_genl_reconfigure()), replaces dead sockets on a running device using nbd_get_socket(), but fails to call nbd_reclassify_socket(sock):
static int nbd_genl_reconnect_sock_cb(struct nbd_device *nbd, unsigned long arg)
{
...
sock = nbd_get_socket(nbd, arg, &err);
if (!sock)
return err;
args = kzalloc_obj(*args);
...
nsock->sock = sock;
...
INIT_WORK(&args->work, recv_work);
queue_work(nbd->recv_workq, &args->work);
...
}
Because nbd_reclassify_socket() is omitted:
- The reconnected socket retains the generic
sk_lock-AF_INET lockdep key.
- When an error or connection reset occurs,
recv_work() acquires mutex_lock(&nsock->tx_lock) and calls nbd_mark_nsock_dead(nbd, nsock, 1).
nbd_mark_nsock_dead() invokes kernel_sock_shutdown(nsock->sock, SHUT_RDWR), which calls inet_shutdown() and acquires lock_sock(sk).- Lockdep records the dependency
&nsock->tx_lock -> sk_lock-AF_INET.
- Later, when an unrelated allocation (such as
resv_map_alloc() in hugetlb_reserve_pages() during hugetlbfs_file_mmap()) executes with GFP_KERNEL under &mm->mmap_lock and inode lock, lockdep traverses the graph and detects the circular lock dependency cycle, issuing the reported splat.