Root Cause Analysis
1. Involved Entities
2. Background and Object Lifecycle
The BPF memory allocator (struct bpf_mem_alloc) allocates memory units through per-CPU cache buckets (struct bpf_mem_cache *c).
- When an object is allocated via
unit_alloc() (kernel/bpf/memalloc.c:819) or bpf_mem_cache_alloc_flags() (kernel/bpf/memalloc.c:1022), the memory unit prepends an 8-byte header (struct llist_node *llnode = ptr - LLIST_NODE_SZ;) storing a pointer to the allocating per-CPU cache:
*(struct bpf_mem_cache **)llnode = c;
- When an object is freed on another (or the same) CPU using
unit_free() or unit_free_rcu(), the allocator reads back this pointer and records it in c->tgt:
/*
* Remember bpf_mem_cache that allocated this object.
* The hint is not accurate.
*/
c->tgt = *(struct bpf_mem_cache **)llnode;
This hint is designed to return freed objects back to the cache of the CPU that originally allocated them when draining local lists, preventing cross-CPU out-of-memory (OOM) imbalances while avoiding atomic overhead in fast paths.
- In
unit_free_rcu(), after updating c->tgt, the freed node is enqueued onto c->free_by_rcu. When check_free_by_rcu(c) runs, pending nodes are transferred to c->waiting_for_gp, c->call_rcu_in_progress is set to 1, and an RCU grace period callback is scheduled via:
call_rcu_hurry(&c->rcu, __free_by_rcu);
- Once the regular RCU grace period expires, RCU invokes
__free_by_rcu(&c->rcu) in softirq context. __free_by_rcu() reads c->tgt to forward the grace-period-expired nodes into tgt->free_by_rcu_ttrace:
static void __free_by_rcu(struct rcu_head *head)
{
struct bpf_mem_cache *c = container_of(head, struct bpf_mem_cache, rcu);
struct bpf_mem_cache *tgt = c->tgt;
...
llist_add_batch(llnode, c->waiting_for_gp_tail, &tgt->free_by_rcu_ttrace);
do_call_rcu_ttrace(tgt);
...
atomic_set(&c->call_rcu_in_progress, 0);
}
3. Technical Mechanism of the Data Race
The data race reported by KCSAN occurs because c->tgt is updated and read using plain C memory accesses without memory barriers, concurrency annotations (READ_ONCE() / WRITE_ONCE()), or synchronization across execution contexts:
-
Unsynchronized Write Outside IRQ-Disabled Section:
In unit_free_rcu():
static void notrace unit_free_rcu(struct bpf_mem_cache *c, void *ptr)
{
struct llist_node *llnode = ptr - LLIST_NODE_SZ;
unsigned long flags;
c->tgt = *(struct bpf_mem_cache **)llnode; // Line 907: Plain store with IRQs enabled
local_irq_save(flags);
...
c->tgt is written while interrupts are enabled in task context.
-
Concurrent Read in Softirq Context:
While an RCU grace period is in progress (c->call_rcu_in_progress == 1) or being dispatched, task context (e.g., executing bpf_map_update_value() -> trie_update_elem() -> bpf_mem_cache_free_rcu() -> unit_free_rcu()) writes to c->tgt.
If a timer interrupt fires on the local CPU right around line 907, the interrupt return path (__irq_exit_rcu()) invokes pending softirqs via handle_softirqs() -> rcu_core() -> rcu_do_batch().
The RCU batch processes __free_by_rcu(), which executes line 394:
struct bpf_mem_cache *tgt = c->tgt; // Line 394: Plain load
This load runs concurrently with or directly interrupts the store in unit_free_rcu().
-
Cross-CPU / RCU Thread Concurrency:
Additionally, because RCU callbacks can execute on other CPUs (such as under CONFIG_RCU_NOCB_CPU kthreads or after CPU-hotplug offline events), __free_by_rcu() can run concurrently on another CPU while CPU 0's task context or IRQ work updates c->tgt.
Because both operations are plain, unannotated memory accesses to the shared field c->tgt, KCSAN flags this as a data race between task context and softirq context.