Root Cause Analysis

Involved Functions, Structs, and Files

• Files:
  • mm/mmap.c
  • mm/memory.c
  • lib/rbtree.c / include/linux/rbtree_augmented.h
• Functions:
  • mmap_region() (mm/mmap.c)
  • unmap_region() (mm/mmap.c)
  • free_pgtables() (mm/memory.c)
  • unlink_file_vma() / __remove_shared_vm_struct() (mm/mmap.c)
  • vma_interval_tree_remove() (mm/interval_tree.c)
  • rb_erase_augmented() / ____rb_erase_color() (lib/rbtree.c, include/linux/rbtree_augmented.h)
• Structs and Fields:
  • struct vm_area_struct (vma->vm_start, vma->vm_end, vma->vm_flags, vma->shared.rb)
  • struct mm_struct (mm->mm_mt)
  • struct address_space (mapping->i_mmap)
  • struct rb_node (__rb_parent_color, rb_left, rb_right)

Technical Mechanism and Triggering Flow

1. Successor VMA Pointer Initialization in mmap_region(): In mmap_region() (mm/mmap.c), after determining the range to map and unmapping any conflicting VMAs, the kernel locates the neighboring VMAs:

   next = mas_next(&mas, ULONG_MAX);
   prev = mas_prev(&mas, 0);
   

   Here, next holds a pointer to the existing successor vm_area_struct located at or above end, if one exists in mm->mm_mt.

2. Clobbering next on vma_merge() Failure: When mapping a file, call_mmap(file, vma) is invoked. Device drivers or subsystems (such as io_uring_mmap() calling remap_pfn_range()) can alter vma->vm_flags (e.g., adding VM_SPECIAL flags like VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP). Because flags changed, mmap_region() checks whether merging with prev is now possible:

   if (unlikely(vm_flags != vma->vm_flags && prev)) {
       next = vma_merge(mm, prev, vma->vm_start, vma->vm_end, vma->vm_flags,
                        NULL, vma->vm_file, vma->vm_pgoff, NULL, NULL_VM_UFFD_CTX, NULL);
       if (next) {
           ...
           goto unmap_writable;
       }
   }
   

   If merging fails (e.g. because vma->vm_flags & VM_SPECIAL prohibits merging in vma_merge()), vma_merge() returns NULL. Storing this result directly into next clobbers the previously found successor VMA pointer, setting next = NULL.

3. Erroneous Range Ceiling during Error Recovery: Later in mmap_region(), if an error occurs—such as mas_preallocate(&mas, vma, GFP_KERNEL) failing with -ENOMEM (triggered by fault injection)—execution jumps to unmap_and_free_vma:

   unmap_and_free_vma:
       fput(vma->vm_file);
       vma->vm_file = NULL;
   
       /* Undo any partial mapping done by a device driver. */
       unmap_region(mm, mas.tree, vma, prev, next, vma->vm_start, vma->vm_end);
   

   unmap_region() receives next == NULL. It then invokes free_pgtables():

   free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
                 next ? next->vm_start : USER_PGTABLES_CEILING);
   

   Because next is NULL, the ceiling argument passed to free_pgtables() defaults to USER_PGTABLES_CEILING.

4. Premature Unlinking of Active VMAs: In free_pgtables() (mm/memory.c), the maple tree iterator mas starts at vma->vm_end and iterates using mas_find(&mas, ceiling - 1). Because ceiling is USER_PGTABLES_CEILING rather than the successor VMA's start address:

   • The loop finds all active VMAs residing above vma->vm_end in mm->mm_mt.
   • For each of those VMAs, free_pgtables() invokes unlink_anon_vmas(vma) and unlink_file_vma(vma), and frees their page table ranges (free_pgd_range()).
   • unlink_file_vma() calls vma_interval_tree_remove(vma, &mapping->i_mmap), removing these active VMAs from their backing file's interval tree (i_mmap).
   • However, because these VMAs were not actually being unmapped, they remain present in mm->mm_mt.

5. Double Unlink and Tree Invariant Violation on Process Exit: When the process subsequently exits, exit_mmap() (mm/mmap.c) runs to tear down the address space:

   free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
   

   free_pgtables() traverses mm->mm_mt and encounters those active VMAs a second time. It invokes unlink_file_vma(vma) again, which calls vma_interval_tree_remove(vma, &mapping->i_mmap) on nodes that were already removed from the red-black tree.

   In include/linux/rbtree_augmented.h and lib/rbtree.c, rb_erase_augmented() attempts to delete and rebalance the node. Because the node was already detached, the red-black tree structural invariants are broken. Rebalancing enters ____rb_erase_color() (lib/rbtree.c:354), which expects a black node undergoing deletion to have a non-NULL sibling. Instead, sibling = parent->rb_left evaluates to NULL. Checking rb_is_red(sibling) dereferences sibling->__rb_parent_color at address 0x0, causing a General Protection Fault under KASAN shadow memory checking (0xdffffc0000000000) and crashing the kernel.