Adding a Memory Shrinking API to the Linux Kernel's Rust Allocator

As a developer new to Linux kernel development, sending your first patch series can be both exciting and daunting. Recently, my 3-patch series [1] was merged into the mainline Linux kernel after six revision cycles. The series adds a new shrink_to() method to the kernel's Rust vector type and uses it to optimize memory consumption in the Rust-based Android Binder driver.

In this post, I will break down the problem I addressed, the design of the solution, and what it was like navigating the kernel review process for the first time.

Background: What is Rust Binder?

Android Binder is the primary Inter-Process Communication (IPC) mechanism for the Android operating system. Because it is highly performance-sensitive and handles a large volume of untrusted data from various sandboxed applications, it has historically been a frequent target for memory-safety vulnerabilities.

To address this, Google engineers spearheaded a rewrite of the Binder driver in Rust (located under drivers/android/binder/).

Inside the driver's context manager (drivers/android/binder/context.rs), the system tracks all registered processes in a dynamic vector named all_procs, represented as a KVVec<Arc<Process>>. KVVec is a kernel-specific vector type backed by the KVmalloc allocator, which can use either kmalloc or vmalloc depending on the allocation size.

The Problem: Memory Footprint of Dynamic Vectors

Like standard vectors, KVVec manages its memory allocations dynamically. When new processes register with a Binder context, the vector grows, typically doubling its capacity to minimize the number of reallocations.

However, during normal system operations, such as device boot or heavy app launch/termination cycles, the number of registered processes spikes and then falls. When processes terminate or close their Binder file descriptors, they are deregistered:

pub(crate) fn deregister_process(self: &Arc<Self>, proc: &Arc<Process>) {
    ...
    let mut manager = self.manager.lock();
    manager.all_procs.retain(|p| !Arc::ptr_eq(p, proc));
    ...
}

The .retain() method updates the logical length of the vector but leaves the allocated physical capacity unchanged. Over time, this results in unused, wasted kernel memory allocated to empty slots in the vector.

Prior to this patch series, KVVec had no mechanism to reduce its capacity at all; there was simply no shrink_to() method available.

The Solution

Part 1: Implementing shrink_to() for KVVec

The first and most substantial patch in the series (commit 47ac2a4b5cd8 [2]) adds a new shrink_to(min_capacity, flags) method to KVVec (i.e., Vec<T, KVmalloc>) in rust/kernel/alloc/kvec.rs.

The implementation has to handle two distinct allocation backends:

• kmalloc allocations: The method delegates to realloc(), letting the kernel's slab allocator decide whether shrinking is worthwhile.
• vmalloc allocations: Since vrealloc does not yet support in-place shrinking, the method only shrinks if at least one full page of memory can be freed, using an explicit allocate-copy-free sequence.

The distinction is made at runtime using is_vmalloc_addr().

Part 2: A Hysteresis-Based Shrinking Strategy in Binder

With shrink_to() available, the third patch (commit 34268365a9e9 [3]) uses it in the Binder driver to reclaim unused memory during process deregistration.

If we were to shrink the vector to its exact length on every deregistration, any subsequent registration would immediately force a new kernel allocation. This "thrashing" would hurt IPC latency.

Instead, the patch implements a conservative shrinking strategy with hysteresis:

1. Trigger Condition: Only shrink when the number of active processes (len) drops below a quarter of the allocated capacity (cap / 4).
2. Target Capacity: Shrink the vector to twice the current length (len * 2) rather than the exact length.

Here is the implementation:

// Shrink the vector if it has significant unused capacity to avoid memory waste,
// but use a conservative strategy to prevent shrink-then-regrow oscillation.
// Only shrink when length drops below 1/4 of capacity, and shrink to twice the length.
let len = manager.all_procs.len();
let cap = manager.all_procs.capacity();
if len < cap / 4 {
    // Shrink to twice the current length. Ignore allocation failures since this
    // is just an optimization; the vector remains valid even if shrinking fails.
    let _ = manager.all_procs.shrink_to(len * 2, GFP_KERNEL);
}

Why This Math Works

Below is a diagram illustrating how the hysteresis shrinking strategy behaves in the scenario where capacity is 32 and active processes drop to 7.

Suppose our vector has grown to a capacity of 32 elements during a peak load, but the number of active processes then drops to 7.

• Since 7 < (32 / 4) = 8, the trigger condition is met.
• We shrink the vector to a capacity of 7 * 2 = 14.
• This immediately frees up the memory associated with 18 unused element slots.
• Crucially, the vector still has 7 empty slots. This allows up to 7 new processes to register before another reallocation is required, providing a buffer against shrink-then-grow oscillation.

Additionally, shrinking memory is purely an optimization. If the system is under extreme memory pressure and the shrink_to allocation fails, we gracefully discard the result using let _ = .... The vector remains valid and continues to function normally.

The Review Process

This patch series went through six revision cycles before being merged. Each version incorporated detailed feedback from kernel maintainers:

• Early versions explored a Shrinkable trait and generic implementations. Through review, the approach was narrowed to a focused KVVec-specific implementation.
• The kmalloc-vs-vmalloc distinction was refined across several iterations, moving from a trait-based approach to a runtime is_vmalloc_addr() check.
• The binder shrinking strategy itself evolved: early versions used a cap > 128 threshold and shrunk to cap / 2. Reviewers guided the design toward the cleaner len < cap / 4 trigger with len * 2 target.

Entering kernel development can feel intimidating due to the mailing-list workflow and strict review standards. However, the feedback loop for this patch series was incredibly constructive.

I want to extend my sincere gratitude to:

• Alice Ryhl (Google), who suggested the optimization, co-designed the shrinking strategy, and reviewed all three patches.
• Danilo Krummrich (Red Hat), who co-suggested the shrink_to API, provided extensive design feedback across all six revisions, including the kmalloc/vmalloc split, SAFETY comment structure, and API scoping, and Acked the patches.
• Greg Kroah-Hartman, who merged the series into the mainline tree.

Their patience and detailed reviews made my first contribution to the Linux kernel a rewarding learning experience. I look forward to working on more kernel and Rust-for-Linux contributions in the future.

Key Takeaways

What's Next

Sharp-eyed readers may have noticed that the shrink_to() implementation includes a workaround for vmalloc-backed allocations: since vrealloc() does not yet support in-place shrinking, the method has to allocate a new buffer, copy, and free the old one.

I have a follow-up patch series currently in linux-next that addresses this at the root by implementing actual page freeing in vrealloc() when shrinking across a page boundary. It touches the core mm/vmalloc subsystem, and if it lands, it would allow the KVVec::shrink_to() workaround to be replaced with a straightforward realloc() call for all allocator backends.

More on that once there's news to share. Stay tuned.

References

• [1] https://lore.kernel.org/all/20260216-binder-shrink-vec-v3-v6-0-ece8e8593e53@zohomail.in/
• [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=47ac2a4b5cd8
• [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34268365a9e9424e38083c8f318cc34b153dcb07