We Replaced MMAP with Io_uring in Our Rust Query Engine. It Got Slower

In the beginning, location was mmap. It was convenient: it fto america lazily publication immense numbers of Arrow IPC files from disk without managing representation ourselves. It fresh our record format perfectly — Arrow IPC’s layout is designed for zero-copy random access, and mmap gives you precisely that.

Then we deployed to production, ran existent concurrent query loads, and mmap became a existent problem.

Our Workload

At Conviva, we analyse trillions of events a time to pinpoint and diagnose end personification experience. At the halfway of our architecture is an arena and shape study motor built connected DataFusion, Arrow, Rust, Rayon, and Tokio. Raw events get transformed, encoded successful a proprietary mostly-numeric format, and stored successful the cloud. We transcript them to section NVMe and publication ample (~3–5 GB) Arrow IPC files. We chose Arrow IPC for simplicity and velocity — its representation and disk layouts are identical, truthful decode costs is minimal, and mmap gives america zero-copy sounds natively supported by arrow-rust. A emblematic query touches 6 columns crossed 8 batch files (one batch per file), ~1.6 GB per batch, ~13 GB full per time of data.

The Test Setup

Hardware: 192-core box, ~750 GB RAM. Two disk configs during the investigation: 2× NVMe LVM-striped (~5.5 GB/s fio ceiling) and 32× NVMe RAID-0 (~21 GB/s fio ceiling). Kernel 5.15 during investigation, 6.x successful production.

The Production Symptom

At lighter loads, mmap worked good — fast, serving queries from earthy events successful seconds. The problem started nether heavier concurrency. Some latency summation nether load is expected — much queries competing for the aforesaid CPU. But we saw p95s and p99s spike good beyond what linear scaling would predict, pinch rows scanned per halfway dropping sharply moreover aft accounting for concurrency:

• OS page cache shrank — each pod consumed much representation arsenic backstage allocations, little arsenic shared cache
• A immense number of page faults
• p95 spiked from ~30s to 150s+ nether existent concurrent load
• Adding pods made it worse, not better

That pointed astatine mmap page-cache thrashing nether representation pressure.

Controlled Benchmark: 1 Pod vs. 4 Pods

To isolate the effect, we ran a controlled test: 1 pod vs. 4 pods connected the aforesaid host, aforesaid concurrent query load. We expected 4 pods to triumph — much parallelism, amended isolation. We were wrong.

For 14-day queries — agelong capable to capable the page cache — 1 pod hit 4 pods by a existent margin: 41% faster astatine max, >20% astatine p95. The mmap page cache lives connected the big and is shared crossed pods, truthful the 4 pods weren’t fighting each different for CPU — they were fighting for page cache. perf grounds connected the aforesaid tally showed 100% fastener contention astatine the kernel level.

The halfway issue: mmap’s page cache is implicit shared state. Every process connected the big shares 1 cache, 1 fastener hierarchy, 1 eviction policy. No azygous pod controls the assets that matters astir for publication latency, and arsenic concurrency rises, everyone’s portion of it shrinks.

A Storm of Page Faults

We didn’t want to conscionable guess, truthful we dug into the mmap mechanics and the page responsibility stats. When you mmap a file, the exertion gets a region of representation backed by the record connected disk. Here’s what happens connected access:

1. The CPU touches a Virtual Memory Address (VMA) pinch nary beingness page attached, and throws a page fault.
2. The kernel handles the exception: looks up the VMA, checks ownership, and acquires a lock, since different thread mightiness beryllium modifying aliases unmapping that virtual abstraction concurrently.
3. Linux 6.4+ has a accelerated per-VMA lock; earlier kernels autumn backmost to the slower mmap_lock.
4. Once the kernel confirms the VMA is file-backed, it triggers a record responsibility — a minor fault if the bytes are already successful lukewarm page cache from read-ahead, aliases a major fault if it must trigger beingness I/O.

Recommended reading: mmap_lock scalability (LWN) and per-VMA locks (LWN, Suren Baghdasaryan’s design).

Under dense page cache contention, read-ahead runs retired of room and awesome faults spike. Here’s what that looked for illustration — pidstat connected 1 process during a stressful run:

RSS grows to 98.91% of RAM → the kernel has nary prime but to evict pages still needed → evicted pages get touched again → a awesome responsibility large wind arsenic they’re publication backmost from disk. Early on, read-ahead keeps faults mostly insignificant and fast; arsenic concurrent queries heap up, read-ahead stops keeping up and awesome faults spike.

Minor faults, meanwhile, ran sustained successful the millions per second:

Each insignificant responsibility touches a cache statement via atomics — astatine 2 cardinal faults/sec, that’s capable to thrash L1/L2 entirely, which is deadly for an exertion that leans connected ample cache-resident lookup tables. Faults tin besides trigger TLB shootdowns, and CPUs only clasp a fewer 1000 TLB entries. (You can’t destruct page faults, but you tin negociate them amended — much connected that successful Part 2.)

An uncontended insignificant responsibility costs astir 0.5–1 microsecond, truthful 2 million/sec is adjacent to the ceiling of what mmap tin prolong — and nether existent contention, the thread-visible hold runs good past that.

Virtual reside space, meanwhile, had grown to ~3 TB from mmap’ing truthful galore Arrow files:

Modern kernels grip ample VMA trees, but not for free — each responsibility does a VMA lookup, and each lookup takes the mmap fastener (fast way notwithstanding).

Context switches told the aforesaid story:

Over 2 cardinal discourse switches/sec, versus 14K/sec connected a warm-cache tally — 150x more. Every thread was perpetually blocking connected page faults, getting descheduled, and rescheduled erstwhile pages arrived.

Perf Top and Off-CPU Analysis

To corroborate the nexus betwixt page faults and fastener contention, we compared perf apical connected acold vs. lukewarm runs of the aforesaid query:

__filemap_add_folio adds a page to the page cache. It hardly shows up warm, since the data’s already there; cold, nether representation pressure, it dominates because pages are perpetually evicted and re-inserted. Our existent query codification drops from ~45% of CPU (warm) to ~5% (cold) — not because it’s doing little work, but because the kernel is doing truthful overmuch more.

Off-CPU clip via bpftrace (actionable clip only, excluding idle Rayon threads):

• Futex: 30.9% (1,172s) — threads blocked connected synchronization, queued down different thread’s page-fault handler
• Preempted: 29.3% (1,109s) — amazingly precocious for 12 threads connected 192 cores; the kernel’s page-fault activity (readahead kthreads) was preempting our worker threads
• Disk I/O: 6.9% (262s) — existent NVMe latency was mini adjacent to the machinery supra it
• mmap_sem: 0.9% (33.5s) — the definitive VMA lock; mini only because it captures the wait, not the cascading futex wakes from threads queued down it

The picture: nether load, page-cache thrashing and kernel-level fastener contention — not disk I/O — were the bottleneck. This isn’t unsocial to mmap; immoderate buffered I/O way tin deed akin page-cache and fastener contention.

The Fio Ceiling

fio pinch the io_uring motor — 4 processes, iodepth 32, 4 MiB blocks, O_DIRECT:

What mmap really delivered, peak, from vmstat during the stressful runs: 3.44 GB/s — astir 16% of what the hardware could do. That spread was the size of the prize.

Enter io_uring

io_uring has earned its hype. Beyond async kernel I/O, portion of its committedness is nonstop personification I/O that bypasses the page cache wholly — the point causing astir of our problems above. Worth reading:

1. “io_uring for precocious capacity DBMS” — a bully overview of optimizations, though focused connected accepted DBMSes pinch 4KB page buffers, truthful galore don’t translate. IOPOLL needs circumstantial block-device entree not really disposable from containers; SQPOLL had nary measurable effect successful our Arrow-based testing.
2. LanceDB’s io_uring post — oriented astir mini 4KB (vector search) reads. Key takeaway: without amended scheduling and concurrency, io_uring by itself doesn’t help.

The plan: bypass the page cache pinch O_DIRECT, taxable sounds via io_uring, coordinate pinch Tokio, decode Arrow inline. We utilized compio, a Rust-native io_uring wrapper (executor + futures + reactor built astir io_uring). The first trim leaned connected compio’s async futures — 1 early per Arrow file read, each 40 columns (8 batches × 5 columns) submitted concurrently, awaiting completions to output decoded Arrow buffers.

Here’s really we expected io_uring to reply mmap’s problems:

Initial laptop testing wasn’t encouraging

Looking back, possibly we should person written this station earlier building thing — our first creation didn’t present connected astir of that past column. Proper io_uring creation takes existent work, and we wanted to iterate fast, truthful we started small.

We started connected macOS, which has nary io_uring — kqueue is simply a different beast wholly — but it fto america sanity-check the compio abstraction and our batching logic: does it compile, are we producing much aliases less page faults than mmap? A laptop can’t beryllium correctness, but it tin drawback evident regressions earlier booking clip connected the large Linux boxes.

Total runtime was slower, but awesome faults collapsed astir 70×. That’s the expected style of bypassing the page cache — nary OS-managed faults, because there’s thing to fault. We told ourselves the other latency was because macOS lacks existent io_uring nether the hood, and that Linux would present the throughput win.

The Linux reality check

Major faults dropped from 128,957 to 3,647 — a 35× reduction. We’d solved precisely the point io_uring is expected to solve: the kernel was nary longer thrashing connected awesome page-ins.

But insignificant faults went up 8×, and full query clip went from 13.6s to 21.8s — io_uring was ~60% slower than the mmap baseline it was expected to replace. We’d traded 1 people of responsibility for different and mislaid connected the trade. The net is afloat of posts declaring io_uring wins complete mmap. We had an implementation that had gone backwards.

Before we get to what went wrong, it’s worthy stepping done what we’d really built — the correction only makes consciousness erstwhile you spot the architecture astir it.

The Batch Materialization Layer

Arrow IPC organizes information arsenic batches — contiguous statement chunks, each pinch its ain on-disk layout, pinch columns stored successful their ain byte ranges. In our setup, each record holds 1 ample batch, and a time of information spans astir 8 files. A query spanning 1 time touches each 8 files, pulls ~5–6 columns from each, and ends up issuing astir 40 individual file reads.

With mmap, the publication count hardly matters — constituent the query motor astatine the file, it appears arsenic memory, and the kernel pages successful immoderate bytes a query touches. With io_uring, each publication has to beryllium submitted explicitly, which is much codification and much places to get the timing wrong. So we built a furniture betwixt the io_uring plumbing and the query motor — the Batch Materialization Layer (BMT successful our logs) — pinch a elemental API:

• prefetch(batch, columns) — occurrence io_uring sounds for a group of columns, populate a per-column cache pinch OnceCell-style slots
• materialize(batch, column) — return cached bytes, aliases await the in-flight read

The creation felt correct connected a whiteboard. Prefetching maps straight to really the query motor wants to hide I/O latency down CPU work: commencement sounds early, do different things while they land, travel backmost for bytes erstwhile needed. The cache decouples the query motor from io_uring semantics wholly — queries conscionable telephone materialize and either get bytes instantly aliases wait.

What we didn’t spot astatine the clip was really overmuch this 1 furniture was doing. Everything ran connected a azygous BMT thread: accepting prefetch/materialize calls, submitting compio futures for each requested column, awaiting completions, decoding bytes into Arrow buffers, populating the cache, and handing materialized columns backmost to the query engine. I/O coordination, Arrow decode, cache management, and query-facing API, each connected 1 telephone stack and 1 async runtime. It felt for illustration cleanable separation of concerns astatine the time. It was really 1 furniture doing 5 jobs, wherever a hole successful 1 would ripple into the others. More connected that successful Part 2.

The first trim fired each ~40 file sounds astatine erstwhile — ~40 concurrent compio futures the infinitesimal a query wanted a file. That’s what “prefetching” meant to america then: occurrence everything, fto async coordinate, hold for it all.

The Meandering

The io_uring papers each stress O_DIRECT, which we weren’t utilizing — our sounds were still flowing done the page cache. So the first point we tried was turning O_DIRECT on: nonstop DMA into our buffers, nary kernel caching, nary of the page-cache machinery that had been our full problem nether mmap. Runtime dropped from 21.8s to astir 19s — real, but modest, and not the leap the blog posts had led america to expect.

Next was Arrow, which was showing up prominently successful perf. Samples sat connected Buffer::from_slice_ref, arrow-rs’s default measurement to build a Buffer from a byte portion — it allocates caller representation and memcpys into it. Every 4 KiB destination page that memcpy touches needs the kernel to zero and representation it — a insignificant responsibility per page — and 8 cardinal insignificant faults complete ~13 GB of sounds lines up almost precisely pinch that math. Our Arrow furniture was, successful effect, forcing the kernel to redo the memory-management activity we thought we’d bypassed by moving to io_uring successful the first place.

We worked astir it by constructing the Buffer straight from the io_uring-owned bytes, skipping the copy. Runtime dropped from 19s to astir 16s, but the codification was disfigured capable that immoderate reviewer would emblem it — manual Buffer building sidesteps arrow-rs invariants successful ways that are difficult to travel six months later. We logged the existent hole arsenic early work: a reusable buffer excavation wherever io_uring writes into pre-allocated destination representation and Arrow copies from that — you’d still salary the memcpy, but into pre-faulted pages, which is adjacent to free.

16 seconds — amended than our first cut, still worse than mmap. We’d done the evident things — O_DIRECT on, Arrow’s default transcript worked astir — and the blogs had promised near-hardware-ceiling throughput. We weren’t close, and astatine that point, thing other successful the creation looked evidently incorrect to us.

To beryllium continued…

We’d tried the evident fixes and talked done much theories pinch Claude than I want to count. We were sitting connected an implementation 60% slower than mmap, pinch a Batch Materialization Layer that looked reasonable connected paper. What we didn’t person — and needed, to make immoderate much advancement — was existent information astir what our io_uring submissions were doing infinitesimal to moment.

Time to adhd logs and instrumentation, and excavation successful astatine a micro level, the old-fashioned way.

Part 2 picks up here: the logs, the arguments pinch Claude, the infinitesimal we realized 40 concurrent SQEs was the existent problem, the architectural rethink that followed, and the memory-management communicative that ended up mattering arsenic overmuch arsenic immoderate of it.

Learn much astatine P99 CONF

I’ll beryllium doing a heavy dive connected this taxable astatine P99 CONF, online October 21–22, 2026 — a convention for developers who attraction astir p99 percentiles and high-performance, low-latency applications.

Register here.