Breaking the WAL

Hi, it’s Carl again. You whitethorn retrieve maine arsenic the feline who taught Claude to usage Antithesis.

Earlier this year, SQLite released (3.51.3), which fixed a longstanding bug successful their Write-Ahead Logging (WAL) subsystem called the WAL-Reset bug. The bug had been hanging astir since 2010, but the SQLite squad had apparently been unaware of its beingness until earlier this twelvemonth (more connected this below). As they wrote astatine the time:

“The bug is simply a information title pinch tight timing constraints. It is improbable to hap successful communal use. The developers person ne'er been capable to reproduce the bug organically and had to adhd typical testing logic to SQLite that deliberately triggers the circumstances of the bug successful bid to verify that the rumor has been fixed.”

I was really connected a roadworthy travel pinch my woman erstwhile I publication astir this, but I’m besides a elephantine database nerd, truthful I was instantly nerd-sniped, hard. Bugs successful SQLite, aft all, are legendarily rare. Moreover, this sounded for illustration conscionable a cleanable brown M&M: a known, challenging bug that we could way down pinch Antithesis (we’ve done this a batch successful our POCs). On apical of that, I’d precocious shipped our skills for Claude.

So, sitting connected a hillside connected the Sunshine Coast, I whipped retired my phone, and asked Claude to get to work. I had it get SQL 3.51.2 – still buggy – group up successful Antithesis, and past instrumentality the codification pinch a bunch of Antithesis assertions. You tin spot the instrumented type here.

Then I asked it to constitute a elemental workload which exercised the WAL insert and checkpoint code. Notably, this is simply a wholly generic workload. It conscionable runs writes and checkpoints concurrently – things you’d expect to really hap successful production, each the time. The assertions are besides generic to the bug, they’re each modular assertions you’d adhd to immoderate database, things for illustration “no mislaid committed writes” and “database is not corrupt” (called integrity cheque successful sqlite).

On my first run, Antithesis caught the bug successful 15 mins. Here’s the report. The portion you’re looking for is:

Then I repeated the workout pinch 3.51.3, pinch the aforesaid workload and Antithesis instrumentation. Sure enough, the tally came backmost green.

I thought astir this coming because Tailscale conscionable wrote an fantabulous blog post astir resolving the uptime issues they’d knowledgeable successful 2025. Those issues were really the SQLite squad discovered the WAL-Reset bug. Tailscale suffered 6 months of shaky uptime, past they and the SQLite squad spent weeks hunting the bug, rolled retired and rolled backmost a hole that collapsed thing else, past had to hold 2 much months to spot if the “real” hole (3.51.3) worked.

To guidelines origin the issue, they had to constitute a caller transaction logging pipeline successful Tailscale, past shim successful a caller debugging instrumentality for the virtual filesystem furniture successful SQLite. In Antithesis, this process isn’t rather down to a azygous click, but 1 click will springiness you a causality analysis that pinpoints the rumor to wrong a fraction of a second, and deterministic, time-travel debugging that allows you to do what-ifs and destructive analysis.

As the Tailscale squad wrote, “nobody wanted america to walk six months looking for bugs successful SQLite. This was an immensely frustrating acquisition for some our customers and staff”.

Finding bugs for illustration the WAL-Reset bug is excruciatingly difficult (perhaps moreover for illustration crawling complete surgery glass) – but pinch uncommon and difficult bugs, the existent torture tin travel erstwhile you’re waiting to spot if your hole really worked. I’ve worked connected capable databases, and person knowledgeable this myself many, galore times.

So it was some sobering and uplifting to recognize conscionable really achy this bug had been successful the wild. By giving agents the skills to usage Antithesis, I’d conscionable recovered and verified it successful for illustration an hour, from my phone, sitting nether pine trees successful the sunshine. I knew our supplier skills worked, but I had nary thought they worked this well. If you person a gnarly database issue, telephone me.