SQLite is the unsung workhorse of modern software. It runs on billions of devices, from smartphones and web browsers to edge routers and client applications. Its testing setup is legendary. The project famously maintains far more test code than library code, achieving 100% branch coverage. Because of this reputation, finding a reproducible database corruption bug in SQLite is rare. Finding one that has existed for sixteen years is almost unheard of.
Tailscale uses SQLite to manage local state on client devices. With millions of active nodes running on everything from Linux servers to macOS laptops, their software encounters rare hardware and software edge cases at scale. Recently, their telemetry flagged a small but persistent number of database corruption errors. The error was a standard SQLite warning: SQLITE_CORRUPT.
Normally, database corruption points to failing hardware, sudden power loss, or direct file tampering. But the frequency of these errors, while low, was too consistent across different hardware configurations to be dismissed as random disk failures. Tailscale engineers set out to isolate the root cause. They built a test harness designed to simulate sudden power losses, crashes, and rapid write cycles.
To understand what they uncovered, you have to look at how SQLite handles writes. By default, SQLite uses a rollback journal to keep transactions safe. For high-performance applications, developers often switch to Write-Ahead Logging (WAL) mode. In WAL mode, SQLite avoids writing changes directly to the main database file. Instead, it appends them to a separate file ending in -wal. Readers can pull data from the main database and the log file concurrently, which prevents writes from blocking reads.
Eventually, the changes in the WAL file must be merged back into the main database. This process is called checkpointing. Once checkpointing completes, the WAL file is logically empty. However, deleting a file and recreating it later is slow because it requires filesystem metadata updates. To maintain performance, SQLite leaves the -wal file on disk and overwrites it from the beginning during the next write cycle.
This optimization introduces a tracking challenge. If a WAL file is 5 megabytes, and a new write session only writes 1 megabyte of data, the remaining 4 megabytes of the file still contain old data from the previous session. SQLite must distinguish new data from the old garbage left behind.
To do this, SQLite writes a header at the start of the WAL file. This header contains two 32-bit integer values called salts, labeled salt-1 and salt-2. Every time the WAL file wraps around and starts writing from the beginning, SQLite generates a new pair of salts. Every frame of data written to the WAL is stamped with these salts and a checksum.
When SQLite opens a database, it checks if the WAL file needs recovery. This recovery process reads the WAL file frame by frame. It validates each frame by checking if its salt matches the salt in the WAL header and if the checksum is correct. The moment it finds a frame where the salt does not match or the checksum is invalid, it stops reading. It assumes it has reached the end of the valid log.
The bug lies in how SQLite handled recovery after a crash. If a writer crashed mid-transaction, it could leave a partially written frame on disk. During recovery, SQLite would try to parse this partial frame. Under normal circumstances, the checksum check would fail, and recovery would stop safely.
However, if the crash left a frame header intact but the payload corrupted, or if the sector size of the drive caused a write to be split, SQLite could read past the crash point. If the stale frames left over from the previous run of the WAL happened to have a matching salt, the recovery code would keep reading. It would mistake those old, stale frames for new, valid transactions.
This happened because the salt generation logic did not guarantee uniqueness across restarts in all environments. If the salt did not change, or if a database was restored from a snapshot where the disk state was partially preserved, the recovery parser would match the old salt values.
The recovery process would then apply these stale pages to the main database file. Because the pages belonged to a previous state of the database, they would overwrite current data with old, out-of-date data. The result was silent database corruption. The next time the application tried to read the database, it would find mismatched page pointers and throw a corruption error.
This bug was introduced in 2010 when WAL mode was first added to SQLite. It went unnoticed for sixteen years because the conditions required to trigger it are incredibly specific. You need an unclean shutdown at the exact moment a transaction is being written, followed by a recovery phase where the newly generated salt matches the salt of the stale frames at the end of the physical file.
Once Tailscale isolated the behavior and provided a reproducible test case, the SQLite team patched the vulnerability. The fix ensures that SQLite is much more conservative when validating frames during recovery. It prevents the engine from reading past a failed write even if stale frames downstream happen to share the same salt values.
For developers running SQLite in production, the takeaways are clear. First, update SQLite to the latest version to pull in the patch. Second, if you use WAL mode, ensure your application handles unclean shutdowns gracefully. Finally, implement regular database integrity checks using PRAGMA integrity_check to catch corruption early before it affects your users.
