# sqlite-js-stream: Litestream, rewritten for Node

SQLite is the most widely used database in the world, but it only runs on a single machine. If that machine fails, your database is lost. Unlike Postgres, you can't solve this with traditional replication, because SQLite is just a file. You're replicating data, not a server.

[Litestream](https://github.com/benbjohnson/litestream) addresses this by streaming the WAL into unchangeable [LTX](https://github.com/superfly/ltx) files on S3 or a local folder, and then restoring from those files. It's a well-designed tool written in Go. However, Platformatic's stack is built on Node.

[sqlite-js-stream](https://github.com/platformatic/sqlite-js-stream) is a TypeScript version of Litestream that runs directly on Node. It uses the built-in `node:sqlite` and Node's native type stripping, so there's no need for a build step.

To be clear about how this was built: the **entire port was written by an AI coding agent called pi, running a DeepSeek V4 Flash vision model on two NVIDIA DGX Spark machines. There was n**o cloud cluster, no massive GPU farm, no cloud model, and no one watching over every line. The design was based on Litestream and rewritten in TypeScript, with a human responsible for the architecture, reviews, and key decisions. The AI sped up the coding, but didn't replace human judgment. Details about the model, how it was used, and its mistakes are covered in [The AI part](https://github.com/platformatic/blog-posts/blob/1c456cc469d4417449358c7d76a3b15331a2671d/sqlite-js-stream/post.md#the-ai-part--what-it-actually-got-us) below.

* * *

## **The real win is distribution**

Litestream is a Go binary. You install it, run it as a separate process alongside your app, write a config file, point it to the database and the storage bucket, and keep that binary updated with every deployment. This adds real operational overhead, and it's the main reason SQLite isn't used more often in production.

A JavaScript version folds most of it away. It's a package:

```plaintext
npm install sqlite-js-stream
```

It runs inside your main process, so there's no sidecar process to manage, no config file to keep up with, and no extra binary to version with your app. It works with your existing package manager, CI, bundler, and deployment pipeline. You can run it behind the app's event loop, in a worker thread, or in a serverless function. You deploy it just like any other part of your Node app.

That's what really matters. SQLite is the most widely used database, but the main reason people avoid it in production isn't the database itself; it's the risk of losing data. If the solution is to run a sidecar binary that needs constant attention, many teams just choose Postgres instead. But if the solution is as simple as adding one npm install to your existing app, SQLite becomes a practical default again, not just a toy.

This is Atwood's Law playing out in real time:

> *Any application that can be written in JavaScript will eventually be written in JavaScript.* [Jeff Atwood](https://blog.codinghorror.com/the-principle-of-least-power/), 2007

Litestream is one of the best Go tools out there, and Atwood's Law suggests it was never going to stay Go-only for long. The real reason is distribution: the JavaScript runtime reaches more people than a compiled binary ever could. That's why this port is important. It turns a genuinely useful tool into something anyone using Node can install and use with their existing SQLite database.

* * *

## **Why this exists**

Backing up a SQLite database is not a copy job. If you just `cp` the file while it is being written, you get a corrupt snapshot. The WAL makes it worse: committed writes might be in `-wal`, not in the main file, and they can be checkpointed and truncated at any time. To get a consistent backup, you need to understand the WAL and hold off checkpointing while you read.

Litestream solves this by turning the WAL into a sequence of LTX files, each one holding the pages changed in a single transaction. To restore, you just replay those files in order. It's a smart approach, and it's fully compatible with litestream's format. Files created by this tool can be read by litestream, and the other way around.

The layout on main:

```plaintext
sqlite-js-stream        core library (root package, pnpm workspace)
packages/backend-s3     S3 upload strategy (replica client + S3Leaser)
```

* * *

## **What's in the box**

The API is deliberately driver-agnostic. You hand a Replica a database path and a replica client; it reads the DB and WAL straight from disk, so your app can use whatever SQLite driver it wants.

```javascript
import { Replica, FileReplicaClient, FileLeaser } from 'sqlite-js-stream'

const replica = new Replica('/data/app.db', new FileReplicaClient('/backups'))
replica.leaser = new FileLeaser('/backups') // only one host is the active writer

await replica.run({ intervalMs: 1000, signal })
```

What has landed so far:

*   **Streaming, end to end.** `sync()/snapshot()` read pages one at a time from a file handle and stream the encoded LTX straight to a temp file. Peak memory is bounded by one page buffer, not the database size. `restore()` overlays incremental files on a snapshot base without materializing the whole DB.
    
*   **The LTX format, faithfully.** The port writes `superfly/ltx v0.5.2` at `ltx/{level}/{min}-{max}.ltx`, snapshots at level 9, with CRC-64 ISO checksums. Same bytes as litestream writes.
    
*   **Compaction + retention, matched to litestream.** Per-level time-bucketed compaction (L1 every 30s, L2 every 5m, L3 every 1h), time-based snapshot retention, TXID-boundary cascade down the levels, and L0 time retention that never deletes the live head.
    
*   **Distributed leasing.** A `Leaser` claims a `lock.json` lease; `run()` makes exactly one host the primary (writer) and the rest idle followers. On S3 it uses real `If-None-Match/If-Match` conditional writes, so two racy acquirers cannot both win. The lease is renewed on a dedicated timer, independent of the run-loop cadence.
    
*   **A pnpm monorepo.** The core lives at the root; the S3 addon is a separate package under `packages/`.
    
    * * *
    
    ## **First numbers**
    
    `sync()` / `restore()` have reproducible benchmarks in [bench/bench-sync-restore.ts](https://github.com/platformatic/sqlite-js-stream/blob/main/bench/bench-sync-restore.ts). These tests run on localhost with a file backend, no network involved, and use a small database (100,000 rows across 100 transactions, with a 3.76 MB WAL). The results show the tool works and that memory use is limited. This isn't a direct comparison with litestream, which is important to note.
    
    ```plaintext
    workload: rows=100000, batch=1000, transactions=100
    
    sync():    wall ~180-250 ms    ~15-21 MB/s (WAL)    ~400-480k rows/s    peak RSS ~106 MB
    restore(): wall ~330-360 ms    ~7-8 MB/s (DB)      ~280-300k rows/s    peak RSS ~114 MB
     integrity: ok, rows: 100000
    ```
    

The number that matters is the memory one. Run it at two sizes with `SCALE=4` (which runs sync at 50k and 200k rows), and the WAL grows 4.1x while peak RSS moves 1.22x:

```plaintext
rows=50000    WAL=1.87 MB   sync=140 ms   peak RSS=99.8 MB
rows=200000   WAL=7.72 MB   sync=542 ms   peak RSS=121.2 MB
WAL grew 4.1x; peak RSS grew 1.22x.
```

So the streaming claim is accurate: memory use is limited by a page buffer, not the size of the database. Peak RSS covers the whole process, including Node's baseline, so the key point is that memory stays flat as the database grows, not the exact value.

What this does **not** prove: it is not litestream performance, it is file-backend localhost, and it is a few-MB database. The real head-to-head needs the same workload run through the Go binary against the same target, and honestly, the pure-JS LZ4 codec is expected to lose to Go's SIMD `pierrec/lz4` on the decompress side. That's next on the list.

* * *

## **Being honest about the limits**

This is a **disaster-recovery tool, not a distributed database.** The lease elects a single writer to storage, but it does not merge, compare, or reconcile the data across hosts. If host A has `row id=1 → "X"` and host B has `row id=1 → "Y"` and they share one prefix, only the primary's value lands in S3. From the backup's perspective, the other is simply gone.

So, it's only safe to point several hosts at one prefix if they all have the same logical data, like a disaster recovery standby restored from the same backup. Different databases should use different prefixes. This warning is now in the README and the architecture documentation, because it's a real risk and no one should have to learn it the hard way.

There is also a genuine gap on failover: a standby host only contains data up to the last backup, not the last few seconds the primary wrote before it died. There is no way around that with a WAL-based replicator.

Status: early. Cloud backends beyond S3 (GCS, Azure, SFTP, NATS), VFS mirroring, and the IPC server are not implemented. The core package is not published yet.

* * *

## **The AI part: what it actually achieved**

This project was a real experiment in building a complex port using only an agent running on **local models**.

The model actually doing the work is [DeepSeek-V4-Flash-Vision-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp), DeepSeek's V4 Flash vision build, served with [vLLM](https://github.com/vllm-project/vllm) using tensor parallelism (`TP=2`) across **two NVIDIA DGX Spark** machines. Each one is a GB10 Grace Blackwell with 128 GB of unified memory at 273 GB/s; two of them linked over ConnectX is how the model fits in local memory. It's an experimental local flash/vision build, not a cloud frontier model. The harness is pi, and a human wrote the PLAN, reviewed the diffs, and made the scope calls.

*   **TDD kept it honest.** Tests were written first, which is how the agent stayed on track. 121 tests now (103 core + 18 S3), all green under Node 24.
    
*   **The agent was good at the mechanical parts**: reading the Go source, mirroring the compaction-level scheduling, getting the LTX encoder/decoder round-trip to pass, and standing up the monorepo + CI.
    
*   **The agent also went down a rabbit hole that had to be cancelled.** At one point it started building a *legacy v0.3.x generation* restore path, a whole different LZ4 frame codec, on the instruction to "upgrade to the newer file format." That turned out to be a misunderstanding of the problem: the write format was already the modern LTX. The agent did the work, then a human looked at it and said no, that's backward-compat for reading old backups nobody needs. That path was cut. The architecture and the scope call were a human decision, not the model's.
    
*   **Local models were a constraint, not a badge.** There was no cloud model to lean on. That meant more careful specs and more relying on the test suite, and it made the "no build step, run `.ts` directly" requirement real rather than aspirational.
    

If you want to poke at how far this can go, clone the repo, read `PLAN.md` (it tracks every milestone and risk), run `pnpm test:all`, and try pointing a replica at your own database.

If you hit something: a format edge case, a compaction bug, an interop gap, open an issue. The hardest part of a project like this is the byte-for-byte LTX compatibility, and the gaps are better found with your database than in production.

SQLite's biggest weakness doesn't have to be limited to Go anymore. This version was written in Node.

*Credit where it's due: the design, the LTX format, and most of the engineering reasoning come from* [*Ben Johnson*](https://github.com/benbjohnson)*'s* [*Litestream*](https://github.com/benbjohnson/litestream) *and the* [*superfly/ltx*](https://github.com/superfly/ltx) *library. This is a port, and the original deserves the credit.*