libSQL, the fork of SQLite, crosses 5k GitHub stars
We are now slightly over a year into our work on libSQL, a fast-growing database built on SQLite designed for modern applications and workloads like serverless and edge computing. As we've recently exceeded 5,000 stars and growing and 46 contributors on GitHub, it makes sense to take a moment to reflect on what we've accomplished so far and look ahead into the future.
The origin story for libSQL is kind of interesting, but we haven't talked much about it in public. The story starts in the late summer of 2022. We were building a backend-as-a-service product and had been using SQLite for the local development version successfully. Production deployments were using a PostgreSQL-based database, but the developer experience suffered compared to SQLite, especially around provisioning times and replication, which was something we wanted to do to achieve low latency at the edge.
We considered using SQLite for production as well. However, we knew SQLite needed to provide a great out-of-the-box experience suitable for modern applications and use cases, but it was lacking in that area. We also understood that contributing to SQLite to support these use cases would be hard. (We knew that a project named dqlite had previously attempted to add replication hooks to SQLite, only to face rejection.)
And then, we came across an innocent-looking post by Phil Eaton.
To which our CEO responded internally:
Early on Glauber insisted that if we do this, it needs to be compatible with SQLite. We looked around and noticed that different projects were using the SQLite VFS layer to hook into the storage engine. But in our view, we would be better off, especially in the long term, with changes to the core database.
Many SQLite+something projects were launched in the past, achieving mild results. We knew that for our version to succeed, we should not be too narrow. But instead, tackle the community problem of SQLite not being open to contributions.
To avoid being associated with a particular tech direction we were taking, we launched the project in early October with no code changes: just a manifesto with a list of possible directions we could take, a CoC, and a proper OSS License (Apache, which we later changed to MIT):
We knew there were bunch of folks interested in community-based version of SQLite, but the initial response was just insane:
So now we knew that we could fully commit to libSQL. Piotr Sarna, who had joined the company earlier that year, had already done some work around supporting WebAssembly user-defined functions in libSQL. But later November 2022, he started working on replication by virtualizing the WAL (a similar approach to Litestream and LiteFS, but more integrated).
In parallel, I was busy looking into ways to let SQLite escape the boundaries of a single machine by implementing a SQLite C API compatible proxy that would speak the PostgreSQL wire protocol:
Of course, it didn't take long for us to discover that this was going to be a hard path going forward because of the type and system table differences between PostgreSQL and SQLite (although they're similar in many ways). Also, Pete Hunt and others to pointed out that providing a HTTP interface for SQLite is much more powerful, especially for serveress applications. So at that point our engineer, Marin Postma, took over the libSQL server development in December, and started working towards that goal. I went to start hammering out the first version of the managed database service infrastructure that you today know as Turso, but that's a story for another post.
Fast forward to today, libSQL is becoming a popular version of SQLite with built-in support for serverless and replication. We're working towards the first official release with SDKs for Rust, TypeScript/JavaScript, Go, and Python. The libSQL project is powering the Turso managed database service, which is running various production apps ranging from multitenant SaaS products like per-user personalized LLMs to international art-tech showcases such as the U2 “space baby” experience at Las Vegas's The Sphere (Exosphere). Although we were initially worried about having a separate tree from SQLite with modifications, back-merging from upstream has turned out to be relatively easy. We're working hard to remain fully compatible with SQLite if you want to use libSQL with the SQLite C API. Of course, if you use the modern features, you're using libSQL interfaces. But we're trying to keep even that as close to drop-in replacement as possible.
That said, we're still very early. Here is a quick look at some of the things we're working on for libSQL in the near and long term future:
- Improve the libSQL SDKs and documentation.
- Add support for local writes that can be synced to remote.
- Integrate with local-first frameworks for multiplayer use cases.
- Work towards native PostgreSQL integration.
- Improve SQLite write concurrency (unless SQLite does that first).
To try out libSQL today, the easiest way is to sign up on the Turso free tier to provision a managed SQLite-based database for your app. For questions and feedback, join our Discord.
