Andrew Quinn replaces 3GB SQLite with 10MB FST for static data

This review analyzes Andrew Quinn's technical process of replacing a large SQLite database with a Finite State Transducer (FST) binary, assessing its implications for indie founders optimizing data storage and lookup performance.

TL;DR

Best for: Read-heavy applications with static, string-keyed data that require minimal memory footprint and fast lookups, such as dictionaries, autocomplete, or URL routing tables. Ideal for indie founders deploying resource-constrained services. Skip if: Your data is dynamic, requires frequent writes, involves complex relational queries, or uses non-string keys. This is not a general-purpose database replacement. Bottom line: Finite State Transducers offer a highly specialized, performant, and space-efficient solution for specific static data lookup problems, dramatically reducing binary size and improving query speed.

METHODOLOGY

This v0 review draws on the founder Andrew Quinn's published claims and technical details in his blog post, "Replacing a 3 GB SQLite db with a 10 MB FST (finite state transducer) binary," accessed on May 10, 2026. The review covers the problem statement, the technical approach using a Finite State Transducer (FST), the specific steps taken, and the reported outcomes regarding file size and lookup performance. We analyze the technical feasibility and potential use cases for this optimization strategy. What is not covered in this v0 review includes independent performance benchmarks, long-term workflow integration, or edge-case behavior under various data distributions. Our update cadence will involve re-testing when claims diverge from observed behavior or when new, comparable data structures emerge. This initial assessment relies on the methodology and results presented by Andrew Quinn.

WHAT IT DOES

Problem: Large SQLite database

Andrew Quinn faced a challenge with a 3 GB SQLite database containing static, read-only data. This database, while functional, contributed significantly to the overall application size and memory footprint, particularly for a service that primarily performed lookups on this data. The core issue was using a general-purpose relational database for a task that only required efficient key-value lookups, leading to unnecessary overhead in storage and potential performance bottlenecks.

Solution: Finite State Transducer structure

Quinn's solution involved replacing the SQLite database with a Finite State Transducer (FST). An FST is a highly compact data structure optimized for storing and querying sets of strings or key-value pairs where keys are strings. It represents a directed acyclic graph where paths from the start node to an end node correspond to keys, and transitions can carry output values. This structure allows for extremely efficient prefix searches, exact key lookups, and minimal storage by sharing common prefixes among keys.

Implementation: Rust fst crate

The implementation involved several steps. First, data was extracted from the SQLite database into a CSV format. Next, a custom Rust program was written using the fst crate to process this CSV data. The program iterated through the sorted key-value pairs, building the FST. The fst crate provides robust tools for constructing and querying these structures. The final FST was serialized into a binary file, which was then integrated into the application. This process transformed the 3 GB SQLite database into a 10 MB FST binary, a 300x reduction in size.

WHAT'S INTERESTING / WHAT'S NOT

What's most interesting here is the dramatic scale of the optimization. Reducing a 3 GB database to a 10 MB binary is a 300x improvement in storage efficiency, which is significant for deployment size, cold start times, and memory usage. This directly translates to lower operational costs and faster application performance, especially for indie founders or small teams where every megabyte and millisecond counts. The use of the fst crate in Rust highlights a practical application of advanced data structures, moving beyond typical database solutions for highly specific, read-heavy workloads. The FST's inherent properties, like shared prefixes, make it exceptionally well-suited for dictionary-like data where string keys are common and lookups need to be fast. This approach is a strong contender for scenarios like spell checkers, autocomplete engines, or routing tables where the dataset is largely static.

What's not interesting, or rather, what needs careful consideration, is the specificity of this solution. An FST is not a drop-in replacement for a general-purpose database. It lacks transactional capabilities, complex query language support (like SQL's JOINs or aggregations), and built-in mechanisms for dynamic updates. The process described is also custom; it requires engineering effort to extract, transform, and load data into the FST, as well as to integrate the FST binary into the application's lookup logic. This is not a plug-and-play tool but a specialized engineering technique. The founder's post implicitly assumes a static dataset; the implications of needing to rebuild and redeploy the FST for frequent data changes are not explored, which would introduce its own set of complexities and overhead.

PRICING

Finite State Transducers (FSTs) are a data structure and a concept, not a commercial product with a price. The fst crate used by Andrew Quinn is an open-source library, freely available under the MIT license. The cost associated with implementing this solution is primarily in engineering time for design, implementation, and maintenance. Pricing snapshot date: May 10, 2026.

VERDICT

Andrew Quinn's approach to replacing a 3 GB SQLite database with a 10 MB FST binary is a highly effective optimization for specific use cases. It is best for applications characterized by static, string-keyed data that demand minimal memory footprint and extremely fast lookups. This solution is particularly compelling for indie founders or projects with tight resource constraints, where reducing binary size and improving lookup performance directly impacts deployment costs and user experience. However, it is crucial to understand that FSTs are not a general-purpose database. They are unsuitable for dynamic data, write-heavy workloads, or scenarios requiring complex relational queries. For those specific, read-optimized scenarios, an FST offers a superior alternative to traditional databases, delivering dramatic space efficiency and lookup speed gains.

WHAT WE'D TEST NEXT

Our next steps would involve independently benchmarking the FST's lookup performance against the original SQLite database, specifically measuring average and worst-case latency for various key distributions. We would also compare the FST's memory footprint during runtime against an in-memory SQLite database. Further tests would include evaluating the FST build time and resource consumption for larger datasets (e.g., 10GB, 100GB source data) and assessing the complexity and performance overhead of updating and redeploying the FST for datasets that change periodically. Finally, a comparison with other compact data structures like compressed key-value stores or specialized embedded databases (e.g., RocksDB, LMDB) for similar static, read-heavy workloads would provide a broader context for optimization choices.

Pull quote: “Reducing a 3 GB database to a 10 MB binary is a 300x improvement in storage efficiency, which is significant for deployment size, cold start times, and memory usage.”