Efficient String Compression for Modern Database Systems

Why It Matters

Halving string footprints reduces cloud storage costs and improves cache efficiency, directly boosting analytical query performance while offering a configurable trade‑off for decompression overhead.

Key Takeaways

• FSST halves string storage in CedarDB.
• Combines token compression with dictionary keys.
• Cold queries run up to 40% faster.
• Hot queries may slow 2‑3× without caching.
• 40% size‑penalty selects FSST automatically.

Pulse Analysis

Strings dominate modern data warehouses, accounting for roughly half of stored values and frequently appearing in filter predicates. Traditional dictionary compression works well for low‑cardinality columns but struggles when distinct strings proliferate. FSST addresses this gap by replacing common substrings with single‑byte symbols, fitting the entire symbol table into L1 cache and enabling rapid encoding and decoding. When paired with a dictionary, FSST retains the benefits of integer‑key comparisons while squeezing additional space out of the dictionary entries themselves, creating a hybrid that balances size and speed.

Integrating FSST into CedarDB required careful engineering. The system serializes the symbol table alongside an offset array, allowing random access to each compressed string. Because FSST‑compressed strings vary in length, direct predicate evaluation is less efficient than integer‑key scans, prompting the developers to compress the dictionary with FSST instead of the raw strings. A configurable penalty—set at 40% in production—ensures FSST is only adopted when it delivers a substantial storage win over the next‑best scheme, mitigating the risk of excessive decompression latency.

Real‑world benchmarks illustrate the practical payoff. On ClickBench, FSST saved about 6 GB (≈20% of total data) and accelerated disk‑bound queries by up to 40%, while TPC‑H saw a 40% overall size cut and a 10% query‑time improvement for key workloads. Hot‑run scenarios that fully decompress strings can experience 2‑3× slowdowns, a cost that can be offset by caching decompressed values. For enterprises, the net effect is lower storage spend, faster data loading, and more predictable query performance, making FSST a compelling addition to modern analytical databases.