Show HN: Spice Cayenne – SQL Acceleration Built on Vortex

Why It Matters

Cayenne gives enterprises a scalable, low‑latency query engine for massive data‑lake environments, reducing infrastructure costs and enabling real‑time AI‑driven applications.

Key Takeaways

• Vortex format delivers up to 100× faster random access
• Cayenne runs 1.4× faster than DuckDB on TPC‑H
• Memory usage drops to one‑third of DuckDB
• Separates metadata (SQLite) from data for scalability
• Supports multi‑TB, low‑latency analytics without extra clusters

Pulse Analysis

The data‑lake landscape is shifting toward object storage as the primary repository for analytical workloads. Traditional embedded engines such as DuckDB or SQLite excel on sub‑terabyte datasets but hit performance and memory ceilings when data scales to multiple terabytes. Spice Cayenne addresses this gap by leveraging Vortex, an open‑source columnar format that promises dramatically faster random access and full‑scan speeds while maintaining zero‑copy compatibility with Apache Arrow. Coupled with a lightweight SQLite metadata store, Cayenne eliminates the single‑file constraints that plague existing accelerators, enabling efficient segment‑level pruning and ACID‑compliant catalog operations.

Benchmark results illustrate the practical impact of this architecture. On a 16‑vCPU, 64‑GiB instance, Cayenne completed TPC‑H SF‑100 queries 1.4× faster than DuckDB and consumed only 11.7 GB of RAM versus DuckDB’s 33.2 GB. ClickBench tests showed a 14% speed advantage and a three‑fold reduction in memory usage. These gains stem from Vortex’s optimized layout, pluggable compression, and the separation of metadata, which reduces the need for costly round‑trips to object storage during query planning. The performance uplift is especially valuable for latency‑sensitive AI applications that require sub‑second response times across petabyte‑scale datasets.

For businesses, Cayenne translates into lower total cost of ownership and faster time‑to‑insight. By removing the necessity for dedicated acceleration clusters or complex ETL pipelines, organizations can serve high‑concurrency analytical queries directly from their data lake. The open‑source nature of Spice.ai and Vortex encourages community contributions and ensures alignment with emerging standards like Apache Iceberg. As the roadmap adds indexing, richer compression, and alternative metadata backends, Cayenne is poised to become a cornerstone of next‑generation, AI‑enabled analytics stacks, supporting both operational and strategic decision‑making at scale.