Exa-D: How to Store the Web in S3

Why It Matters

By reducing recomputation and storage overhead, exa‑d accelerates search‑engine indexing and lowers infrastructure costs, giving Exa a competitive edge in web‑scale retrieval.

Key Takeaways

• Typed columns enforce strict data contracts
• Dependency graph automates execution order
• Lance on S3 enables fragment‑level column updates
• Ray Data pipelines provide CPU‑GPU parallelism
• Surgical patches reduce recompute for bug fixes

Pulse Analysis

Storing the public web at petabyte scale forces search engines to grapple with heterogeneous content, rapid update cycles, and massive derived signal sets. Exa‑d tackles these challenges by abstracting every piece of information into typed columns and declaring their relationships rather than scripting procedural steps. This declarative model mirrors spreadsheet formulas, catching type mismatches at compile time and allowing engineers to focus on signal quality while the framework resolves state, retries, and scheduling automatically. The resulting dependency graph provides a single source of truth for execution order, ensuring that new signals or schema changes propagate predictably across billions of documents.

The storage layer builds on Lance, a columnar format optimized for S3. By fragmenting the dataset and tracking column presence at the fragment level, exa‑d can write, replace, or delete a single column without rewriting entire files. This granular approach dramatically cuts I/O and storage costs when fixing bugs, rolling out new embedding models, or handling hourly news updates. Metadata stored alongside fragments eliminates the need for auxiliary tables, simplifying consistency checks and enabling precise backfills that touch only the affected data slices.

Execution is orchestrated through Ray Data pipelines that translate the topologically sorted dependency graph into parallel tasks. Ray actors hold stateful resources such as GPU‑resident models, while separate stages keep CPUs, GPUs, and network I/O busy simultaneously. The system computes only missing or invalid columns, skipping cached results and automatically resuming after failures. As web volume and signal complexity grow, exa‑d’s modular design—typed contracts, fragment‑level storage, and scalable DAG execution—positions it as a reusable blueprint for any organization seeking real‑time, cost‑effective web‑scale indexing.