WEKA Reports 10x Higher AI Inference Throughput with NeuralMesh on OCI

Why It Matters

By eliminating the GPU memory bottleneck, WEKA’s solution makes long‑context inference economically viable, enabling enterprises to scale AI services without proportional hardware spend. This shifts the cost structure of generative AI toward higher utilization and lower per‑token expense.

Key Takeaways

• NeuralMesh serves 5,000 users vs 600 DRAM baseline
• Token throughput reaches ~2 million per second, 10× higher
• 7× more tokens per GPU cuts cost per token
• NVMe cache expands usable memory from 8.64 TiB to 287 TiB
• Solution now on Oracle Marketplace, ready for OCI deployment

Pulse Analysis

The benchmark results underscore a broader industry challenge: GPU memory limits have become the primary constraint on large‑context inference workloads. Traditional DRAM‑centric designs force frequent KV‑cache evictions, inflating latency and driving up token costs. WEKA’s Augmented Memory Grid decouples the cache from GPU memory, leveraging high‑performance NVMe to create a shared token warehouse. This architectural shift not only expands the effective memory pool to hundreds of terabytes but also enables any host in the cluster to pick up any session without losing cache state, dramatically improving load balancing and scaling potential.

For enterprises deploying generative AI services—search, summarization, code assistance, or multi‑turn agents—the ability to serve thousands of concurrent users on a fixed GPU footprint translates directly into revenue upside. The ten‑fold increase in token throughput means faster response times and higher user satisfaction, while the seven‑fold boost in tokens per GPU slashes the cost per token, improving ROI on existing hardware investments. Companies that have struggled with the "memory wall" can now consider longer context windows (100k tokens or more) without prohibitive infrastructure spend.

WEKA’s partnership with Oracle Cloud Infrastructure positions the solution for rapid adoption across cloud‑first AI strategies. With the product listed on the Oracle Marketplace, organizations can provision the NeuralMesh‑Augmented Memory Grid stack in minutes, leveraging Oracle’s bare‑metal H100 instances for optimal performance. As AI workloads continue to grow in complexity, solutions that unlock latent GPU capacity while reducing operational costs will become a decisive competitive advantage for cloud providers and enterprise AI teams alike.