The Same 16 GPUs, Twice the Users: Inference-Aware Routing for LLM Clusters

Why It Matters

Intelligent routing transforms expensive LLM inference into a scalable, cost‑effective service, unlocking higher user capacity on existing hardware. This shift is critical for enterprises seeking to deploy generative AI at production scale without ballooning GPU spend.

Key Takeaways

• Inference scheduler doubles concurrent users on same 16 H100 GPUs
• TTFT drops from ~80 s to ~150 ms with cache‑aware routing
• Throughput improves up to 109 % versus vanilla Kubernetes service
• Scheduler routes requests using KV cache state and queue depth
• Red Hat AI Enterprise provides 60‑day free trial of llm‑d

Pulse Analysis

Large language model inference is fundamentally different from typical microservice traffic. Requests vary widely in compute time, memory pressure, and token generation phases, making round‑robin load balancing inefficient. A single long request can monopolize a pod while others sit idle, and shared‑prefix caching—crucial for reducing prefill work—fails when traffic is indiscriminately routed. These characteristics create bottlenecks that inflate GPU‑hour costs and limit user concurrency.

llm‑d addresses the problem with an inference‑aware routing layer built on Envoy and the Kubernetes Gateway API. The Inference Scheduler continuously monitors every pod’s KV‑cache contents, queue depth, and current load, scoring each instance before directing a request to the optimal node. In benchmark tests on a cluster of eight vLLM pods across 16 NVIDIA H100 GPUs, this approach delivered up to 109 % higher throughput and cut time‑to‑first‑token from roughly 80 seconds to 150 milliseconds. The result is a stable service that can sustain around 200 concurrent users at SLA targets, compared with just 20 users on a conventional Kubernetes deployment.

For enterprises, the economic impact is immediate: the same hardware investment supports twice the workload, reducing the need for additional GPU purchases. The solution integrates seamlessly with vLLM, preserving node‑level optimizations while adding cluster‑wide intelligence. Red Hat AI Enterprise bundles llm‑d with a 60‑day free trial, enabling organizations to evaluate the technology on‑premise or in the cloud. As generative AI moves from pilot projects to core business applications, inference‑aware scheduling will become a cornerstone of cost‑effective, high‑performance AI infrastructure.