LightSeek Foundation Releases TokenSpeed, an Open-Source LLM Inference Engine Targeting TensorRT-LLM-Level Performance for Agentic Workloads

Why It Matters

TokenSpeed delivers faster, more cost‑effective inference for large‑scale coding agents, expanding open‑source options and reducing reliance on expensive proprietary NVIDIA solutions.

Key Takeaways

• TokenSpeed open‑source MIT license targets agentic inference
• C++ FSM scheduler enforces KV cache safety at compile time
• Pluggable kernel layer supports non‑NVIDIA accelerators
• Outperforms TensorRT‑LLM 9% latency, 11% throughput on B200
• MLA kernel halves decode latency, adopted by vLLM

Pulse Analysis

Inference efficiency has become a silent choke point as AI‑driven coding assistants move from niche tools to core development infrastructure. Unlike traditional chatbots, agentic workloads juggle context windows exceeding 50 K tokens and sustain dozens of conversational turns, pressuring both per‑GPU tokens‑per‑minute and per‑user tokens‑per‑second metrics. TokenSpeed tackles this dual challenge by architecting a pipeline that maximizes throughput while guaranteeing a responsive user experience, a balance rarely captured in public benchmarks.

At the heart of TokenSpeed are five interlocking subsystems. A compiler‑backed SPMD modeling layer automatically generates collective communication code, freeing developers from manual parallelism plumbing. The scheduler’s control plane, written in C++ as a finite‑state machine, embeds KV‑cache ownership rules into the type system, catching errors at compile time. Meanwhile, the execution plane remains in Python for rapid iteration. The kernel layer treats GPU kernels as first‑class plugins, offering a public API and registry that support NVIDIA, AMD, and emerging accelerator architectures. Integration with SMG, a PyTorch‑native entrypoint, slashes CPU‑to‑GPU handoff latency, further tightening the end‑to‑end inference loop.

The performance gains translate into tangible business value. In head‑to‑head tests on NVIDIA’s B200 GPU with the Kimi K2.5 model, TokenSpeed delivered roughly 9% lower minimum latency and 11% higher throughput at 100 TPS per user, while its MLA decode kernel halved latency on speculative decoding tasks. By offering these advantages under an open‑source license, TokenSpeed lowers the barrier for enterprises to deploy high‑throughput coding agents without locking into costly proprietary stacks, fostering broader adoption and accelerating innovation across the AI development ecosystem.