LinkedIn’s MixLM: 10x Faster LLM Ranking via Embedding Injection

Key Takeaways

• MixLM compresses 900‑token documents to 1‑2 embedding tokens
• Reduces LLM reranking latency, enabling 100% job search traffic
• Achieves 10× throughput boost with only 0.02 NDCG drop
• Shared‑prefix KV cache cuts per‑candidate cost dramatically
• Daily active users rose 0.47% after deployment

Pulse Analysis

MixLM’s core insight is to move the heavy lifting of document understanding offline. By encoding full job descriptions into dense vectors once and storing them in a near‑line cache, LinkedIn eliminates the quadratic attention cost associated with feeding long texts into a cross‑encoder. This strategy mirrors trends in vector‑search systems, but pushes the compression further by feeding the embeddings directly into the LLM’s input layer, effectively turning the ranker into a lightweight relevance scorer.

The engineering gains are equally compelling. Leveraging shared‑prefix KV caching, the system computes the query and system prompt once per request, then reuses that cache across thousands of candidate embeddings. Combined with a prefill‑only inference path that discards the KV cache after scoring, the solution slashes VRAM usage and allows batch sizes that were previously impossible for LLM‑based rerankers. The result is a tenfold throughput increase, making full‑traffic deployment feasible in 2025—a milestone for production AI.

From a business perspective, the modest 0.02 NDCG dip is outweighed by the 0.47% rise in daily active users, demonstrating that users value faster, more responsive search experiences even if relevance scores dip slightly. MixLM’s success may inspire other platforms—e‑commerce, media, and social networks—to adopt similar embedding‑injection pipelines, reshaping how large language models are integrated into high‑volume, latency‑sensitive services.