How Netflix Serves ML Predictions to 250M Users at 1 Million Requests Per Second
Netflix has built Switchboard, a custom ML serving router that handles over 1 million requests per second for its 250 million global users. The system routes hundreds of model types—recommendations, fraud detection, search embeddings, and artwork scoring—across shared infrastructure while allowing rapid experimentation and instant rollbacks. Switchboard abstracts model versioning from client services, enabling multiple concurrent A/B tests per user. By focusing on the serving layer rather than model training, Netflix achieves sub‑second latency at massive scale.
You Can't Fix What You Can't See
The post outlines six observability patterns essential for debugging microservice architectures, drawing on the Microservices Patterns book by Chris Richardson and real‑world implementations at Netflix, Uber and Discord. It explains why monolithic debugging is simple compared to the fragmented logs,...
You're Ahead of 90% of People If You Know These 5 AI Terms
The post breaks down five core AI concepts—tokens, context windows, temperature, hallucination, and retrieval‑augmented generation (RAG)—that most users overlook. Tokens are the smallest text units and drive pricing, limits, and model performance. Context windows define how much information a model...
System Design Deep Dives: Part - 1
The post explains how scalability and availability shape modern system design. It contrasts vertical scaling—quick but limited—with horizontal scaling that requires stateless services, load balancers, and distributed data stores, citing Twitter’s evolution from a monolith to a sharded architecture. It...
Your Token Was Stolen. Now What?
The article warns that stolen JWTs let attackers impersonate users until the token expires, exposing a critical weakness in many API authentication flows. It outlines the typical login sequence, then highlights how tokens stored in insecure locations or with long...