Turning Chaos Into Push-Button Provisioning with Dhiraj Suri| Ep. 14

Dhiraj Suri’s career illustrates how low-level system languages still shape cloud engineering. Starting at NetApp, he wrote C and C++ storage software for hardware appliances, then moved to Brightroll where a billion-request ad-serving stack was rebuilt in C for performance. Those experiences taught him that language choice impacts scalability, yet he now works primarily with Java for Confluent’s Schema Registry and Go for control-plane services, while keeping an eye on Rust’s emerging role. This blend of legacy expertise and modern cloud stacks informs his leadership on the Stream Governance Platform, where schema enforcement guarantees data compatibility across polyglot environments.

The most demanding project Dhiraj describes was at Splunk, where disparate engineering and IT systems slowed enterprise customer provisioning. He architected an asynchronous, event-driven pipeline that stitched Salesforce, MuleSoft, AWS SQS/SNS, and internal queues into a push-button provisioning workflow. By decoupling components with a service mesh and deploying Kubernetes operators, the solution achieved fault tolerance, back-pressure handling, and exponential retry logic without manual intervention. This design demonstrates how streaming platforms like Kafka—or its SQS-based predecessor—can transform non-deterministic processes into reliable, observable services, dramatically reducing time-to-value for large customers.

Beyond the specific implementation, Dhiraj stresses that robust schema management is the glue holding heterogeneous microservices together. OpenAPI contracts, Avro, and Protobuf enable forward and backward compatibility, but his low-latency background leads him to champion Protobuf and gRPC for performance-critical pipelines. He advises engineers to stay current through blogs, Confluent’s developer portal, and the Apache open-source community, leveraging off-the-shelf tools like Argo Workflows rather than reinventing foundations. As data streaming converges with AI agents, the principles of asynchronous design, fault-tolerant operators, and clear schema contracts will remain essential for building scalable, push-button cloud services.