Building Banking Systems with Kafka Streams with Mateo Rojas | Ep. 28

In the early days of Kafka Streams, developers like Mateo Rojas wrestled with windowed joins that demanded absurdly long time windows—sometimes years—to guarantee that disparate events could be correlated. Without mature exactly‑once semantics, teams resorted to oversized windows and K‑tables for deduplication, storing each incoming record to prevent double processing. These work‑arounds highlighted the steep learning curve of event‑driven microservices and the fragility of early stream orchestration, especially when coordinating KYC, AML, and banking approvals across multiple topics.

Security proved another formidable obstacle. The immutable nature of Kafka topics made at‑rest encryption a persistent challenge; rotating keys required re‑processing entire logs, an impractical task for a system treated as the source of truth. Teams settled for TLS in transit and short retention periods, while a dedicated decryption service handled occasional privileged reads. This approach balanced regulatory compliance with operational feasibility, yet underscored the trade‑offs between data confidentiality and the convenience of Kafka‑centric state reconstruction.

Today, the industry favors workflow engines over intricate stream joins for complex banking orchestration. Modern platforms provide built‑in state management, visual modeling, and reliable compensation mechanisms, reducing the need for custom window logic and manual deduplication. The debate over using Kafka as a primary source of truth continues, with practitioners weighing eventual consistency against strong consistency guarantees and security requirements. By learning from early missteps—oversized windows, ad‑hoc deduplication, and incomplete encryption—organizations can design more resilient, auditable, and secure streaming architectures that align with contemporary best practices.