AI Isn’t Replacing Curious Developers
In this episode of Data Engineering Central, host Dan Beach chats with front‑end veteran Neil Roberts about his journey from early BASIC on an Atari to modern AI‑augmented development. Neil reflects on the hacker‑culture of the late‑90s, the importance of curiosity, and how AI tools amplify a developer’s existing habits—making the diligent craftier and the careless sloppier. He argues that AI won’t replace developers; it simply accelerates whatever approach you already take, and that staying curious lets you peel back AI‑generated code to learn deeper concepts. The conversation also touches on the shifting need for full‑stack skills as AI brings front‑end work back into focus for traditionally back‑end engineers.
Spark, AI, and the Future of Data Engineering with Daniel Aronovich
In this episode, host Dan Beach chats with data engineering veteran Daniel Aronovich about his 15‑year journey from MATLAB‑based signal processing at Intel to Python, Spark, and his current startup, True Data Flynn. Daniel explains how he transitioned from data...
Atomic Transactions in Databricks Spark SQL
Databricks announced that Unity Catalog now supports atomic transactions for managed Delta tables, entering public preview, while Iceberg tables remain in private preview. The feature introduces classic SQL transaction commands—BEGIN TRANSACTION, COMMIT, and ROLLBACK—directly in Spark SQL, extending the platform’s...
DuckDB, AI, and the Future of Data Engineering
In this episode, Dan Beach chats with State Farm staff engineer Matt Martin about his journey from industrial engineering to data engineering, his deep involvement with DuckDB, and the evolving landscape of data platforms. Matt shares how early automation with...
Polars Powerful Streaming Engine
Polars’ new streaming engine offers a single‑node, Rust‑based alternative to heavyweight distributed frameworks like Spark. By applying lazy query optimisation and batch‑wise materialisation, it delivers low‑latency ETL pipelines while dramatically cutting hardware costs. Early adopters have swapped Spark jobs for...