Big Data Videos
Big Data Videos
Big Data Pulse
EMAIL DIGESTS
Daily
Every morning
Weekly
Tuesday recap
EMAIL DIGESTS
Daily
Every morning
Weekly
Tuesday recap
The video outlines a complete CI/CD workflow for Spark batch data pipelines running on AWS, detailing how code moves from a developer’s machine to production clusters such as EMR or Glue. In the continuous‑integration stage, GitHub Actions automatically run linting, formatting checks, unit tests, and code‑coverage analysis, then compile the code with SBT or pip to produce a deployable artifact. The continuous‑deployment stage uploads that artifact to an S3 bucket and triggers its deployment to a development EMR/Glue cluster. When the code passes dev validation, the same artifact is promoted to a staging environment where integration tests run; a pull‑request merge then promotes the artifact to production without rebuilding. The presenter notes that a PDF of the pipeline diagram will be sent via direct message. By automating testing and deployment across isolated environments, the process reduces manual errors, shortens release cycles, and gives data engineering teams a reproducible, version‑controlled path to push Spark jobs into production at scale.
The video walks viewers through creating a conversational AI agent that answers plain‑English questions about the U.S. economy, all built and run inside Snowflake’s unified cloud data platform. By the end, users have a functional Streamlit interface that queries CPI,...
The video introduces a proactive artificial‑intelligence platform that continuously scans enterprise data to automatically detect anomalies. By flagging irregular patterns—such as sudden jumps in marketing spend tied to a particular vendor—the system surfaces insights that would otherwise remain hidden until...
The episode of Railnatter focuses on the Plain Line Pattern Recognition (PLPR) train, a 20‑year‑old measurement unit that has become a cornerstone of Britain’s rail‑track inspection regime. Host Gareth and guest Alex, a veteran Network Rail engineer, explain why this...
The video introduces the Department of Energy’s Orchestrated Platform for Autonomous Laboratories (Opel), a cross‑lab initiative designed to accelerate AI‑enabled biological discovery and support the broader Genesis mission. Four national laboratories—Oak Ridge, Argonne, Pacific Northwest, and Lawrence Berkeley—are pooling expertise...
The video introduces Dimensional Insight’s new data‑wellness offering, emphasizing that robust data governance is essential before organizations deploy AI. James Curtley and Julie Learu explain how their approach embeds governance at every stage of the data pipeline—from source extraction to...
The video provides a concise comparison of Lambda and Kappa architectures, two dominant paradigms for processing large‑scale data streams. Lambda, introduced to marry batch accuracy with real‑time speed, relies on separate batch and streaming pipelines, whereas Kappa streamlines the stack...
In this tutorial Brian Kafki steps back from dimensional modeling to outline the full data‑warehouse ETL pipeline, from source systems through raw ingestion, pre‑staging, staging, an operational data store (ODS) snapshot, and finally the data mart that powers BI tools. He...
The video argues that AI is not eliminating data engineers but redefining the role, outlining a 2026 AI‑enhanced data engineering stack that promises to keep the top 1% of engineers job‑proof. Founder Chris Garzone of Data Engineer Academy walks viewers...
The speaker outlines a proactive ‘data engineering roadmap’ strategy for orchestrating career and salary growth over a multi‑year horizon. Using a friend’s five‑year plan as an example—mapping promotions and job switches that boosted total compensation from about $100k to roughly...
The video argues that frequent job-hopping can dramatically increase lifetime compensation for data engineers, claiming switchers can earn 10–20 times more than long-tenured employees. Using a hypothetical 10-year comparison, the speaker says a stay-at-home trajectory might grow from $100k to...
The video presents a comprehensive 2026‑edition roadmap for becoming an AWS‑focused data engineer, detailing the essential services across six logical layers—from foundational infrastructure to AI‑enabled analytics. It emphasizes that product‑centric firms predominantly run on AWS, making the transition from Azure...
The video walks viewers through Amazon SageMaker’s newly refreshed Unified Studio, a fully managed, browser‑based environment that requires no local installation. By logging in with existing IAM permissions, users instantly access data catalogs, S3 buckets, and a suite of connection...
Lyft’s Lift Pink subscription, offering monthly discounts, faced a surge of failed renewal payments as many users entered faulty credit‑card information during free‑trial sign‑ups. The churn threatened revenue and highlighted a systemic data‑quality problem. The engineering team discovered that transaction, payment,...
The adviser tells a job seeker that their main problem is targeting the wrong role—applying as a performance engineer when they should be pursuing data engineering or AI roles—and that performance engineering may be becoming less in demand. They warn...
