Free Lab: Building Spark Declarative Pipelines & Lakeflow Designer

Why It Matters

By offering a no‑code, free‑access lab with AI‑assisted SQL, Databricks democratizes complex pipeline building, enabling faster, cost‑effective data engineering for teams without deep programming expertise.

Key Takeaways

• Free lab teaches Databricks Lakeflow declarative pipelines.
• No‑code UI lets you build bronze, silver, gold layers via SQL.
• Genie AI generates and answers SQL queries within the designer.
• Course accessible at learn-data-engineering.com without Databricks subscription.
• Visual pipeline graph shows real‑time data flow and transformations.

Summary

The video introduces a free, self‑paced lab that walks users through Databricks’ new Lakeflow Designer, focusing on declarative pipelines built entirely with SQL. The instructor highlights how the no‑code interface lets data engineers define bronze, silver, and gold layers, set up streaming tables, and create materialized views without writing Python code. Key insights include the ability to construct end‑to‑end pipelines using simple SQL statements, visual pipeline graphs that update in real time, and the integration of Genie AI, which can auto‑generate queries and answer natural‑language questions about the data. The lab uses an online retail dataset to demonstrate table creation, data cleaning, and sales‑by‑day aggregation, showcasing both the UI and the underlying SQL logic. Notable examples feature a streaming table for raw orders, a cleaned‑orders table, and a materialized view summarizing daily revenue. The presenter also shows Genie’s chat and agent modes, where the tool can both provide raw SQL code and concise answers, such as identifying the day with highest sales. Visualizations like bar charts are created directly from the pipeline outputs. The significance lies in lowering the barrier to advanced data‑engineering workflows: users can experiment with Databricks’ lakehouse capabilities for free, accelerate pipeline development, and leverage AI assistance, making sophisticated ETL processes accessible to a broader audience.

Original Description

Next Monday, we’re launching a new free Databricks lab, and in this livestream I’ll walk you through it step by step.

Something changed recently in Databricks that makes building pipelines much easier: you can now create full data pipelines using Spark Declarative Pipelines SQL.

And on top of that, with Lakeflow Designer + Genie (Databricks Assistant) you can even build pipelines in a visual, almost no-code way.

I’ll walk through the lab and explain what’s actually going on.

► Get right into the FREE DATABRICKS LAB here:

https://learndataengineering.com/p/spark-declarative-pipelines-lakeflow-designer-on-databricks

We’re going to look at:

- building a Bronze → Silver → Gold pipeline (Medallion architecture)

- creating declarative pipelines using SQL

- understanding how Delta Lake and Unity Catalog fit into this

- orchestrating and running everything with Lakeflow Designer

- using Genie to speed up development and iteration

So you’ll see how to go from raw data → clean data → business-level aggregations in a structured way.

And you can ask questions directly if something is unclear. That’s usually the fastest way to really understand this.

📅 Monday, April 27 | 10am EST / 4pm CEST

🎥 Recording will be available afterwards

If you want to understand how modern Databricks pipelines are built now, this will help.

See you there.

► Find MORE FREE LABS here: https://learndataengineering.com/p/free-labs

► Learn Data Engineering with my DATA ENGINEERING ACADEMY:

https://learndataengineering.com/p/academy