Building Data Pipelines

What you will be able to do

• Choose between ETL and ELT for a given workload and defend the decision on cost, latency, and scale
• Write advanced SQL with window functions, CTEs, and set-based merges that transform large tables efficiently
• Build Python extraction and processing jobs that survive flaky APIs, retries, and datasets larger than memory
• Model a warehouse in dbt with layered, tested, incremental transformations you and analysts can trust
• Design star-schema dimensional models with the correct grain and slowly changing dimension handling
• Orchestrate multi-step pipelines in Airflow using dependencies, retries, backfills, and idempotent tasks
• Make pipelines observable and reliable in production with data quality tests, sensors, and alerting
• Ship an end-to-end pipeline that extracts, loads, transforms with dbt, and runs on an Airflow schedule

What is inside

4 modules, 8 lessons. Each module ends in a graded quiz and most carry an assignment.

1. 01

   Pipeline Foundations: ETL and ELT

   The mental model every senior data engineer works from. This module compares ETL and ELT head to head, so you can pick an architecture on cost, latency, and where the compute lives, then frames every pipeline around idempotency, incremental loading, and data contracts. These are the foundations the rest of the course builds on: pipelines that are reliable, reprocessable, and cheap to run at scale.

   2 lessons · 5 quiz questions

2. 02

   SQL and Python for Data Processing

   This module is the hands-on core of the course: the SQL and Python that actually move and reshape data at scale. You will push SQL past the basics into window functions, CTEs, and set-based incremental merges, then use Python for extraction and processing that stays resilient and memory-safe on large, messy inputs.

   2 lessons · 5 quiz questions · assignment

3. 03

   Transformations with dbt and Dimensional Modeling

   Raw loaded data is not a warehouse yet. In this module you build layered, tested transformations in dbt (models, refs, generic and singular tests, macros, and incremental builds), then design the dimensional layer that downstream reporting depends on: star schemas, a correctly declared grain, the right fact and measure types, and slowly changing dimensions. The result is analytics engineering that makes numbers reliable, joins safe, and history accurate.

   2 lessons · 5 quiz questions

4. 04

   Orchestration and Reliability with Airflow

   Operationalize your pipeline so it runs without you watching. This module builds Airflow DAGs with real scheduling, dependencies, retries, and backfills, then adds the idempotency, sensors, data quality checks, and alerting that make a pipeline observable and dependable in production. It ends by wiring Python and dbt steps into one scheduled workflow.

   2 lessons · 5 quiz questions · assignment