Data Engineering Foundations

What you will be able to do

• You will be able to explain what a data engineer owns and how the modern data stack fits together across ingestion, storage, transformation, orchestration, and serving.
• You will be able to design an ETL or ELT pipeline and justify which pattern fits a given pair of source and destination systems.
• You will be able to build reliability into a pipeline using idempotency, schema and data quality checks, testing, and basic observability.
• You will be able to choose between batch and streaming for a workload and defend the trade-offs in latency, cost, and complexity.
• You will be able to explain streaming fundamentals such as event logs, windowing, and delivery guarantees, and say when low latency is worth it.
• You will be able to compare data warehouses, data lakes, and lakehouses and pick the right store for a given use case and budget.
• You will be able to model data for analytics using dimensional modeling and a medallion (bronze, silver, gold) layout that downstream teams can trust.
• You will be able to trace an end-to-end data flow from a source system to a serving layer and explain it to a non-technical stakeholder.

What is inside

4 modules, 8 lessons. Each module ends in a graded quiz and most carry an assignment.

1. 01

   The Data Engineer's Job and the Modern Data Stack

   Map the data engineer's role and the modern data stack before you build. This module defines what you own between raw sources and downstream consumers, then walks the stack layer by layer (ingestion, storage, transformation, orchestration, and serving) to give you the vocabulary and mental model the rest of the course relies on.

   2 lessons · 5 quiz questions

2. 02

   Data Pipelines: Moving and Shaping Data

   Pipelines are the core of data engineering. This module breaks down how data moves from a source system into a usable destination: you compare ETL and ELT, learn the four-stage anatomy of a pipeline (extract, load, transform, orchestrate), and see how scheduling and dependencies hold the whole thing together. Then you focus on what separates a hobby script from production work: idempotency, data quality checks, testing, and observability that tells you when a run goes wrong.

   2 lessons · 5 quiz questions · assignment

3. 03

   Batch Versus Streaming: Choosing How Data Flows

   Not every workload should run the same way, so this module contrasts batch and streaming processing and the real trade-offs between them. You will study batch patterns, scheduling, and incremental loads for large periodic jobs, then move to streaming fundamentals such as event logs, windowing, and delivery guarantees. By the end you can reason about latency, cost, and complexity, and apply a clear framework for when low latency is worth it and when a nightly batch is the smarter call.

   2 lessons · 5 quiz questions

4. 04

   Where Data Lives: Warehouses, Lakes, and Lakehouses

   Every pipeline ends by writing data somewhere, and that landing choice quietly decides your query speed, your monthly bill, and whether analysts can trust what they find. This module compares data warehouses, data lakes, and the lakehouse pattern, then goes under the hood on the columnar formats (Parquet), partitioning, and file layout that make queries fast or ruinously expensive. You will finish by modeling data for analytics with dimensional modeling and a medallion (Bronze, Silver, Gold) architecture, so downstream teams inherit clean, trusted tables instead of a swamp.

   2 lessons · 5 quiz questions · assignment