Home / Learn / Data & AI / Pipeline to Dashboard
All lessons

Pipeline to Dashboard: ETL, warehouses and BI, step by step

Build the pipelines that move data from email, shared drives and APIs into a warehouse, clean it, model it and put it on dashboards people trust: Python, Polars, DuckDB, Dagster, Azure SQL, dbt and Power BI, with a project at the end of every module.

For analysts moving into data engineering, BI developers who want to own the pipeline, and students. Basic Python and SQL help; every step is explained. Free, no sign-up. All 49 lessons, about 31.4 hours.

What you will be able to do

  • Get data in from anywhere: shared drives, email attachments, SharePoint and Google Drive, SFTP, APIs and source databases, landed safely and only once.
  • Clean it so people trust it: profiling, data contracts, Polars and DuckDB transforms, duplicates and entity matching, and quality gates that quarantine bad rows.
  • Run it on its own with Dagster: assets, partitions, backfills, schedules, sensors, retries and alerts.
  • Load and model a warehouse: Azure SQL, Postgres, DuckDB and ClickHouse, star schemas, slowly changing dimensions and tested dbt models.
  • Deliver dashboards that add up: fast Power BI models, correct DAX, refresh and security, and reconciliation that earns trust.

Every lesson has a step-by-step animation of the key operation, a plain-words meaning box, live problems with fresh numbers, a short quiz and a summary sheet to download. Each module ends with a project you can show an employer. Take pandas and Statistics first if they are new; go on to Data at Scale when you finish.

0 of 49 lessons done

Without the tick, your progress lasts until you close this tab. Either way it stays in this browser: there is no sign-up and nothing is sent to us.

All modules

Part A: Get the data in and clean

1 Ingest anything 7 lessons

  1. 1.1 The shape of a data platform
  2. 1.2 An engineering setup that survives a real job
  3. 1.3 Landing files from folders and shared drives
  4. 1.4 Extracting data from email
  5. 1.5 SharePoint, OneDrive, Google Drive, SFTP and object storage
  6. 1.6 APIs and source databases
  7. 1.7 Project 1: a multi-source landing service

2 Clean and trust the data 7 lessons

  1. 2.1 Profiling and data contracts
  2. 2.2 Transforming with Polars and DuckDB
  3. 2.3 The messy-data field guide
  4. 2.4 Cleaning with Claude Code and open models
  5. 2.5 Duplicates, keys and entity resolution
  6. 2.6 Quality gates, quarantine and the data quality report
  7. 2.7 Project 2: a trustworthy cleaning pipeline

Part B: Run it and store it

3 Orchestrate with Dagster 7 lessons

  1. 3.1 What an orchestrator really does
  2. 3.2 Dagster assets and your first real project
  3. 3.3 Resources, config, IO managers and environments
  4. 3.4 Partitions and backfills
  5. 3.5 Schedules, sensors, retries and alerts
  6. 3.6 Asset checks, testing and deploying Dagster
  7. 3.7 Project 3: a self-running hotel pipeline

4 Load to the warehouse 7 lessons

  1. 4.1 Choosing where the data lives
  2. 4.2 Azure SQL Database, hands on
  3. 4.3 Loading patterns that survive reruns
  4. 4.4 Postgres, DuckDB and ClickHouse side by side
  5. 4.5 T-SQL for people who run warehouses
  6. 4.6 Security, cost and operations in Azure
  7. 4.7 Project 4: one pipeline, any warehouse

Part C: Model it and show it

5 Model it in SQL 7 lessons

  1. 5.1 Dimensional modelling and grain
  2. 5.2 Dimensions done right: keys, dates and SCD2
  3. 5.3 SQL the way senior analysts write it
  4. 5.4 dbt: SQL models with software discipline
  5. 5.5 dbt inside Dagster, and the alternatives
  6. 5.6 Metrics, additivity and the semantic layer
  7. 5.7 Project 5: a tested star schema

6 Deliver BI people trust 7 lessons

  1. 6.1 Power BI data models that stay fast
  2. 6.2 DAX that is correct, not just working
  3. 6.3 Dashboards people read in ten seconds
  4. 6.4 Refresh, security and report delivery
  5. 6.5 Open-source BI without Microsoft licences
  6. 6.6 Reconciliation, the skill that earns trust
  7. 6.7 Project 6: BI people can trust

Part D: Production

7 Run it in production 7 lessons

  1. 7.1 Observability and incident response
  2. 7.2 CI/CD, environments and safe change
  3. 7.3 Choosing tools with a written rationale
  4. 7.4 A low-cost open-source stack for medium companies
  5. 7.5 Performance, scale and when to change gear
  6. 7.6 Capstone: a daily data platform
  7. 7.7 Ship it and tell the story