Become a data engineer, starting from zero.

433 steps of 5-10 minutes. Python, SQL, pipelines, cloud and streaming, each step with practice and a check.

Step 1, live: the journey of one order

SELECT * FROM raw.orders;  -- shop DB + marketplace CSV
ordercreated_atamountcurcustomer
10412026-11-02 09:14 CET42.50EUR7
10422026-11-02T08:40:00Z19.99USD3
10422026-11-02T08:40:00Z19.99USD3
104302.11.2026 12:05emptyEUR9
10442026-11-01 23:30 PST120.00usd7
10452026-11-01 18:02 CET64.00EUR3

6 rows2 sources. One duplicate, one empty amount, 3 time formats, 2 currencies.

SELECT DISTINCT order_id, to_utc(created_at),
       amount * fx.rate AS amount_eur
FROM raw.orders JOIN fx USING (currency)
WHERE amount IS NOT NULL;
ordercreated_at (UTC)amount_eur
10412026-11-02 08:1442.50
10422026-11-02 08:4018.39
10442026-11-02 07:30110.40
10452026-11-01 17:0264.00

4 rows, 1 quarantinedDuplicate removed, order 1043 sent to quarantine, times in UTC, USD converted at a sample rate of 0.92.

SELECT o.*, c.country
FROM stg_orders o
JOIN dim_customer c USING (customer_id);
ordercreated_at (UTC)amount_eurcountry
10412026-11-02 08:1442.50DE
10422026-11-02 08:4018.39FR
10442026-11-02 07:30110.40DE
10452026-11-01 17:0264.00FR

fct_orders: 4 rowsEach order joined to its customer. One row is one order: that is the grain.

SELECT order_date, COUNT(*), SUM(amount_eur)
FROM fct_orders GROUP BY 1;
day (UTC)ordersrevenue_eurchart
2026-11-01164.00
2026-11-023171.29

235.29 EUROrder 1044 looked like November 1 in the raw file. In UTC it belongs to November 2.

The route, drawn to scale

Each segment is as wide as its number of steps. Python and SQL come first, because almost every data job asks for both.

0-83 Start from zero 84-140 Data foundations 141-237 Pipelines 238-277 Reliable data 278-323 Cloud 324-347 Streaming and real time 348-362 Data for ML and AI 363-432 Senior practice

One step is one idea and 5-10 minutes

Short steps keep you going on a busy evening. New words appear only after a step has explained them.

  • One shop dataset from step 1 to the final project: orders, customers, clicks, payments, tickets.
  • Every 10-15 steps, a checkpoint or capstone you can show.
  • Hard tools arrive when you feel the need. Spark comes after pandas runs out of memory.
  1. 2-3 min

    Read

    A situation from real work, the idea in plain words, one diagram.

  2. 1-2 min

    See

    Code or SQL on the shop dataset, with real output.

  3. 3-5 min

    Do

    One small task in your terminal or editor.

  4. instant

    Check

    The checker runs your query, test or pipeline and shows a green tick.

What you will have built

27 capstones and checkpoints, then a final project. A selection:

  • step 386An on-call week: 5 production incidents fixed in a row
  • step 158Incremental ingestion from 3 sources, safe to run twice
  • step 228A dbt project with tests, docs and CI
  • step 299The same shop pipeline on AWS, then on GCP and Azure
  • step 347A live orders dashboard fed by change data capture
  • step 370A warehouse bill cut by 40%, with the numbers

Syllabus

Every step, in order. Steps 0-30 are free.

Part 1 Start from zero 0-83

What data engineering is 0-7
  1. 0What a data engineer does all day free
  2. 1The journey of one order: from click to dashboard free
  3. 2Sources, pipelines, storage, consumers free
  4. 3Batch and streaming in one picture free
  5. 4OLTP and OLAP: two kinds of databases free
  6. 5Data engineer, analyst, analytics engineer, ML engineer free
  7. 6The modern data stack on one map free
  8. 7What you will build in this course free
Terminal and git 8-18
  1. 8Open the terminal and run 3 commands free
  2. 9Files, folders and paths free
  3. 10Look inside a CSV: head, tail, wc free
  4. 11Find rows with grep free
  5. 12Pipes and redirection free
  6. 13Environment variables free
  7. 14Install tools with a package manager free
  8. 15git: your first commit free
  9. 16Branches and merges free
  10. 17GitHub: push and open a pull request free
  11. 18Checkpoint: a repo with a cleaned CSV free
Python from zero 19-43
  1. 19Install Python and run your first script free
  2. 20Variables and types free
  3. 21Strings and f-strings free
  4. 22Lists free
  5. 23Dictionaries free
  6. 24Conditions free
  7. 25Loops free
  8. 26Functions free
  9. 27Modules and imports free
  10. 28Reading and writing files free
  11. 29CSV with the csv module free
  12. 30JSON in and out free
  13. 31Errors and try/except
  14. 32Virtual environments and uv
  15. 33Installing packages
  16. 34Classes in 10 minutes
  17. 35Dataclasses and type hints
  18. 36Dates, times and time zones
  19. 37Calling an HTTP API
  20. 38Pagination and rate limits
  21. 39Logging instead of print
  22. 40Your first test with pytest
  23. 41A command-line script with arguments
  24. 42Generators: handling files bigger than memory
  25. 43Capstone: daily API-to-CSV collector
SQL from zero 44-73
  1. 44What a database is
  2. 45Tables, rows, columns
  3. 46Run Postgres and connect
  4. 47SELECT
  5. 48WHERE
  6. 49ORDER BY and LIMIT
  7. 50COUNT, SUM, AVG, MIN, MAX
  8. 51GROUP BY
  9. 52HAVING
  10. 53NULL and why it breaks comparisons
  11. 54INNER JOIN
  12. 55LEFT JOIN
  13. 56Join fan-out: when totals double
  14. 57Subqueries
  15. 58CTEs with WITH
  16. 59CASE WHEN
  17. 60Dates in SQL
  18. 61Text in SQL
  19. 62Window functions: ROW_NUMBER
  20. 63Running totals and LAG/LEAD
  21. 64Deduplicate with a window
  22. 65INSERT, UPDATE, DELETE
  23. 66CREATE TABLE and data types
  24. 67Primary and foreign keys
  25. 68Indexes
  26. 69Transactions
  27. 70Reading EXPLAIN
  28. 71Views and materialized views
  29. 72SQL from Python
  30. 73Capstone: 10 business questions on a shop database
Advanced SQL 74-83
  1. 74Gaps and islands
  2. 75Sessionization: grouping clicks into visits
  3. 76Recursive CTEs
  4. 77ROLLUP, CUBE, GROUPING SETS
  5. 78Pivot and unpivot
  6. 79Top N per group, medians, percentiles
  7. 80Isolation levels and MVCC
  8. 81Join algorithms: nested loop, hash, merge
  9. 82Table partitioning in Postgres
  10. 83Checkpoint: 10 interview-style SQL problems

Part 2 Data foundations 84-140

Data modeling 84-105
  1. 84Why the shape of data matters
  2. 85Normalization in plain words
  3. 86Drawing an ER diagram
  4. 87Facts and dimensions
  5. 88Grain: one row means what
  6. 89Star schema
  7. 90Snowflake schema
  8. 91Surrogate and natural keys
  9. 92The date dimension
  10. 93Slowly changing dimensions: type 1 and 2
  11. 94SCD types 3, 4 and 6
  12. 95Fact table types: transaction, periodic snapshot, accumulating
  13. 96Factless facts
  14. 97Bridge tables for many-to-many
  15. 98Conformed dimensions and the bus matrix
  16. 99Late-arriving dimensions
  17. 100One big table: when it wins
  18. 101Data Vault in one step
  19. 102Modeling event data
  20. 103Semi-structured JSON in a warehouse
  21. 104Choosing a model for a team
  22. 105Capstone: model a subscription business
Storage and file formats 106-117
  1. 106Row storage and column storage
  2. 107CSV traps: commas, quotes, encodings
  3. 108JSON Lines
  4. 109Parquet
  5. 110Avro
  6. 111Compression codecs
  7. 112Partitioning files by date
  8. 113The small files problem
  9. 114Object storage: buckets and keys
  10. 115MinIO: S3 on your laptop
  11. 116Lake, warehouse, lakehouse
  12. 117Capstone: raw CSV to partitioned Parquet in MinIO
Docker and servers for data 118-127
  1. 118Why containers
  2. 119Your first Dockerfile
  3. 120docker compose with Postgres
  4. 121Volumes: data that survives restarts
  5. 122Container networking
  6. 123Config and secrets in containers
  7. 124cron: the simplest scheduler
  8. 125SSH to a server
  9. 126Makefile as the project's front door
  10. 127Capstone: a one-command local data stack
Distributed systems basics 128-135
  1. 128Why one machine stops being enough
  2. 129Replication
  3. 130Partitioning and sharding
  4. 131Consistency models
  5. 132CAP and PACELC
  6. 133Clocks and the order of events
  7. 134Consensus in one step
  8. 135At-most-once, at-least-once, exactly-once
Other databases 136-140
  1. 136Document stores: MongoDB
  2. 137Key-value and wide-column: DynamoDB and Cassandra
  3. 138Redis
  4. 139Search engines: Elasticsearch and OpenSearch
  5. 140Getting data out of NoSQL into a warehouse

Part 3 Pipelines 141-237

Ingestion 141-158
  1. 141Kinds of sources
  2. 142Full load and incremental load
  3. 143Watermarks: remembering where you stopped
  4. 144Idempotency: safe to run twice
  5. 145API ingestion with retries
  6. 146Many API calls at once with asyncio
  7. 147Webhooks
  8. 148Files from SFTP and email
  9. 149Extracting from a production database
  10. 150Change data capture: the idea
  11. 151dlt: ingestion as Python code
  12. 152Airbyte
  13. 153Managed connectors: Fivetran and friends
  14. 154Schema drift
  15. 155Late and duplicate data
  16. 156Backfills
  17. 157Web scraping, legally
  18. 158Capstone: incremental ingestion from 3 sources
Batch processing on one machine 159-167
  1. 159pandas basics
  2. 160pandas: group, join, reshape
  3. 161Where pandas runs out of memory
  4. 162Polars
  5. 163DuckDB
  6. 164DuckDB over Parquet in S3
  7. 165Benchmark: pandas vs Polars vs DuckDB
  8. 166Choosing the tool by data size
  9. 167Capstone: 50 GB on a laptop
Spark 168-183
  1. 168Why distributed processing
  2. 169Driver, executors, cluster
  3. 170Spark locally
  4. 171The DataFrame API
  5. 172Lazy evaluation: transformations and actions
  6. 173Spark SQL
  7. 174Partitions
  8. 175Shuffles
  9. 176Broadcast joins
  10. 177Data skew
  11. 178Why Python UDFs are slow
  12. 179Reading the Spark UI
  13. 180Caching
  14. 181Memory and executor tuning
  15. 182Writing partitioned output
  16. 183Capstone: a 3x faster Spark job
JVM for Spark optional 184-186
  1. 184Reading Spark code in Scala
  2. 185JVM memory and garbage collection in Spark
  3. 186Java and Scala in job postings: what to learn
Lakehouse tables 187-195
  1. 187Table formats: why files are not enough
  2. 188Delta Lake
  3. 189Apache Iceberg
  4. 190MERGE and upserts
  5. 191Schema evolution
  6. 192Idempotent partition overwrite
  7. 193Time travel and rollback
  8. 194Table maintenance: compaction and vacuum
  9. 195Capstone: a daily-updated table with no duplicates
Orchestration 196-213
  1. 196Why cron stops being enough
  2. 197DAGs
  3. 198Airflow: install and first DAG
  4. 199Operators and tasks
  5. 200Schedules, catchup, data intervals
  6. 201Retries, timeouts, SLAs
  7. 202Sensors and dependencies between DAGs
  8. 203Passing data between tasks
  9. 204Dagster: thinking in assets
  10. 205Prefect and others: how to choose
  11. 206Backfills in an orchestrator
  12. 207Alerts that people read
  13. 208Testing DAGs
  14. 209Dynamic task mapping
  15. 210Data-aware scheduling
  16. 211Executors: local, Celery, Kubernetes
  17. 212Running Airflow in production
  18. 213Capstone: the daily pipeline, orchestrated
Transformation with dbt 214-228
  1. 214ETL and ELT
  2. 215A dbt project from scratch
  3. 216Models and ref()
  4. 217Sources and freshness
  5. 218Materializations
  6. 219Incremental models
  7. 220Tests
  8. 221Docs and lineage graph
  9. 222Jinja and macros
  10. 223Snapshots for SCD type 2
  11. 224Seeds and packages
  12. 225Layers: staging, intermediate, marts
  13. 226dbt in CI: slim builds
  14. 227Semantic layer and metrics
  15. 228Capstone: a dbt project for the shop
Testing data code 229-233
  1. 229Unit tests for transformations
  2. 230Testing Spark code
  3. 231dbt unit tests
  4. 232Integration tests with Testcontainers
  5. 233Test data: fixtures and synthetic rows
Serving data 234-237
  1. 234BI dashboards with Superset
  2. 235Self-serve analytics with Metabase
  3. 236A data API with FastAPI
  4. 237Caching and limits for data APIs

Part 4 Reliable data 238-277

Data quality and contracts 238-249
  1. 238What bad data costs
  2. 239Quality dimensions you can measure
  3. 240Great Expectations and Soda
  4. 241Data contracts
  5. 242Schema registry
  6. 243Anomaly checks on volume and freshness
  7. 244The quarantine pattern
  8. 245Reconciliation with the source
  9. 246Freshness SLAs
  10. 247Entity resolution: one customer from 3 systems
  11. 248Golden records and master data
  12. 249Capstone: a pipeline that stops bad data
DataOps 250-262
  1. 250Git flow for data projects
  2. 251CI for pipelines
  3. 252Dev, staging, prod for data
  4. 253Terraform: infrastructure as code
  5. 254Terraform for buckets and warehouses
  6. 255Secrets management
  7. 256Pipeline observability: logs, metrics, traces
  8. 257Lineage with OpenLineage
  9. 258Runbooks
  10. 259On-call and incidents
  11. 260Writing a postmortem
  12. 261AI coding agents in data work
  13. 262Capstone: CI/CD for the whole stack
Kubernetes for data 263-268
  1. 263Pods, deployments, services
  2. 264Helm charts
  3. 265Spark on Kubernetes
  4. 266Airflow with KubernetesPodOperator
  5. 267Resource requests, limits, autoscaling
  6. 268Capstone: the data stack on a local kind cluster
Governance, privacy, security 269-277
  1. 269Data catalogs
  2. 270Finding and classifying PII
  3. 271Masking and pseudonymization
  4. 272Access control: roles and grants
  5. 273Row-level and column-level security
  6. 274GDPR: deleting a person from a data lake
  7. 275Retention policies
  8. 276Audit logs
  9. 277Capstone: a privacy review of your platform

Part 5 Cloud 278-323

Cloud basics 278-288
  1. 278Open a cloud account safely
  2. 279Budget alerts before anything else
  3. 280IAM: users, roles, policies
  4. 281Regions and zones
  5. 282VPC, subnets and routes
  6. 283Private endpoints: data that never touches the internet
  7. 284NAT gateways: the hidden cost
  8. 285Encryption with KMS
  9. 286How clouds charge for data
  10. 287The cloud CLI
  11. 288Teardown: leave nothing running
AWS 289-299
  1. 289S3 for data
  2. 290IAM roles for pipelines
  3. 291Glue Data Catalog
  4. 292Athena
  5. 293Glue jobs and EMR Serverless
  6. 294Lambda for small jobs
  7. 295Step Functions and MWAA
  8. 296Redshift Serverless
  9. 297Kinesis
  10. 298Lake Formation
  11. 299Capstone: the shop pipeline on AWS
Google Cloud 300-307
  1. 300Cloud Storage
  2. 301BigQuery
  3. 302BigQuery partitioning and clustering
  4. 303BigQuery cost: on-demand and slots
  5. 304Pub/Sub
  6. 305Dataflow
  7. 306Composer and Dataproc
  8. 307Capstone: the shop pipeline on GCP
Azure 308-314
  1. 308ADLS Gen2
  2. 309Data Factory
  3. 310Microsoft Fabric
  4. 311Event Hubs
  5. 312Entra ID and access
  6. 313Purview
  7. 314Capstone: the shop pipeline on Azure
Warehouses and lakehouses 315-323
  1. 315Snowflake architecture
  2. 316Snowflake warehouses and credits
  3. 317Time travel and zero-copy clones
  4. 318Databricks and the lakehouse
  5. 319Unity Catalog and open catalogs
  6. 320Trino: one SQL over many sources
  7. 321ClickHouse for fast analytics
  8. 322Choosing a warehouse
  9. 323Capstone: the same marts on Snowflake and Databricks

Part 6 Streaming and real time 324-347

Streaming 324-347
  1. 324When real time is worth it
  2. 325Events and logs
  3. 326Kafka: brokers, topics, partitions
  4. 327Producers
  5. 328Consumers and consumer groups
  6. 329Offsets and replays
  7. 330Consumer lag and how to watch it
  8. 331Rebalancing
  9. 332Retention and log compaction
  10. 333Delivery guarantees
  11. 334Schemas for events
  12. 335Schema compatibility modes
  13. 336Kafka Connect
  14. 337CDC with Debezium
  15. 338Windows: tumbling, sliding, session
  16. 339Event time and watermarks
  17. 340State in stream processing
  18. 341Spark Structured Streaming
  19. 342Apache Flink
  20. 343Exactly-once, end to end
  21. 344Streaming into a lakehouse
  22. 345Applying CDC changes to an Iceberg table
  23. 346Real-time OLAP
  24. 347Capstone: a live orders dashboard with CDC

Part 7 Data for ML and AI 348-362

ML and AI data pipelines 348-362
  1. 348What ML teams need from data engineers
  2. 349Features and feature pipelines
  3. 350Point-in-time correctness
  4. 351Feature stores with Feast
  5. 352Training datasets and versioning
  6. 353Embeddings for data engineers
  7. 354Vector search with pgvector
  8. 355Unstructured data: PDFs, images, audio
  9. 356A document pipeline for RAG
  10. 357LLMs inside pipelines: extract and classify
  11. 358Cost and caching of LLM calls in batch
  12. 359Text-to-SQL over your warehouse
  13. 360Data for AI agents: MCP over the warehouse
  14. 361Handing off to MLOps
  15. 362Capstone: support tickets to a RAG-ready index

Part 8 Senior practice 363-432

Performance and cost 363-370
  1. 363Profiling a slow query
  2. 364Partition pruning
  3. 365Clustering and sort keys
  4. 366Incremental everything
  5. 367FinOps for data
  6. 368Storage tiers and lifecycle rules
  7. 369Spot instances and autoscaling
  8. 370Capstone: cut a warehouse bill by 40%
Production incidents 371-386
  1. 371Reading an incident: alert, logs, timeline
  2. 372Duplicates after a backfill
  3. 373The daylight saving time bug
  4. 374Upstream renamed a column
  5. 375Silent nulls after a source change
  6. 376A skewed join kills Spark
  7. 377Small files slow every query
  8. 378SELECT * burns the monthly budget
  9. 379Kafka consumer lag grows overnight
  10. 380Late data missing from yesterday's report
  11. 381A DAG that ran twice
  12. 382A stale exchange rate cached for a week
  13. 383A deleted partition, restored with time travel
  14. 384A permissions change breaks the nightly load
  15. 385Dashboard numbers disagree with finance
  16. 386Capstone: an on-call week, 5 incidents in a row
Architecture 387-396
  1. 387Requirements before tools
  2. 388Lambda and Kappa architectures
  3. 389Medallion: bronze, silver, gold
  4. 390Data mesh and data products
  5. 391Build, buy, or open source
  6. 392Reverse ETL
  7. 393Multi-tenant data platforms
  8. 394Disaster recovery for data
  9. 395Migrating a legacy warehouse
  10. 396Architecture decision records
Data engineering in business 397-406
  1. 397Auditing a company's data
  2. 398Finding manual work to automate
  3. 399Defining metrics people agree on
  4. 400Estimating ROI and TCO
  5. 401Writing a proposal
  6. 402A 90-day data roadmap
  7. 403Working with stakeholders
  8. 404Replacing Excel processes
  9. 405Presenting to leadership
  10. 406Capstone: an audit and proposal for a real business
Senior engineering skills 407-412
  1. 407Writing a design doc
  2. 408Estimating data work
  3. 409Prioritizing tech debt
  4. 410Code review for data code
  5. 411Mentoring juniors
  6. 412Capstone: take over a legacy pipeline repo and fix it
Interviews and career 413-424
  1. 413A data engineer CV
  2. 414Portfolio from your capstones
  3. 415SQL interview problems
  4. 416Python coding interview for data roles
  5. 417Data modeling interview
  6. 418System design: e-commerce analytics
  7. 419System design: real-time fraud detection
  8. 420System design: clickstream at 1 billion events a day
  9. 421Behavioral interview
  10. 422Take-home assignments
  11. 423Salary negotiation
  12. 424Your first 90 days on the job
Final project 425-432
  1. 425Final project: brief and requirements
  2. 426Final project: architecture and ADRs
  3. 427Final project: ingestion and storage
  4. 428Final project: transformation and quality
  5. 429Final project: streaming path
  6. 430Final project: cloud deploy with Terraform
  7. 431Final project: observability and cost report
  8. 432Final project: write-up and demo for your portfolio

Tools you will use

Ideas first, tools as examples. A new tool changes the example, the step stays.

Basics
Terminal, git, Python, SQL, Postgres, Docker, MongoDB, Redis
Processing
pandas, Polars, DuckDB, Spark
Pipelines
dlt, Airbyte, Airflow, Dagster, dbt
Storage
Parquet, Avro, S3, MinIO, Delta Lake, Iceberg
Cloud
AWS, Google Cloud, Azure, Snowflake, Databricks, Terraform, Kubernetes
Streaming
Kafka, Debezium, Flink, Spark Streaming, ClickHouse
Quality and ops
Great Expectations, Soda, OpenLineage, data contracts
ML and AI
Feast, pgvector, RAG pipelines, LLMs in batch, MCP

Who it is for

Career changers

You have never written code. Part 1 starts with the terminal and ends with Python and SQL you use every day.

Analysts and developers

You know SQL or a language. Skim part 1, then go deep into pipelines, cloud and streaming.

Engineers aiming for senior

Performance, cost, architecture, governance, and how to pitch a data platform to a business.

FAQ

Do I need to know programming?

No. Steps 0-83 teach the terminal, git, Python and SQL from the first command.

What level will I reach?

Mid to senior. You build and run pipelines, choose between tools with numbers, fix production incidents, design a data platform and defend the design. Part 8 and the hard blocks of parts 1-6 cover the senior level: distributed systems, Kubernetes, cost, architecture.

Where does the experience come from?

From failures you fix yourself. Block 8.2 rebuilds 14 real classes of data incidents small: duplicates after a backfill, a daylight saving bug, a renamed upstream column, a skewed Spark join, a SELECT * that burns the budget. You get the alert and the logs, find the cause, fix it and write a postmortem. Step 412 hands you a legacy repo with no tests to bring into shape.

Which clouds?

AWS, Google Cloud and Azure, one block each, plus Snowflake and Databricks. You deploy the same shop pipeline to each, so you see what changes between clouds.

Who pays for the cloud?

You use your own accounts. The first cloud step sets a budget alert, and every cloud practice ends with a teardown. Plan for about $30 across the whole course. Before part 5, everything runs on your laptop in Docker.

What computer do I need?

macOS, Linux or Windows with WSL, 8 GB of memory and 30 GB of free disk. 16 GB helps in the Spark block.

Why so many steps?

Each step holds one idea, so steps stay short. 433 steps at 5-10 minutes come to 36-72 hours of reading and practice, plus capstones.

Is ML and AI included?

Yes. Part 7 covers feature pipelines, point-in-time correctness, vector search, document pipelines for RAG, and LLM calls inside batch jobs.

Interviews?

Block 8.6 covers SQL, Python and data modeling interviews, 3 system design problems, take-homes and salary negotiation.

Languages?

English and Russian, mirrored step by step.

What does it cost?

Steps 0-30 are free. After that, a yearly plan or lifetime access. The waitlist hears about prices first.

Get the first steps when they open

One email when steps 0-30 go live.