Learn · Data Pipeline
Build a data pipeline
Real data is messy. A pipeline turns raw, unreliable input into clean, trustworthy tables through a sequence of stages. Toggle stages on and off, hit run, and watch exactly which rows get cleaned, which get quarantined, and which make it to the warehouse.
✓ Loaded to warehouse
⚠ Quarantine (rejected rows)
Why stage it?
Each stage does one job, so failures are isolated and observable. Validate rejects rows that break the rules (bad email, impossible age, negative amount, malformed date) rather than letting them poison downstream tables. Deduplicate drops repeated keys. Transform standardises and derives fields (here: lower-casing emails and tagging a spend tier). Rejected rows go to a quarantine so nothing is silently lost — you can inspect and fix them. Skip a stage and you can see precisely what it was protecting you from. This is the everyday shape of data engineering.