Case Study
Olist End-to-End Data Engineering Pipeline
A Medallion pipeline on GCP that processes 112K+ e-commerce transactions with PySpark, dbt, and Airflow, running entirely on the Always Free tier.
- Role
- Data Engineer (solo project)
- Timeline
- 2026
- Stack
- PythonSQLPySparkdbtApache AirflowGCP (BigQuery, Cloud Storage)
- Links
- transactions processed
- 112K+transactions processed
- less BigQuery data scanned
- ~90%less BigQuery data scanned
- cloud cost (Always Free tier)
- $0cloud cost (Always Free tier)

01 / Problem
The public Olist dataset (Brazilian e-commerce) looks clean but hides a real-world trap: customer_id changes on every purchase, so a naive model counts one returning customer as many different people, and every Customer Lifetime Value (LTV) figure downstream is wrong. I wanted a pipeline that is correct by construction, cheap to run, and observable end to end.
02 / Approach
Bronze: ingest raw files with PySpark using explicit StructType schemas as data contracts, failing fast on schema drift. Silver: clean and conform the data, and resolve every order back to customer_unique_id to fix "The Olist Trap". Gold: dbt models with an incremental load strategy, table partitioning, and clustering by customer_unique_id produce analytics-ready marts. Idempotent Apache Airflow DAGs orchestrate every run, and a custom JobLogger writes run metadata to BigQuery for end-to-end observability.
03 / Outcome
Partitioning, clustering, and incremental models cut BigQuery data scanned by about 90%. Gold-layer LTV marts are accurate after entity resolution, any DAG can be re-run safely with the same result, and the whole pipeline runs within GCP's Always Free tier.
Source code, dbt models, and Airflow DAGs are on GitHub.