Ingest
Python lands Garmin source payloads.
Ingestion jobs authenticate to Garmin Connect, download FIT and health payloads, and write source files into the raw S3 landing zone before analytical assumptions are applied.
Live analytics engineering project
Running Signals
A Garmin-to-signal-marts lakehouse project that turns personal running data into reliable outputs for frontend explorers and downstream ML use cases, with Python ingestion, Databricks storage, dbt modeling, SQL, and data quality checks kept explicit.
Gold mart status
Production-shaped demo
Checking modeled outputs
Loading the latest completed day from the gold signal marts.
Analysis pillars
Three signal families, no coaching claims.
Running Signals focuses on explainable training signals that are straightforward to validate from the modeled data.
Consistency
Measures whether training is happening regularly: active days, active weeks, streaks, and missed-week patterns.
Volume
Tracks accumulated training load through weekly and monthly distance, rolling totals, and long-run contribution.
Fitness
Keeps the claims descriptive: pace versus heart rate, recovery heart rate, resting heart rate, HRV, and sleep context.
Methodology
Medallion Architecture from Garmin payloads to analytical marts.
The pipeline keeps ingestion, bronze preservation, silver standardization, and gold signal marts separate so lineage, data quality, and analytical definitions remain explicit.
Bronze
Bronze preserves source-shaped evidence.
Databricks bronze tables keep raw Garmin payloads recoverable with payload lineage intact, so source data can be replayed when parsers, schemas, or signal definitions improve.
Silver
Silver standardizes reusable entities.
dbt silver models clean, type, deduplicate, and standardize run, record, date, week, and health-day entities into tested building blocks for downstream analytics.
Gold
Gold publishes analytical signal marts.
Gold models encode consistency, volume, route, and descriptive fitness definitions as marts for frontend explorers and downstream ML feature work. The website reads those marts through server-side Databricks SQL calls.
Stack
Tools chosen for a realistic analytics engineering workflow.
The project keeps the stack recognizable and explainable: Python for ingestion, S3 and Databricks for lakehouse storage, dbt and SQL for transformations, and Next.js for frontend explorers.
Python
Ingestion jobs and Garmin payload handling.
AWS S3
AWS S3
Raw FIT and health file landing zone.
Databricks
External volumes, bronze tables, and SQL access.
dbt
dbt
Silver and gold transformations, tests, and docs.
SQL
SQL
Readable analytical definitions for marts.
Next.js
Server-rendered frontend explorers.
Vercel
Hosted site surface for reviewers.
Garmin
Source system for activity and health data.
Explore pages