Orchestration & Pipelines
Scheduling and coordinating pipeline steps so they run reliably.
Backend Engineer track
Junior
Write correct code, ship small changes safely, ask good questions.
Nothing here yet.
Mid-level
Own a feature end to end without hand-holding.
- BackfillFilling in data for existing rows after a change.
- Data PipelineA sequence of steps that moves and transforms data.
Senior
Own a system, its failure modes, and its trade-offs.
- Pipeline Orchestration (Airflow)Scheduling and managing data pipelines.
Staff
Shape how many teams build, across systems.
Nothing here yet.
Principal
Set technical direction for the organization.
Nothing here yet.
Data Analyst track
Junior
Write correct SQL, build trusted dashboards, ask good questions.
Nothing here yet.
Mid-level
Own an analysis end to end, from vague question to recommendation.
- Data PipelineA sequence of steps that moves and transforms data.
- Pipeline DAGA pipeline expressed as tasks and their dependencies.
- Pipeline SchedulingRunning pipelines on a time schedule or when upstream data arrives.
- Task DependenciesWhich steps must finish before others can start.
Senior
Own experimentation and metrics design; call out bad numbers.
Nothing here yet.
Staff
Shape how the organization measures and decides.
Nothing here yet.
Principal
Set measurement strategy across the company.
Nothing here yet.
Data Engineer track
Junior
Build and fix pipelines from clear specs; write correct SQL.
Core: start here
- BackfillFilling in data for existing rows after a change.
- Data PipelineA sequence of steps that moves and transforms data.
- Pipeline DAGA pipeline expressed as tasks and their dependencies.
- Pipeline Orchestration (Airflow)Scheduling and managing data pipelines.
- Pipeline SchedulingRunning pipelines on a time schedule or when upstream data arrives.
- Retries and Failure HandlingRetrying transient failures and alerting on real ones.
1 more junior concepts
- Task DependenciesWhich steps must finish before others can start.
Mid-level
Own pipelines and models end to end, including their quality.
Core: start here
- Data-Aware / Event-Driven SchedulingTriggering work when data lands instead of at fixed times.
- Idempotent PipelinesPipelines that produce the same result when rerun, so retries and backfills are safe.
- Partitioned Pipeline RunsEach run processing one time slice, like one day of data.
- Reruns and Catch-UpRe-running past intervals after a failure or code change.
1 more mid-level concepts
- SensorsTasks that wait for a condition, like a file arriving.
Senior
Design the platform's storage, processing and modeling choices.
- Asset-Based OrchestrationOrchestrating the datasets you want to exist, not just the tasks.
- Choosing an Orchestrator (Airflow, Dagster, Prefect)Task-centric vs asset-centric orchestration and their trade-offs.
Staff
Shape how the whole organization produces and uses data.
Nothing here yet.
Principal
Set data strategy and architecture across the company.
Nothing here yet.