Ingestion
Getting data from source systems into your platform reliably.
Backend Engineer track
Junior
Write correct code, ship small changes safely, ask good questions.
- CSV Import / ExportMoving tabular data in and out, with all its edge cases.
Mid-level
Own a feature end to end without hand-holding.
- Bulk Loading (COPY)Loading large amounts of data much faster than row-by-row inserts.
- Messages vs StreamsTransient queued messages vs a durable, replayable log of events.
- UpsertInsert, or update if the row exists, in one statement.
Senior
Own a system, its failure modes, and its trade-offs.
- Change Data CaptureStreaming database changes out as events.
- Schema EvolutionChanging data formats so old and new readers and writers keep working.
Staff
Shape how many teams build, across systems.
Nothing here yet.
Principal
Set technical direction for the organization.
Nothing here yet.
Data Engineer track
Junior
Build and fix pipelines from clear specs; write correct SQL.
Core: start here
- Batch vs Streaming IngestionLoading data on a schedule vs continuously as it's produced.
- CSV Import / ExportMoving tabular data in and out, with all its edge cases.
- Data IngestionMoving data from source systems into storage you control.
- DeduplicationRemoving duplicate records that at-least-once delivery and retries create.
- File-Based IngestionLoading CSV, JSON or Parquet files dropped in storage or SFTP.
- Full vs Incremental LoadReloading everything each time vs only what changed.
- Ingesting from APIsPulling data from REST APIs with pagination, rate limits and retries.
- Landing Zone / Raw LayerWhere ingested data is stored untouched before any transformation.
- UpsertInsert, or update if the row exists, in one statement.
2 more junior concepts
- Bulk Loading (COPY)Loading large amounts of data much faster than row-by-row inserts.
- Managed Ingestion ConnectorsTools like Fivetran or Airbyte that sync SaaS and database sources for you.
Mid-level
Own pipelines and models end to end, including their quality.
Core: start here
- Append vs Upsert (MERGE)Adding new rows vs updating existing ones by key.
- Change Data CaptureStreaming database changes out as events.
- High-Water MarkRemembering the last loaded timestamp or ID to know where to resume.
- Push vs Pull IngestionThe source sending data to you vs you fetching it.
- Schema DriftA source adding, renaming or retyping fields without telling you.
- Schema EvolutionChanging data formats so old and new readers and writers keep working.
2 more mid-level concepts
- MERGE StatementSQL that inserts, updates or deletes rows in one pass based on a match.
- Messages vs StreamsTransient queued messages vs a durable, replayable log of events.
Senior
Design the platform's storage, processing and modeling choices.
- Log-Based CDCCapturing changes from a database's write-ahead log instead of querying tables.
Staff
Shape how the whole organization produces and uses data.
Nothing here yet.
Principal
Set data strategy and architecture across the company.
Nothing here yet.
Frontend Engineer track
Junior
Build UI that works, ship small changes safely, ask good questions.
- CSV Import / ExportMoving tabular data in and out, with all its edge cases.
Mid-level
Own a feature end to end without hand-holding.
Nothing here yet.
Senior
Own an app's architecture, performance, and failure modes.
- Schema EvolutionChanging data formats so old and new readers and writers keep working.
Staff
Shape how many teams build, across apps.
Nothing here yet.
Principal
Set technical direction for the organization.
Nothing here yet.