Data Engineering Foundations
What data engineering is, the lifecycle, and the shapes data comes in.
Backend Engineer track
Junior
Write correct code, ship small changes safely, ask good questions.
- Data Engineer vs Analyst vs Scientist vs ML EngineerWho builds pipelines, who answers questions, who models, and who ships models.
- DatasetA named collection of related data, like a table or a set of files.
- Structured, Semi-Structured and Unstructured DataTables, JSON-like records, and free-form text, images or audio.
Mid-level
Own a feature end to end without hand-holding.
- OLTP vs OLAPTransaction processing vs analytical queries.
Senior
Own a system, its failure modes, and its trade-offs.
Nothing here yet.
Staff
Shape how many teams build, across systems.
Nothing here yet.
Principal
Set technical direction for the organization.
Nothing here yet.
Data Analyst track
Junior
Write correct SQL, build trusted dashboards, ask good questions.
- Data Engineer vs Analyst vs Scientist vs ML EngineerWho builds pipelines, who answers questions, who models, and who ships models.
- Data ProductA dataset treated as a product, with an owner, documentation, quality guarantees and users.
- Data Science Hierarchy of NeedsCollect, move, store, clean, analyze, then learn: why reliable plumbing comes before AI.
- DatasetA named collection of related data, like a table or a set of files.
- Modern Data StackCloud warehouse, managed ingestion, SQL transformations and BI tools wired together.
- Source SystemsWhere data originates: application databases, APIs, logs, files, SaaS tools and devices.
- Structured, Semi-Structured and Unstructured DataTables, JSON-like records, and free-form text, images or audio.
Mid-level
Own an analysis end to end, from vague question to recommendation.
Nothing here yet.
Senior
Own experimentation and metrics design; call out bad numbers.
Nothing here yet.
Staff
Shape how the organization measures and decides.
- Data LiteracyAn organization's ability to read, question and use data.
Principal
Set measurement strategy across the company.
Nothing here yet.
Data Engineer track
Junior
Build and fix pipelines from clear specs; write correct SQL.
Core: start here
- Data Engineer vs Analyst vs Scientist vs ML EngineerWho builds pipelines, who answers questions, who models, and who ships models.
- Data Engineering LifecycleGeneration, ingestion, storage, transformation and serving, plus the undercurrents beneath them.
- OLTP vs OLAPTransaction processing vs analytical queries.
- Source SystemsWhere data originates: application databases, APIs, logs, files, SaaS tools and devices.
- Structured, Semi-Structured and Unstructured DataTables, JSON-like records, and free-form text, images or audio.
4 more junior concepts
- Data EngineeringBuilding the systems that collect, move, store and prepare data for analysis and ML.
- Data Science Hierarchy of NeedsCollect, move, store, clean, analyze, then learn: why reliable plumbing comes before AI.
- DatasetA named collection of related data, like a table or a set of files.
- Volume, Velocity, VarietyThe three dimensions that make data "big", and which one is actually your problem.
Mid-level
Own pipelines and models end to end, including their quality.
- Bounded vs Unbounded DataA finite dataset vs a stream that never ends.
- Modern Data StackCloud warehouse, managed ingestion, SQL transformations and BI tools wired together.
Senior
Design the platform's storage, processing and modeling choices.
Core: start here
- Data ProductA dataset treated as a product, with an owner, documentation, quality guarantees and users.
1 more senior concepts
- Data Engineering UndercurrentsSecurity, data management, DataOps, architecture, orchestration and software engineering across every stage.
Staff
Shape how the whole organization produces and uses data.
Nothing here yet.
Principal
Set data strategy and architecture across the company.
Nothing here yet.