Contents

Track:Backend EngineerData AnalystData EngineerFrontend Engineer

Data Engineer

Every concept a data engineer meets, from collecting events to serving trusted tables, in the order you actually need it.

1704 concepts, 227 of them core · jump toJuniorMid-levelSeniorStaffPrincipal

Junior

Build and fix pipelines from clear specs; write correct SQL.

Core: start here

Version Control (Git)

  • .gitignoreFiles Git should never track, like build output and .env.
  • AmendChanging the most recent commit's content or message.
  • BranchA movable pointer to a line of commits, for isolated work.
  • CloneCopying a remote repository, with its full history, to your machine.
  • CommitA snapshot of changes with a message explaining them.
  • Commit MessageA summary line plus a body explaining why a change was made.
  • DiffSeeing exactly what changed between commits, branches or your working copy.
  • GitThe distributed version control system almost everyone uses.
  • Log and HistoryBrowsing and filtering commit history.
  • Merge ConflictTwo branches changed the same lines and Git needs you to decide.
  • RebaseReplaying commits on top of another base for a linear history.

Pull Requests & Code Review

Debugging

Testing

  • API TestingCalling endpoints and checking status codes, bodies and side effects.
  • Integration TestA test of several components working together.
  • MockA test double that verifies how it was called.
  • Regression TestA test ensuring a fixed bug doesn't come back.
  • Test CoverageThe share of code executed by tests, and why 100% isn't the goal.
  • Test Runner / Test FrameworkTools like pytest, JUnit, Jest or Vitest that find, run and report tests.
  • Unit TestA fast test of one small piece of code in isolation.

Clean Code & Principles

Refactoring

  • RefactoringChanging code's structure without changing its behavior.

Developer Tooling

  • Development Environment SetupGetting a project running locally from a fresh machine.
  • Environment VariableA key-value setting passed to processes from their environment.
  • FormatterA tool that rewrites code to a consistent style automatically.
  • LinterA tool that flags likely bugs and style issues.
  • LockfileA record of exact dependency versions so every install is identical.
  • Package ManagerA tool for installing and versioning dependencies, like npm, pip or cargo.
  • Reading DocumentationGoing to the official docs first, and knowing how to navigate them.
  • Searching EffectivelyFinding answers fast in error messages, issues, docs and forums.
  • SSHSecurely logging into and running commands on remote machines.

AI-Assisted Development

Documentation & Writing

  • MarkdownThe plain-text formatting syntax used for READMEs, docs and PRs.
  • READMEThe front page of a project: what it is, how to run it, how to contribute.

Math for Programmers

Backend Basics

  • LoggingRecording what the application does so you can debug it later.

Relational Databases & SQL

Schema Migrations

  • BackfillFilling in data for existing rows after a change.

Files & Media

System Design Fundamentals

  • PartitioningDividing data into parts, within one machine or across many.

Events & Integration

Data Engineering Foundations

Collection & Instrumentation

Ingestion

Storage, Formats & Lakehouse

Data Modeling for Analytics

  • Date Dimension / Date SpineA table with one row per day and its calendar attributes.
  • Dimension TableA table describing the who, what, where of facts, like customers or products.
  • Fact TableA table of measurable events, like orders or page views, at a fixed grain.
  • GrainWhat one row of a table represents; the first decision in any model.
  • Star SchemaOne fact table joined directly to its dimension tables.

Batch & Distributed Processing

Stream Processing

Transformation & Analytics SQL

Orchestration & Pipelines

Data Quality & Observability

  • Data FreshnessHow recently a table was updated, and whether that's recent enough.
  • Data Quality DimensionsAccuracy, completeness, consistency, timeliness, validity and uniqueness.
  • Data TestsAutomated checks like not-null, unique and accepted values on tables.

Data Governance & Privacy

Serving & Analytics

  • Metric DefinitionsPrecisely defining what a number means, so two dashboards don't disagree.

DataOps & Platform

  • Query CostWhy scanning a whole table can cost real money in a cloud warehouse.

Working as a Data Engineer

Data Engineering Basics

Agile & Delivery Process

Estimation & Planning

  • Breaking Down TasksSplitting work into pieces small enough to finish and estimate.
  • EstimationPredicting how long work will take, and communicating the uncertainty.

Communication

Junior Habits & First Job

Career Growth

426 more junior concepts

Programming Basics

  • ASCIIThe original 7-bit character set that UTF-8 is backward compatible with.
  • Base CaseThe condition that stops recursion.
  • Block Scope vs Function ScopeWhether a variable lives until the end of its block or of the whole function.
  • BooleanA true/false value.
  • Boolean Flag ArgumentsPassing true/false to switch a function's behavior, and why it hurts readability.
  • Character EncodingHow characters map to bytes: ASCII, UTF-8, UTF-16.
  • Code CommentText for humans in code, best used to explain why rather than what.
  • Conditional (if/else)Running different code depending on a condition.
  • ConstantA name bound to a value that cannot be reassigned.
  • Control FlowThe order in which statements execute: branches, loops, returns.
  • Data TypeThe kind of value something is, which determines what operations are valid on it.
  • Dates and TimesTime zones, UTC, ISO 8601, and why date bugs are everywhere.
  • Default ParameterA parameter value used when the caller omits it.
  • Expression vs StatementAn expression produces a value; a statement performs an action.
  • Floating-Point NumberA binary approximation of real numbers, and why 0.1 + 0.2 != 0.3.
  • FunctionA named, reusable block of code that takes inputs and returns an output.
  • Global VariableA variable visible everywhere, and why it makes code hard to reason about.
  • IntegerA whole-number type, usually with a fixed size and range.
  • ISO 8601The standard text format for dates and times, like 2026-10-10T09:00:00Z.
  • IterationStepping through the items of a collection one at a time.
  • LoopRepeating code with for, while, do-while or for-each.
  • Magic NumberAn unexplained literal in code that should be a named constant.
  • Naming ThingsChoosing names that reveal intent; one of the hardest parts of programming.
  • Null / None / nilA value meaning "nothing here", and the source of countless crashes.
  • Off-by-One ErrorA loop or index that runs one step too many or too few.
  • OperatorA symbol that performs an operation on values, like +, == or &&.
  • Operator PrecedenceThe rules for which operators bind first in an expression.
  • Parameter vs ArgumentParameters are in the definition; arguments are the values passed at the call.
  • Primitive TypeA built-in basic type such as integer, float, boolean or character.
  • RecursionA function solving a problem by calling itself on smaller inputs.
  • Regular ExpressionA pattern language for matching and extracting text.
  • Return ValueThe result a function hands back to its caller.
  • ScopeThe region of code where a name is visible.
  • ShadowingAn inner variable hiding an outer one with the same name.
  • Short-Circuit Evaluation&& and || stopping as soon as the result is known.
  • Stack OverflowCrashing when the call stack runs out of space, usually from runaway recursion.
  • StringA sequence of characters, usually immutable.
  • String InterpolationEmbedding values directly inside a string literal.
  • Switch / MatchChoosing between many branches based on a value.
  • Ternary OperatorA compact inline if/else expression.
  • Time ZoneOffsets from UTC that change with location and daylight saving.
  • Truthy and FalsyNon-boolean values that act as true or false in conditions.
  • Type CoercionImplicit conversion by the language, like "5" + 1 in JavaScript.
  • Type ConversionTurning a value of one type into another, explicitly or implicitly.
  • UnicodeThe universal character set, and why string length is trickier than it looks.
  • Unix TimestampSeconds since 1970-01-01 UTC; a simple, unambiguous point in time.
  • UTF-8The dominant variable-length encoding of Unicode.
  • VariableA named reference to a value stored in memory.
  • Variadic FunctionA function that accepts any number of arguments.

Collections

Object-Oriented Programming

Functional Programming

Error Handling

Type Systems

Concurrency & Async

  • async / awaitSyntax for writing asynchronous code that reads like synchronous code.
  • CallbackA function passed in to be called when work finishes.
  • Callback HellDeeply nested callbacks that make async code unreadable.
  • Promise / FutureAn object representing a value that will be available later.

Version Control (Git)

  • Atomic CommitOne commit doing one logical thing, easy to review and revert.
  • BlameSeeing who last changed each line and in which commit.
  • Branch Naming ConventionsNames like feat/login-page that tell people what a branch is for.
  • Conventional CommitsA commit message convention like "feat:" and "fix:" that tools can parse.
  • Detached HEADChecking out a commit directly instead of a branch.
  • Force PushOverwriting remote history; use --force-with-lease and never on shared branches.
  • ForkYour own copy of someone else's repository, for contributing via pull requests.
  • Git ConfigYour name, email, aliases and defaults, at global or repository level.
  • HEADGit's pointer to the commit you currently have checked out.
  • MergeCombining the histories of two branches.
  • Push, Pull, FetchSending commits, downloading and merging, or only downloading.
  • RemoteAnother copy of the repository, usually "origin" on GitHub or GitLab.
  • RepositoryA project's files plus their complete history.
  • ResetMoving a branch pointer back, in soft, mixed or hard mode.
  • RevertCreating a new commit that undoes an earlier one.
  • SquashCombining several commits into one.
  • Staging AreaWhere you choose which changes go into the next commit.
  • StashTemporarily shelving uncommitted changes.
  • TagA named pointer to a commit, usually marking a release.
  • Version ControlTracking every change to code so you can review, revert and collaborate.

Branching & Releases

  • ChangelogA human-readable list of notable changes per release.
  • Feature BranchA short-lived branch for one change.
  • GitHub FlowBranch from main, open a PR, merge back to main.
  • HotfixAn urgent fix shipped outside the normal release cycle.
  • Protected BranchA branch that requires reviews and passing checks before merging.
  • Semantic VersioningMAJOR.MINOR.PATCH, and what each number promises about compatibility.

Pull Requests & Code Review

Debugging

Testing

Clean Code & Principles

Refactoring

Design Patterns

Developer Tooling

AI-Assisted Development

  • AI Coding AssistantTools like Copilot, Cursor or Claude Code that suggest and write code.
  • HallucinationAn AI confidently producing APIs, facts or code that don't exist.
  • Prompting for CodeGiving an AI enough context and constraints to produce useful code.
  • Vibe CodingAccepting AI code without reading it: fine for prototypes, risky in production.

Documentation & Writing

Data Structures

  • Binary TreeA tree where each node has at most two children.
  • Data StructureA way of organizing data so certain operations are efficient.
  • Dynamic ArrayAn array that grows by reallocating, with amortized O(1) appends.
  • Hash FunctionA function mapping data of any size to a fixed-size value.
  • Hash TableKey-value storage with average O(1) lookup via a hash function.
  • Linked ListNodes pointing to the next node: fast inserts, slow random access.
  • QueueA first-in, first-out collection.
  • StackA last-in, first-out collection.
  • TreeA hierarchy of nodes with one root and no cycles.

Algorithms

Math for Programmers

Computer Architecture

  • CPUThe processor that executes instructions.
  • RAMFast, temporary memory for running programs.
  • SSD vs HDDStorage types and their very different speeds.

Operating Systems

Networking Fundamentals

  • DNSThe internet's phone book, turning names into IP addresses.
  • Domain NameA human-readable address like example.com.
  • IP AddressA numeric address identifying a device on a network.
  • Latency vs BandwidthHow long data takes to arrive vs how much can flow at once.
  • localhost / 127.0.0.1The address that always means "this machine".
  • ping and tracerouteChecking reachability and the path packets take.
  • PortA number identifying a specific service on a host.

HTTP

TLS & Certificates

  • Let's EncryptA free, automated certificate authority.
  • TLS CertificateA file proving a server controls a domain, signed by a trusted authority.

API Styles & Formats

  • APIAn interface that lets programs talk to each other.
  • EndpointA specific URL and method an API exposes, like POST /orders.
  • JSONThe text data format most APIs speak.
  • Request and Response BodyThe payload sent with a request or returned in a response, usually JSON.
  • Resource NamingPlural nouns, nested paths and consistent URL design.
  • RESTAn API style built on resources, URLs and HTTP methods.
  • SerializationConverting objects to bytes or text for storage or transfer.
  • YAMLA human-friendly data format common in configuration files.

Backend Basics

API Design

Relational Databases & SQL

Indexing & Query Performance

  • Database IndexA lookup structure that speeds up queries at the cost of slower writes.
  • N+1 Query ProblemOne query for a list, then one more query for every item in it.

Transactions & Concurrency Control

  • TransactionA group of operations that succeed or fail together.

NoSQL & Other Data Stores

  • Document DatabaseStoring JSON-like documents, as in MongoDB.
  • Key-Value StoreStoring values by key, as in Redis or DynamoDB.
  • NoSQLDatabases not built on the relational table model.
  • Object StorageStoring files as objects, as in S3.
  • RedisAn in-memory data store used for caching, queues and more.

Schema Migrations

  • Migration ToolsTools like Flyway, Alembic, Prisma Migrate and Rails migrations.
  • Schema MigrationA versioned script that changes the database schema.
  • Seed DataInitial data loaded for development or tests.

Database Operations

Caching

Queues & Async Processing

  • Background JobWork done outside the request, like sending an email.
  • MessageA self-contained unit of data sent from one component to another through a broker.

Architecture Styles

  • MonolithOne deployable application containing all the features.

System Design Fundamentals

Reliability & Resilience

  • BackupsCopies of data for restoring, and why untested backups don't count.

Performance & Scalability

  • LatencyThe time a single operation takes.

Cryptography Basics

Secure Development

Privacy & Compliance

  • PIIPersonally identifiable information that needs special care.

Linux & Servers

Containers

Cloud Computing

CI/CD & Deployment

Observability

Incidents & SRE

  • IncidentAn unplanned event that disrupts or degrades service.

Working in Production

Data Engineering Foundations

  • Data EngineeringBuilding the systems that collect, move, store and prepare data for analysis and ML.
  • Data Science Hierarchy of NeedsCollect, move, store, clean, analyze, then learn: why reliable plumbing comes before AI.
  • DatasetA named collection of related data, like a table or a set of files.
  • Volume, Velocity, VarietyThe three dimensions that make data "big", and which one is actually your problem.

Ingestion

Storage, Formats & Lakehouse

  • JSON LinesOne JSON object per line, easy to stream and append.

Batch & Distributed Processing

  • Notebooks (Jupyter)Interactive documents mixing code, output and notes.
  • pandasPython's standard DataFrame library for data analysis.

Transformation & Analytics SQL

Orchestration & Pipelines

Serving & Analytics

LLM & AI Engineering

Agile & Delivery Process

  • AgileDelivering in small increments and adapting to feedback.
  • BacklogThe prioritized list of work that hasn't started yet.
  • Backlog RefinementClarifying and sizing upcoming work.
  • Daily StandupA short daily sync on progress and blockers.
  • EpicA large body of work split into smaller stories.
  • KanbanVisualizing work on a board and limiting work in progress.
  • RetrospectiveA regular meeting to reflect on how the team works and improve it.
  • ScrumAn agile framework with sprints, roles and ceremonies.
  • Software Development LifecycleThe stages from idea to production to maintenance.
  • SprintA fixed period, often two weeks, to deliver planned work.
  • Sprint PlanningChoosing the work for the next sprint.
  • Sprint Review / DemoShowing what was built to stakeholders.
  • User Story"As a user, I want… so that…": a requirement from the user's point of view.
  • WaterfallSequential phases from requirements to release.

Estimation & Planning

Communication

Product Thinking

Junior Habits & First Job

Career Growth

Mid-level

Own pipelines and models end to end, including their quality.

Core: start here

API Styles & Formats

  • Schema EvolutionChanging data formats so old and new readers and writers keep working.

Relational Databases & SQL

Queues & Async Processing

Distributed Systems

Privacy & Compliance

Ingestion

Storage, Formats & Lakehouse

Data Modeling for Analytics

Batch & Distributed Processing

Stream Processing

Transformation & Analytics SQL

  • Incremental ModelsProcessing only new or changed rows instead of rebuilding a whole table.

Orchestration & Pipelines

Data Quality & Observability

  • Data IncidentA data quality failure that reaches users, and how to respond.
  • Data ObservabilityMonitoring freshness, volume, schema and distributions to catch silent breakage.
  • Data ReconciliationChecking that totals match between source and destination.

Data Governance & Privacy

  • Data CatalogA searchable inventory of datasets, their meaning and their owners.
  • Data LineageWhere data came from and everything it flows into.
  • Data MaskingHiding sensitive values while keeping data usable.

Serving & Analytics

DataOps & Platform

Working as a Data Engineer

599 more mid-level concepts

Programming Basics

  • Bitwise OperationOperating on individual bits: AND, OR, XOR and shifts.
  • Integer OverflowA value exceeding its type's range and wrapping around or failing.
  • Pattern MatchingBranching on the shape of data and destructuring it at the same time.
  • Tail RecursionRecursion where the call is the last action, which some languages turn into a loop.

Collections

  • GeneratorA function that lazily yields a sequence of values.
  • IteratorAn object that yields items one at a time from a sequence.
  • Lazy EvaluationComputing values only when they're actually needed.

Object-Oriented Programming

Functional Programming

Error Handling

Type Systems

  • Discriminated UnionA union whose members are told apart by a tag field.
  • Duck TypingIf it has the right methods, it's the right type.
  • Finite State MachineA model with a fixed set of states and allowed transitions, e.g. an order going from paid to shipped.
  • GenericsCode that works over many types while keeping type safety.
  • Runtime ValidationChecking that untrusted data matches a type at runtime, e.g. with Zod or Pydantic.
  • Type InferenceThe compiler working out types without annotations.
  • Type NarrowingRefining a broad type to a specific one through checks.
  • Union TypeA value that can be one of several types.

Concurrency & Async

Version Control (Git)

  • BisectBinary-searching history to find the commit that introduced a bug.
  • Cherry-PickApplying one specific commit onto another branch.
  • Fast-Forward vs Merge CommitMoving a branch pointer ahead vs recording a merge commit.
  • Git HooksScripts that run on Git events, like linting before a commit.
  • Git LFSStoring large binary files outside normal Git history.
  • Interactive RebaseRewriting commits: squash, reorder, edit, drop.
  • Merge vs RebasePreserving history as it happened vs rewriting it to be linear.
  • MonorepoMany projects living in one repository.
  • ReflogGit's log of where HEAD has been; your undo history after disasters.
  • Removing Secrets from Git HistoryPurging a committed secret with git filter-repo, and why you must rotate it anyway.
  • Rewriting Shared HistoryWhy rebasing or force-pushing shared branches breaks teammates.
  • Signed CommitsCryptographically proving who authored a commit.
  • SubmoduleA repository embedded inside another at a fixed commit.

Branching & Releases

Pull Requests & Code Review

Debugging

Testing

Clean Code & Principles

Refactoring

Design Patterns

  • Anti-PatternA common solution that looks right but causes problems.
  • Dependency InjectionPassing dependencies in instead of creating them inside.
  • Design PatternA named, reusable solution to a recurring design problem.
  • Factory MethodLetting subclasses or functions decide which class to instantiate.
  • SingletonEnsuring a class has exactly one instance; often an anti-pattern.
  • StrategySwapping algorithms behind a common interface.

Developer Tooling

AI-Assisted Development

Documentation & Writing

  • API DocumentationReference docs for an API's endpoints, parameters and errors.
  • Architecture DiagramA box-and-arrow picture of a system's components and data flow.
  • Class DiagramA diagram of classes, their fields and relationships.
  • Diagrams as CodeWriting diagrams as text, with tools like Mermaid or PlantUML.
  • Docs as CodeKeeping docs in the repository and reviewing them like code.
  • Onboarding DocumentationDocs that get a new teammate productive quickly.
  • RunbookStep-by-step instructions for operating or fixing a system.
  • Sequence DiagramA diagram of messages passed between components over time.
  • State DiagramA diagram of states and the transitions between them.
  • UMLA standard visual language for software diagrams.

Data Structures

  • Adjacency List vs MatrixTwo ways to store a graph's edges.
  • B-TreeA wide, shallow tree optimized for disks; how database indexes work.
  • Binary Search TreeA binary tree ordered so each lookup can halve the search.
  • DequeA double-ended queue that adds and removes at both ends.
  • Directed Acyclic Graph (DAG)A graph with directed edges and no cycles, as in build systems and pipelines.
  • Doubly Linked ListA linked list with pointers in both directions.
  • GraphNodes connected by edges; models networks, dependencies and maps.
  • Hash CollisionTwo keys hashing to the same slot, and how tables handle it.
  • HeapA tree that keeps the min or max at the root; backs priority queues.
  • LRU CacheA cache evicting the least recently used item, built from a hash map and a linked list.
  • Priority QueueA queue that always returns the highest-priority item first.
  • Tree TraversalVisiting tree nodes in pre-order, in-order, post-order or level order.
  • TrieA tree of characters for fast prefix lookups.

Algorithms

Math for Programmers

  • CombinatoricsCounting arrangements and combinations, e.g. how many IDs a format allows.
  • Discrete MathLogic, sets, graphs and combinatorics: the math of computing.
  • Probability BasicsReasoning about chance, for sampling, A/B tests and failure rates.
  • Set TheoryUnions, intersections and differences; the basis of SQL.

Computer Architecture

Operating Systems

Networking Fundamentals

HTTP

TLS & Certificates

API Styles & Formats

  • API Client / SDKA library wrapping an API so callers don't hand-write HTTP.
  • AvroA binary serialization format whose schemas are designed to evolve.
  • GraphQLA query language that lets clients ask for exactly the data they need.
  • gRPCA fast RPC framework built on HTTP/2 and Protocol Buffers.
  • JSON SchemaA vocabulary for validating the structure of JSON.
  • OpenAPIA standard format for describing REST APIs.
  • Protocol BuffersA compact, schema-based binary serialization format.
  • Resource ModelingDeciding what your API's resources are and how they relate.
  • REST ConstraintsStatelessness, uniform interface, cacheability and REST's other rules.
  • RPCCalling a function on another machine as if it were local.
  • XMLA verbose markup format still common in enterprise and legacy systems.

UI/UX for Engineers

  • A/B TestingComparing two variants with real users to see which performs better.
  • Product AnalyticsTracking how people use features to inform decisions.

Backend Basics

API Design

Relational Databases & SQL

Indexing & Query Performance

Transactions & Concurrency Control

NoSQL & Other Data Stores

Schema Migrations

Database Operations

Caching

Queues & Async Processing

Files & Media

  • Data ExportLetting users download their data, often as a background job.
  • Presigned URLA temporary URL letting clients upload or download directly from storage.
  • Streaming Large FilesProcessing files in chunks instead of loading them into memory.

Product Building Blocks

Architecture Styles

System Design Fundamentals

Distributed Systems

  • BASEBasically Available, Soft state, Eventual consistency: the counterpart to ACID.
  • CAP TheoremDuring a network partition, you must choose consistency or availability.
  • Eventual ConsistencyReplicas converge once updates stop, but reads may be stale in the meantime.

Reliability & Resilience

Performance & Scalability

Events & Integration

Cryptography Basics

Secure Development

  • Attack SurfaceEvery point where an attacker could try to get in.
  • Audit LoggingRecording who did what and when, for accountability.
  • CVEA public identifier for a known vulnerability.
  • Defense in DepthSeveral layers of security, so one failure isn't fatal.
  • Dependency ScanningFinding known vulnerabilities in your dependencies.
  • SASTStatic analysis that looks for security bugs in source code.
  • Secrets ManagementKeeping API keys and passwords in a vault, not in code.
  • TyposquattingMalicious packages named like popular ones.
  • Zero-DayA vulnerability exploited before a fix exists.

Privacy & Compliance

Linux & Servers

Containers

Kubernetes & Orchestration

  • ConfigMap and SecretConfiguration and sensitive values injected into pods.
  • Container OrchestrationAutomating the deployment, scaling and healing of containers.
  • CrashLoopBackOffA pod that keeps crashing and restarting.
  • DeploymentDeclares how many replicas of a pod should run and how to update them.
  • HelmA package manager for Kubernetes.
  • IngressRouting external HTTP traffic into the cluster.
  • Job and CronJobRunning one-off and scheduled tasks in Kubernetes.
  • kubectlThe command-line tool for Kubernetes.
  • KubernetesThe dominant container orchestration platform.
  • Liveness and Readiness ProbesHow Kubernetes checks whether a container is healthy.
  • NamespaceDividing cluster resources between teams or apps.
  • NodeA machine in the cluster that runs pods.
  • PodThe smallest deployable unit in Kubernetes: one or more containers.
  • ReplicaSetKeeps a set number of identical pods running.
  • ServiceA stable network address for a set of pods.

Cloud Computing

Infrastructure as Code

CI/CD & Deployment

Observability

Incidents & SRE

Working in Production

Data Engineering Foundations

Collection & Instrumentation

Ingestion

  • MERGE StatementSQL that inserts, updates or deletes rows in one pass based on a match.
  • Messages vs StreamsTransient queued messages vs a durable, replayable log of events.

Storage, Formats & Lakehouse

Data Modeling for Analytics

Batch & Distributed Processing

Stream Processing

Transformation & Analytics SQL

Orchestration & Pipelines

  • SensorsTasks that wait for a condition, like a file arriving.

Data Quality & Observability

Data Governance & Privacy

Serving & Analytics

Working as a Data Engineer

Data Engineering Basics

  • Reverse ETLSyncing warehouse data back into operational tools.

Machine Learning Basics

LLM & AI Engineering

Agile & Delivery Process

Estimation & Planning

Communication

Product Thinking

Career Growth

Senior

Design the platform's storage, processing and modeling choices.

Core: start here

Documentation & Writing

Queues & Async Processing

Architecture Styles

System Design Fundamentals

Data Engineering Foundations

  • Data ProductA dataset treated as a product, with an owner, documentation, quality guarantees and users.

Ingestion

  • Log-Based CDCCapturing changes from a database's write-ahead log instead of querying tables.

Storage, Formats & Lakehouse

Data Modeling for Analytics

  • Conformed DimensionOne shared dimension used by many fact tables so numbers line up.
  • Data VaultModeling with hubs, links and satellites for auditable, change-friendly history.
  • Inmon vs KimballA normalized enterprise warehouse first vs dimensional marts first.

Batch & Distributed Processing

  • Broadcast JoinSending a small table to every worker to avoid shuffling the big one.
  • Data SkewA few keys holding most of the data, so one task runs forever.

Stream Processing

Orchestration & Pipelines

Data Quality & Observability

Data Governance & Privacy

Serving & Analytics

  • Feature StoreA shared store of ML features, consistent between training and serving.

DataOps & Platform

Technical Leadership

  • MentoringHelping less experienced engineers grow.
383 more senior concepts

Functional Programming

  • CurryingTurning a multi-argument function into a chain of single-argument ones.
  • MonadA pattern for chaining computations that carry context, like Option or Promise.
  • Referential TransparencyAn expression can be replaced by its value without changing behavior.

Error Handling

Type Systems

Concurrency & Async

  • Actor ModelConcurrency through isolated actors that communicate by messages.
  • BackpressureA slow consumer signalling a fast producer to slow down.
  • ChannelA typed pipe for passing values between concurrent tasks.
  • Green Threads / GoroutinesLightweight threads scheduled by the runtime rather than the OS.
  • LivelockTasks keep reacting to each other without making progress.
  • SemaphoreA counter limiting how many tasks can use a resource at once.
  • StarvationA task never gets the resources it needs to run.

Version Control (Git)

  • Git InternalsBlobs, trees, commits and refs: what Git actually stores.
  • Monorepo vs PolyrepoOne repository for everything vs one per project.
  • Patch FilesSharing changes as diff files instead of branches.
  • WorktreeSeveral working directories checked out from one repository.

Pull Requests & Code Review

  • Stacked PRsA chain of dependent pull requests, each reviewable on its own.

Debugging

  • Core DumpA snapshot of a crashed process's memory for later analysis.
  • Debugging in ProductionFinding issues with logs, traces and metrics when you can't attach a debugger.
  • Flame GraphA visualization of where a program spends its time.

Testing

  • Contract TestingVerifying that services agree on the API contract between them.
  • FuzzingFeeding random inputs to find crashes and vulnerabilities.
  • Mutation TestingChanging code on purpose to check that the tests notice.
  • Property-Based TestingGenerating many inputs to check that properties always hold.
  • Soak TestRunning under load for hours to find leaks and slow degradation.
  • Stress TestingPushing beyond capacity to find the breaking point.
  • Testing in ProductionValidating safely with real traffic using flags, canaries and monitoring.

Clean Code & Principles

Refactoring

Documentation & Writing

  • C4 ModelDiagramming software at four zoom levels: context, containers, components, code.
  • DiátaxisOrganizing docs into tutorials, how-to guides, reference and explanation.

Data Structures

  • Balanced TreeTrees like AVL or red-black that stay shallow for guaranteed O(log n).
  • Bloom FilterA compact structure answering "definitely not" or "probably yes" for set membership.
  • HyperLogLogEstimating the number of distinct items with tiny memory.
  • K-D TreeA tree for searching points in multi-dimensional space.
  • Merkle TreeA tree of hashes that verifies large data efficiently.
  • Persistent Data StructureAn immutable structure that shares unchanged parts between versions.
  • Ring BufferA fixed-size buffer that wraps around.
  • Segment TreeA tree for fast range queries over arrays.
  • Skip ListA layered linked list with O(log n) search.
  • Union-FindTracking which elements belong to the same group.

Algorithms

Math for Programmers

  • Amdahl's LawThe speedup from parallelism is limited by the part that stays sequential.
  • Linear Algebra BasicsVectors and matrices, the foundation of graphics and ML.
  • Little's LawItems in a system = arrival rate × time each spends in it.

Operating Systems

Networking Fundamentals

HTTP

TLS & Certificates

  • Certificate ChainLeaf, intermediate and root certificates linking to a trusted root.
  • TLS HandshakeHow client and server agree on keys before sending data.

API Styles & Formats

Relational Databases & SQL

Indexing & Query Performance

Transactions & Concurrency Control

Database Internals

NoSQL & Other Data Stores

Schema Migrations

Database Operations

Caching

Queues & Async Processing

Product Building Blocks

Architecture Styles

System Design Fundamentals

Distributed Systems

Reliability & Resilience

Performance & Scalability

Events & Integration

Cryptography Basics

Secure Development

  • Bug BountyPaying outside researchers to report vulnerabilities.
  • Container SecurityMinimal images, non-root users and image scanning.
  • CVSSA score rating how severe a vulnerability is.
  • DASTTesting a running app for vulnerabilities from the outside.
  • Penetration TestingAuthorized simulated attacks to find vulnerabilities.
  • Responsible DisclosureReporting vulnerabilities privately before going public.
  • SBOMA software bill of materials listing every component.
  • Secret RotationRegularly replacing credentials.
  • Security ReviewReviewing a design or change specifically for security risks.
  • Shift-Left SecurityFinding security issues early in development.
  • Software Supply Chain SecurityProtecting against compromised dependencies and build pipelines.
  • STRIDEA way to categorize threats: spoofing, tampering, repudiation, disclosure, denial of service, elevation.
  • Threat ModelingSystematically asking what could go wrong and how to prevent it.

Privacy & Compliance

  • CCPACalifornia's consumer privacy law.
  • HIPAAUS rules for protecting health information.
  • PCI DSSRules for handling payment card data.
  • Privacy by DesignBuilding privacy in from the start.
  • Right to ErasureDeleting a user's data on request, across every system.

Linux & Servers

Containers

  • Minimal Base ImagesAlpine, distroless and scratch images for a smaller attack surface.
  • OCIThe open standards for container images and runtimes.
  • Union FilesystemStacking read-only image layers with a writable layer on top.

Kubernetes & Orchestration

Cloud Computing

Infrastructure as Code

CI/CD & Deployment

Observability

Incidents & SRE

Working in Production

  • Break-Glass AccessEmergency elevated access that is logged, time-limited and reviewed afterwards.

Data Engineering Foundations

Collection & Instrumentation

  • Identity ResolutionStitching anonymous visitors, devices and accounts into one person.
  • IoT and Sensor DataHigh-frequency readings from devices, with gaps, clock drift and duplicates.
  • Telemetry PipelineThe path from emitted logs, metrics and events to where they're stored and queried.

Storage, Formats & Lakehouse

  • Apache ArrowAn in-memory columnar format for moving data between tools without conversion.
  • BucketingHashing rows into a fixed number of files by key to speed joins.
  • Data SharingGiving another team or company live access to data without copying it.
  • ORCA columnar format common in the Hadoop ecosystem.
  • Zero-Copy CloneCopying a table instantly by sharing its underlying files.

Data Modeling for Analytics

Batch & Distributed Processing

Stream Processing

  • Apache FlinkA stream-first processing engine with strong event-time support.
  • Stream JoinsJoining streams with each other or with tables, within time bounds.

Transformation & Analytics SQL

Data Quality & Observability

Data Governance & Privacy

  • TokenizationReplacing sensitive values with tokens that map back only through a secure vault.

Serving & Analytics

  • Embedded AnalyticsShowing charts and reports inside your product to customers.
  • OLAP CubePre-aggregated data for fast slicing by dimensions.

Machine Learning Basics

LLM & AI Engineering

  • Choosing a ModelTrading off quality, speed and cost across model sizes and providers.
  • ChunkingSplitting documents into pieces for embedding and retrieval.
  • EvalsSystematically measuring the quality of LLM output.
  • Fine-TuningFurther training a model on your own examples.
  • GuardrailsChecks on model inputs and outputs for safety and correctness.
  • LLM Cost and LatencyManaging tokens, model choice and caching.
  • LLM-as-JudgeUsing one model to grade another model's output.
  • Prompt CachingReusing processed prompt prefixes to save cost and time.
  • RerankingRe-ordering retrieved results with a stronger model.

Agile & Delivery Process

Estimation & Planning

Communication

Product Thinking

Career Growth

Technical Leadership

Staff

Shape how the whole organization produces and uses data.

Clean Code & Principles

  • Conway's LawSystems mirror the communication structure of the organizations that build them.

Documentation & Writing

  • RFCA request for comments: proposing a significant change for wide review.

System Design Fundamentals

Reliability & Resilience

Performance & Scalability

Events & Integration

Secure Development

Privacy & Compliance

Kubernetes & Orchestration

Cloud Computing

  • Multi-CloudUsing several cloud providers, and whether it's worth it.

CI/CD & Deployment

  • DORA MetricsFour delivery metrics: deploy frequency, lead time, change failure rate and recovery time.

Observability

  • Wide EventsRich, high-cardinality events instead of pre-aggregated metrics.

Incidents & SRE

Data Modeling for Analytics

  • Bus MatrixA planning grid of business processes and the dimensions they share.

Data Governance & Privacy

  • Data FabricAn integration layer that connects data across systems using metadata.
  • Data MeshDomain teams owning their data as products, on a self-serve platform.
  • EU AI ActEU rules on AI systems and the data used to build them.

DataOps & Platform

Communication

Product Thinking

Career Growth

Technical Leadership

Principal

Set data strategy and architecture across the company.

Concurrency & Async

Data Structures

  • Suffix ArrayA sorted array of a string's suffixes for fast substring search.

Algorithms

  • Maximum FlowFinding the most that can flow through a network (Ford-Fulkerson).

Database Internals

Distributed Systems

  • Byzantine FaultNodes that behave arbitrarily or maliciously.
  • Fencing TokenA counter that stops stale lock holders from writing.
  • Hedged RequestsSending a duplicate request when the first is slow, to cut tail latency.
  • LinearizabilityOperations appear to happen instantly, in real-time order.
  • Logical ClocksLamport and vector clocks for ordering events without real time.
  • PaxosThe classic, notoriously hard-to-understand consensus algorithm.
  • Total Order BroadcastDelivering the same messages in the same order to every node.

Performance & Scalability

Linux & Servers

  • eBPFRunning safe, sandboxed programs inside the Linux kernel for tracing and networking.

Career Growth

Technical Leadership