Stream Processing
Processing data continuously as it arrives.
Backend Engineer track
Junior
Write correct code, ship small changes safely, ask good questions.
- Real-Time vs Near-Real-TimeMilliseconds vs minutes, and which one the business actually needs.
Mid-level
Own a feature end to end without hand-holding.
Nothing here yet.
Senior
Own a system, its failure modes, and its trade-offs.
- Apache KafkaA distributed, durable log for high-volume event streaming.
- Stream ProcessingComputing over events as they arrive, as with Flink.
- Stream WindowingGrouping events by time windows: tumbling, sliding and session windows.
Staff
Shape how many teams build, across systems.
Nothing here yet.
Principal
Set technical direction for the organization.
Nothing here yet.
Data Engineer track
Junior
Build and fix pipelines from clear specs; write correct SQL.
- Real-Time vs Near-Real-TimeMilliseconds vs minutes, and which one the business actually needs.
Mid-level
Own pipelines and models end to end, including their quality.
Core: start here
- Apache KafkaA distributed, durable log for high-volume event streaming.
- Event Time vs Processing TimeWhen something happened vs when your system saw it.
- Late-Arriving DataEvents that show up after their window was already computed.
- Stream ProcessingComputing over events as they arrive, as with Flink.
2 more mid-level concepts
- Micro-BatchingProcessing a stream as a series of tiny batches.
- Stream WindowingGrouping events by time windows: tumbling, sliding and session windows.
Senior
Design the platform's storage, processing and modeling choices.
Core: start here
- Kappa ArchitectureEverything as a stream, with reprocessing by replaying the log.
- Lambda ArchitectureParallel batch and streaming paths merged at query time.
- Stateful Stream ProcessingKeeping running state like counts or sessions across events.
- Stream-Table DualityA table is the latest state of a stream; a stream is a table's changelog.
- WatermarkA stream's estimate of how far event time has progressed, used to close windows.
2 more senior concepts
- Apache FlinkA stream-first processing engine with strong event-time support.
- Stream JoinsJoining streams with each other or with tables, within time bounds.
Staff
Shape how the whole organization produces and uses data.
Nothing here yet.
Principal
Set data strategy and architecture across the company.
Nothing here yet.