Queues & Async Processing
Moving work out of the request and between services.
Backend Engineer
Junior
Write correct code, ship small changes safely, ask good questions.
Core: start here
- Background JobWork done outside the request, like sending an email.
3 more junior concepts
- Scheduled JobsTasks that run on a schedule, like nightly cleanups.
- MessageA self-contained unit of data sent from one component to another through a broker.
- WebhookAn HTTP callback: a service calls your URL when something happens.
Mid-level
Own a feature end to end without hand-holding.
Core: start here
- Message QueueA buffer where producers leave messages for consumers to process later.
- Retry with Exponential BackoffRetrying with growing delays so you don't hammer a failing service.
- At-Least-Once DeliveryMessages may arrive more than once, so consumers must be idempotent.
- Dead Letter QueueWhere messages go after failing repeatedly.
- Idempotent ConsumerA consumer that handles duplicate messages safely.
16 more mid-level concepts
- IdempotenceDoing something twice has the same effect as doing it once.
- Publish-SubscribeSenders publish to topics; receivers subscribe without knowing each other.
- Real-Time Pub/SubBroadcasting events to subscribed clients through a broker.
- EventA record that something happened.
- Producer and ConsumerThe two sides of a queue.
- WorkerA process that pulls jobs from a queue and runs them.
- Job QueueA queue of background tasks with workers, like Sidekiq, Celery or BullMQ.
- Message BrokerMiddleware like RabbitMQ that routes messages.
- Queue Depth and Consumer LagHow far behind consumers are; a key health signal.
- Queue vs TopicPoint-to-point (each message to one consumer) vs pub/sub (each message to every subscriber).
- Delayed JobsJobs scheduled to run at a later time.
- Message Envelope and HeadersMetadata travelling with a message: IDs, timestamps, correlation and type.
- SubscriptionA consumer's registration to receive messages from a topic.
- Acknowledgement (ack and nack)A consumer confirming a message was processed, or asking for it to be redelivered.
- Competing ConsumersSeveral workers pulling from one queue to share the load.
- Visibility TimeoutHow long a received message stays hidden before it's delivered again.
Senior
Own a system, its failure modes, and its trade-offs.
Core: start here
- Saga PatternA chain of local transactions with compensating actions instead of a distributed transaction.
- Transactional OutboxSaving events in the same database transaction, then publishing them reliably.
22 more senior concepts
- Event StreamingProcessing a continuous flow of events.
- Stream ProcessingComputing over events as they arrive, as with Flink.
- Change Data CaptureStreaming database changes out as events.
- Consumer GroupConsumers sharing the work from a topic.
- Topics and PartitionsHow Kafka-style logs split and order messages.
- Apache KafkaA distributed, durable log for high-volume event streaming.
- Delivery GuaranteesAt-most-once, at-least-once and exactly-once delivery.
- Designing WebhooksSignatures, retries, ordering and idempotency for outgoing webhooks.
- Claim CheckPutting a large payload in storage and sending only a reference through the queue.
- Message TTL and ExpiryDropping or dead-lettering messages that are too old to matter.
- Queue-Based Load LevelingA queue absorbing traffic spikes so a service sees steady load.
- Exchanges and Routing KeysHow brokers like RabbitMQ route a message to the right queues.
- Fan-OutSending one message to many consumers.
- Workflow EngineDurable orchestration of long multi-step processes, like Temporal.
- Consumer OffsetA consumer's position in a log, which it commits as it goes.
- BackpressureA slow consumer signalling a fast producer to slow down.
- Message OrderingWhen and how messages arrive in order.
- Inbox PatternRecording received messages so duplicates can be ignored.
- JitterRandomizing retry delays so clients don't all retry in sync.
- Push vs Pull ConsumersThe broker delivering messages vs consumers fetching them at their own pace.
- Exactly-Once SemanticsThe hard, often misunderstood guarantee of processing each message once.
- Poison MessageA message that crashes every consumer that tries to process it.
Data Engineer
Junior
Build and fix pipelines from clear specs; write correct SQL.
- Scheduled JobsTasks that run on a schedule, like nightly cleanups.
- MessageA self-contained unit of data sent from one component to another through a broker.
- Background JobWork done outside the request, like sending an email.
- WebhookAn HTTP callback: a service calls your URL when something happens.
Mid-level
Own pipelines and models end to end, including their quality.
Core: start here
- Stream ProcessingComputing over events as they arrive, as with Flink.
- Change Data CaptureStreaming database changes out as events.
- Apache KafkaA distributed, durable log for high-volume event streaming.
22 more mid-level concepts
- Message QueueA buffer where producers leave messages for consumers to process later.
- IdempotenceDoing something twice has the same effect as doing it once.
- Retry with Exponential BackoffRetrying with growing delays so you don't hammer a failing service.
- EventA record that something happened.
- Producer and ConsumerThe two sides of a queue.
- WorkerA process that pulls jobs from a queue and runs them.
- Job QueueA queue of background tasks with workers, like Sidekiq, Celery or BullMQ.
- Message BrokerMiddleware like RabbitMQ that routes messages.
- Queue Depth and Consumer LagHow far behind consumers are; a key health signal.
- Queue vs TopicPoint-to-point (each message to one consumer) vs pub/sub (each message to every subscriber).
- Delayed JobsJobs scheduled to run at a later time.
- Message Envelope and HeadersMetadata travelling with a message: IDs, timestamps, correlation and type.
- SubscriptionA consumer's registration to receive messages from a topic.
- Acknowledgement (ack and nack)A consumer confirming a message was processed, or asking for it to be redelivered.
- At-Least-Once DeliveryMessages may arrive more than once, so consumers must be idempotent.
- Visibility TimeoutHow long a received message stays hidden before it's delivered again.
- Dead Letter QueueWhere messages go after failing repeatedly.
- Idempotent ConsumerA consumer that handles duplicate messages safely.
- Event StreamingProcessing a continuous flow of events.
- Consumer GroupConsumers sharing the work from a topic.
- Topics and PartitionsHow Kafka-style logs split and order messages.
- Delivery GuaranteesAt-most-once, at-least-once and exactly-once delivery.
Senior
Design the platform's storage, processing and modeling choices.
Core: start here
- Exactly-Once SemanticsThe hard, often misunderstood guarantee of processing each message once.
13 more senior concepts
- Saga PatternA chain of local transactions with compensating actions instead of a distributed transaction.
- Message TTL and ExpiryDropping or dead-lettering messages that are too old to matter.
- Transactional OutboxSaving events in the same database transaction, then publishing them reliably.
- Exchanges and Routing KeysHow brokers like RabbitMQ route a message to the right queues.
- Fan-OutSending one message to many consumers.
- Workflow EngineDurable orchestration of long multi-step processes, like Temporal.
- Consumer OffsetA consumer's position in a log, which it commits as it goes.
- BackpressureA slow consumer signalling a fast producer to slow down.
- Message OrderingWhen and how messages arrive in order.
- Inbox PatternRecording received messages so duplicates can be ignored.
- JitterRandomizing retry delays so clients don't all retry in sync.
- Push vs Pull ConsumersThe broker delivering messages vs consumers fetching them at their own pace.
- Poison MessageA message that crashes every consumer that tries to process it.
Frontend Engineer
Mid-level
Own a feature end to end without hand-holding.
- WebhookAn HTTP callback: a service calls your URL when something happens.
- IdempotenceDoing something twice has the same effect as doing it once.
- Publish-SubscribeSenders publish to topics; receivers subscribe without knowing each other.
- Real-Time Pub/SubBroadcasting events to subscribed clients through a broker.
- Retry with Exponential BackoffRetrying with growing delays so you don't hammer a failing service.
- EventA record that something happened.
Senior
Own an app's architecture, performance, and failure modes.
- BackpressureA slow consumer signalling a fast producer to slow down.
- JitterRandomizing retry delays so clients don't all retry in sync.