Apache Flink

A stream-first processing engine with strong event-time support.

Also known as: Flink, Apache Flink, Flink SQL

Apache Flink is a stream-processing engine built around unbounded streams, with strong support for event time and stateful processing. It treats a batch job as a stream with a known end, so the same operators work for both.

The classic mistake is choosing Flink for its reputation when a simpler engine fits. Flink’s strengths — exactly-once state, event-time windows, and low-latency processing with backpressure — come with real operational work: state backends, checkpoints, savepoints, and tuning parallelism. A team that only needs results every few minutes may be better served by micro-batching on an engine they already run.

Core ideas

Event time. Flink can group events by when they happened, not when they arrived, and uses watermarks to decide when a time window is complete (event time vs processing time).
State. Operators keep keyed state — running counts, sessions, dedup sets — and Flink manages it (stateful stream processing).
Fault tolerance. It takes periodic checkpoints of operator state; on failure it restarts from the last one and replays from a replayable source such as Kafka.
Exactly-once. With checkpointing plus sources and sinks that support it, Flink can give exactly-once results rather than at-least-once.
Stream–table duality. A table is the latest state of a stream and a stream is a table’s changelog, which is how Flink SQL joins and updates data (stream–table duality).

Flink offers a DataStream API and a Table API/SQL layer; which one you use depends on how much of the logic is plain SQL. Names and APIs evolve quickly, so check the version you’re on.

When not to use it

Flink is not the lightest thing to operate. If your latency budget is minutes rather than seconds, a scheduled batch job or a micro-batch engine may be simpler and cheaper. If your team already runs Apache Spark for batch, adding Flink means a second engine, a second set of skills, and a second thing to monitor. And if the data fits on one machine, single-node engines beat any cluster.

See stream processing for the broader picture.