Key Technologies
Flink
Learn about how you can use Flink to solve a large number of problems in System Design.
Many system design problems will require stream processing. You have a continuous flow of data and you want to process, transform, or analyze it in real-time.
The most basic example of this might be a service reading clicks from a Kafka topic, doing a trivial transformation (maybe reformatting the data for ingestion), and writing to a database. Easy.
Simple Kafka Stream Processing
But things can get substantially more complex from here. Imagine we want to keep track of the count of clicks per user in the last 5 minutes. Because of that 5 minute window, we've introduced
state
to our problem. Each message can't be processed independently because we need to remember the count from previous messages. While we can definitely do this in our service by just holding counters in memory, we've introduced a bunch of new problems.
-
As one example of a problem, if our new service crashes it will lose all of its state: basically the count for the preceding 5 minutes is gone. Our service could hypothetically recover from this by re-reading all the messages from the Kafka topic, but this is slow and expensive.
-
Or another problem is scaling. If we want to add a new service instance because we're handling more clicks, we need to figure out how to re-distribute the state from existing instances to the new ones. This is a complicated dance with a lot of failure scenarios!
-
Or what if events come in out of order or late! This is likely to happen and will impact the accuracy of our counts.
And things only get
harder
from here as we add complexity and more statefulness. Fortunately, engineers have been building these systems for decades and have come up with useful abstractions. Enter one of the most powerful stream processing engines:
Apache Flink
.
Flink is a framework for building stream processing applications that solves some of the tricky problems like those we've discussed above and more. While we could talk about Flink for days, in this deep dive we're going to focus on two different perspectives to understanding Flink:
-
First, we're going to talk about how Flink is used. There's a good chance you'll encounter a stream-oriented problem in your interview and Flink is a powerful, flexible tool for the job when it applies.
-
Secondly, you'll learn how Flink works , at a high-level, under the hood. Flink solves a lot of problems for you, but for interviews it's important you understand how it does that so you can answer deep-dive questions and support your design. We'll cover the important bits.
Let's get to it!