tillrohrmann commented on a change in pull request #427:
URL: https://github.com/apache/flink-web/pull/427#discussion_r604266500
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
Review comment:
Font is not sharp on my machine.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
Review comment:
```suggestion
Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale** a Flink job. You can
stop-and-restore a job with a different parallelism. If your job was running
with a parallelism of p=100, and your load increased, you can restart it with
p=200 to cope with the additional data.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
Review comment:
Meta comment: What helps when reviewing blog posts via Github is to put
every sentence in a separate line. That way one can easily make suggestions and
it is much more review friendly.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
Review comment:
> Reactive Mode is a JobManager mode where your job is always using all
TaskManagers available.
Maybe say that the new Reactive Mode will try to use all available
resources/slots or so.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
Review comment:
```suggestion
cp ./examples/streaming/TopSpeedWindowing.jar usrlib/
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
Review comment:
```suggestion
# Put Job into usrlib/ directory
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
+
+The whole setup is presented here:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/arch.png"
width="640px" alt="Reactive Mode Demo Architecture"/>
+</div>
+
+Let's discuss the components:
+
+**Flink**
+
+- The **JobManager** is deployed as a [Kubernetes
job](https://kubernetes.io/docs/concepts/workloads/controllers/job/). We are
submitting a container that is based on the official Flink Docker image, but
has the jar file of our job added to it. The Flink job simply reads data from a
Kafka topic and does some expensive math operations per event received. We use
these math operations to generate high CPU loads, without requiring a large
Kafka deployment.
+- The **TaskManager(s)** are deployed as a Kubernetes deployment, which is
scaled through a [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
In this experiment, the autoscaler is monitoring the CPU load of the pods in
the deployment. The number of pods is adjusted between 1 and 15 pods by the
autoscaler.
+
+**Additional Components**:
+
+- We have a **Zookeeper** and **Kafka** deployment (each with one pod) to
provide a Kafka topic that serves as the input for the Flink job.
+- **Data Generator** pod which produces messages to the Kafka topic. The data
generator is generating simple string messages at a adjustable rate. In this
experiment, the rate is following a sine wave.
+- For monitoring, we are deploying **Prometheus** and **Grafana**.
+
+The entire setup is [available on
GitHub](https://github.com/rmetzger/flink-reactive-mode-k8s-demo) if you want
to try this out yourself.
+
+### Results
+
+We've deployed above components on a hosted Kubernetes cluster, running it for
several days. The results are best examined based on a simple Grafana dashboard:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/result.png"
alt="Reactive Mode Demo Result"/>
+ <p class="align-center">Reactive Mode Experiment Results</p>
+</div>
+
+Let's take a look at the dashboard:
+
+- On the top left, we see the **Kafka consumer lag**, reported by Flink's
Kafka consumer (source). This metric reports basically the queue size of
unprocessed messages. A high lag means that Flink is not processing messages as
fast as they are produced: we need to scale up.
+
+ In the chart you see that the lag is usually following the throughput of
data coming from Kafka. When the throughput is the highest, the reported lag is
at \~75k messages. In low throughput times, it is basically at zero.
+
+- On the top right, you see the **throughput**, measured in Records per
second, as reported by Flink. You see that the throughput is roughly following
a sine wave, peaking at 6k messages per second, and going down to almost zero.
+
+- The bottom left chart is showing the **CPU load** per TaskManager. We've
added this metric to the dashboard, since this is the metric used by Kubernetes
to decide on the replica count of the TaskManager deployment. We can see that
as soon as a certain CPU load is reached, additional TaskManagers are started.
+
+- The bottom right chart is showing the **TaskManager count** over time. We
see when throughput (and CPU load) is peaking, that we usually run on 5
TaskManagers (with some peaks up to even 8). On low throughput, we are running
the minimal number of just one TaskManager. This chart is showing nicely that
Reactive Mode is working as expected in this experiment: The number of
TaskManagers is adjusting to the load on the system.
+
+
+### Lessons Learned
+
+When we initially started with the experiment, we noticed some anomalies in
the behavior of Flink, depicted in this chart:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl
}}/img/blog/2021-04-reactive-mode/high-timeout.png" alt="Reactive Mode Demo
Lessons Learned"/>
+ <p class="align-center">Reactive Mode not scaling down properly</p>
+</div>
+
+In all the charts, we see sudden spikes or drops: The consumer lag is going to
up to 600k messages (that's 8 times more than the usual 75k lag we observe at
peak), the throughput seems to peak (and drop). On the "Number of TaskManagers"
chart, we see that we are not following the throughput line very nicely. We are
wasting resources by allocating too many TaskManagers for the given at rate.
+
+We see that these issues are only occurring when the load is decreasing, and
Reactive Mode is supposed to scale down. So what is happening here?
+
+The Flink JobManager is sending periodic heartbeats to the TaskManagers, to
check if they are still alive. These heartbeats have a default timeout of 50
seconds. This value might seem high, but in high load scenarios, there might me
network congestions, garbage collection pauses or other disruptions that cause
slow heartbeats. We don't want to consider a TaskManager dead just because of a
temporary disruption.
Review comment:
```suggestion
The Flink JobManager is sending periodic heartbeats to the TaskManagers, to
check if they are still alive. These heartbeats have a default timeout of 50
seconds. This value might seem high, but in high load scenarios, there might be
network congestions, garbage collection pauses or other disruptions that cause
slow heartbeats. We don't want to consider a TaskManager dead just because of a
temporary disruption.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
Review comment:
In the graphs it looks that the workers are always ahead of the load.
Usually I would expect that you have some over and under provisioning when
changing the workloads.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
Review comment:
Not sure whether the claim that databases don't support this actually
holds true in the general case.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
Review comment:
The first sentence seem to be missing something. What happens if we
compare static vs. dynamic number of workers?
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
Review comment:
Not sure about "Reactive Mode" vs. the reactive mode.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
Review comment:
Maybe
```suggestion
The [Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
Review comment:
```suggestion
If you want to try out Reactive Mode yourself locally, just follow these
steps using a Flink 1.13.0 distribution:
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
Review comment:
```suggestion
The big benefit of Reactive Mode is that you don't need any specific
knowledge to scale Flink anymore. Flink basically behaves like a fleet of
servers (webservers, caches, batch processing) that you can grow and shrink as
you wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide utilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
Review comment:
```suggestion
What is interesting as a side note is that unlike most autoscalable "fleets
of servers", Flink is a stateful system, often processing valuable data
requiring strong correctness guarantees, comparable to a database. But unlike a
database, Flink is resilient enough (through checkpointing and state backups)
to adjust to changing workloads by just adding or removing resources, with very
little requirements (simple blob store for state backups).
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
Review comment:
```suggestion
Similarly, Kubernetes has [Horizontal Pod
Autoscalers](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
+
+The whole setup is presented here:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/arch.png"
width="640px" alt="Reactive Mode Demo Architecture"/>
Review comment:
Maybe somebody can help us beautify this graphic a bit.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
Review comment:
```suggestion
# These instructions assume you are in the root directory of a Flink
distribution.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
Review comment:
Not sure about the first comma.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
+
+The whole setup is presented here:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/arch.png"
width="640px" alt="Reactive Mode Demo Architecture"/>
+</div>
+
+Let's discuss the components:
+
+**Flink**
+
+- The **JobManager** is deployed as a [Kubernetes
job](https://kubernetes.io/docs/concepts/workloads/controllers/job/). We are
submitting a container that is based on the official Flink Docker image, but
has the jar file of our job added to it. The Flink job simply reads data from a
Kafka topic and does some expensive math operations per event received. We use
these math operations to generate high CPU loads, without requiring a large
Kafka deployment.
+- The **TaskManager(s)** are deployed as a Kubernetes deployment, which is
scaled through a [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
In this experiment, the autoscaler is monitoring the CPU load of the pods in
the deployment. The number of pods is adjusted between 1 and 15 pods by the
autoscaler.
+
+**Additional Components**:
+
+- We have a **Zookeeper** and **Kafka** deployment (each with one pod) to
provide a Kafka topic that serves as the input for the Flink job.
+- **Data Generator** pod which produces messages to the Kafka topic. The data
generator is generating simple string messages at a adjustable rate. In this
experiment, the rate is following a sine wave.
+- For monitoring, we are deploying **Prometheus** and **Grafana**.
+
+The entire setup is [available on
GitHub](https://github.com/rmetzger/flink-reactive-mode-k8s-demo) if you want
to try this out yourself.
+
+### Results
+
+We've deployed above components on a hosted Kubernetes cluster, running it for
several days. The results are best examined based on a simple Grafana dashboard:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/result.png"
alt="Reactive Mode Demo Result"/>
+ <p class="align-center">Reactive Mode Experiment Results</p>
+</div>
+
+Let's take a look at the dashboard:
+
+- On the top left, we see the **Kafka consumer lag**, reported by Flink's
Kafka consumer (source). This metric reports basically the queue size of
unprocessed messages. A high lag means that Flink is not processing messages as
fast as they are produced: we need to scale up.
+
+ In the chart you see that the lag is usually following the throughput of
data coming from Kafka. When the throughput is the highest, the reported lag is
at \~75k messages. In low throughput times, it is basically at zero.
+
+- On the top right, you see the **throughput**, measured in Records per
second, as reported by Flink. You see that the throughput is roughly following
a sine wave, peaking at 6k messages per second, and going down to almost zero.
+
+- The bottom left chart is showing the **CPU load** per TaskManager. We've
added this metric to the dashboard, since this is the metric used by Kubernetes
to decide on the replica count of the TaskManager deployment. We can see that
as soon as a certain CPU load is reached, additional TaskManagers are started.
+
+- The bottom right chart is showing the **TaskManager count** over time. We
see when throughput (and CPU load) is peaking, that we usually run on 5
TaskManagers (with some peaks up to even 8). On low throughput, we are running
the minimal number of just one TaskManager. This chart is showing nicely that
Reactive Mode is working as expected in this experiment: The number of
TaskManagers is adjusting to the load on the system.
+
+
+### Lessons Learned
+
+When we initially started with the experiment, we noticed some anomalies in
the behavior of Flink, depicted in this chart:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl
}}/img/blog/2021-04-reactive-mode/high-timeout.png" alt="Reactive Mode Demo
Lessons Learned"/>
+ <p class="align-center">Reactive Mode not scaling down properly</p>
Review comment:
Same here. The font is quite small.
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
+
+The whole setup is presented here:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/arch.png"
width="640px" alt="Reactive Mode Demo Architecture"/>
+</div>
+
+Let's discuss the components:
+
+**Flink**
+
+- The **JobManager** is deployed as a [Kubernetes
job](https://kubernetes.io/docs/concepts/workloads/controllers/job/). We are
submitting a container that is based on the official Flink Docker image, but
has the jar file of our job added to it. The Flink job simply reads data from a
Kafka topic and does some expensive math operations per event received. We use
these math operations to generate high CPU loads, without requiring a large
Kafka deployment.
+- The **TaskManager(s)** are deployed as a Kubernetes deployment, which is
scaled through a [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
In this experiment, the autoscaler is monitoring the CPU load of the pods in
the deployment. The number of pods is adjusted between 1 and 15 pods by the
autoscaler.
+
+**Additional Components**:
+
+- We have a **Zookeeper** and **Kafka** deployment (each with one pod) to
provide a Kafka topic that serves as the input for the Flink job.
+- **Data Generator** pod which produces messages to the Kafka topic. The data
generator is generating simple string messages at a adjustable rate. In this
experiment, the rate is following a sine wave.
+- For monitoring, we are deploying **Prometheus** and **Grafana**.
+
+The entire setup is [available on
GitHub](https://github.com/rmetzger/flink-reactive-mode-k8s-demo) if you want
to try this out yourself.
+
+### Results
+
+We've deployed above components on a hosted Kubernetes cluster, running it for
several days. The results are best examined based on a simple Grafana dashboard:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/result.png"
alt="Reactive Mode Demo Result"/>
+ <p class="align-center">Reactive Mode Experiment Results</p>
+</div>
+
+Let's take a look at the dashboard:
+
+- On the top left, we see the **Kafka consumer lag**, reported by Flink's
Kafka consumer (source). This metric reports basically the queue size of
unprocessed messages. A high lag means that Flink is not processing messages as
fast as they are produced: we need to scale up.
+
+ In the chart you see that the lag is usually following the throughput of
data coming from Kafka. When the throughput is the highest, the reported lag is
at \~75k messages. In low throughput times, it is basically at zero.
+
+- On the top right, you see the **throughput**, measured in Records per
second, as reported by Flink. You see that the throughput is roughly following
a sine wave, peaking at 6k messages per second, and going down to almost zero.
+
+- The bottom left chart is showing the **CPU load** per TaskManager. We've
added this metric to the dashboard, since this is the metric used by Kubernetes
to decide on the replica count of the TaskManager deployment. We can see that
as soon as a certain CPU load is reached, additional TaskManagers are started.
+
+- The bottom right chart is showing the **TaskManager count** over time. We
see when throughput (and CPU load) is peaking, that we usually run on 5
TaskManagers (with some peaks up to even 8). On low throughput, we are running
the minimal number of just one TaskManager. This chart is showing nicely that
Reactive Mode is working as expected in this experiment: The number of
TaskManagers is adjusting to the load on the system.
+
+
+### Lessons Learned
+
+When we initially started with the experiment, we noticed some anomalies in
the behavior of Flink, depicted in this chart:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl
}}/img/blog/2021-04-reactive-mode/high-timeout.png" alt="Reactive Mode Demo
Lessons Learned"/>
+ <p class="align-center">Reactive Mode not scaling down properly</p>
+</div>
+
+In all the charts, we see sudden spikes or drops: The consumer lag is going to
up to 600k messages (that's 8 times more than the usual 75k lag we observe at
peak), the throughput seems to peak (and drop). On the "Number of TaskManagers"
chart, we see that we are not following the throughput line very nicely. We are
wasting resources by allocating too many TaskManagers for the given at rate.
+
+We see that these issues are only occurring when the load is decreasing, and
Reactive Mode is supposed to scale down. So what is happening here?
+
+The Flink JobManager is sending periodic heartbeats to the TaskManagers, to
check if they are still alive. These heartbeats have a default timeout of 50
seconds. This value might seem high, but in high load scenarios, there might me
network congestions, garbage collection pauses or other disruptions that cause
slow heartbeats. We don't want to consider a TaskManager dead just because of a
temporary disruption.
+
+However, this default value is causing problems in this experiment: When the
Kubernetes autoscaler notices that the CPU load has gone down, it will reduce
the replica count of the TaskManager deployment, stopping at least one
TaskManager instance. Flink will almost immediately stop processing messages,
because the connection loss in the data transport layer of Flink. However, the
JobManager will wait for 50 seconds (the default heartbeat timeout) before the
TaskManager is considered dead.
+
+During this waiting period, the throughput is at zero and messages are queuing
in Kafka (causing spikes in the consumer lag). Once Flink is running again,
Flink will try to catch up on the queued messages, causing a spike in CPU load.
The autoscaler notices this load spike and allocates more TaskManagers.
+
+We are only seeing this effect on scale down, because scale down is much more
disruptive than scaling up. Scale up, which means adding TaskManagers, is
disrupting the processing only for the duration of a job restart (which is fast
since our application state are just a few bytes for the Kafka offsets), while
scaling down is disrupting the processing for roughly 50 seconds.
+
+To mitigate this issue, we've reduced the heartbeat timeout in our experiment
to 8 seconds. Additionally, we are looking into improving the behavior of the
JobManager to handle TaskManager losses better.
Review comment:
```suggestion
To mitigate this issue, we have reduced the heartbeat timeout in our
experiment to 8 seconds. Additionally, we are looking into improving the
behavior of the JobManager to detect TaskManager losses better and faster.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
+
+The whole setup is presented here:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/arch.png"
width="640px" alt="Reactive Mode Demo Architecture"/>
+</div>
+
+Let's discuss the components:
+
+**Flink**
+
+- The **JobManager** is deployed as a [Kubernetes
job](https://kubernetes.io/docs/concepts/workloads/controllers/job/). We are
submitting a container that is based on the official Flink Docker image, but
has the jar file of our job added to it. The Flink job simply reads data from a
Kafka topic and does some expensive math operations per event received. We use
these math operations to generate high CPU loads, without requiring a large
Kafka deployment.
+- The **TaskManager(s)** are deployed as a Kubernetes deployment, which is
scaled through a [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
In this experiment, the autoscaler is monitoring the CPU load of the pods in
the deployment. The number of pods is adjusted between 1 and 15 pods by the
autoscaler.
+
+**Additional Components**:
+
+- We have a **Zookeeper** and **Kafka** deployment (each with one pod) to
provide a Kafka topic that serves as the input for the Flink job.
+- **Data Generator** pod which produces messages to the Kafka topic. The data
generator is generating simple string messages at a adjustable rate. In this
experiment, the rate is following a sine wave.
+- For monitoring, we are deploying **Prometheus** and **Grafana**.
+
+The entire setup is [available on
GitHub](https://github.com/rmetzger/flink-reactive-mode-k8s-demo) if you want
to try this out yourself.
+
+### Results
+
+We've deployed above components on a hosted Kubernetes cluster, running it for
several days. The results are best examined based on a simple Grafana dashboard:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/result.png"
alt="Reactive Mode Demo Result"/>
+ <p class="align-center">Reactive Mode Experiment Results</p>
+</div>
+
+Let's take a look at the dashboard:
+
+- On the top left, we see the **Kafka consumer lag**, reported by Flink's
Kafka consumer (source). This metric reports basically the queue size of
unprocessed messages. A high lag means that Flink is not processing messages as
fast as they are produced: we need to scale up.
+
+ In the chart you see that the lag is usually following the throughput of
data coming from Kafka. When the throughput is the highest, the reported lag is
at \~75k messages. In low throughput times, it is basically at zero.
+
+- On the top right, you see the **throughput**, measured in Records per
second, as reported by Flink. You see that the throughput is roughly following
a sine wave, peaking at 6k messages per second, and going down to almost zero.
+
+- The bottom left chart is showing the **CPU load** per TaskManager. We've
added this metric to the dashboard, since this is the metric used by Kubernetes
to decide on the replica count of the TaskManager deployment. We can see that
as soon as a certain CPU load is reached, additional TaskManagers are started.
+
+- The bottom right chart is showing the **TaskManager count** over time. We
see when throughput (and CPU load) is peaking, that we usually run on 5
TaskManagers (with some peaks up to even 8). On low throughput, we are running
the minimal number of just one TaskManager. This chart is showing nicely that
Reactive Mode is working as expected in this experiment: The number of
TaskManagers is adjusting to the load on the system.
+
+
+### Lessons Learned
+
+When we initially started with the experiment, we noticed some anomalies in
the behavior of Flink, depicted in this chart:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl
}}/img/blog/2021-04-reactive-mode/high-timeout.png" alt="Reactive Mode Demo
Lessons Learned"/>
+ <p class="align-center">Reactive Mode not scaling down properly</p>
+</div>
+
+In all the charts, we see sudden spikes or drops: The consumer lag is going to
up to 600k messages (that's 8 times more than the usual 75k lag we observe at
peak), the throughput seems to peak (and drop). On the "Number of TaskManagers"
chart, we see that we are not following the throughput line very nicely. We are
wasting resources by allocating too many TaskManagers for the given at rate.
+
+We see that these issues are only occurring when the load is decreasing, and
Reactive Mode is supposed to scale down. So what is happening here?
+
+The Flink JobManager is sending periodic heartbeats to the TaskManagers, to
check if they are still alive. These heartbeats have a default timeout of 50
seconds. This value might seem high, but in high load scenarios, there might me
network congestions, garbage collection pauses or other disruptions that cause
slow heartbeats. We don't want to consider a TaskManager dead just because of a
temporary disruption.
+
+However, this default value is causing problems in this experiment: When the
Kubernetes autoscaler notices that the CPU load has gone down, it will reduce
the replica count of the TaskManager deployment, stopping at least one
TaskManager instance. Flink will almost immediately stop processing messages,
because the connection loss in the data transport layer of Flink. However, the
JobManager will wait for 50 seconds (the default heartbeat timeout) before the
TaskManager is considered dead.
Review comment:
```suggestion
However, this default value is causing problems in this experiment: When the
Kubernetes autoscaler notices that the CPU load has gone down, it will reduce
the replica count of the TaskManager deployment, stopping at least one
TaskManager instance. Flink will almost immediately stop processing messages,
because of the connection loss in the data transport layer of Flink. However,
the JobManager will wait for 50 seconds (the default heartbeat timeout) before
the TaskManager is considered dead.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
Review comment:
```suggestion
The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines. On low load, it
should stop TaskManagers to save resources.
```
##########
File path: _posts/2021-04-05-reactive-mode.md
##########
@@ -0,0 +1,149 @@
+---
+layout: post
+title: "Scaling Flink automatically with Reactive Mode"
+date: 2021-04-5T00:00:00.000Z
+authors:
+- rob:
+ name: "Robert Metzger"
+ twitter: "rmetzger_"
+excerpt: Apache Flink 1.13 introduced Reactive Mode, a big step forward in
Flink's ability to dynamically adjust to changing workloads, reducing resource
utilization and overall costs. The blog post is demonstrating Reactive Mode on
Kubernetes, including some lessons learned.
+
+---
+
+{% toc %}
+
+## Introduction
+
+Streaming jobs which run for several days or longer usually experience changes
in their workload during their lifetime. These changes can originate from
seasonal spikes, such as day vs. night, weekdays vs. weekend or holidays vs.
non-holidays, sudden events or simply growing popularity of your product. Some
of these changes are more predictable than others but what all have in common
is that they change the resource demand of your job if you want to keep the
same service quality for your customers.
+
+When comparing a scenario where the number of workers is static, while the
workload is changing over time against the same workload, but with the number
of workers adjusting to the workload. The area between the worker allocation
and the load is the loss, which is generally much higher with a static
allocation.
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/intro.png"
width="640px" alt="Reactive Mode Intro"/>
+</div>
+
+
+Since Flink 1.2 introduced [rescalable
state](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html),
it has been possible to **manually rescale**. You can stop-and-restore a job
with a different parallelism. If your job was running with a parallelism of
p=100, and your load increased, you can restart it with p=200 to cope with the
additional data.
+
+The problem with this approach is that you have to orchestrate a rescale
operation with custom tools, including error handling etc. by yourself.
+
+[Reactive
Mode](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/)
introduces a new option in Flink 1.13: You monitor your Flink cluster and add
or remove resources depending on some metrics, Flink will do the rest. Reactive
Mode is a JobManager mode where your job is always using all TaskManagers
available.
+
+The big benefit of Reactive Mode is that you don't need any specific knowledge
to scale Flink anymore. Flink basically behaves like a fleet of servers
(webservers, caches, batch processing) that you can grow and shrink as you
wish. Since this is such a common pattern, there is a lot of infrastructure
available for handling this case. All major cloud providers provide facilities
to monitor a metric and automatically scale a set of machines accordingly. This
is for example called [Auto Scaling
group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)
in AWS, or [Managed Instance
group](https://cloud.google.com/compute/docs/instance-groups) in Google Cloud.
+Similarly, Kubernetes has [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)s.
+
+What's interesting as a side note is that unlike most autoscalable "fleets of
servers", Flink is a stateful system, often processing valuable data requiring
strong correctness guarantees, comparable to a database. But unlike a database,
Flink is resilient enough (through checkpointing and state backups) to adjust
to changing workloads by just adding or removing resources, with very little
requirements (simple blob store for state backups).
+
+## Getting Started
+
+If you want to try out Reactive Mode yourself locally, just follow these steps
on a Flink 1.13.0 distribution:
+
+```bash
+# these instructions assume you are in the root directory of a Flink
distribution.
+# Put Job into lib/ directory
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
+# Submit Job in Reactive Mode
+./bin/standalone-job.sh start -Dscheduler-mode=reactive
-Dexecution.checkpointing.interval="10s" -j
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+# Start first TaskManager
+./bin/taskmanager.sh start
+```
+
+You have now started a Flink job in Reactive Mode. The [web
interface](http://localhost:8081) shows that the job is running on one
TaskManager. If you want to scale up the job, simply add another TaskManager to
the cluster:
+
+```bash
+# Start additional TaskManager
+./bin/taskmanager.sh start
+```
+
+To scale down, remove a TaskManager instance.
+
+```bash
+# Remove a TaskManager
+./bin/taskmanager.sh stop
+```
+
+Reactive Mode also works when deploying [Flink on
Docker](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/)
or using the [standalone Kubernetes
deployment](https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/)
(both only as application clusters).
+
+## Demo on Kubernetes
+
+In this section, we want to demonstrate Reactive Mode in a real-world
scenario. You can use this demonstration as a starting point for your own
scalable deployment of Flink on Kubernetes, or as a template for building your
own deployment on another infrastructure.
+
+### The Setup
+
+The central idea of this demo is to use a Kubernetes [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/),
which monitors the CPU load of all TaskManager pods and adjusts their
replication factor accordingly. On high CPU load, the autoscaler should add
more TaskManagers, distributing the load across more machines, on low load, it
should stop TaskManagers to save resources.
+
+The whole setup is presented here:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/arch.png"
width="640px" alt="Reactive Mode Demo Architecture"/>
+</div>
+
+Let's discuss the components:
+
+**Flink**
+
+- The **JobManager** is deployed as a [Kubernetes
job](https://kubernetes.io/docs/concepts/workloads/controllers/job/). We are
submitting a container that is based on the official Flink Docker image, but
has the jar file of our job added to it. The Flink job simply reads data from a
Kafka topic and does some expensive math operations per event received. We use
these math operations to generate high CPU loads, without requiring a large
Kafka deployment.
+- The **TaskManager(s)** are deployed as a Kubernetes deployment, which is
scaled through a [Horizontal Pod
Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
In this experiment, the autoscaler is monitoring the CPU load of the pods in
the deployment. The number of pods is adjusted between 1 and 15 pods by the
autoscaler.
+
+**Additional Components**:
+
+- We have a **Zookeeper** and **Kafka** deployment (each with one pod) to
provide a Kafka topic that serves as the input for the Flink job.
+- **Data Generator** pod which produces messages to the Kafka topic. The data
generator is generating simple string messages at a adjustable rate. In this
experiment, the rate is following a sine wave.
+- For monitoring, we are deploying **Prometheus** and **Grafana**.
+
+The entire setup is [available on
GitHub](https://github.com/rmetzger/flink-reactive-mode-k8s-demo) if you want
to try this out yourself.
+
+### Results
+
+We've deployed above components on a hosted Kubernetes cluster, running it for
several days. The results are best examined based on a simple Grafana dashboard:
+
+<div class="row front-graphic">
+ <img src="{{ site.baseurl }}/img/blog/2021-04-reactive-mode/result.png"
alt="Reactive Mode Demo Result"/>
+ <p class="align-center">Reactive Mode Experiment Results</p>
Review comment:
Can we increase the font of the different graphs a bit? It is quite hard
to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
