Hi Madan,
The next release of Flink Kubernetes Operator will come with a built-in
autoscaler. Unfortunately, there's no official support for autoscaling Flink
applications on YARN yet. You can take a look at
https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling and
try building one on your own.
FLIP-271: Autoscaling - Apache Flink - Apache Software
Foundation<https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling>
Motivation. One of the most challenging aspects of running always-on streaming
pipeline is the correct sizing of Flink deployments. Too few resources lead to
unstable Flink jobs which requires users to scale up their deployments manually.
cwiki.apache.org
Best,
Zhanghao Chen
From: Madan D via user
Sent: Saturday, January 28, 2023 8:56
To: user@flink.apache.org
Subject: Rescale Flink - Adjust Parallelism based on TPS
Hello Team,
I would like to understand if there's a way flink can rescale automatically by
adjusting parallelism based on incoming traffic.
For example, when a job starts incoming TPS is around 100 which can be
processed with parallelism 4, and when TPS increased 1000 I might need to
rescale flink to use parallelism 10 or 20.
Is there a way we can set min and max parallelism to flink application which
can adjust automatically based on incoming load with stop and starting job with
new config.
I assume Kubernetes automatically does by changing replicas but all our
applications are running on yarn as of today.
Regards,
Madan
