Thank you both. Looks like with the upcoming Flink 1.18 release, we should
build an auto-scaler service to monitor the job and properly adjust the
allocated resource from YARN.
Leon
On Wed, Jun 28, 2023 at 9:08 AM Madan D wrote:
> Hello Leon,
>
> As described by Chen below Adaptive Scheduler d
Hello Leon,
As described by Chen below Adaptive Scheduler doesn't perform auto scale a
Flink Job other than allocating the requested slots based on availability.
Recently we implemented this with EMR managed scaling by combining adaptive
scheduler since there's no direct support of auto scaling
Hi Leon,
Adaptive scheduler alone cannot autoscale a Flink job. It simply adjusts the
parallelism of a job based on available slots [1]. To autoscale a job, we
further need a policy to suggest the recommended resources for the job and a
mechanism to adjust the allocated resources of the job (ak
