Hi, I'm playing with a Flink 1.18 demo with the auto-scaler and the adaptive scheduler.
The operator can correctly collect data and order the job to scale up, but it'll take the job several times to reach the required parallelism. E.g. The original parallelism for each vertex is something like below: Vertex : write hourly_ads s3 || Parallelism : 100 Vertex : Source: ads-kafkasource -> Timestamps/Watermarks -> ads-filter-action -> ads-cast || Parallelism : 100 Vertex : Commit hourly_ads || Parallelism : 1 Then the operator decides to scale the job to this: Vertex : write hourly_ads s3 || Parallelism : 200 Vertex : Source: ads-kafkasource -> Timestamps/Watermarks -> ads-filter-action -> ads-cast || Parallelism : 100 Vertex : Commit hourly_ads || Parallelism : 1 But when scaling, the vertex write hourly_ads s3 does scale directly from 100 to 200, but first scales to a number, say 120, then 150, then 180. It'll take 3~4 times till the job to reach the required parallelism. I'm wondering how to avoid this issue ? Thanks in advanced.
