Yes, exposing an API to adjust the parallelism of individual operators is definitely a good step towards the auto-scaling feature which we will consider. The missing piece is persisting this information so that in case of recovery you don't recover with a completely different parallelism.
I also agree that it will be very hard for Flink to decide on the best parallelism in general. For that to do you usually need to know a bit about the application logic. Hence, outsourcing this problem to the user who can do better decisions is a very good idea. The community will keep improving this feature so that with next releases it should become more powerful. Cheers, Till On Thu, May 6, 2021 at 2:38 PM vishalovercome <vis...@moengage.com> wrote: > Thank you for answering all my questions. My suggestion would be to start > off > with exposing an API to allow dynamically changing operator parallelism as > the users of flink will be better able to decide the right scaling policy. > Once this functionality is there, its just a matter of providing policies > (ratio based, throughput based, back-pressure based). The web UI could be > used for setting parallelism as well. > > An analogy would be autoscaling provided by cloud providers. The features > provided are: > 1. Web UI for manually overriding parallelism (min, max, desired) > 2. Metric based scaling policies > > It will be difficult for developers to think of a reasonable value for > maxParallelism for each operator and like I explained above, sometimes even > a small increase in parallelism is enough to bring things down. A UI / > external policy based approach will allow for quick experimentation and > fine > tuning. I don't think it will be possible for flink developers to build one > size fits all solution. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
