Hi Samrat Do you mean to create an independent module for flink scaling in flink-k8s-operator? How about creating a project such as `flink-auto-scaling` which is completely independent? Besides resource managers such as k8s and yarn, we can do more things in the project, for example, updating config in the user's `job submission system` after scaling flink jobs. WDYT?
Best, Shammon On Thu, Feb 16, 2023 at 7:38 PM Maximilian Michels <m...@apache.org> wrote: > Hi Samrat, > > The autoscaling module is now pluggable but it is still tightly > coupled with Kubernetes. It will take additional work for the logic to > work independently of the cluster manager. > > -Max > > On Thu, Feb 16, 2023 at 11:14 AM Samrat Deb <decordea...@gmail.com> wrote: > > > > Oh! yesterday it got merged. > > Apologies , I missed the recent commit @Gyula. > > > > Thanks for the update > > > > > > > > On Thu, Feb 16, 2023 at 3:17 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > > > > > Max recently moved the autoscaler logic in a separate submodule, did > you > > > see that? > > > > > > > > > > https://github.com/apache/flink-kubernetes-operator/commit/5bb8e9dc4dd29e10f3ba7c8ce7cefcdffbf92da4 > > > > > > Gyula > > > > > > On Thu, Feb 16, 2023 at 10:27 AM Samrat Deb <decordea...@gmail.com> > wrote: > > > > > > > Hi , > > > > > > > > *Context:* > > > > Auto Scaling was introduced in Flink as part of FLIP-271[1]. > > > > It discusses one of the important aspects to provide a robust default > > > > scaling algorithm. > > > > a. Ensure scaling yields effective usage of assigned task > slots. > > > > b. Ramp up in case of any backlog to ensure it gets processed > in a > > > > timely manner > > > > c. Minimize the number of scaling decisions to prevent costly > > > rescale > > > > operation > > > > The flip intends to add an auto scaling framework based on 6 major > > > metrics > > > > and contains different types of threshold to trigger the scaling. > > > > > > > > Thread[2] discusses a different problem: why autoscaler is part of > the > > > > operator instead of jobmanager at runtime. > > > > The Community decided to keep the autoscaling logic in the > > > > flink-kubernetes-operator. > > > > > > > > *Proposal: * > > > > In this discussion, I want to put forward a thought of extracting > out the > > > > auto scaling logic into a new submodule in flink-kubernetes-operator > > > > repository[3], > > > > which will be independent of any resource manager/Operator. > > > > Currently the Autoscaling algorithm is very tightly coupled with the > > > > kubernetes API. > > > > This makes the autoscaling core algorithm not so easily extensible > for > > > > different available resource managers like YARN, Mesos etc. > > > > A Separate autoscaling module inside the flink kubernetes operator > will > > > > help other resource managers to leverage the autoscaling logic. > > > > > > > > [1] > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling > > > > [2] https://lists.apache.org/thread/pvfb3fw99mj8r1x8zzyxgvk4dcppwssz > > > > [3] https://github.com/apache/flink-kubernetes-operator > > > > > > > > > > > > Bests, > > > > Samrat > > > > > > > >
