@Pedro: The current design focuses on record processing time metrics. In most cases when we need to scale (such as too much state per operator), record processing time actually slows, so it would detect that. Of course in the future we can add new logic if we see something missing.
@ConradJam: We agree that the on/off config for the actual scaling is very important. The autoscaler module would then publish all metrics and parallelism advice without actually taking action. We should definitely have this. Thanks for the pointer to the talk, we have considered that approach but it's mostly suitable for very simple pipelines. We hope that our approach would generalize better to complex applications and would also cover the simple case in a similar way. Gyula On Sun, Nov 6, 2022 at 7:25 AM Zheng Yu Chen <jam.gz...@gmail.com> wrote: > Hi Max > Thank you for dirver this flip，I have some advice for this flip > > Do we not only exist in the (on/off) switch, but also have one more option > for (advcie). > After the user opens (advcie), it does not actually perform AutoScaling. It > only outputs the notification form of tuning suggestions for the user's > reference. It is up to the user to decide whether to trigger the adjustment > of the parallelism.I believe that this function is very useful in the > debugging phase or the observation phase. When the user observes a certain > period of time, he thinks it is feasible and then turns on the switch. > > at the same time, I found that FFA 2020 Netflix has a related topic > discussing the automatic tuning function > Attach the video address: Autoscaling Flink at Netflix - Timothy Farkas > <https://www.youtube.com/watch?v=NV0jvA5ZDNc> This may be helpful for us > to > complete this function > > Here is a description of using some prediction functions to predict the > operator traffic of this job. Can we provide some interfaces for users to > customize and implement some tuning algorithms? > > > Maximilian Michels <m...@apache.org> 于2022年11月5日周六 02:37写道： > > > Hi, > > > > I would like to kick off the discussion on implementing autoscaling for > > Flink as part of the Flink Kubernetes operator. I've outlined an approach > > here which I find promising: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling > > > > I've been discussing this approach with some of the operator > contributors: > > Gyula, Marton, Matyas, and Thomas (all in CC). We started prototyping an > > implementation based on the current FLIP design. If that goes well, we > > would like to contribute this to Flink based on the results of the > > discussion here. > > > > I'm curious to hear your thoughts. > > > > -Max > > > > > -- > Best > > ConradJam >
