Hi all,
So far, I think we have reached an agreement on this FLIP. I have started
the voting thread [1]. Please cast your vote there or ask additional
questions here.
[1] https://lists.apache.org/thread/58yyxvnygw7oy5t556g8rm9y2xzb1l66
Best,
Lijie
Lijie Wang 于2021年11月5日周五 下午4:04写道：
> Hi all,
Hi all,
Thanks for all the comments on this FLIP.
Based on the discussion in the mailing list and comments in the wiki, I
updated the FLIP doc, the mainly changes include:
1. Added the limitation that currently only supports ALL-EDGES-BLOCKING
batch jobs*.*
2. Adopted some suggestions in the
I have to admit that I cannot think of a better name for the adaptive batch
scheduler atm. Maybe it is good enough to call the two schedulers
AdaptiveBatchScheduler and AdaptiveStreamingScheduler to tell which
scheduler is used for which execution mode. It is true, though, that the
former is
Hi David,
Thanks for your comments.
I personally think that "Adaptive" means: Flink automatically determines
the appropriate scheduling and execution plan based on some information.
The information can include both resource information and workload
information, rather than being limited to a
Hi, thanks for drafting the FLIP, Lijie and Zhu Zhu. It already looks
pretty solid and it will be a really great improvement to the batch
scheduling. I'd second to the Till's feedback, especially when it comes to
the consistent behavior between different deployment types / schedulers.
What I'm
Hi, Till & Zhu
Thanks for your feedback. Also thanks for your comments and suggestions
on wiki, which are very helpful for perfecting the FLIP.
I also agree to provide our users with consistent and easy-to-understand
deployment options. Regarding the three options proposed by Till, my
opinion
Hi Till,
Thanks for the comments!
I agree with you that we should avoid an auto-scaled job not able to be
scheduled
in standalone/reactive mode. And I think it's great if we can expose a
deployment
option that is consistent for streaming and batch jobs, which can be easier
to
understand. Just
Hi Lijie,
Thanks for drafting this FLIP together with Zhu Zhu :-)
I like the idea of making the parallelism of operators of a bounded job
dependent on the data size. This makes the job adjust automatically when
the data sources/sizes change.
I can see this work well in combination with the
Hi all,
Zhu Zhu and I propose to introduce a new job scheduler to Flink: adaptive
batch job scheduler. The new scheduler can automatically decide
parallelisms of job vertices for batch jobs, according to the size of data
volume each vertex needs to process.
Major benefits of this scheduler
