Hi Junrui, Thanks for driving this FLIP, + 1 for this proposal. I believe it will greatly improve the experiences of batch users.
Best, Lijie Zhu Zhu <reed...@gmail.com> 于2022年12月30日周五 12:40写道： > Hi Junrui, > > Thanks for creating this FLIP! > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch > scheduling, also with some must-have features like speculative execution. > It will be great that users can easily use it, without required to knowing > the underlying scheduler and configuring some advanced items. > > So generally +1 for this proposal. > > Regarding the configuration key renaming, like yuxia mentioned, we should > deprecate the old ones and add new ones with new names, to guarantee > compatibility. > > Thanks, > Zhu > > yuxia <luoyu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道： > > > > Hi, JunRui Lee. > > > > Thanks for driving this FLIP. It must a good improvement for batch > users' experiences. > > I have few questions about this FLIP: > > 1: About the configuration renaming. The old configurations will be > deprecated or removed directly? if user upgrade their Flink version, these > old configuration will still be considered or just ignored? If ignore, the > users may need to modify their configurations after they upgrade their > Flink. > > > > 2: I'm cursion in which case users will disable auto parallelism > derivation if they have enabled adaptive batch scheduler. IIUC, auto > parallelism derivation is what adaptive batch scheduler aim to do. If use > want to diable auto parallelism derivation, can they just disable adaptive > batch scheduler.? > > > > Best regards, > > Yuxia > > > > ----- 原始邮件 ----- > > 发件人: "JunRui Lee" <jrlee....@gmail.com> > > 收件人: "dev" <dev@flink.apache.org> > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default > scheduler for batch jobs > > > > Hi, devs, > > > > I'd like to start a discussion about FLIP-283: Use adaptive batch > > scheduler as default scheduler for batch jobs[1]. > > > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive > > batch scheduler has stronger batch scheduling capabilities, including > > automatically deciding parallelisms of job vertices for batch > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], > > and speculative execution (FLIP-168)[4]. To further use the adaptive > > batch scheduler to improve flink's batch capability, in this FLIP > > we aim to make the adaptive batch scheduler as the default batch > > scheduler. > > > > Currently, users have to set some configuration of the adaptive > > batch scheduler, which is not very convenient. To use the adaptive > > batch scheduler as the default batch scheduler, we need to improve > > the user's out-of-the-box experience. Therefore, we also need to > > optimize the current adaptive batch scheduler configuration. > > > > Looking forward to your feedback. > > > > [1]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs > > [2]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler > > [3]:https://issues.apache.org/jira/browse/FLINK-29663 > > [4]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > > > > Best regards, > > JunRui Lee >
