Thanks for the proposal. Another potential benefit I see in this FLIP is that it may reduce the complexity and maintenance overhead of the scheduler. During developing hybrid shuffle, we had to re-implement some similar logic to make both default and adaptive batch schedulers support the new shuffle type. For batch workloads, the functionality provided by adaptive batch scheduler should already be a superset of those provided by the default scheduler. If that can be confirmed from user feedback after making the adaptive batch scheduler default, we may consider making default scheduler streaming dedicated. That should significantly simplify the default scheduler, benefiting both the maintenance of the default scheduler and the development of new features in the batch area.
So +1 from my side. Best, Xintong On Wed, Jan 4, 2023 at 1:14 PM JunRui Lee <jrlee....@gmail.com> wrote: > Hi, Yuxia > > Thanks for the reply. > > > 1. About the configuration renaming. The old configurations ... > > Thanks for your suggestion. And this explanation has been updated > to the FLIP to describe this change more clearly. The old configurations > will be deprecated after the configuration renaming, and it still will be > considered to guarantee compatibility. > > > 2. I'm cursion in which case users will disable auto parallelism... > > Actually, the adaptive batch scheduler is not only used for automatic > parallelism derivation, but also for speculative execution. If users > want to use speculative execution and do not enable automatic > parallelism, they can configure the configuration > "execution.batch.adaptive.auto-parallelism.enabled" instead of > configuring the configuration "jobmanager.scheduler". > > In addition, “adaptive” represents a function, not just the automatic > derivation of parallelism, and more functions may be added in the > future. So we want users to focus on enabling or disabling the > parallelism derivation function itself rather than replacing the > scheduler type. > > Best, > JunRui > > Lijie Wang <wangdachui9...@gmail.com> 于2023年1月3日周二 11:03写道： > > > Hi Junrui, > > > > Thanks for driving this FLIP, + 1 for this proposal. I believe it will > > greatly improve the experiences of batch users. > > > > Best, > > Lijie > > > > Zhu Zhu <reed...@gmail.com> 于2022年12月30日周五 12:40写道： > > > > > Hi Junrui, > > > > > > Thanks for creating this FLIP! > > > > > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch > > > scheduling, also with some must-have features like speculative > execution. > > > It will be great that users can easily use it, without required to > > knowing > > > the underlying scheduler and configuring some advanced items. > > > > > > So generally +1 for this proposal. > > > > > > Regarding the configuration key renaming, like yuxia mentioned, we > should > > > deprecate the old ones and add new ones with new names, to guarantee > > > compatibility. > > > > > > Thanks, > > > Zhu > > > > > > yuxia <luoyu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道： > > > > > > > > Hi, JunRui Lee. > > > > > > > > Thanks for driving this FLIP. It must a good improvement for batch > > > users' experiences. > > > > I have few questions about this FLIP: > > > > 1: About the configuration renaming. The old configurations will be > > > deprecated or removed directly? if user upgrade their Flink version, > > these > > > old configuration will still be considered or just ignored? If ignore, > > the > > > users may need to modify their configurations after they upgrade their > > > Flink. > > > > > > > > 2: I'm cursion in which case users will disable auto parallelism > > > derivation if they have enabled adaptive batch scheduler. IIUC, auto > > > parallelism derivation is what adaptive batch scheduler aim to do. If > use > > > want to diable auto parallelism derivation, can they just disable > > adaptive > > > batch scheduler.? > > > > > > > > Best regards, > > > > Yuxia > > > > > > > > ----- 原始邮件 ----- > > > > 发件人: "JunRui Lee" <jrlee....@gmail.com> > > > > 收件人: "dev" <dev@flink.apache.org> > > > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > > > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default > > > scheduler for batch jobs > > > > > > > > Hi, devs, > > > > > > > > I'd like to start a discussion about FLIP-283: Use adaptive batch > > > > scheduler as default scheduler for batch jobs[1]. > > > > > > > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive > > > > batch scheduler has stronger batch scheduling capabilities, including > > > > automatically deciding parallelisms of job vertices for batch > > > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], > > > > and speculative execution (FLIP-168)[4]. To further use the adaptive > > > > batch scheduler to improve flink's batch capability, in this FLIP > > > > we aim to make the adaptive batch scheduler as the default batch > > > > scheduler. > > > > > > > > Currently, users have to set some configuration of the adaptive > > > > batch scheduler, which is not very convenient. To use the adaptive > > > > batch scheduler as the default batch scheduler, we need to improve > > > > the user's out-of-the-box experience. Therefore, we also need to > > > > optimize the current adaptive batch scheduler configuration. > > > > > > > > Looking forward to your feedback. > > > > > > > > [1]: > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs > > > > [2]: > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler > > > > [3]:https://issues.apache.org/jira/browse/FLINK-29663 > > > > [4]: > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > > > > > > > > Best regards, > > > > JunRui Lee > > > > > >
