Hi, all. I have started voting[1] for this FLIP. If you have any questions, you can reply in this discussion thread.
[1]: https://lists.apache.org/thread/gdymm7pr2slzy9gqkfo97vn73496w0cj Best regards, Junrui Xintong Song <tonysong...@gmail.com> 于2023年1月4日周三 18:16写道： > Thanks for the proposal. > > Another potential benefit I see in this FLIP is that it may reduce the > complexity and maintenance overhead of the scheduler. During developing > hybrid shuffle, we had to re-implement some similar logic to make both > default and adaptive batch schedulers support the new shuffle type. For > batch workloads, the functionality provided by adaptive batch scheduler > should already be a superset of those provided by the default scheduler. If > that can be confirmed from user feedback after making the adaptive batch > scheduler default, we may consider making default scheduler streaming > dedicated. That should significantly simplify the default scheduler, > benefiting both the maintenance of the default scheduler and the > development of new features in the batch area. > > So +1 from my side. > > Best, > > Xintong > > > > On Wed, Jan 4, 2023 at 1:14 PM JunRui Lee <jrlee....@gmail.com> wrote: > > > Hi, Yuxia > > > > Thanks for the reply. > > > > > 1. About the configuration renaming. The old configurations ... > > > > Thanks for your suggestion. And this explanation has been updated > > to the FLIP to describe this change more clearly. The old configurations > > will be deprecated after the configuration renaming, and it still will be > > considered to guarantee compatibility. > > > > > 2. I'm cursion in which case users will disable auto parallelism... > > > > Actually, the adaptive batch scheduler is not only used for automatic > > parallelism derivation, but also for speculative execution. If users > > want to use speculative execution and do not enable automatic > > parallelism, they can configure the configuration > > "execution.batch.adaptive.auto-parallelism.enabled" instead of > > configuring the configuration "jobmanager.scheduler". > > > > In addition, “adaptive” represents a function, not just the automatic > > derivation of parallelism, and more functions may be added in the > > future. So we want users to focus on enabling or disabling the > > parallelism derivation function itself rather than replacing the > > scheduler type. > > > > Best, > > JunRui > > > > Lijie Wang <wangdachui9...@gmail.com> 于2023年1月3日周二 11:03写道： > > > > > Hi Junrui, > > > > > > Thanks for driving this FLIP, + 1 for this proposal. I believe it will > > > greatly improve the experiences of batch users. > > > > > > Best, > > > Lijie > > > > > > Zhu Zhu <reed...@gmail.com> 于2022年12月30日周五 12:40写道： > > > > > > > Hi Junrui, > > > > > > > > Thanks for creating this FLIP! > > > > > > > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in > batch > > > > scheduling, also with some must-have features like speculative > > execution. > > > > It will be great that users can easily use it, without required to > > > knowing > > > > the underlying scheduler and configuring some advanced items. > > > > > > > > So generally +1 for this proposal. > > > > > > > > Regarding the configuration key renaming, like yuxia mentioned, we > > should > > > > deprecate the old ones and add new ones with new names, to guarantee > > > > compatibility. > > > > > > > > Thanks, > > > > Zhu > > > > > > > > yuxia <luoyu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道： > > > > > > > > > > Hi, JunRui Lee. > > > > > > > > > > Thanks for driving this FLIP. It must a good improvement for batch > > > > users' experiences. > > > > > I have few questions about this FLIP: > > > > > 1: About the configuration renaming. The old configurations will be > > > > deprecated or removed directly? if user upgrade their Flink version, > > > these > > > > old configuration will still be considered or just ignored? If > ignore, > > > the > > > > users may need to modify their configurations after they upgrade > their > > > > Flink. > > > > > > > > > > 2: I'm cursion in which case users will disable auto parallelism > > > > derivation if they have enabled adaptive batch scheduler. IIUC, auto > > > > parallelism derivation is what adaptive batch scheduler aim to do. If > > use > > > > want to diable auto parallelism derivation, can they just disable > > > adaptive > > > > batch scheduler.? > > > > > > > > > > Best regards, > > > > > Yuxia > > > > > > > > > > ----- 原始邮件 ----- > > > > > 发件人: "JunRui Lee" <jrlee....@gmail.com> > > > > > 收件人: "dev" <dev@flink.apache.org> > > > > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > > > > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default > > > > scheduler for batch jobs > > > > > > > > > > Hi, devs, > > > > > > > > > > I'd like to start a discussion about FLIP-283: Use adaptive batch > > > > > scheduler as default scheduler for batch jobs[1]. > > > > > > > > > > In FLIP-187, we introduced an adaptive batch scheduler. The > adaptive > > > > > batch scheduler has stronger batch scheduling capabilities, > including > > > > > automatically deciding parallelisms of job vertices for batch > > > > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], > > > > > and speculative execution (FLIP-168)[4]. To further use the > adaptive > > > > > batch scheduler to improve flink's batch capability, in this FLIP > > > > > we aim to make the adaptive batch scheduler as the default batch > > > > > scheduler. > > > > > > > > > > Currently, users have to set some configuration of the adaptive > > > > > batch scheduler, which is not very convenient. To use the adaptive > > > > > batch scheduler as the default batch scheduler, we need to improve > > > > > the user's out-of-the-box experience. Therefore, we also need to > > > > > optimize the current adaptive batch scheduler configuration. > > > > > > > > > > Looking forward to your feedback. > > > > > > > > > > [1]: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs > > > > > [2]: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler > > > > > [3]:https://issues.apache.org/jira/browse/FLINK-29663 > > > > > [4]: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > > > > > > > > > > Best regards, > > > > > JunRui Lee > > > > > > > > > >
