[ https://issues.apache.org/jira/browse/FLINK-22677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17375218#comment-17375218 ]
Zhu Zhu commented on FLINK-22677: --------------------------------- One thing need to mention is that I did not change {{AdaptiveScheduler}} to support async partition registration, because otherwise we will need to introduce async callback process to {{AdaptiveScheduler}}, which was intentionally avoided. Given that customized {{ShuffleMaster}} is mainly for batch jobs on external shuffle purpose and reactive mode is for streaming jobs only, I think for reactive mode we can keep the assumption that the {{ShuffleMaster}} will complete partition registration immediately. WDYT? [~trohrmann] > Scheduler should invoke ShuffleMaster#registerPartitionWithProducer by a real > asynchronous fashion > -------------------------------------------------------------------------------------------------- > > Key: FLINK-22677 > URL: https://issues.apache.org/jira/browse/FLINK-22677 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination > Affects Versions: 1.14.0 > Reporter: Jin Xing > Assignee: Zhu Zhu > Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > Current scheduler enforces a synchronous registration though the API of > ShuffleMaster#registerPartitionWithProducer returns a CompletableFuture. In > scenario of remote shuffle service, the talk between ShuffleMaster and remote > cluster tends to be expensive. A synchronous registration risks to block main > thread potentially and might cause negative side effects like heartbeat > timeout. Additionally, expensive synchronous invokes to remote could > bottleneck the throughput for applying shuffle resource, especially for batch > jobs with complicated DAGs; -- This message was sent by Atlassian Jira (v8.3.4#803005)