Thanks for updating! +1 for supporting the pipelined region scheduling. Although we could not prevent resource deadlock in all scenarios, it is really a big step.
The design generally LGTM. One minor thing I want to make sure. If I understand correctly, the blocking edge will not be consumable before the upstream is finished. Without it, when the failure occurs in the upstream region, there is still possible to have a resource deadlock. I don't know whether it is an explicit protocol now. But after this FLIP, I think it should not be broken. I'm also wondering could we execute the upstream and downstream regions at the same time if we have enough resources. It can shorten the running time of large job. We should not break the protocol of blocking edge. But if it is possible to change the data exchange mode of two regions dynamically? Best, Yangze Guo On Fri, Mar 27, 2020 at 1:15 PM Zhu Zhu <reed...@gmail.com> wrote: > > Thanks for reporting this Yangze. > I have update the permission to those images. Everyone are able to view them > now. > > Thanks, > Zhu Zhu > > Yangze Guo <karma...@gmail.com> 于2020年3月27日周五 上午11:25写道： >> >> Thanks for driving this discussion, Zhu Zhu & Gary. >> >> I found that the image link in this FLIP is not working well. When I >> open that link, Google doc told me that I have no access privilege. >> Could you take a look at that issue? >> >> Best, >> Yangze Guo >> >> On Fri, Mar 27, 2020 at 1:38 AM Gary Yao <g...@apache.org> wrote: >> > >> > Hi community, >> > >> > In the past releases, we have been working on refactoring Flink's scheduler >> > with the goal of making the scheduler extensible [1]. We have rolled out >> > most of the intended refactoring in Flink 1.10, and we think it is now time >> > to leverage our newly introduced abstractions to implement a new resource >> > optimized scheduling strategy: Pipelined Region Scheduling. >> > >> > This scheduling strategy aims at: >> > >> > * avoidance of resource deadlocks when running batch jobs >> > >> > * tunable with respect to resource consumption and throughput >> > >> > More details can be found in the Wiki [2]. We are looking forward to your >> > feedback. >> > >> > Best, >> > >> > Zhu Zhu & Gary >> > >> > [1] https://issues.apache.org/jira/browse/FLINK-10429 >> > >> > [2] >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-119+Pipelined+Region+Scheduling
