[
https://issues.apache.org/jira/browse/FLINK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu reassigned FLINK-21332:
-------------------------------
Assignee: Zhilong Hong
> Optimize releasing result partitions in RegionPartitionReleaseStrategy
> ----------------------------------------------------------------------
>
> Key: FLINK-21332
> URL: https://issues.apache.org/jira/browse/FLINK-21332
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Zhilong Hong
> Assignee: Zhilong Hong
> Priority: Major
> Fix For: 1.13.0
>
>
> RegionPartitionReleaseStrategy is responsible for releasing result partitions
> when all the downstream tasks finish.
> The current implementation is:
> {code:java}
> for each consumed SchedulingResultPartition of current finished
> SchedulingPipelinedRegion:
> for each consumer SchedulingPipelinedRegion of the
> SchedulingResultPartition:
> if all the regions are finished:
> release the partitions
> {code}
> The time complexity of releasing a result partition is O(N^2). However,
> considering that during the entire stage, all the result partitions need to
> be released, the time complexity is actually O(N^3).
> After the optimization of DefaultSchedulingTopology, the consumed result
> partitions are grouped. Since the result partitions in one group are
> isomorphic, we can just cache the finished status of result partition groups
> and the corresponding pipeline regions.
> The optimized implementation is:
> {code:java}
> for each ConsumedPartitionGroup of current finished SchedulingPipelinedRegion:
> if all consumer SchedulingPipelinedRegion of the ConsumedPartitionGroup are
> finished:
> set the ConsumePartitionGroup to be fully consumed
> for result partition in the ConsumePartitionGroup:
> if all the ConsumePartitionGroups it belongs to are fully consumed:
> release the result partition
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
