[jira] [Updated] (FLINK-10653) Introduce Pluggable Shuffle Service Architecture
[ https://issues.apache.org/jira/browse/FLINK-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-10653: - Summary: Introduce Pluggable Shuffle Service Architecture (was: Introduce Pluggable Shuffle Manager Architecture) > Introduce Pluggable Shuffle Service Architecture > > > Key: FLINK-10653 > URL: https://issues.apache.org/jira/browse/FLINK-10653 > Project: Flink > Issue Type: New Feature > Components: Runtime / Network >Reporter: zhijiang >Assignee: zhijiang >Priority: Major > > This is the umbrella issue for improving shuffle architecture. > Shuffle is the process of data transfer between stages, which involves in > writing outputs on sender side and reading data on receiver side. In flink > implementation, it covers three parts of writer, transport layer and reader > separately which are uniformed for both streaming and batch jobs. > In detail, the current ResultPartitionWriter interface on upstream side only > supports in-memory outputs for streaming job and local persistent file > outputs for batch job. If we extend to implement another writer such as > DfsWriter, RdmaWriter, SortMergeWriter, etc based on ResultPartitionWriter > interface, it has not the unified mechanism to extend the reader side > accordingly. > In order to make the shuffle architecture more flexible and support more > scenarios especially for batch jobs, a high level shuffle architecture is > necessary to manage and extend both writer and reader sides together. > Refer to the design doc for more details. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-10653) Introduce Pluggable Shuffle Service Architecture
[ https://issues.apache.org/jira/browse/FLINK-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijiang updated FLINK-10653: - Fix Version/s: 1.9.0 > Introduce Pluggable Shuffle Service Architecture > > > Key: FLINK-10653 > URL: https://issues.apache.org/jira/browse/FLINK-10653 > Project: Flink > Issue Type: New Feature > Components: Runtime / Network >Reporter: Zhijiang >Assignee: Zhijiang >Priority: Major > Fix For: 1.9.0 > > > This is the umbrella issue for improving shuffle architecture. > Shuffle is the process of data transfer between stages, which involves in > writing outputs on sender side and reading data on receiver side. In flink > implementation, it covers three parts of writer, transport layer and reader > separately which are uniformed for both streaming and batch jobs. > In detail, the current ResultPartitionWriter interface on upstream side only > supports in-memory outputs for streaming job and local persistent file > outputs for batch job. If we extend to implement another writer such as > DfsWriter, RdmaWriter, SortMergeWriter, etc based on ResultPartitionWriter > interface, it has not the unified mechanism to extend the reader side > accordingly. > In order to make the shuffle architecture more flexible and support more > scenarios especially for batch jobs, a high level shuffle architecture is > necessary to manage and extend both writer and reader sides together. > Refer to the design doc for more details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
