[
https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324189#comment-17324189
]
Jiayi Liao commented on FLINK-13247:
------------------------------------
Actually we've already implemented Yarn shuffle service in our internal
version, and also can be contributed to Flink community. I can summarize our
implementation and write a detailed design before 5th May, if the community
still needs this. What do you think [~zjwang]?
> Implement external shuffle service for YARN
> -------------------------------------------
>
> Key: FLINK-13247
> URL: https://issues.apache.org/jira/browse/FLINK-13247
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Reporter: MalcolmSanders
> Priority: Minor
>
> Flink batch job users could achieve better cluster utilization and job
> throughput throught external shuffle service because the producers of
> intermedia result partitions can be released once intermedia result
> partitions have been persisted on disks. In
> [FLINK-10653|https://issues.apache.org/jira/browse/FLINK-10653], [~zjwang]
> has introduced pluggable shuffle manager architecture which abstracts the
> process of data transfer between stages from flink runtime as shuffle
> service. I propose to YARN implementation for flink external shuffle service
> since YARN is widely used in various companies.
> The basic idea is as follows:
> (1) Producers write intermedia result partitions to local disks assigned by
> NodeManager;
> (2) Yarn shuffle servers, deployed on each NodeManager as an auxiliary
> service, are acknowledged of intermedia result partition descriptions by
> producers;
> (3) Consumers fetch intermedia result partition from yarn shuffle servers;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
