GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/21293
[SPARK-24237][SS] Continuous shuffle dependency and map output tracker
## What changes were proposed in this pull request?
As our disscussion in [jira
comment](https://issues.apache.org/jira/browse/SPARK-24036?focusedCommentId=16470067&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16470067)
and [design
comment](https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE/edit?disco=AAAAB4X1H_E)
and [design
doc](https://docs.google.com/document/d/14cGJ75v9myznywtB35ytEqL9wHy9xfZRv06B6g2tUgI/edit#bookmark=id.2lfv2glj7ny0),
this pr including the following changes:
1. Add ContinuousShuffleDependency support, which can allow shuffleId
generated from epoch and re-register shuffleHandle with new shuffleId.
2. Add ContinuousMapOutputTrackerWorker, which can get shuffle status by a
blocking way, and support register shuffle\map ouput in tracker worker.
3. Add ContinuousShuffleMapTask.
## How was this patch tested?
Add a new UT for ContinuousMapOutputTracker.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuanyuanking/spark SPARK-24237
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21293.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21293
----
commit fbcc88bfb0d2fb6dbaae9664d6f0852b71e64f2b
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-10T09:18:16Z
commit for continuous map output tracker
commit 44ae9d917c354d780071a8e112a118674865143d
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-04T03:58:06Z
INF-SPARK-1382: Continuous shuffle map task implementation and output
trackder support
commit af2d60854856e669f40a03b76fffe02dac7b79c2
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-10T13:23:39Z
Address comments
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
