GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/21353
[SPARK-24036][SS] Scheduler changes for continuous processing shuffle
support
## What changes were proposed in this pull request?
This is the last part of the preview PRs, the mainly change is in
DAGScheduler, we added a interface to submit whole job at once.
In order to work and compile properly, this also including several parts in
#21332 and #21293. I will marked the duplicated part for convenient review.
Sorry about this...
## How was this patch tested?
Added new UT
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuanyuanking/spark SPARK-24304
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21353.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21353
----
commit fbcc88bfb0d2fb6dbaae9664d6f0852b71e64f2b
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-10T09:18:16Z
commit for continuous map output tracker
commit 44ae9d917c354d780071a8e112a118674865143d
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-04T03:58:06Z
INF-SPARK-1382: Continuous shuffle map task implementation and output
trackder support
commit af2d60854856e669f40a03b76fffe02dac7b79c2
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-10T13:23:39Z
Address comments
commit 56442dc1c7450518d9bc84b4bfeddb017daa967b
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-11T12:12:29Z
Fix mima test
commit 2ac9980f30b8b50809aa780035281f6a62ad9573
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-17T13:01:33Z
add inteface to submit all stages of one job in DAGScheduler
commit 0f070fc5b26a3e2d5c30fdd7d59c4dd8896255ac
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-17T13:05:29Z
fix
commit 42e2a63f68ba12264f001ed53ff33c27c1287de8
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-05-17T13:36:32Z
Merge SPARK-24304
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
