GitHub user ifilonenko opened a pull request:
https://github.com/apache/spark/pull/22777
[SPARK-25299][K8S][WIP] Enabling Remote Shuffle Backup
## What changes were proposed in this pull request?
TODO
## How was this patch tested?
TODO
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ifilonenko/spark SPARK-25299-experimental
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22777.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22777
----
commit 73d3ac7d0dd876d436aa5e6723dbd75952f5a93c
Author: mcheah <mcheah@...>
Date: 2018-10-02T23:21:59Z
Initial experiment for backing up shuffle files.
Instead of expecting the shuffle service to just pick up the data written
to local disk, executors can upload the data explicitly to the shuffle service.
Shuffle services do not need to be colocated with executors this way.
This can extend to arbitrary numbers of replicas.
commit 8281e1b7d736a72d1e3bbb9114136380bf5c4488
Author: mcheah <mcheah@...>
Date: 2018-10-10T16:54:14Z
merge conflicts
commit 1aa6d656e0e38099e576f8de4550c0d8fcace4ca
Author: Ilan Filonenko <ifilondz@...>
Date: 2018-10-19T22:44:25Z
fixing issue of client connections and refactoring for heartbeat
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
