GitHub user weixiuli opened a pull request:
https://github.com/apache/spark/pull/23243
add initRegisteredExecutorsDB
## What changes were proposed in this pull request?
As we all know that spark on Yarn uses DB to record RegisteredExecutors
information, when the ExternalShuffleService restart and it can be reload,
which will be used as well .
While neither spark's standalone nor spark on k8s can record it's
RegisteredExecutors information
by db or other ,so when ExternalShuffleService restart ,which
RegisteredExecutors information will be lost,this is't what we want to.
This commit add initRegisteredExecutorsDB which can be used either
spark standalone or spark on k8s to record RegisteredExecutors information ,
when the ExternalShuffleService restart and it can be reload, which will
be used as well .
(Please fill in changes proposed in this fix)
## How was this patch tested?
test("test initRegisteredExecutorsDB ") {
val sparkConf = new SparkConf()
Utils.loadDefaultSparkProperties(sparkConf)
val securityManager = new SecurityManager(sparkConf)
sparkConf.set(config.SHUFFLE_SERVICE_DB_ENABLED.key, "true")
sparkConf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
sparkConf.set("spark.local.dir", "/tmp")
val externalShuffleService = new ExternalShuffleService(sparkConf,
securityManager)
}
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/weixiuli/spark branch-2.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23243.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23243
----
commit 3591c16dc758b762ead253be490a67748c33078a
Author: éç§å© <weixiuli@...>
Date: 2018-12-06T06:20:43Z
add initRegisteredExecutorsDB
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
