GitHub user weixiuli opened a pull request: https://github.com/apache/spark/pull/23243
add initRegisteredExecutorsDB ## What changes were proposed in this pull request? As we all know that spark on Yarn uses DB to record RegisteredExecutors information, when the ExternalShuffleService restart and it can be reload, which will be used as well . While neither spark's standalone nor spark on k8s can record it's RegisteredExecutors information by db or other ,so when ExternalShuffleService restart ,which RegisteredExecutors information will be lost,this is't what we want to. This commit add initRegisteredExecutorsDB which can be used either spark standalone or spark on k8s to record RegisteredExecutors information , when the ExternalShuffleService restart and it can be reload, which will be used as well . (Please fill in changes proposed in this fix) ## How was this patch tested? test("test initRegisteredExecutorsDB ") { val sparkConf = new SparkConf() Utils.loadDefaultSparkProperties(sparkConf) val securityManager = new SecurityManager(sparkConf) sparkConf.set(config.SHUFFLE_SERVICE_DB_ENABLED.key, "true") sparkConf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true") sparkConf.set("spark.local.dir", "/tmp") val externalShuffleService = new ExternalShuffleService(sparkConf, securityManager) } (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/weixiuli/spark branch-2.4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23243.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23243 ---- commit 3591c16dc758b762ead253be490a67748c33078a Author: éç§å© <weixiuli@...> Date: 2018-12-06T06:20:43Z add initRegisteredExecutorsDB ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org