[GitHub] spark pull request #23243: add initRegisteredExecutorsDB
GitHub user weixiuli opened a pull request:
https://github.com/apache/spark/pull/23243
add initRegisteredExecutorsDB
## What changes were proposed in this pull request?
As we all know that spark on Yarn uses DB to record RegisteredExecutors
information, when the ExternalShuffleService restart and it can be reload,
which will be used as well .
While neither spark's standalone nor spark on k8s can record it's
RegisteredExecutors information
by db or other ,so when ExternalShuffleService restart ,which
RegisteredExecutors information will be lost,this is't what we want to.
This commit add initRegisteredExecutorsDB which can be used either
spark standalone or spark on k8s to record RegisteredExecutors information ,
when the ExternalShuffleService restart and it can be reload, which will
be used as well .
(Please fill in changes proposed in this fix)
## How was this patch tested?
test("test initRegisteredExecutorsDB ") {
val sparkConf = new SparkConf()
Utils.loadDefaultSparkProperties(sparkConf)
val securityManager = new SecurityManager(sparkConf)
sparkConf.set(config.SHUFFLE_SERVICE_DB_ENABLED.key, "true")
sparkConf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
sparkConf.set("spark.local.dir", "/tmp")
val externalShuffleService = new ExternalShuffleService(sparkConf,
securityManager)
}
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/weixiuli/spark branch-2.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23243.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23243
commit 3591c16dc758b762ead253be490a67748c33078a
Author: éç§å©
Date: 2018-12-06T06:20:43Z
add initRegisteredExecutorsDB
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23207: [SPARK-26193][SQL] Implement shuffle write metrics in SQ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23207 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99752/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23243: add initRegisteredExecutorsDB
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23243 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23243: add initRegisteredExecutorsDB
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23243 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23243: add initRegisteredExecutorsDB
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23243 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22305 I can help if this looks good to @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23241: [SPARK-26283][CORE] Enable reading from open frames of z...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23241 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23241: [SPARK-26283][CORE] Enable reading from open frames of z...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/23241 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23241: [SPARK-26283][CORE] Enable reading from open frames of z...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23241 **[Test build #99758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99758/testReport)** for PR 23241 at commit [`29f618e`](https://github.com/apache/spark/commit/29f618e682282f38ff56369eadab0baff3895180). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23241: [SPARK-26283][CORE] Enable reading from open frames of z...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23241 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5797/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23241: [SPARK-26283][CORE] Enable reading from open frames of z...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23241 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23243: [branch-2.4][ExternalShuffleService]add initRegisteredEx...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23243 Backport from which JIRA @weixiuli? Usually the fix should go to master first and it's backported to other branches when it's needed. If it should be fixed in master branch as well, let's file a JIRA and switch the branch to the master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredE...
Github user weixiuli commented on the issue: https://github.com/apache/spark/pull/23243 @HyukjinKwon OK ,thank you! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22141: [SPARK-25154][SQL] Support NOT IN sub-queries inside nes...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22141 @maropu Very sorry. I haven't had the time to come back to it. I have some stuff on my plate. I will get to this after i am done. Thanks !! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
