spark git commit: [SPARK-4994][network]Cleanup removed executors' ShuffleInfo in yarn shuffle service
Repository: spark
Updated Branches:
refs/heads/master 2bda1c1d3 - 61073f832
[SPARK-4994][network]Cleanup removed executors' ShuffleInfo in yarn shuffle
service
when the application is completed, yarn's nodemanager can remove application's
local-dirs.but all executors' metadata of completed application havenot be
removed. now it lets yarn ShuffleService to have much more memory to store
Executors' ShuffleInfo. so these metadata need to be removed.
Author: lianhuiwang lianhuiwan...@gmail.com
Closes #3828 from lianhuiwang/SPARK-4994 and squashes the following commits:
f3ba1d2 [lianhuiwang] Cleanup removed executors' ShuffleInfo
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/61073f83
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/61073f83
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/61073f83
Branch: refs/heads/master
Commit: 61073f832128845a76469fc37376483b784c927b
Parents: 2bda1c1
Author: lianhuiwang lianhuiwan...@gmail.com
Authored: Fri Feb 6 14:47:52 2015 -0800
Committer: Andrew Or and...@databricks.com
Committed: Fri Feb 6 14:48:30 2015 -0800
--
.../org/apache/spark/network/yarn/YarnShuffleService.java | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/61073f83/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
--
diff --git
a/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
b/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
index a34aabe..63b2122 100644
---
a/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
+++
b/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
@@ -76,6 +76,9 @@ public class YarnShuffleService extends AuxiliaryService {
// The actual server that serves shuffle files
private TransportServer shuffleServer = null;
+ // Handles registering executors and opening shuffle blocks
+ private ExternalShuffleBlockHandler blockHandler;
+
public YarnShuffleService() {
super(spark_shuffle);
logger.info(Initializing YARN shuffle service for Spark);
@@ -99,7 +102,8 @@ public class YarnShuffleService extends AuxiliaryService {
// If authentication is enabled, set up the shuffle server to use a
// special RPC handler that filters out unauthenticated fetch requests
boolean authEnabled = conf.getBoolean(SPARK_AUTHENTICATE_KEY,
DEFAULT_SPARK_AUTHENTICATE);
-RpcHandler rpcHandler = new ExternalShuffleBlockHandler(transportConf);
+blockHandler = new ExternalShuffleBlockHandler(transportConf);
+RpcHandler rpcHandler = blockHandler;
if (authEnabled) {
secretManager = new ShuffleSecretManager();
rpcHandler = new SaslRpcHandler(rpcHandler, secretManager);
@@ -136,6 +140,7 @@ public class YarnShuffleService extends AuxiliaryService {
if (isAuthenticationEnabled()) {
secretManager.unregisterApp(appId);
}
+ blockHandler.applicationRemoved(appId, false /* clean up local dirs */);
} catch (Exception e) {
logger.error(Exception when stopping application {}, appId, e);
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-4994][network]Cleanup removed executors' ShuffleInfo in yarn shuffle service
Repository: spark
Updated Branches:
refs/heads/branch-1.3 caca15a4c - af6ddf8b6
[SPARK-4994][network]Cleanup removed executors' ShuffleInfo in yarn shuffle
service
when the application is completed, yarn's nodemanager can remove application's
local-dirs.but all executors' metadata of completed application havenot be
removed. now it lets yarn ShuffleService to have much more memory to store
Executors' ShuffleInfo. so these metadata need to be removed.
Author: lianhuiwang lianhuiwan...@gmail.com
Closes #3828 from lianhuiwang/SPARK-4994 and squashes the following commits:
f3ba1d2 [lianhuiwang] Cleanup removed executors' ShuffleInfo
(cherry picked from commit 61073f832128845a76469fc37376483b784c927b)
Signed-off-by: Andrew Or and...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/af6ddf8b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/af6ddf8b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/af6ddf8b
Branch: refs/heads/branch-1.3
Commit: af6ddf8b61c34259984386aa3415157e17bfd49a
Parents: caca15a
Author: lianhuiwang lianhuiwan...@gmail.com
Authored: Fri Feb 6 14:47:52 2015 -0800
Committer: Andrew Or and...@databricks.com
Committed: Fri Feb 6 14:48:46 2015 -0800
--
.../org/apache/spark/network/yarn/YarnShuffleService.java | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/af6ddf8b/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
--
diff --git
a/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
b/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
index a34aabe..63b2122 100644
---
a/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
+++
b/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
@@ -76,6 +76,9 @@ public class YarnShuffleService extends AuxiliaryService {
// The actual server that serves shuffle files
private TransportServer shuffleServer = null;
+ // Handles registering executors and opening shuffle blocks
+ private ExternalShuffleBlockHandler blockHandler;
+
public YarnShuffleService() {
super(spark_shuffle);
logger.info(Initializing YARN shuffle service for Spark);
@@ -99,7 +102,8 @@ public class YarnShuffleService extends AuxiliaryService {
// If authentication is enabled, set up the shuffle server to use a
// special RPC handler that filters out unauthenticated fetch requests
boolean authEnabled = conf.getBoolean(SPARK_AUTHENTICATE_KEY,
DEFAULT_SPARK_AUTHENTICATE);
-RpcHandler rpcHandler = new ExternalShuffleBlockHandler(transportConf);
+blockHandler = new ExternalShuffleBlockHandler(transportConf);
+RpcHandler rpcHandler = blockHandler;
if (authEnabled) {
secretManager = new ShuffleSecretManager();
rpcHandler = new SaslRpcHandler(rpcHandler, secretManager);
@@ -136,6 +140,7 @@ public class YarnShuffleService extends AuxiliaryService {
if (isAuthenticationEnabled()) {
secretManager.unregisterApp(appId);
}
+ blockHandler.applicationRemoved(appId, false /* clean up local dirs */);
} catch (Exception e) {
logger.error(Exception when stopping application {}, appId, e);
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
