HyukjinKwon commented on code in PR #58619:
URL: https://github.com/apache/spark/pull/58619#discussion_r3985794979
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -908,6 +908,22 @@ package object config {
.stringConf
.createWithDefault("spark_shuffle")
+ private[spark] val SHUFFLE_SERVICE_REQUIRE_APP_SCOPED_LOCAL_DIRS =
+ ConfigBuilder("spark.shuffle.service.requireAppScopedLocalDirs")
+ .doc("Whether the external shuffle service requires every local
directory an executor " +
+ "reports at registration to lie inside the registering application's
own " +
+ "per-application directory: the application id must appear as a path
segment of the " +
+ "directory's canonical path, under one of the service's configured
local directory " +
+ "roots. This keeps each application's shuffle and RDD blocks within
its own directory " +
+ "scope at registration and cleanup time. Only affects standalone mode.
Enable it only " +
+ "after every Worker in the cluster creates executor local directories
under a " +
+ "per-application directory (Workers on this version do); executors
launched by older " +
+ "Workers report unscoped paths and their registrations are rejected
while this is " +
+ "enabled.")
+ .version("4.4.0")
Review Comment:
4.3.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
