holdenk opened a new pull request, #58619: URL: https://github.com/apache/spark/pull/58619
### What changes were proposed in this pull request? Nest executor local directories under <root>/<appId>/executor-* on the Worker, and add an opt-in config spark.shuffle.service.requireAppScopedLocalDirs (default false) that makes the external shuffle service reject executor registrations whose local directories do not lie inside the registering application's own per-application directory. The check is opt-in for rolling-upgrade compatibility: enable it only after every Worker is upgraded. The underlying LocalDirValidator already supported this mode (used by the YARN shuffle service); this wires standalone to use it. ### Why are the changes needed? Possible race condition. ### Does this PR introduce _any_ user-facing change? There is a new, administrator configurable, config flag. ### How was this patch tested? New test ### Was this patch authored or co-authored using generative AI tooling? Yes, claude and cursor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
