dongjoon-hyun commented on PR #58619:
URL: https://github.com/apache/spark/pull/58619#issuecomment-5733884489
A few additional comments.
- The latest CI run has new `core` failures in `local-cluster` suites, while
recent `master` commits are green:
- `WorkerDecommissionExtendedSuite`: "Decommission 2 executors from 3
executors in total"
- `BlockManagerDecommissionIntegrationSuite`: "SPARK-46957: Migrated
shuffle files should be able to cleanup from executor"
- `CoarseGrainedSchedulerBackendSuite`: "compute max number of concurrent
tasks can be launched" (and "... when some executors are busy")
Some of them fail after a ~1 minute timeout, which may mean executors
didn't come up. Could you check whether they are related to this PR (e.g., the
`<appId>` directory shared by multiple Workers in the same JVM in
`local-cluster` mode)?
- `spark.shuffle.service.requireAppScopedLocalDirs` is only checked at
`RegisterExecutor`. When `spark.shuffle.service.db.enabled=true`, executors
recovered via `reloadRegisteredExecutors` after a shuffle service restart are
not re-validated, so enabling the config and restarting the service still
serves the existing unscoped registrations. Could you mention this in the
config doc and `docs/spark-standalone.md`?
- nit: The actual layout is `<root>/spark-*/<appId>/executor-*` because
`Utils.getOrCreateLocalRootDirs` creates a `spark-*` directory under each
configured root. The docs and test comments say `<root>/<appId>/executor-*`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
