Yang Jie created SPARK-58693:
--------------------------------
Summary: Do not abort K8s shuffle data recovery when a directory
listing returns null
Key: SPARK-58693
URL: https://issues.apache.org/jira/browse/SPARK-58693
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 5.0.0
Reporter: Yang Jie
`KubernetesLocalDiskShuffleExecutorComponents.recoverDiskStore` collects
candidate files with a strict `Array.map`/`flatMap` chain that never handles
`File.listFiles` returning null. A single bad entry throws,
`initializeExecutor` swallows the exception via `Utils.tryLogNonFatalError`,
and the executor starts with nothing recovered.
Two reachable cases:
1. A configured local directory with fewer than three path components. The scan
walks two levels up with `new File(new File(new File(s).getParent).getParent)`;
for `/data` the second `getParent` returns null and the `File` constructor
throws. `/data` is the mount path used by the Local Storage example in
`running-on-kubernetes.md`, two blocks above the recovery instructions. Since
`map` is strict, one such entry also discards every correctly nested directory
configured alongside it. An empty string behaves the same way, and the existing
`filter(_ != null)` does not stop it because `String.split(",")` yields `""`
rather than null.
2. A directory in the walk whose `listFiles()` returns null: removed while the
walk is in progress, or present but not readable by the executor uid.
The only null check in the method, `if (files != null)` applied to the result
of `Array.flatMap`, can never fire, so the walk reads as null-safe when it is
not.
In both cases the executor logs one NPE stack trace that does not mention
shuffle recovery, and the job recomputes every map output on the reused PVC.
`SPARK-40459` already established that a single file should not block the rest
of the recovery; the collection phase is still all-or-nothing.
Fix: route every listing through a helper that logs and skips when `listFiles`
returns null, tolerate a shallow local directory by skipping it with a warning,
and drop the dead guard. This mirrors what `SPARK-57530` did for
`SparkFileUtils.recursiveList`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
