[
https://issues.apache.org/jira/browse/SPARK-58693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie resolved SPARK-58693.
------------------------------
Target Version/s: 4.3.0, 4.2.1
Assignee: Yang Jie
Resolution: Fixed
Issue resolved by pull request 57905
https://github.com/apache/spark/pull/57905
> Do not abort K8s shuffle data recovery when a directory listing returns null
> ----------------------------------------------------------------------------
>
> Key: SPARK-58693
> URL: https://issues.apache.org/jira/browse/SPARK-58693
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Assignee: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> `KubernetesLocalDiskShuffleExecutorComponents.recoverDiskStore` collects
> candidate files with a strict `Array.map`/`flatMap` chain that never handles
> `File.listFiles` returning null. A single bad entry throws,
> `initializeExecutor` swallows the exception via `Utils.tryLogNonFatalError`,
> and the executor starts with nothing recovered.
> Two reachable cases:
> 1. A configured local directory with fewer than three path components. The
> scan walks two levels up with `new File(new File(new
> File(s).getParent).getParent)`; for `/data` the second `getParent` returns
> null and the `File` constructor throws. `/data` is the mount path used by the
> Local Storage example in `running-on-kubernetes.md`, two blocks above the
> recovery instructions. Since `map` is strict, one such entry also discards
> every correctly nested directory configured alongside it. An empty string
> behaves the same way, and the existing `filter(_ != null)` does not stop it
> because `String.split(",")` yields `""` rather than null.
> 2. A directory in the walk whose `listFiles()` returns null: removed while
> the walk is in progress, or present but not readable by the executor uid.
> The only null check in the method, `if (files != null)` applied to the result
> of `Array.flatMap`, can never fire, so the walk reads as null-safe when it is
> not.
> In both cases the executor logs one NPE stack trace that does not mention
> shuffle recovery, and the job recomputes every map output on the reused PVC.
> `SPARK-40459` already established that a single file should not block the
> rest of the recovery; the collection phase is still all-or-nothing.
> Fix: route every listing through a helper that logs and skips when
> `listFiles` returns null, tolerate a shallow local directory by skipping it
> with a warning, and drop the dead guard. This mirrors what `SPARK-57530` did
> for `SparkFileUtils.recursiveList`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
