vamsikarnika opened a new pull request, #19904:
URL: https://github.com/apache/hudi/pull/19904
### Describe the issue this Pull Request addresses
On Spark on Kubernetes, `FileIOUtils#getConfiguredLocalDirs()` only
special-cases
YARN's `LOCAL_DIRS` before falling back to `java.io.tmpdir`, i.e. `/tmp`
inside
the executor container. Spark itself already resolves a mounted scratch
directory on Kubernetes and publishes it via the `SPARK_LOCAL_DIRS`
environment
variable (`LocalDirsFeatureStep`), but Hudi never looks at it.
The practical effect: Spark shuffle correctly lands on a large mounted disk
via
`SPARK_LOCAL_DIRS`, while a Hudi spillable map (`ExternalSpillableMap`, used
by
merge/compaction/clustering paths) for the same executor silently spills to
the
small container root filesystem instead, since it resolves its default base
path from `java.io.tmpdir` rather than `SPARK_LOCAL_DIRS`.
### Summary and Changelog
Add a `SPARK_LOCAL_DIRS`-aware branch to `getConfiguredLocalDirs()`,
immediately
after the existing YARN check and before the `java.io.tmpdir` fallback:
```java
} else if (System.getenv("SPARK_LOCAL_DIRS") != null) {
return System.getenv("SPARK_LOCAL_DIRS").split(",");
}
```
YARN keeps precedence when both are set (this is a pure addition, not a
reordering). No engine dependency is added to `hudi-io`/`hudi-common`: this
reads a well-known environment variable by name, exactly as the existing
branch already does for YARN's `LOCAL_DIRS`. Flink and plain Java/standalone
callers are unaffected, since neither environment variable is set in those
contexts.
Deliberately not adding a writability check on the resolved directory:
`HoodieWriteConfig#getSpillableMapBasePath()` is resolved on every
`ExternalSpillableMap` construction — effectively per merge handle / per
partition on the write path, and uncached — so a filesystem stat here would
add a syscall on the hot path across potentially thousands of partitions. The
existing YARN branch performs no such validation either, so this keeps that
behavior consistent. If `SPARK_LOCAL_DIRS` is set but unwritable, Hudi now
fails loudly at spill time instead of silently falling back to `/tmp`.
Added `testGetConfiguredLocalDirsPrefersSparkLocalDirs` covering:
`SPARK_LOCAL_DIRS` alone, comma-separated multi-directory values, YARN taking
precedence when both are set, and the unchanged fallback when neither is set.
### Impact
No public API change. Behavior change only when
`hoodie.memory.spillable.map.path`
is unset (an advanced config with no default) and the process is running
under
Spark on Kubernetes: the effective spillable-map location moves from `/tmp`
to
whatever directory Spark's own Kubernetes local-dirs feature step publishes.
YARN and non-Spark engines see no behavior change.
### Risk Level
low - purely additive branch in an existing if/else chain; does not touch the
YARN or `java.io.tmpdir` fallback paths. The new branch is gated entirely on
an
environment variable that is unset for every engine and deployment mode
except
Spark on Kubernetes.
### Documentation Update
none - this changes only the default used when
`hoodie.memory.spillable.map.path`
is unset; that config's existing documentation is unaffected.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
