This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 24cfbbc12530 [SPARK-57945][K8S] Avoid persisting Java options to disk
in `entrypoint.sh`
24cfbbc12530 is described below
commit 24cfbbc125300c773371998431b87787fb28a7b0
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Jul 5 20:21:47 2026 -0700
[SPARK-57945][K8S] Avoid persisting Java options to disk in `entrypoint.sh`
### What changes were proposed in this pull request?
This PR removes the temporary file `java_opts.txt` from the K8s container
`entrypoint.sh` by reading `SPARK_JAVA_OPT_*` environment variables directly
into the array via process substitution, which works in both `bash` and `zsh`.
### Why are the changes needed?
The `java_opts.txt` file was introduced at Apache Spark 2.3.0.
- https://github.com/apache/spark/pull/20192
`SPARK_JAVA_OPT_*` carries `spark.executor.extraJavaOptions` verbatim,
which may contain sensitive values (e.g., keystore passwords). The file was
written in plaintext to the working directory and never deleted, so it could
outlive the process on persistent volumes (e.g., PV/hostPath).
Note that the scope of this PR is only preventing a persistent files. The
pods still have `env` variables .
### Does this PR introduce _any_ user-facing change?
No, the resulting Java options are identical.
### How was this patch tested?
Manually verified in both `bash` and `zsh` that options with spaces, the
empty case under `set -e`, and the numeric ordering are preserved.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57019 from dongjoon-hyun/SPARK-57945.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit f5227a64c2f3b9a6e53098b9d1a2ae425fa3f341)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
index f9561b9aa4ed..a2ae7e356694 100755
---
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
+++
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
@@ -41,12 +41,10 @@ if [ -z "$JAVA_HOME" ]; then
fi
SPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*"
-env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' >
java_opts.txt
-if [ "$(command -v readarray)" ]; then
- readarray -t SPARK_EXECUTOR_JAVA_OPTS < java_opts.txt
-else
- SPARK_EXECUTOR_JAVA_OPTS=("${(@f)$(< java_opts.txt)}")
-fi
+SPARK_EXECUTOR_JAVA_OPTS=()
+while IFS= read -r opt; do
+ SPARK_EXECUTOR_JAVA_OPTS+=("$opt")
+done < <(env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed
's/[^=]*=\(.*\)/\1/g')
if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then
SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
