This is an automated email from the ASF dual-hosted git repository. yikun pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark-docker.git
The following commit(s) were added to refs/heads/master by this push: new 006e8fa [SPARK-43793] Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug 006e8fa is described below commit 006e8fade69f148a05fc73f591f52c7678e48f04 Author: Yikun Jiang <yikunk...@gmail.com> AuthorDate: Thu May 25 19:05:26 2023 +0800 [SPARK-43793] Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug ### What changes were proposed in this pull request? In previous code, this is susceptible to a few bugs particularly around newlines in values. ``` env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txt ``` ### Why are the changes needed? To address DOI comments: https://github.com/docker-library/official-images/pull/13089#issuecomment-1533540388 , question 6. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? 1. Test mannully ``` export SPARK_JAVA_OPT_0="foo=bar" export SPARK_JAVA_OPT_1="foo1=bar1" for v in "${!SPARK_JAVA_OPT_}"; do SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" ) done for v in ${SPARK_EXECUTOR_JAVA_OPTS[]}; do echo $v done # foo=bar # foo1=bar1 ``` 2. CI passed Closes #42 from Yikun/SPARK-43793. Authored-by: Yikun Jiang <yikunk...@gmail.com> Signed-off-by: Yikun Jiang <yikunk...@gmail.com> --- 3.4.0/scala2.12-java11-ubuntu/entrypoint.sh | 5 +++-- entrypoint.sh.template | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh b/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh index 4bb1557..716f1af 100755 --- a/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh +++ b/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh @@ -38,8 +38,9 @@ if [ -z "$JAVA_HOME" ]; then fi SPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*" -env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt -readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txt +for v in "${!SPARK_JAVA_OPT_@}"; do + SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" ) +done if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH" diff --git a/entrypoint.sh.template b/entrypoint.sh.template index 4bb1557..716f1af 100644 --- a/entrypoint.sh.template +++ b/entrypoint.sh.template @@ -38,8 +38,9 @@ if [ -z "$JAVA_HOME" ]; then fi SPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*" -env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt -readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txt +for v in "${!SPARK_JAVA_OPT_@}"; do + SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" ) +done if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH" --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org