[
https://issues.apache.org/jira/browse/SPARK-58426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100202#comment-18100202
]
Nicholas Barton commented on SPARK-58426:
-----------------------------------------
I filed
[apache/spark-kubernetes-operator#778|https://github.com/apache/spark-kubernetes-operator/pull/778]
with a proposed fix
> Operator Helm chart passes `jvmArgs` as a single argv element, so JVM options
> are ignored
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-58426
> URL: https://issues.apache.org/jira/browse/SPARK-58426
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: kubernetes-operator-0.9.0, kubernetes-operator-1.0.0
> Reporter: Nicholas Barton
> Priority: Major
> Labels: pull-request-available
>
> h2. Summary:
> Since SPARK-56107 removed {{{}docker-entrypoint.sh{}}}, the Helm chart passes
> the whole {{operatorDeployment.operatorPod.operatorContainer.jvmArgs}} string
> to the operator container as a [single command
> element|https://github.com/apache/spark-kubernetes-operator/blob/7ccacb14c006cee191fcbb48e248ca4d1d291bd1/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml#L79]:
> {code:java}
> command: ["java", "-cp", "./spark-kubernetes-operator.jar", "$(LOG_CONFIG)",
> "$(OPERATOR_JAVA_OPTS)", "org.apache.spark.k8s.operator.SparkOperator"]
> {code}
> Kubernetes {{$(VAR)}} expansion substitutes into a single argv element and
> never performs whitespace splitting ({{{}ExpandContainerCommandAndArgs{}}}
> calls {{expansion.Expand}} once per {{command}} entry and appends one string
> per entry). The entire options string therefore arrives as one argument.
> Because it begins with {{{}-D{}}}, the JVM parses it as a single system
> property: it sets {{file.encoding}} to the whole string and silently ignores
> the remaining options.
> Per the Kubernetes API reference for
> [Container.command|https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/],
> the entrypoint array is "Not executed within a shell", and "Variable
> references $(VAR_NAME) are expanded using the container's environment. If a
> variable cannot be resolved, the reference in the input string is unchanged."
> Expansion therefore substitutes into each array element individually and
> never splits it on whitespace, so a multi-option jvmArgs string passed as one
> element reaches the JVM as one argument.
> There is no error, warning, or log line; the operator starts and runs
> normally.
> With the shipped defaults, this means the {{{}UseParallelGC{}}},
> {{{}InitialRAMPercentage{}}},
> {{{}MaxRAMPercentage{}}}, {{{}AlwaysPreTouch{}}},
> {{{}CrashOnOutOfMemoryError{}}}, {{{}ErrorFile{}}}, and
> {{UseCompactObjectHeaders}} args all have no effect. The operator runs with
> the JVM's default 25% {{MaxRAMPercentage}} instead of the intended 80%, and
> with SerialGC rather than ParallelGC.
> h2. Reproduction
> Verified against a kind cluster (Kubernetes v1.36.1) with the unmodified
> chart 1.8.0 and default values, image
> {{apache/spark-kubernetes-operator:1.0.0}} (Zulu JDK 26.0.1), operator
> container memory limit at its 2Gi default.
> {code:java}
> helm install op ./build-tools/helm/spark-kubernetes-operator -n spark
> --create-namespace
> POD=$(kubectl -n spark get pods -o jsonpath='{.items[0].metadata.name}')
> kubectl -n spark exec $POD -- sh -c "tr '\0' '\n' < /proc/1/cmdline | cat -n"
> {code}
> Actual argv — note element 5:
> {code:java}
> 1 java
> 2 -cp
> 3 ./spark-kubernetes-operator.jar
> 4 -Dlog4j.configurationFile=/opt/spark-operator/conf/log4j2.properties
> 5 -Dfile.encoding=UTF8 -XX:+CrashOnOutOfMemoryError
> -XX:ErrorFile=/dev/stderr -XX:+UseParallelGC -XX:InitialRAMPercentage=80
> -XX:MaxRAMPercentage=80 -XX:+AlwaysPreTouch -XX:+UseCompactObjectHeaders
> 6 org.apache.spark.k8s.operator.SparkOperator
> {code}
> The operator's own metrics endpoint can show the effect on the running JVM:
> {code:java}
> kubectl -n spark exec $POD -- sh -c \
> 'wget -qO- http://localhost:19090/prometheus | grep -E
> "^jvm_memoryusage_heap_max|^jvm_gc_[a-z_]+_count"'
> jvm_memoryusage_heap_max 536805376 # ~512Mi = 25% of the 2Gi limit,
> not the intended 80%
> jvm_gc_copy_count 24 # "Copy"/"MarkSweepCompact" =
> SerialGC, not ParallelGC
> {code}
> The same check with each option as its own argv element gives
> {{jvm_memoryusage_heap_max 1648361472}} (~1.57Gi) and
> {{jvm_gc_ps_scavenge_count}} / {{jvm_gc_ps_marksweep_count}} (ParallelGC).
> I also tested the repro using k3s/k3d and confirmed the same outcome.
> h2. Proposed fix
> Render each option as its own {{command}} element so no {{$(VAR)}}
> indirection is involved.
> h2. Disclosure
> I used Claude Code (Opus 5) to investigate this issue and help put together
> the report and repro.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
