Nicholas Barton created SPARK-58426:
---------------------------------------
Summary: Operator Helm chart passes `jvmArgs` as a single argv
element, so JVM options are ignored
Key: SPARK-58426
URL: https://issues.apache.org/jira/browse/SPARK-58426
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: kubernetes-operator-1.0.0, kubernetes-operator-0.9.0
Reporter: Nicholas Barton
h2. Summary:
Since SPARK-56107 removed {{{}docker-entrypoint.sh{}}}, the Helm chart passes
the whole {{operatorDeployment.operatorPod.operatorContainer.jvmArgs}} string
to the operator container as a [single command
element|https://github.com/apache/spark-kubernetes-operator/blob/7ccacb14c006cee191fcbb48e248ca4d1d291bd1/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml#L79]:
{code:java}
command: ["java", "-cp", "./spark-kubernetes-operator.jar", "$(LOG_CONFIG)",
"$(OPERATOR_JAVA_OPTS)", "org.apache.spark.k8s.operator.SparkOperator"]
{code}
Kubernetes {{$(VAR)}} expansion substitutes into a single argv element and
never performs whitespace splitting ({{{}ExpandContainerCommandAndArgs{}}}
calls {{expansion.Expand}} once per {{command}} entry and appends one string
per entry). The entire options string therefore arrives as one argument.
Because it begins with {{{}-D{}}}, the JVM parses it as a single system
property: it sets {{file.encoding}} to the whole string and silently ignores
the remaining options.
Per the Kubernetes API reference for
[Container.command|https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/],
the entrypoint array is "Not executed within a shell", and "Variable
references $(VAR_NAME) are expanded using the container's environment. If a
variable cannot be resolved, the reference in the input string is unchanged."
Expansion therefore substitutes into each array element individually and never
splits it on whitespace, so a multi-option jvmArgs string passed as one element
reaches the JVM as one argument.
There is no error, warning, or log line; the operator starts and runs normally.
With the shipped defaults, this means the {{{}UseParallelGC{}}},
{{{}InitialRAMPercentage{}}},
{{{}MaxRAMPercentage{}}}, {{{}AlwaysPreTouch{}}},
{{{}CrashOnOutOfMemoryError{}}}, {{{}ErrorFile{}}}, and
{{UseCompactObjectHeaders}} args all have no effect. The operator runs with the
JVM's default 25% {{MaxRAMPercentage}} instead of the intended 80%, and with
SerialGC rather than ParallelGC.
h2. Reproduction
Verified against a kind cluster (Kubernetes v1.36.1) with the unmodified chart
1.8.0 and default values, image {{apache/spark-kubernetes-operator:1.0.0}}
(Zulu JDK 26.0.1), operator container memory limit at its 2Gi default.
{code:java}
helm install op ./build-tools/helm/spark-kubernetes-operator -n spark
--create-namespace
POD=$(kubectl -n spark get pods -o jsonpath='{.items[0].metadata.name}')
kubectl -n spark exec $POD -- sh -c "tr '\0' '\n' < /proc/1/cmdline | cat -n"
{code}
Actual argv — note element 5:
{code:java}
1 java
2 -cp
3 ./spark-kubernetes-operator.jar
4 -Dlog4j.configurationFile=/opt/spark-operator/conf/log4j2.properties
5 -Dfile.encoding=UTF8 -XX:+CrashOnOutOfMemoryError
-XX:ErrorFile=/dev/stderr -XX:+UseParallelGC -XX:InitialRAMPercentage=80
-XX:MaxRAMPercentage=80 -XX:+AlwaysPreTouch -XX:+UseCompactObjectHeaders
6 org.apache.spark.k8s.operator.SparkOperator
{code}
The operator's own metrics endpoint can show the effect on the running JVM:
{code:java}
kubectl -n spark exec $POD -- sh -c \
'wget -qO- http://localhost:19090/prometheus | grep -E
"^jvm_memoryusage_heap_max|^jvm_gc_[a-z_]+_count"'
jvm_memoryusage_heap_max 536805376 # ~512Mi = 25% of the 2Gi limit, not
the intended 80%
jvm_gc_copy_count 24 # "Copy"/"MarkSweepCompact" =
SerialGC, not ParallelGC
{code}
The same check with each option as its own argv element gives
{{jvm_memoryusage_heap_max 1648361472}} (~1.57Gi) and
{{jvm_gc_ps_scavenge_count}} / {{jvm_gc_ps_marksweep_count}} (ParallelGC).
I also tested the repro using k3s/k3d and confirmed the same outcome.
h2. Proposed fix
Render each option as its own {{command}} element so no {{$(VAR)}} indirection
is involved.
h2. Disclosure
I used Claude Code (Opus 5) to investigate this issue and help put together the
report and repro.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
