This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 383f8bc [SPARK-58101] Remove misleading
`spark.submit.deployMode=cluster` from
`spark-connect-server-with-spark-cluster.yaml`
383f8bc is described below
commit 383f8bc9c3eceb6a2137d11d32659e614e781e54
Author: yalin.sahin <[email protected]>
AuthorDate: Mon Jul 13 09:15:52 2026 -0700
[SPARK-58101] Remove misleading `spark.submit.deployMode=cluster` from
`spark-connect-server-with-spark-cluster.yaml`
### What changes were proposed in this pull request?
Remove the `spark.submit.deployMode: "cluster"` line from
`examples/spark-connect-server-with-spark-cluster.yaml`.
### Why are the changes needed?
The example declares cluster deploy mode for a `SparkApplication` whose
main class is `org.apache.spark.sql.connect.service.SparkConnectServer`, but
the setting is silently ignored: the operator launches the main class inside
the driver pod it creates, so the server effectively runs in client mode
regardless (verified on-cluster with operator 0.9.0 / chart 1.7.0, Spark 4.1.2
— runtime behavior is identical with `deployMode` set to `cluster` and
`client`). Upstream `SparkSubmit` also e [...]
Note this is distinct from `examples/submit-spark-connect-server.sh`
(SPARK-51382), which submits via the standalone REST API on port 6066 and
bypasses the SparkSubmit-side check; that example is not touched.
See https://issues.apache.org/jira/browse/SPARK-58101.
### Does this PR introduce _any_ user-facing change?
No functional change — documentation/example fix only. The example now
reflects the actual runtime behavior.
### How was this patch tested?
Manually verified on a minikube cluster with spark-kubernetes-operator
0.9.0 (Helm chart 1.7.0) and Spark 4.1.2 attached to a standalone
`SparkCluster` master: applying the example with `deployMode` set to `cluster`
and to `client` produces identical behavior (Connect server runs in the
operator-managed driver pod in client mode in both cases).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)
Closes #753 from yalindogusahin/SPARK-58101.
Authored-by: yalin.sahin <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/spark-connect-server-with-spark-cluster.yaml | 1 -
1 file changed, 1 deletion(-)
diff --git a/examples/spark-connect-server-with-spark-cluster.yaml
b/examples/spark-connect-server-with-spark-cluster.yaml
index ee0988a..9f6655e 100644
--- a/examples/spark-connect-server-with-spark-cluster.yaml
+++ b/examples/spark-connect-server-with-spark-cluster.yaml
@@ -20,7 +20,6 @@ spec:
mainClass: "org.apache.spark.sql.connect.service.SparkConnectServer"
sparkConf:
spark.master: "spark://prod-master-svc:7077"
- spark.submit.deployMode: "cluster"
spark.executor.cores: "1"
spark.cores.max: "3"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
