Jonathan Lafleche created SPARK-35174:
-----------------------------------------
Summary: Avoid opening watch when waitAppCompletion is false
Key: SPARK-35174
URL: https://issues.apache.org/jira/browse/SPARK-35174
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 3.1.1
Reporter: Jonathan Lafleche
In spark-submit, we currently [open a pod watch for any spark
submission|https://github.com/apache/spark/blame/0494dc90af48ce7da0625485a4dc6917a244d580/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L150-L167].
If WAIT_FOR_APP_COMPLETION is false, we then immediately ignore the result of
the watcher and break out of the watcher.
When submitting spark applications at scale, this is a source of operational
pain, since opening the watch relies on opening a websocket, which tends to run
into subtle networking issues around negotiating the websocket connection.
I'd like to change this behaviour so that we eagerly check whether we are
waiting on app completion, and avoid opening the watch altogether when
WAIT_FOR_APP_COMPLETION is false.
Would you accept a contribution for that change, or are there any concerns I've
overlooked?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
