Hi, I am not using Airflow but I assume your application is deployed in cluster mode and in this case the class you are looking for is *org.apache.spark.deploy.k8s.submit.Client* [1].
If we are talking about the first "spark-submit" used to start the application and not "spark-submit --status" then it contains loop where the application status is logged. This loop stops when the *LoggingPodStatusWatcher* reports the app is completed [2] or when "spark.kubernetes.submission.waitAppCompletion" [3] is false. And you are right the monitoring (POD state watching) is done via REST (HTTPS) and should be detected by "io.fabric8.kubernetes.client.Watcher.onClose()" method so by the kubernetes client. I hope this helps. Some further questions if you need some more help: 1. What is the Spark version you are running? 2. Does it contain SPARK-24266 [4]? 3. If yes can you reproduce the issue without airflow and do you have the logs about the issue? Best regards, Attila [1] https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L88-L103 [2] https://github.com/apache/spark/blob/8604db28b87b387bbdb3761df85fae292cd402a1/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L162-L166 [3] https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala#L112-L114 [4] https://issues.apache.org/jira/browse/SPARK-24266 -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org