[
https://issues.apache.org/jira/browse/SPARK-27426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999035#comment-16999035
]
jugosag commented on SPARK-27426:
---------------------------------
We are observing the exact same problem on Windows with Spark 2.4.4 (patched to
resolve SPARK-28921). It works when running in local mode.
>From what we understand about the mechanism in place is that SparkLauncher
>opens a server port on which someone (SparkSubmit?) should report back the
>state of the application based on which the SparkAppHandle info is then
>updated.
When running in local mode, we see such a connection from SparkSubmit to our
application, but nothing when running the application in k8s mode. Is this just
not expected to work for K8s (a.k.a. missing feature) or is this a defect?
> SparkAppHandle states not getting updated in Kubernetes
> -------------------------------------------------------
>
> Key: SPARK-27426
> URL: https://issues.apache.org/jira/browse/SPARK-27426
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.0
> Environment: # Cent os 7
> # Kubernetes 1.14
> # Spark 2.4.0
> Reporter: Nishant Ranjan
> Priority: Major
> Labels: SparkAppHandle, kubernetes, sparklauncher
>
> While launching Spark application through "startApplication()" ,
> SparkAppHandle state is not getting updated.
> sparkLaunch = new SparkLauncher()
> .setSparkHome("/root/test/spark-2.4.0-bin-hadoop2.7")
> .setMaster("k8s://https://172.16.23.30:6443";)
> .setVerbose(true)
> .addSparkArg("--verbose")
>
> .setAppResource("local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar")
> .setConf("spark.app.name","spark-pi")
> .setMainClass("org.apache.spark.examples.SparkPi")
> .setConf("spark.executor.instances","5")
> .setConf("spark.kubernetes.container.image","registry.renovite.com/spark:v2")
> .setConf("spark.kubernetes.driver.pod.name","spark-pi-driver")
> .setConf("spark.kubernetes.container.image.pullSecrets","dev-registry-key")
> .setConf("spark.kubernetes.authenticate.driver.serviceAccountName","spark")
> .setDeployMode("cluster")
> ;
> SparkAppHandle handle = sparkLaunch.startApplication();
> Observations:
> # Now, I tried listeners etc but handle.getState() returns UNKNOWN and when
> Spark application is completed. state changes to LOST.
> # SparkAppHandle is not null
> # handle.getAppId() is always null.
> My best guess is that communication is not working properly between listener
> and Spark driver in Kubernetes.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
