[ https://issues.apache.org/jira/browse/SPARK-47556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sundeep K updated SPARK-47556: ------------------------------ Description: h3. Issue: We noticed that sometimes K8s executor pods go in a crash loop. Reason being 'Error: MountVolume.SetUp failed for volume "spark-conf-volume-exec"'. Upon investigation we noticed that there are 2 spark jobs that launched with same application id and when one of them finishes first it deletes all it's resources and deletes the resources of other job too. -> Spark application ID is created using this [code|https://github.com/apache/spark/blob/36126a5c1821b4418afd5788963a939ea7f64078/core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala#L38] "spark-application-" + System.currentTimeMillis This means if 2 applications launch at the same milli second they could end up having same AppId -> [spark-app-selector|https://github.com/apache/spark/blob/93f98c0a61ddb66eb777c3940fbf29fc58e2d79b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L23] label is added to all resource created by driver and it's value is application Id. Kubernetes Scheduler deletes all the apps with same [label|https://github.com/apache/spark/blob/2a8bb5cdd3a5a2d63428b82df5e5066a805ce878/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L162C1-L172C6] upon termination. This results in deletion of config map and executor pods of job that's still running, driver tries to relaunch the executor pods, but config map is not present, so it's in crash loop h3. Context We are using [Spark of Kubernetes |https://spark.apache.org/docs/latest/running-on-kubernetes.html]and launch our spark jobs using PySpark. We launch multiple Spark Jobs within a given k8s namespace. Each Spark job can be launched from different pods or from different processes in a pod. Every time a job is launched it has a unique app name. Here is how the job is launched (omitting irrelevant details): {code:java} # spark_conf has settings required for spark on k8s sp = SparkSession.builder \ .config(conf=spark_conf) \ .appName('testapp') sp.master(f'k8s://{kubernetes_host}') session = sp.getOrCreate() with session: session.sql('SELECT 1'){code} h3. Repro Set same app id in spark config, run 2 different jobs, one that finishes fast, one that runs slow. Slower job goes into crash loop {code:java} "spark.app.id": "<same Id for 2 spark job>"{code} h3. Workaround Set unique spark.app.id for all the jobs that run on k8s eg: {code:java} "spark.app.id": f'{AppName}-{CurrTimeInMilliSecs}-{UUId}'[:63]{code} h3. Fix Add unique hash add the end of Application ID: [https://github.com/apache/spark/pull/45712] was: h3. Issue: We noticed that sometimes K8s executor pods go in a crash loop. Reason being 'Error: MountVolume.SetUp failed for volume "spark-conf-volume-exec"'. Upon investigation we noticed that there are 2 spark jobs that launched with same application id and when one of them finishes first it deletes all it's resources and deletes the resources of other job too. -> Spark application ID is created using this [code|https://affirm.slack.com/archives/C06Q2GWLWKH/p1711132115304449?thread_ts=1711123500.783909&cid=C06Q2GWLWKH] "spark-application-" + System.currentTimeMillis This means if 2 applications launch at the same milli second they could end up having same AppId -> [spark-app-selector|https://github.com/apache/spark/blob/93f98c0a61ddb66eb777c3940fbf29fc58e2d79b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L23] label is added to all resource created by driver and it's value is application Id. Kubernetes Scheduler deletes all the apps with same [label|https://github.com/apache/spark/blob/2a8bb5cdd3a5a2d63428b82df5e5066a805ce878/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L162C1-L172C6] upon termination. This results in deletion of config map and executor pods of job that's still running, driver tries to relaunch the executor pods, but config map is not present, so it's in crash loop h3. Context We are using [Spark of Kubernetes |https://spark.apache.org/docs/latest/running-on-kubernetes.html]and launch our spark jobs using PySpark. We launch multiple Spark Jobs within a given k8s namespace. Each Spark job can be launched from different pods or from different processes in a pod. Every time a job is launched it has a unique app name. Here is how the job is launched (omitting irrelevant details): {code:java} # spark_conf has settings required for spark on k8s sp = SparkSession.builder \ .config(conf=spark_conf) \ .appName('testapp') sp.master(f'k8s://{kubernetes_host}') session = sp.getOrCreate() with session: session.sql('SELECT 1'){code} h3. Repro Set same app id in spark config, run 2 different jobs, one that finishes fast, one that runs slow. Slower job goes into crash loop {code:java} "spark.app.id": "<same Id for 2 spark job>"{code} h3. Workaround Set unique spark.app.id for all the jobs that run on k8s eg: {code:java} "spark.app.id": f'{AppName}-{CurrTimeInMilliSecs}-{UUId}'[:63]{code} h3. Fix Add unique hash add the end of Application ID: [https://github.com/apache/spark/pull/45712] > [K8] Spark App ID collision resulting in deleting wrong resources > ----------------------------------------------------------------- > > Key: SPARK-47556 > URL: https://issues.apache.org/jira/browse/SPARK-47556 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core > Affects Versions: 3.1 > Reporter: Sundeep K > Priority: Major > Fix For: 3.3.0 > > > h3. Issue: > We noticed that sometimes K8s executor pods go in a crash loop. Reason being > 'Error: MountVolume.SetUp failed for volume "spark-conf-volume-exec"'. Upon > investigation we noticed that there are 2 spark jobs that launched with same > application id and when one of them finishes first it deletes all it's > resources and deletes the resources of other job too. > -> Spark application ID is created using this > [code|https://github.com/apache/spark/blob/36126a5c1821b4418afd5788963a939ea7f64078/core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala#L38] > "spark-application-" + System.currentTimeMillis > This means if 2 applications launch at the same milli second they could end > up having same AppId > -> > [spark-app-selector|https://github.com/apache/spark/blob/93f98c0a61ddb66eb777c3940fbf29fc58e2d79b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L23] > label is added to all resource created by driver and it's value is > application Id. Kubernetes Scheduler deletes all the apps with same > [label|https://github.com/apache/spark/blob/2a8bb5cdd3a5a2d63428b82df5e5066a805ce878/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L162C1-L172C6] > upon termination. > This results in deletion of config map and executor pods of job that's still > running, driver tries to relaunch the executor pods, but config map is not > present, so it's in crash loop > h3. Context > We are using [Spark of Kubernetes > |https://spark.apache.org/docs/latest/running-on-kubernetes.html]and launch > our spark jobs using PySpark. We launch multiple Spark Jobs within a given > k8s namespace. Each Spark job can be launched from different pods or from > different processes in a pod. Every time a job is launched it has a unique > app name. Here is how the job is launched (omitting irrelevant details): > {code:java} > # spark_conf has settings required for spark on k8s > sp = SparkSession.builder \ > .config(conf=spark_conf) \ > .appName('testapp') > sp.master(f'k8s://{kubernetes_host}') > session = sp.getOrCreate() > with session: > session.sql('SELECT 1'){code} > h3. Repro > Set same app id in spark config, run 2 different jobs, one that finishes > fast, one that runs slow. Slower job goes into crash loop > {code:java} > "spark.app.id": "<same Id for 2 spark job>"{code} > h3. Workaround > Set unique spark.app.id for all the jobs that run on k8s > eg: > {code:java} > "spark.app.id": f'{AppName}-{CurrTimeInMilliSecs}-{UUId}'[:63]{code} > h3. Fix > Add unique hash add the end of Application ID: > [https://github.com/apache/spark/pull/45712] > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org