[ https://issues.apache.org/jira/browse/SPARK-33201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amandeep updated SPARK-33201: ----------------------------- Description: In MesosCluster mode, all of the spark jobs fail to run because by default "--py-files" is set to an empty string which causes spark-submit to use wrong jar name. This issue is caused by [Git commit | https://github.com/apache/spark/commit/38f030725c561979ca98b2a6cc7ca6c02a1f80ed] For ex: {quote}--total-executor-cores 2 --py-files --conf spark.driver.maxResultSize=15g {quote} so the very next --conf is used as the value for --py-files so `spark.driver.maxResultSize=15g` is used as jar name which causes error as below: {quote}20/10/19 20:19:18 WARN DependencyUtils: Local jar \{dir}/slaves/d9971b08-4929-4b00-9677-b14088c38603-S13/frameworks/86ff1f10-79fd-44f0-b807-71741091b457-0000/executors/driver-20201019200043-0014-retry-9/runs/3697919f-c3dc-4a13-8928-2fb3a20ac98d/spark.driver.maxResultSize=15g does not exist, skipping. 20/10/19 20:19:18 WARN SparkSubmit$$anon$2: Failed to load org.apache.spark.examples.SparkPi. java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi {quote} In order to reproduce the bug run any spark provided examples on mesos in cluster mode: {code:java} ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://<server>:7077 --deploy-mode cluster --conf spark.master.rest.enabled=true --executor-memory 5G --total-executor-cores 2 <home>/examples/jars/spark-examples_2.12-3.0.1.jar 100 {code} was: In MesosCluster mode, all of the spark jobs fail to run because by default "--py-files" is set to an empty string which causes spark-submit to use wrong jar name. This issue is caused by [https://github.com/apache/spark/commit/38f030725c561979ca98b2a6cc7ca6c02a1f80ed] For ex: {quote} --total-executor-cores 2 --py-files --conf spark.driver.maxResultSize=15g {quote} so the very next --conf is used as the value for --py-files so `spark.driver.maxResultSize=15g` is used as jar name which causes error as below: {quote} 20/10/19 20:19:18 WARN DependencyUtils: Local jar \{dir}/slaves/d9971b08-4929-4b00-9677-b14088c38603-S13/frameworks/86ff1f10-79fd-44f0-b807-71741091b457-0000/executors/driver-20201019200043-0014-retry-9/runs/3697919f-c3dc-4a13-8928-2fb3a20ac98d/spark.driver.maxResultSize=15g does not exist, skipping. 20/10/19 20:19:18 WARN SparkSubmit$$anon$2: Failed to load org.apache.spark.examples.SparkPi. java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi {quote} In order to reproduce the bug run any spark provided examples on mesos in cluster mode: {code} ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://<server>:7077 --deploy-mode cluster --conf spark.master.rest.enabled=true --executor-memory 5G --total-executor-cores 2 <home>/examples/jars/spark-examples_2.12-3.0.1.jar 100 {code} > Mesos dispatcher service is not working due to empty --pyFiles conf in > cluster mode which is the default > -------------------------------------------------------------------------------------------------------- > > Key: SPARK-33201 > URL: https://issues.apache.org/jira/browse/SPARK-33201 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 3.0.0, 3.0.1 > Reporter: Amandeep > Priority: Major > > In MesosCluster mode, all of the spark jobs fail to run because by default > "--py-files" is set to an empty string which causes spark-submit to use wrong > jar name. This issue is caused by [Git commit | > https://github.com/apache/spark/commit/38f030725c561979ca98b2a6cc7ca6c02a1f80ed] > For ex: > {quote}--total-executor-cores > 2 > --py-files > --conf > spark.driver.maxResultSize=15g > {quote} > so the very next --conf is used as the value for --py-files so > `spark.driver.maxResultSize=15g` is used as jar name > which causes error as below: > {quote}20/10/19 20:19:18 WARN DependencyUtils: Local jar > \{dir}/slaves/d9971b08-4929-4b00-9677-b14088c38603-S13/frameworks/86ff1f10-79fd-44f0-b807-71741091b457-0000/executors/driver-20201019200043-0014-retry-9/runs/3697919f-c3dc-4a13-8928-2fb3a20ac98d/spark.driver.maxResultSize=15g > does not exist, skipping. > 20/10/19 20:19:18 WARN SparkSubmit$$anon$2: Failed to load > org.apache.spark.examples.SparkPi. > java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi > {quote} > In order to reproduce the bug run any spark provided examples on mesos in > cluster mode: > {code:java} > ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master > mesos://<server>:7077 --deploy-mode cluster --conf > spark.master.rest.enabled=true --executor-memory 5G --total-executor-cores 2 > <home>/examples/jars/spark-examples_2.12-3.0.1.jar 100 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org