[ https://issues.apache.org/jira/browse/SPARK-32675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun reassigned SPARK-32675: ------------------------------------- Assignee: Farhan Khan > --py-files option is appended without passing value for it > ---------------------------------------------------------- > > Key: SPARK-32675 > URL: https://issues.apache.org/jira/browse/SPARK-32675 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 3.0.0 > Reporter: Farhan Khan > Assignee: Farhan Khan > Priority: Major > > Submitted application passing --py-files option in a hardcoded manner for a > Mesos Cluster in cluster mode using REST Submission API. It is causing a > simple Java-based SparkPi job to fail. > This Bug is introduced by SPARK-26466. > Here is the example job submission: > {code:bash} > curl -X POST http://localhost:7077/v1/submissions/create --header > "Content-Type:application/json" --data '{ > "action": "CreateSubmissionRequest", > "appResource": > "file:///opt/spark-3.0.0-bin-3.2.0/examples/jars/spark-examples_2.12-3.0.0.jar", > "clientSparkVersion": "3.0.0", > "appArgs": ["30"], > "environmentVariables": {}, > "mainClass": "org.apache.spark.examples.SparkPi", > "sparkProperties": { > "spark.jars": > "file:///opt/spark-3.0.0-bin-3.2.0/examples/jars/spark-examples_2.12-3.0.0.jar", > "spark.driver.supervise": "false", > "spark.executor.memory": "512m", > "spark.driver.memory": "512m", > "spark.submit.deployMode": "cluster", > "spark.app.name": "SparkPi", > "spark.master": "mesos://localhost:5050" > }}' > {code} > Expected Driver log would contain: > {code:bash} > 20/08/20 20:19:57 WARN DependencyUtils: Local jar > /var/lib/mesos/slaves/e6779377-08ec-4765-9bfc-d27082fbcfa1-S0/frameworks/e6779377-08ec-4765-9bfc-d27082fbcfa1-0000/executors/driver-20200820201954-0002/runs/d9d734e8-a299-4d87-8f33-b134c65c422b/spark.driver.memory=512m > does not exist, skipping. > Error: Failed to load class org.apache.spark.examples.SparkPi. > 20/08/20 20:19:57 INFO ShutdownHookManager: Shutdown hook called > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org