Alexandru Tofan created SPARK-39617:
---------------------------------------
Summary: Driver cores must be a positive number
Key: SPARK-39617
URL: https://issues.apache.org/jira/browse/SPARK-39617
Project: Spark
Issue Type: Bug
Components: Mesos
Affects Versions: 3.2.1
Environment: Spark 3.2.1
Mesos
Java 8 OpenJDK
Reporter: Alexandru Tofan
When updating Spark 3.0.2 to Spark 3.2.1 I get the following error:
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Driver cores must
be a positive number
at
org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
at
org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257)
at
org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
at
org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
at
org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
Obviously I set the driver-cores property but it doesn't matter, it is not
used. I'm suspecting that it is something about the Mesos Dispatcher, it
receives the property because it is being displayed correctly in the
Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and
therefore fails.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
