[ https://issues.apache.org/jira/browse/SPARK-39617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexandru Tofan updated SPARK-39617: ------------------------------------ Description: When updating Spark 3.0.2 to Spark 3.2.1 I get the following error: {code:java} Exception in thread "main" org.apache.spark.SparkException: Driver cores must be a positive number at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634) at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257) at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119) at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026) at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code} Obviously I set the driver-cores property but it doesn't matter, it is not used. I'm suspecting that it is something about the Mesos Dispatcher, it receives the property because it is being displayed correctly in the Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and therefore fails. Update: It happens when submitting a *Spark 3.2.1* job with *Mesos* in {*}Cluster mode{*}. was: When updating Spark 3.0.2 to Spark 3.2.1 I get the following error: {code:java} Exception in thread "main" org.apache.spark.SparkException: Driver cores must be a positive number at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634) at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257) at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119) at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026) at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code} Obviously I set the driver-cores property but it doesn't matter, it is not used. I'm suspecting that it is something about the Mesos Dispatcher, it receives the property because it is being displayed correctly in the Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and therefore fails. > Driver cores must be a positive number > -------------------------------------- > > Key: SPARK-39617 > URL: https://issues.apache.org/jira/browse/SPARK-39617 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 3.2.1 > Environment: Spark 3.2.1 > Mesos > Java 8 OpenJDK > Reporter: Alexandru Tofan > Priority: Major > Labels: mesos, spark-submit > > When updating Spark 3.0.2 to Spark 3.2.1 I get the following error: > {code:java} > Exception in thread "main" org.apache.spark.SparkException: Driver cores must > be a positive number > at > org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634) > at > org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257) > at > org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234) > at > org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119) > at > org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code} > Obviously I set the driver-cores property but it doesn't matter, it is not > used. I'm suspecting that it is something about the Mesos Dispatcher, it > receives the property because it is being displayed correctly in the > Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and > therefore fails. > > Update: > It happens when submitting a *Spark 3.2.1* job with *Mesos* in {*}Cluster > mode{*}. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org