[ https://issues.apache.org/jira/browse/SPARK-22574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260954#comment-16260954 ]
Apache Spark commented on SPARK-22574: -------------------------------------- User 'Gschiavon' has created a pull request for this issue: https://github.com/apache/spark/pull/19793 > Wrong request causing Spark Dispatcher going inactive > ----------------------------------------------------- > > Key: SPARK-22574 > URL: https://issues.apache.org/jira/browse/SPARK-22574 > Project: Spark > Issue Type: Bug > Components: Mesos, Spark Submit > Affects Versions: 2.2.0 > Reporter: German Schiavon Matteo > Priority: Minor > Fix For: 2.0.0, 2.1.0, 2.2.0 > > > When submitting a wrong _CreateSubmissionRequest_ to Spark Dispatcher is > causing a bad state of Dispatcher and making it inactive as a mesos framework. > The class CreateSubmissionRequest initialise its arguments to null as follows: > {code:title=CreateSubmissionRequest.scala|borderStyle=solid} > var appResource: String = null > var mainClass: String = null > var appArgs: Array[String] = null > var sparkProperties: Map[String, String] = null > var environmentVariables: Map[String, String] = null > {code} > There are some checks of this variable but not in all of them, for example in > appArgs and environmentVariables. > If you don't set _appArgs_ it will cause the following error: > {code:title=error|borderStyle=solid} > 17/11/21 14:37:24 INFO MesosClusterScheduler: Reviving Offers. > Exception in thread "Thread-22" java.lang.NullPointerException > at > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444) > at > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451) > at > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538) > at > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570) > at > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555) > at > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621) > {code} > Because it's trying to access to it without checking whether is null or not. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org