[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Saisai Shao resolved SPARK-23830. --------------------------------- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21168 [https://github.com/apache/spark/pull/21168] > Spark on YARN in cluster deploy mode fail with NullPointerException when a > Spark application is a Scala class not object > ------------------------------------------------------------------------------------------------------------------------ > > Key: SPARK-23830 > URL: https://issues.apache.org/jira/browse/SPARK-23830 > Project: Spark > Issue Type: Improvement > Components: YARN > Affects Versions: 2.3.0 > Reporter: Jacek Laskowski > Priority: Trivial > Fix For: 2.4.0 > > > As reported on StackOverflow in [Why does Spark on YARN fail with “Exception > in thread ”Driver“ > java.lang.NullPointerException”?|https://stackoverflow.com/q/49564334/1305344] > the following Spark application fails with {{Exception in thread "Driver" > java.lang.NullPointerException}} with Spark on YARN in cluster deploy mode: > {code} > class MyClass { > def main(args: Array[String]): Unit = { > val c = new MyClass() > c.process() > } > def process(): Unit = { > val sparkConf = new SparkConf().setAppName("my-test") > val sparkSession: SparkSession = > SparkSession.builder().config(sparkConf).getOrCreate() > import sparkSession.implicits._ > .... > } > ... > } > {code} > The exception is as follows: > {code} > 18/03/29 20:07:52 INFO ApplicationMaster: Starting the user application in a > separate Thread > 18/03/29 20:07:52 INFO ApplicationMaster: Waiting for spark context > initialization... > Exception in thread "Driver" java.lang.NullPointerException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637) > {code} > I think the reason for the exception {{Exception in thread "Driver" > java.lang.NullPointerException}} is due to [the following > code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L700-L701]: > {code} > val mainMethod = userClassLoader.loadClass(args.userClass) > .getMethod("main", classOf[Array[String]]) > {code} > So when {{mainMethod}} is used in [the following > code|https://github.com/apache/spark/blob/v2.3.0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L706] > it simply gives NPE. > {code} > mainMethod.invoke(null, userArgs.toArray) > {code} > That could be easily avoided with an extra check if the {{mainMethod}} is > initialized and give a user a message what may have been a reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org