Hi, I am trying to get my Spark application to run on YARN and by now I have managed to build a fat jar as described on < http://markmail.org/message/c6no2nyaqjdujnkq> (which is the only really usable manual on how to get such a jar file). My code runs fine using "sbt test" and "sbt run", but when running
~/spark-1.1.0-bin-hadoop2.4/bin/spark-submit \ --class my.spark.MyClass --master "local[3]" \ target/scala-2.10/myclass-assembly-1.0.jar I get: Spark assembly has been built with Hive, including Datanucleus jars on classpath Exception in thread "main" java.lang.NoClassDefFoundError: com/typesafe/scalalogging/slf4j/Logger at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2615) at java.lang.Class.getMethod0(Class.java:2856) at java.lang.Class.getMethod(Class.java:1668) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:325) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: com.typesafe.scalalogging.slf4j.Logger at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 7 more ABRT problem creation: 'success' It seems to run into an error before it does anything with my jar? I am using "com.typesafe.scala-logging" %% "scala-logging-slf4j" % "2.1.2" instead of "com.typesafe" %% "scalalogging-slf4j" % "1.1.0" in my SBT file, could that be a reason? Thanks Tobias
