The question is the spark dependency is marked as provided or is included in the fat jar.
For example, we are compiling the spark distro separately for java 8 + scala 2.11 + hadoop 2.6 (with maven) and marking it as provided in sbt. -adrian From: Raghuveer Chanda Date: Wednesday, October 21, 2015 at 2:14 PM To: Jean-Baptiste Onofré Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" Subject: Re: Spark on Yarn Hi, So does this mean I can't run spark 1.4 fat jar on yarn without installing spark 1.4. I am including spark 1.4 in my pom.xml so doesn't this mean its compiling in 1.4. On Wed, Oct 21, 2015 at 4:38 PM, Jean-Baptiste Onofré <j...@nanthrax.net<mailto:j...@nanthrax.net>> wrote: Hi The compiled version (master side) and client version diverge on spark network JavaUtils. You should use the same/aligned version. Regards JB Sent from my Samsung device -------- Original message -------- From: Raghuveer Chanda <raghuveer.cha...@gmail.com<mailto:raghuveer.cha...@gmail.com>> Date: 21/10/2015 12:33 (GMT+01:00) To: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Spark on Yarn Hi all, I am trying to run spark on yarn in quickstart cloudera vm.It already has spark 1.3 and Hadoop 2.6.0-cdh5.4.0 installed.(I am not using spark-submit since I want to run a different version of spark). I am able to run spark 1.3 on yarn but get the below error for spark 1.4. The log shows its running on spark 1.4 but still gives a error on a method which is present in 1.4 and not 1.3. Even the fat jar contains the class files of 1.4. As far as running in yarn the installed spark version shouldnt matter, but still its running on the other version. Hadoop Version: Hadoop 2.6.0-cdh5.4.0 Subversion http://github.com/cloudera/hadoop -r c788a14a5de9ecd968d1e2666e8765c5f018c271 Compiled by jenkins on 2015-04-21T19:18Z Compiled with protoc 2.5.0 From source with checksum cd78f139c66c13ab5cee96e15a629025 This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.4.0.jar Error: LogType:stderr Log Upload Time:Tue Oct 20 21:58:56 -0700 2015 LogLength:2334 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/filecache/10/simple-yarn-app-1.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/10/20 21:58:50 INFO spark.SparkContext: Running Spark version 1.4.0 15/10/20 21:58:53 INFO spark.SecurityManager: Changing view acls to: yarn 15/10/20 21:58:53 INFO spark.SecurityManager: Changing modify acls to: yarn 15/10/20 21:58:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn); users with modify permissions: Set(yarn) Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.network.util.JavaUtils.timeStringAsSec(Ljava/lang/String;)J at org.apache.spark.util.Utils$.timeStringAsSeconds(Utils.scala:1027) at org.apache.spark.SparkConf.getTimeAsSeconds(SparkConf.scala:194) at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:68) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982) at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:245) at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:52) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:247) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:188) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267) at org.apache.spark.SparkContext.<init>(SparkContext.scala:424) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) at com.hortonworks.simpleyarnapp.HelloWorld.main(HelloWorld.java:50) 15/10/20 21:58:53 INFO util.Utils: Shutdown hook called Please help :) -- Regards and Thanks, Raghuveer Chanda -- Regards, Raghuveer Chanda Computer Science and Engineering IIT Kharagpur +91-9475470374
