Somewhere in here, you are not actually running vs Hadoop 2 binaries. Your cluster is certainly Hadoop 2, but your client is not using the Hadoop libs you think it is (or your compiled binary is linking against Hadoop 1, which is the default for Spark -- did you change it?)
On Wed, Jul 16, 2014 at 5:45 PM, Andrew Milkowski <amgm2...@gmail.com> wrote: > Hello community, > > tried to run storm app on yarn, using cloudera hadoop and spark distro (from > http://archive.cloudera.com/cdh5/cdh/5) > > hadoop version: hadoop-2.3.0-cdh5.0.3.tar.gz > spark version: spark-0.9.0-cdh5.0.3.tar.gz > > DEFAULT_YARN_APPLICATION_CLASSPATH is part of hadoop-api-yarn jar ... > > thanks for any replies! > > [amilkowski@localhost spark-streaming]$ ./test-yarn.sh > 14/07/16 12:47:17 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/07/16 12:47:17 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 14/07/16 12:47:17 INFO yarn.Client: Got Cluster metric info from > ApplicationsManager (ASM), number of NodeManagers: 1 > 14/07/16 12:47:17 INFO yarn.Client: Queue info ... queueName: root.default, > queueCurrentCapacity: 0.0, queueMaxCapacity: -1.0, > queueApplicationCount = 0, queueChildQueueCount = 0 > 14/07/16 12:47:17 INFO yarn.Client: Max mem capabililty of a single resource > in this cluster 8192 > 14/07/16 12:47:17 INFO yarn.Client: Preparing Local resources > 14/07/16 12:47:18 INFO yarn.Client: Uploading > file:/opt/local/cloudera/spark/cdh5/spark-0.9.0-cdh5.0.3/examples/target/scala-2.10/spark-examples-assembly-0.9.0-cdh5.0.3.jar > to > hdfs://localhost:8020/user/amilkowski/.sparkStaging/application_1405528355264_0004/spark-examples-assembly-0.9.0-cdh5.0.3.jar > 14/07/16 12:47:19 INFO yarn.Client: Uploading > file:/opt/local/cloudera/spark/cdh5/spark-0.9.0-cdh5.0.3/assembly/target/scala-2.10/spark-assembly-0.9.0-cdh5.0.3-hadoop2.3.0-cdh5.0.3.jar > to > hdfs://localhost:8020/user/amilkowski/.sparkStaging/application_1405528355264_0004/spark-assembly-0.9.0-cdh5.0.3-hadoop2.3.0-cdh5.0.3.jar > 14/07/16 12:47:19 INFO yarn.Client: Setting up the launch environment > Exception in thread "main" java.lang.NoSuchFieldException: > DEFAULT_YARN_APPLICATION_CLASSPATH > at java.lang.Class.getField(Class.java:1579) > at > org.apache.spark.deploy.yarn.ClientBase$.getDefaultYarnApplicationClasspath(ClientBase.scala:403) > at > org.apache.spark.deploy.yarn.ClientBase$$anonfun$5.apply(ClientBase.scala:386) > at > org.apache.spark.deploy.yarn.ClientBase$$anonfun$5.apply(ClientBase.scala:386) > at scala.Option.getOrElse(Option.scala:120) > at > org.apache.spark.deploy.yarn.ClientBase$.populateHadoopClasspath(ClientBase.scala:385) > at > org.apache.spark.deploy.yarn.ClientBase$.populateClasspath(ClientBase.scala:444) > at > org.apache.spark.deploy.yarn.ClientBase$class.setupLaunchEnv(ClientBase.scala:274) > at org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:41) > at org.apache.spark.deploy.yarn.Client.runApp(Client.scala:77) > at org.apache.spark.deploy.yarn.Client.run(Client.scala:98) > at org.apache.spark.deploy.yarn.Client$.main(Client.scala:183) > at org.apache.spark.deploy.yarn.Client.main(Client.scala) > [amilkowski@localhost spark-streaming]$ >
