Andrew, Are you running on a CM-managed cluster? I just checked, and there is a bug here (fixed in 1.0), but it's avoided by having yarn.application.classpath defined in your yarn-site.xml.
-Sandy On Wed, Jul 16, 2014 at 10:02 AM, Sean Owen <so...@cloudera.com> wrote: > Somewhere in here, you are not actually running vs Hadoop 2 binaries. > Your cluster is certainly Hadoop 2, but your client is not using the > Hadoop libs you think it is (or your compiled binary is linking > against Hadoop 1, which is the default for Spark -- did you change > it?) > > On Wed, Jul 16, 2014 at 5:45 PM, Andrew Milkowski <amgm2...@gmail.com> > wrote: > > Hello community, > > > > tried to run storm app on yarn, using cloudera hadoop and spark distro > (from > > http://archive.cloudera.com/cdh5/cdh/5) > > > > hadoop version: hadoop-2.3.0-cdh5.0.3.tar.gz > > spark version: spark-0.9.0-cdh5.0.3.tar.gz > > > > DEFAULT_YARN_APPLICATION_CLASSPATH is part of hadoop-api-yarn jar ... > > > > thanks for any replies! > > > > [amilkowski@localhost spark-streaming]$ ./test-yarn.sh > > 14/07/16 12:47:17 WARN util.NativeCodeLoader: Unable to load > native-hadoop > > library for your platform... using builtin-java classes where applicable > > 14/07/16 12:47:17 INFO client.RMProxy: Connecting to ResourceManager at > > /0.0.0.0:8032 > > 14/07/16 12:47:17 INFO yarn.Client: Got Cluster metric info from > > ApplicationsManager (ASM), number of NodeManagers: 1 > > 14/07/16 12:47:17 INFO yarn.Client: Queue info ... queueName: > root.default, > > queueCurrentCapacity: 0.0, queueMaxCapacity: -1.0, > > queueApplicationCount = 0, queueChildQueueCount = 0 > > 14/07/16 12:47:17 INFO yarn.Client: Max mem capabililty of a single > resource > > in this cluster 8192 > > 14/07/16 12:47:17 INFO yarn.Client: Preparing Local resources > > 14/07/16 12:47:18 INFO yarn.Client: Uploading > > > file:/opt/local/cloudera/spark/cdh5/spark-0.9.0-cdh5.0.3/examples/target/scala-2.10/spark-examples-assembly-0.9.0-cdh5.0.3.jar > > to > > > hdfs://localhost:8020/user/amilkowski/.sparkStaging/application_1405528355264_0004/spark-examples-assembly-0.9.0-cdh5.0.3.jar > > 14/07/16 12:47:19 INFO yarn.Client: Uploading > > > file:/opt/local/cloudera/spark/cdh5/spark-0.9.0-cdh5.0.3/assembly/target/scala-2.10/spark-assembly-0.9.0-cdh5.0.3-hadoop2.3.0-cdh5.0.3.jar > > to > > > hdfs://localhost:8020/user/amilkowski/.sparkStaging/application_1405528355264_0004/spark-assembly-0.9.0-cdh5.0.3-hadoop2.3.0-cdh5.0.3.jar > > 14/07/16 12:47:19 INFO yarn.Client: Setting up the launch environment > > Exception in thread "main" java.lang.NoSuchFieldException: > > DEFAULT_YARN_APPLICATION_CLASSPATH > > at java.lang.Class.getField(Class.java:1579) > > at > > > org.apache.spark.deploy.yarn.ClientBase$.getDefaultYarnApplicationClasspath(ClientBase.scala:403) > > at > > > org.apache.spark.deploy.yarn.ClientBase$$anonfun$5.apply(ClientBase.scala:386) > > at > > > org.apache.spark.deploy.yarn.ClientBase$$anonfun$5.apply(ClientBase.scala:386) > > at scala.Option.getOrElse(Option.scala:120) > > at > > > org.apache.spark.deploy.yarn.ClientBase$.populateHadoopClasspath(ClientBase.scala:385) > > at > > > org.apache.spark.deploy.yarn.ClientBase$.populateClasspath(ClientBase.scala:444) > > at > > > org.apache.spark.deploy.yarn.ClientBase$class.setupLaunchEnv(ClientBase.scala:274) > > at org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:41) > > at org.apache.spark.deploy.yarn.Client.runApp(Client.scala:77) > > at org.apache.spark.deploy.yarn.Client.run(Client.scala:98) > > at org.apache.spark.deploy.yarn.Client$.main(Client.scala:183) > > at org.apache.spark.deploy.yarn.Client.main(Client.scala) > > [amilkowski@localhost spark-streaming]$ > > >
