Hi, All,
I want to run Spark on our Hadoop 2.0 Cluster(YARN). After building use
source code, I follow the instruction from
http://spark.incubator.apache.org/docs/latest/running-on-yarn.html . But
when I run:
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar
./spark-class org.apache.spark.deploy.yarn.Client --jar
examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar
--class org.apache.spark.examples.SparkPi --args yarn-standalone
--num-workers 3 --master-memory 4g --worker-memory 2g
--worker-cores 1
I got following exception:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.Client
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.spark.deploy.yarn.Client.
Program will exit.
I checked my directory and can find the
spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar file.
[10.10.0.141:hadoop@lxnamenode02:/home/hadoop/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3]$
ll
total 173412
drwxrwxr-x 3 hadoop hadoop 4096 Dec 18 18:45 cache
-rw-rw-r-- 1 hadoop hadoop 89756622 Dec 18 19:29
spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar
-rw-rw-r-- 1 hadoop hadoop 87809175 Sep 17 16:40
spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar
[10.10.0.141:hadoop@lxnamenode02:/home/hadoop/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3]$
So what's the possible reason that I can not run the sample?
--
You received this message because you are subscribed to the Google Groups
"Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
