does sbt "show full-classpath" show spark-core on the classpath? I am still pretty new to scala but it seems like you have val sparkCore = "org.apache.spark" %% "spark-core" % V.spark % "provided" -- I believe the "provided" part means it's in your classpath. Spark-shell script sets up a lot of stuff for you so...
On Tue, Mar 11, 2014 at 9:02 AM, Gino Mathews <gin...@thinkpalm.com> wrote: > Hi, > > I am new to spark. > I would like to run jobs in Spark stand alone cluster mode. > > No cluser managers other than spark is used. > (https://spark.apache.org/docs/0.9.0/spark-standalone.html) > I have tried wordcount from spark shell and stand alone scala app. > > The code reads input from HDFS and writes the results to HDFS. uses 2 worker > nodes. > > In spark-shell the wordcount is successful, how ever my effort to run stand > alone programmes are in vain. > > > > My environement > > Ubuntu 12.04 - 32 bit > > JAVA 1.7.0_51 > > I have installed spark @ $HOME/Downloads/spark-0.9.0-incubating > installed hadoop 2.2.0 as separate hduser and given permission to other > users. > installed scala 2.10.3 > installed sbt 0.13.1 > > Spark master act as HDFS master > I have one master and 2 worker nodes and HDFS is accessible in all nodes. > I downloaded example project and modified to use my spark cluster. > I started the sparkcluster at spark://192.168.0.138:7077 > and hdfs://master:9000/ > When I run the project as SPARK_HADOOP_VERSION=2.2.0 sbt run, I get > following error > > gino@master:~/Test/spark-example-project$ SPARK_HADOOP_VERSION=2.2.0 sbt run > [info] Loading project definition from > /home/gino/Test/spark-example-project/project > [info] Set current project to spark-example-project (in build > file:/home/gino/Test/spark-example-project/) > [info] Running com.Thinkpalm.spark.WordCountHDFS > [error] (run-main-0) java.lang.NoClassDefFoundError: > org/apache/spark/SparkContext > java.lang.NoClassDefFoundError: org/apache/spark/SparkContext > at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12) > at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12) > at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > [trace] Stack trace suppressed: run last compile:run for the full output. > java.lang.RuntimeException: Nonzero exit code: 1 > at scala.sys.package$.error(package.scala:27) > [trace] Stack trace suppressed: run last compile:run for the full output. > [error] (compile:run) Nonzero exit code: 1 > [error] Total time: 0 s, completed Mar 11, 2014 2:54:54 PM > > Could anyone give some pointers ... I have attached the project for > reference. > > > > Thanks and regards > > Gino Mathews