Creating simple.sbt and src/ in $SPARK_HOME allows me to run a standalone scala
program in the downloaded spark code tree. For example my directory layout is:
$ ls spark-0.9.0-incubating-bin-hadoop2
…
simple.sbt
src
…
$ tree src
src
`-- main
`-- scala
`— SimpleApp.scala
— Bharath
On 25-Mar-2014, at 1:46 am, Diana Carroll <dcarr...@cloudera.com> wrote:
> Thanks Ongen.
>
> Unfortunately I'm not able to follow your instructions either. In particular:
>
> sbt compile
> sbt run <arguments if any>
>
> This doesn't work for me because there's no program on my path called "sbt".
> The instructions in the Quick Start guide are specific that I should call
> "$SPARK_HOME/sbt/sbt". I don't have any other executable on my system called
> "sbt".
>
> Did you download and install sbt separately? In following the Quick Start
> guide, that was not stated as a requirement, and I'm trying to run through
> the guide word for word.
>
> Diana
>
>
> On Mon, Mar 24, 2014 at 4:12 PM, Ognen Duzlevski
> <og...@plainvanillagames.com> wrote:
> Diana,
>
> Anywhere on the filesystem you have read/write access (you need not be in
> your spark home directory):
>
> mkdir myproject
> cd myproject
> mkdir project
> mkdir target
> mkdir -p src/main/scala
> cp $mypath/$mymysource.scala src/main/scala/
> cp $mypath/myproject.sbt .
>
> Make sure that myproject.sbt has the following in it:
>
> name := "I NEED A NAME!"
>
> version := "I NEED A VERSION!"
>
> scalaVersion := "2.10.3"
>
> libraryDependencies += "org.apache.spark" % "spark-core_2.10" %
> "0.9.0-incubating"
>
> If you will be using Hadoop/HDFS functionality you will need the below line
> also
>
> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.2.0"
>
> The above assumes you are using Spark 0.9 and Scala 2.10.3. If you are using
> 0.8.1 - adjust appropriately.
>
> That's it. Now you can do
>
> sbt compile
> sbt run <arguments if any>
>
> You can also do
> sbt package to produce a jar file of your code which you can then add to the
> SparkContext at runtime.
>
> In a more complicated project you may need to have a bit more involved
> hierarchy like com.github.dianacarroll which will then translate to
> src/main/scala/com/github/dianacarroll/ where you can put your multiple
> .scala files which will then have to be a part of a package
> com.github.dianacarroll (you can just put that as your first line in each of
> these scala files). I am new to Java/Scala so this is how I do it. More
> educated Java/Scala programmers may tell you otherwise ;)
>
> You can get more complicated with the sbt project subrirectory but you can
> read independently about sbt and what it can do, above is the bare minimum.
>
> Let me know if that helped.
> Ognen
>
>
> On 3/24/14, 2:44 PM, Diana Carroll wrote:
> Has anyone successfully followed the instructions on the Quick Start page of
> the Spark home page to run a "standalone" Scala application? I can't, and I
> figure I must be missing something obvious!
>
> I'm trying to follow the instructions here as close to "word for word" as
> possible:
> http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala
>
> 1. The instructions don't say what directory to create my test application
> in, but later I'm instructed to run "sbt/sbt" so I conclude that my working
> directory must be $SPARK_HOME. (Temporarily ignoring that it is a little
> weird to be working directly in the Spark distro.)
>
> 2. Create $SPARK_HOME/mysparktest/src/main/scala/SimpleApp.scala.
> Copy&paste in the code from the instructions exactly, replacing
> YOUR_SPARK_HOME with my spark home path.
>
> 3. Create $SPARK_HOME/mysparktest/simple.sbt. Copy&paste in the sbt file
> from the instructions
>
> 4. From the $SPARK_HOME I run "sbt/sbt package". It runs through the ENTIRE
> Spark project! This takes several minutes, and at the end, it says "Done
> packaging". unfortunately, there's nothing in the $SPARK_HOME/mysparktest/
> folder other than what I already had there.
>
> (Just for fun, I also did what I thought was more logical, which is set my
> working directory to $SPARK_HOME/mysparktest, and but $SPARK_HOME/sbt/sbt
> package, but that was even less successful: I got an error:
> awk: cmd. line:1: fatal: cannot open file `./project/build.properties' for
> reading (No such file or directory)
> Attempting to fetch sbt
> /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or
> directory
> /usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or
> directory
> Our attempt to download sbt locally to sbt/sbt-launch-.jar failed. Please
> install sbt manually from http://www.scala-sbt.org/
>
>
> So, help? I'm sure these instructions work because people are following them
> every day, but I can't tell what they are supposed to do.
>
> Thanks!
> Diana
>
>
