Diana,
Anywhere on the filesystem you have read/write access (you need not be
in your spark home directory):
mkdir myproject
cd myproject
mkdir project
mkdir target
mkdir -p src/main/scala
cp $mypath/$mymysource.scala src/main/scala/
cp $mypath/myproject.sbt .
Make sure that myproject.sbt has the following in it:
name := "I NEED A NAME!"
version := "I NEED A VERSION!"
scalaVersion := "2.10.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" %
"0.9.0-incubating"
If you will be using Hadoop/HDFS functionality you will need the below
line also
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.2.0"
The above assumes you are using Spark 0.9 and Scala 2.10.3. If you are
using 0.8.1 - adjust appropriately.
That's it. Now you can do
sbt compile
sbt run <arguments if any>
You can also do
sbt package to produce a jar file of your code which you can then add to
the SparkContext at runtime.
In a more complicated project you may need to have a bit more involved
hierarchy like com.github.dianacarroll which will then translate to
src/main/scala/com/github/dianacarroll/ where you can put your multiple
.scala files which will then have to be a part of a package
com.github.dianacarroll (you can just put that as your first line in
each of these scala files). I am new to Java/Scala so this is how I do
it. More educated Java/Scala programmers may tell you otherwise ;)
You can get more complicated with the sbt project subrirectory but you
can read independently about sbt and what it can do, above is the bare
minimum.
Let me know if that helped.
Ognen
On 3/24/14, 2:44 PM, Diana Carroll wrote:
Has anyone successfully followed the instructions on the Quick Start
page of the Spark home page to run a "standalone" Scala application?
I can't, and I figure I must be missing something obvious!
I'm trying to follow the instructions here as close to "word for word"
as possible:
http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala
1. The instructions don't say what directory to create my test
application in, but later I'm instructed to run "sbt/sbt" so I
conclude that my working directory must be $SPARK_HOME. (Temporarily
ignoring that it is a little weird to be working directly in the Spark
distro.)
2. Create $SPARK_HOME/mysparktest/src/main/scala/SimpleApp.scala.
Copy&paste in the code from the instructions exactly, replacing
YOUR_SPARK_HOME with my spark home path.
3. Create $SPARK_HOME/mysparktest/simple.sbt. Copy&paste in the sbt
file from the instructions
4. From the $SPARK_HOME I run "sbt/sbt package". It runs through the
ENTIRE Spark project! This takes several minutes, and at the end, it
says "Done packaging". unfortunately, there's nothing in the
$SPARK_HOME/mysparktest/ folder other than what I already had there.
(Just for fun, I also did what I thought was more logical, which is
set my working directory to $SPARK_HOME/mysparktest, and but
$SPARK_HOME/sbt/sbt package, but that was even less successful: I got
an error:
awk: cmd. line:1: fatal: cannot open file `./project/build.properties'
for reading (No such file or directory)
Attempting to fetch sbt
/usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or
directory
/usr/lib/spark/sbt/sbt: line 33: sbt/sbt-launch-.jar: No such file or
directory
Our attempt to download sbt locally to sbt/sbt-launch-.jar failed.
Please install sbt manually from http://www.scala-sbt.org/
So, help? I'm sure these instructions work because people are
following them every day, but I can't tell what they are supposed to do.
Thanks!
Diana
