You are specifying the spark master in the jar
.setMaster("spark://hadoop-1.certus.com:7077")
so sbt run is deploying the jar into the master cluster and running it.
Regards
Mayur
Mayur Rustagi
Ph: +919632149971
h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
https://twitter.com/mayur_rustagi
On Thu, Feb 20, 2014 at 7:22 AM, Nan Zhu <[email protected]> wrote:
> I’m not sure if I understand your question correctly
>
> do you mean you didn’t see the application information in Spark Web UI
> even it generates the expected results?
>
> Best,
>
> --
> Nan Zhu
>
> On Thursday, February 20, 2014 at 10:13 AM, Tao Xiao wrote:
>
> My application source file, *SimpleDistributedApp.scala*, is as follows:
>
> __________________________________________________________________
> import org.apache.spark.{SparkConf, SparkContext}
>
> object SimpleDistributedApp {
> def main(args: Array[String]) = {
> val filepath = "hdfs://
> hadoop-1.certus.com:54310/user/root/samples/data"
>
> val conf = new SparkConf()
> .setMaster("spark://hadoop-1.certus.com:7077")
> .setAppName("**SimpleDistributedApp**")
>
> .setSparkHome("/home/xt/soft/spark-0.9.0-incubating-bin-hadoop1")
>
> .setJars(Array("target/scala-2.10/simple-distributed-app_2.10-1.0.jar"))
> .set("spark.executor.memory", "1g")
>
> val sc = new SparkContext(conf)
> val text = sc.textFile(filepath, 3)
>
> val numOfHello = text.filter(line =>
> line.contains("hello")).count()
>
> println("number of lines containing 'hello' is " + numOfHello)
> println("down")
> }
> }
> ______________________________________________________________________
>
>
>
> The corresponding sbt file, *$SPARK_HOME/simple.sbt*, is as follows:
> _________________________________________________________________
>
> name := "Simple Distributed App"
>
> version := "1.0"
>
> scalaVersion := "2.10.3"
>
> libraryDependencies += "org.apache.spark" %% "spark-core" %
> "0.9.0-incubating"
>
> resolvers += "Akka Repository" at "http://repo.akka.io/releases/";
> _________________________________________________________________
>
>
> I built the application into
> *$SPARK_HOME/target/scala-2.10/simple-distributed-app_2.10-1.0.jar*,
> using the command
> SPARK_HADOOP_VERSION=1.2.1 sbt/sbt package
>
> I ran it using the command "sbt/sbt run" and it finished running
> successfully.
>
> But I'm not sure what's the correct and general way to submit and run a
> job in Spark cluster. To be specific,after having built a job into a JAR
> file, say *simpleApp.jar*, where should I put it and how should I submit
> it to Spark cluster?
>
>
>
>
>
>
>
>
>
>
