Currently, we save the result in HDFS, and read it back in our application. Since Clinet.run is blocking call, it's easy to do it in this way.
We are now working on using akka to bring back the result to app without going through the HDFS, and we can use akka to track the log, and stack trace, etc. Sincerely, DB Tsai ------------------------------------------------------- My Blog: https://www.dbtsai.com LinkedIn: https://www.linkedin.com/in/dbtsai On Thu, Jun 19, 2014 at 12:08 PM, Koert Kuipers <ko...@tresata.com> wrote: > db tsai, > if in yarn-cluster mode the driver runs inside yarn, how can you do a > rdd.collect and bring the results back to your application? > > > On Thu, Jun 19, 2014 at 2:33 PM, DB Tsai <dbt...@stanford.edu> wrote: >> >> We are submitting the spark job in our tomcat application using >> yarn-cluster mode with great success. As Kevin said, yarn-client mode >> runs driver in your local JVM, and it will have really bad network >> overhead when one do reduce action which will pull all the result from >> executor to your local JVM. Also, since you can only have one spark >> context object in one JVM, it will be tricky to run multiple spark >> jobs concurrently with yarn-clinet mode. >> >> This is how we submit spark job with yarn-cluster mode. Please use the >> current master code, otherwise, after the job is finished, spark will >> kill the JVM and exit your app. >> >> We setup the configuration of spark in a scala map. >> >> def getArgsFromConf(conf: Map[String, String]): Array[String] = { >> Array[String]( >> "--jar", conf.get("app.jar").getOrElse(""), >> "--addJars", conf.get("spark.addJars").getOrElse(""), >> "--class", conf.get("spark.mainClass").getOrElse(""), >> "--num-executors", conf.get("spark.numWorkers").getOrElse("1"), >> "--driver-memory", conf.get("spark.masterMemory").getOrElse("1g"), >> "--executor-memory", conf.get("spark.workerMemory").getOrElse("1g"), >> "--executor-cores", conf.get("spark.workerCores").getOrElse("1")) >> } >> >> System.setProperty("SPARK_YARN_MODE", "true") >> val sparkConf = new SparkConf >> val args = getArgsFromConf(conf) >> new Client(new ClientArguments(args, sparkConf), hadoopConfig, >> sparkConf).run >> >> Sincerely, >> >> DB Tsai >> ------------------------------------------------------- >> My Blog: https://www.dbtsai.com >> LinkedIn: https://www.linkedin.com/in/dbtsai >> >> >> On Thu, Jun 19, 2014 at 11:22 AM, Kevin Markey <kevin.mar...@oracle.com> >> wrote: >> > Yarn client is much like Spark client mode, except that the executors >> > are >> > running in Yarn containers managed by the Yarn resource manager on the >> > cluster instead of as Spark workers managed by the Spark master. The >> > driver >> > executes as a local client in your local JVM. It communicates with the >> > workers on the cluster. Transformations are scheduled on the cluster by >> > the >> > driver's logic. Actions involve communication between local driver and >> > remote cluster executors. So, there is some additional network >> > overhead, >> > especially if the driver is not co-located on the cluster. In >> > yarn-cluster >> > mode -- in contrast, the driver is executed as a thread in a Yarn >> > application master on the cluster. >> > >> > In either case, the assembly JAR must be available to the application on >> > the >> > cluster. Best to copy it to HDFS and specify its location by exporting >> > its >> > location as SPARK_JAR. >> > >> > Kevin Markey >> > >> > >> > On 06/19/2014 11:22 AM, Koert Kuipers wrote: >> > >> > i am trying to understand how yarn-client mode works. i am not using >> > spark-submit, but instead launching a spark job from within my own >> > application. >> > >> > i can see my application contacting yarn successfully, but then in yarn >> > i >> > get an immediate error: >> > >> > Application application_1403117970283_0014 failed 2 times due to AM >> > Container for appattempt_1403117970283_0014_000002 exited with exitCode: >> > -1000 due to: File file:/home/koert/test-assembly-0.1-SNAPSHOT.jar does >> > not >> > exist >> > .Failing this attempt.. Failing the application. >> > >> > why is yarn trying to fetch my jar, and why as a local file? i would >> > expect >> > the jar to be send to yarn over the wire upon job submission? >> > >> > > >
