Re: Is there programmatic way running Spark job on Yarn cluster without using spark-submit script ?

> This is not independent programmatic way of running of Spark job on Yarn cluster. The example I created simply demonstrates how to wire up the classpath so that spark submit can be called programmatically. For my use case, I wanted to hold open a connection so I could send tasks to the executors

Re: Is there programmatic way running Spark job on Yarn cluster without using spark-submit script ?

Hi Elkhan, There are couple of ways to do this. 1) Spark-jobserver is a popular web server that is used to submit spark jobs. https://github.com/spark-jobserver/spark-jobserver 2) Spark-submit script sets the classpath for the job. Bypassing

Re: Is there programmatic way running Spark job on Yarn cluster without using spark-submit script ?

This is not independent programmatic way of running of Spark job on Yarn cluster. That example demonstrates running on *Yarn-client* mode, also will be dependent of Jetty. Users writing Spark programs do not want to depend on that. I found this SparkLauncher class introduced in Spark 1.4 version

Re: Is there programmatic way running Spark job on Yarn cluster without using spark-submit script ?

An example of being able to do this is provided in the Spark Jetty Server project [1] [1] https://github.com/calrissian/spark-jetty-server On Wed, Jun 17, 2015 at 8:29 PM, Elkhan Dadashov wrote: > Hi all, > > Is there any way running Spark job in programmatic way on Yarn cluster > without using

Is there programmatic way running Spark job on Yarn cluster without using spark-submit script ?

Hi all, Is there any way running Spark job in programmatic way on Yarn cluster without using spark-submit script ? I cannot include Spark jars on my Java application (due o dependency conflict and other reasons), so I'll be shipping Spark assembly uber jar (spark-assembly-1.3.1-hadoop2.3.0.jar) t