Re: Programmatically connect to remote YARN in yarn-client mode

2015-10-14 Thread Marcelo Vanzin
On Wed, Oct 14, 2015 at 10:01 AM, Florian Kaspar wrote: > we are working on a project running on Spark. Currently we connect to a > remote Spark-Cluster in Standalone mode to obtain the SparkContext using > > new JavaSparkContext(new >

Re: Programmatically connect to remote YARN in yarn-client mode

2015-10-14 Thread Florian Kaspar
Thank you, Marcelo, so it is possible to simply copy the YARN configuration from the remote cluster to the local machine (assuming, the local machine can resolve the YARN host etc.) and just letting Spark do the rest? This would actually be great! Our "local" machine is just another virtual

Re: Programmatically connect to remote YARN in yarn-client mode

2015-10-14 Thread Marcelo Vanzin
On Wed, Oct 14, 2015 at 10:29 AM, Florian Kaspar wrote: > so it is possible to simply copy the YARN configuration from the remote > cluster to the local machine (assuming, the local machine can resolve the > YARN host etc.) and just letting Spark do the rest? > Yes,

Programmatically connect to remote YARN in yarn-client mode

2015-10-14 Thread Florian Kaspar
Hey everyone, we are working on a project running on Spark. Currently we connect to a remote Spark-Cluster in Standalone mode to obtain the SparkContext using new JavaSparkContext(new SparkConf().setAppName("").setMaster("spark://:7077")); Currently, we try to connect to a remote (!) YARN