Re: Submit Spark driver on Yarn Cluster in client mode
Not sure if it will work, but you can try creating a dummy hadoop conf directory and put those files (*-site.xml) files inside it and hopefully spark will pick it up and submit it on that remote cluster. (If there isn't any network/firewall issues). Thanks Best Regards On Mon, Nov 24, 2014 at 4:16 PM, Naveen Kumar Pokala < npok...@spcapitaliq.com> wrote: > Hi Akhil, > > > > But driver and yarn both are in different networks, How to specify (export > HADOOP_CONF_DIR=XXX) path. > > > > Like driver is from my windows machine and yarn is some unix machine on > different network. > > > > -Naveen. > > > > *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com > ] > *Sent:* Monday, November 24, 2014 4:08 PM > *To:* Naveen Kumar Pokala > *Cc:* user@spark.apache.org > *Subject:* Re: Submit Spark driver on Yarn Cluster in client mode > > > > You can export the hadoop configurations dir (export > HADOOP_CONF_DIR=XXX) in the environment and then submit it like: > > > > ./bin/spark-submit \ > > --class org.apache.spark.examples.SparkPi \ > > --master yarn-cluster \ # can also be `yarn-client` for client mode > > --executor-memory 20G \ > > --num-executors 50 \ > > /path/to/examples.jar \ > > 1000 > > > > More details over here > https://spark.apache.org/docs/1.1.0/submitting-applications.html#launching-applications-with-spark-submit > > > Thanks > > Best Regards > > > > On Mon, Nov 24, 2014 at 4:01 PM, Naveen Kumar Pokala < > npok...@spcapitaliq.com> wrote: > > Hi, > > > > I want to submit my spark program from my machine on a YARN Cluster in > yarn client mode. > > > > How to specify al l the required details through SPARK submitter. > > > > Please provide me some details. > > > > -Naveen. > > >
Re: Submit Spark driver on Yarn Cluster in client mode
Hi Akhil, But driver and yarn both are in different networks, How to specify (export HADOOP_CONF_DIR=XXX) path. Like driver is from my windows machine and yarn is some unix machine on different network. -Naveen. From: Akhil Das [mailto:ak...@sigmoidanalytics.com] Sent: Monday, November 24, 2014 4:08 PM To: Naveen Kumar Pokala Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: Submit Spark driver on Yarn Cluster in client mode You can export the hadoop configurations dir (export HADOOP_CONF_DIR=XXX) in the environment and then submit it like: ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master yarn-cluster \ # can also be `yarn-client` for client mode --executor-memory 20G \ --num-executors 50 \ /path/to/examples.jar \ 1000 More details over here https://spark.apache.org/docs/1.1.0/submitting-applications.html#launching-applications-with-spark-submit Thanks Best Regards On Mon, Nov 24, 2014 at 4:01 PM, Naveen Kumar Pokala mailto:npok...@spcapitaliq.com>> wrote: Hi, I want to submit my spark program from my machine on a YARN Cluster in yarn client mode. How to specify al l the required details through SPARK submitter. Please provide me some details. -Naveen.
Re: Submit Spark driver on Yarn Cluster in client mode
You can export the hadoop configurations dir (export HADOOP_CONF_DIR=XXX) in the environment and then submit it like: ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master yarn-cluster \ # can also be `yarn-client` for client mode --executor-memory 20G \ --num-executors 50 \ /path/to/examples.jar \ 1000 More details over here https://spark.apache.org/docs/1.1.0/submitting-applications.html#launching-applications-with-spark-submit Thanks Best Regards On Mon, Nov 24, 2014 at 4:01 PM, Naveen Kumar Pokala < npok...@spcapitaliq.com> wrote: > Hi, > > > > I want to submit my spark program from my machine on a YARN Cluster in > yarn client mode. > > > > How to specify al l the required details through SPARK submitter. > > > > Please provide me some details. > > > > -Naveen. >
Submit Spark driver on Yarn Cluster in client mode
Hi, I want to submit my spark program from my machine on a YARN Cluster in yarn client mode. How to specify al l the required details through SPARK submitter. Please provide me some details. -Naveen.
