I think this IS possible?
You must set the HADOOP_CONF_DIR variable on the machine you’re running the
Java process that creates the SparkContext. The Hadoop configuration specifies
the YARN ResourceManager IPs, and Spark will use that configuration.
mn
> On Nov 21, 2014, at 8:10 AM, Prannoy <pran...@sigmoidanalytics.com> wrote:
>
> Hi naveen,
>
> I dont think this is possible. If you are setting the master with your
> cluster details you cannot execute any job from your local machine. You have
> to execute the jobs inside your yarn machine so that sparkconf is able to
> connect with all the provided details.
>
> If this is not the case such give a detail explaintation of what exactly you
> are trying to do :)
>
> Thanks.
>
> On Fri, Nov 21, 2014 at 8:11 PM, Naveen Kumar Pokala [via Apache Spark User
> List] <[hidden email]
> <x-msg://4/user/SendEmail.jtp?type=node&node=19484&i=0>> wrote:
> Hi,
>
>
>
> I am executing my spark jobs on yarn cluster by forming conf object in the
> following way.
>
>
>
> SparkConf conf = new
> SparkConf().setAppName("NewJob").setMaster("yarn-cluster");
>
>
>
> Now I want to execute spark jobs from my local machine how to do that.
>
>
>
> What I mean is there a way to give IP address, port all the details to
> connect a master(YARN) on some other network from my local spark Program.
>
>
>
> -Naveen
>
>
>
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Execute-Spark-programs-from-local-machine-on-Yarn-hadoop-cluster-tp19482.html
>
> <http://apache-spark-user-list.1001560.n3.nabble.com/Execute-Spark-programs-from-local-machine-on-Yarn-hadoop-cluster-tp19482.html>
> To start a new topic under Apache Spark User List, email [hidden email]
> <x-msg://4/user/SendEmail.jtp?type=node&node=19484&i=1>
> To unsubscribe from Apache Spark User List, click here
> <applewebdata://63F90BD1-F0D8-41D0-A1DE-B7ACE896B9D9>.
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: Execute Spark programs from local machine
> on Yarn-hadoop cluster
> <http://apache-spark-user-list.1001560.n3.nabble.com/Execute-Spark-programs-from-local-machine-on-Yarn-hadoop-cluster-tp19482p19484.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
