Hi Everyone,
Recently we have installed spark on yarn in hortonworks cluster. Now I am
trying to run a wordcount program in my eclipse and I
did setMaster("local") and I see the results that's as expected. Now I want
to submit the same job to my yarn cluster from my eclipse. In storm
basically I was doing the same by using StormSubmitter class and by passing
nimbus & zookeeper host to Config object. I was looking for something
exactly the same.
When I went through the documentation online, it read that I am suppose to
"export HADOOP_HOME_DIR=path to the conf dir". So now I copied the conf
folder from one of sparks gateway node to my local Unix box. Now I did
export that dir...
export HADOOP_HOME_DIR=/Users/user1/Documents/conf/
And I did the same in .bash_profile too. Now when I do echo
$HADOOP_HOME_DIR, I see the path getting printed in the command prompt. Now
my assumption is, in my program when I change setMaster("local") to
setMaster("yarn-client") my program should pick up the resource mangers i.e
yarn cluster info from the directory which I have exported and the job
should get submitted to resolve manager from my eclipse. But somehow it's
not happening. Please tell me if my assumption is wrong or if I am missing
anything here.
I have attached the word count program that I was using. Any help is highly
appreciated.
Thank you,
Ajay
submit_spark_job
Description: Binary data
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
