Hi Divya, Please find below code to invoke spark from oozie.
Oozie file: +++++++++++++ <workflow-app xmlns="uri:oozie:workflow:0.2" name="DE_2_workflow_US"> <start to="spark-scala"/> <action name="mysparkscala"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>maprfs:///</job-tracker> <name-node>maprfs:///</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>dileep</value> </property> </configuration> <exec>sparkshell.sh</exec> <file>/ggg/gms/gmsrffr/dev/dileep/sparkshell.sh</file> <file>/axp/gms/gmsrffr/dev/dileep/sparkshell.scala</file> </shell> <ok to="end" /> <error to="fail" /> </action> <kill name="fail"> <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end" /> </workflow-app> ############# SparkShell.sh ############## /opt/mapr/spark/spark-1.2.1/bin/spark-shell --name perf108pret1 --num-executors 1 --executor-cores 1 --executor-memory 1G --driver-memory 2G -i sparkshell.scala exit 0 On Mon, Mar 7, 2016 at 9:32 PM, Chandeep Singh <c...@chandeep.com> wrote: > As a work around you could put your spark-submit statement in a shell > script and then use Oozie’s SSH action to execute that script. > > On Mar 7, 2016, at 3:58 PM, Neelesh Salian <nsal...@cloudera.com> wrote: > > Hi Divya, > > This link should have the details that you need to begin using the Spark > Action on Oozie: > https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html > > Thanks. > > On Mon, Mar 7, 2016 at 7:52 AM, Benjamin Kim <bbuil...@gmail.com> wrote: > >> To comment… >> >> At my company, we have not gotten it to work in any other mode than >> local. If we try any of the yarn modes, it fails with a “file does not >> exist” error when trying to locate the executable jar. I mentioned this to >> the Hue users group, which we used for this, and they replied that the >> Spark Action is very basic implementation and that they will be writing >> their own for production use. >> >> That’s all I know... >> >> On Mar 7, 2016, at 1:18 AM, Deepak Sharma <deepakmc...@gmail.com> wrote: >> >> There is Spark action defined for oozie workflows. >> Though I am not sure if it supports only Java SPARK jobs or Scala jobs as >> well. >> https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html >> Thanks >> Deepak >> >> On Mon, Mar 7, 2016 at 2:44 PM, Divya Gehlot <divya.htco...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Could somebody help me by providing the steps /redirect me to >>> blog/documentation on how to run Spark job written in scala through Oozie. >>> >>> Would really appreciate the help. >>> >>> >>> >>> Thanks, >>> Divya >>> >> >> >> >> -- >> Thanks >> Deepak >> www.bigdatabig.com >> www.keosha.net >> >> >> > > > -- > Neelesh Srinivas Salian > Customer Operations Engineer > > > > > -- Regards Dileep Kumar +91 9742443302