RE: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter

...@chandeep.com] Sent: Monday, March 07, 2016 11:02 AM To: Neelesh Salian Cc: Benjamin Kim; Deepak Sharma; Divya Gehlot; user @spark; u...@hadoop.apache.org Subject: Re: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter As a work around you could put your spark-submit statement

Re: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter

As a work around you could put your spark-submit statement in a shell script and then use Oozie’s SSH action to execute that script. > On Mar 7, 2016, at 3:58 PM, Neelesh Salian wrote: > > Hi Divya, > > This link should have the details that you need to begin using the

Re: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter

Hi Divya, This link should have the details that you need to begin using the Spark Action on Oozie: https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html Thanks. On Mon, Mar 7, 2016 at 7:52 AM, Benjamin Kim wrote: > To comment… > > At my company, we have not

Re: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter

To comment… At my company, we have not gotten it to work in any other mode than local. If we try any of the yarn modes, it fails with a “file does not exist” error when trying to locate the executable jar. I mentioned this to the Hue users group, which we used for this, and they replied that

Re: Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter

There is Spark action defined for oozie workflows. Though I am not sure if it supports only Java SPARK jobs or Scala jobs as well. https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html Thanks Deepak On Mon, Mar 7, 2016 at 2:44 PM, Divya Gehlot wrote: > Hi, >

Steps to Run Spark Scala job from Oozie on EC2 Hadoop clsuter

Hi, Could somebody help me by providing the steps /redirect me to blog/documentation on how to run Spark job written in scala through Oozie. Would really appreciate the help. Thanks, Divya