Re: Tools to manage workflows on Spark

r leverage caching or temp table, going into another >>> Oozie action. You could either save output to file or put all Spark >>> processing into one Oozie action. >>> >>> --- Original Message --- >>> >>> From: "Mayur Rustagi" >

Re: Tools to manage workflows on Spark

gt; Sent: February 28, 2015 7:07 PM >> To: "Qiang Cao" >> Cc: "Ted Yu" , "Ashish Nigam" < >> ashnigamt...@gmail.com>, "user" >> Subject: Re: Tools to manage workflows on Spark >> >> Sorry not really. Spork is a way to migrate

Re: Tools to manage workflows on Spark

processing into one Oozie action. > > --- Original Message --- > > From: "Mayur Rustagi" > Sent: February 28, 2015 7:07 PM > To: "Qiang Cao" > Cc: "Ted Yu" , "Ashish Nigam" , > "user" > Subject: Re: Tools to manage work

Re: Tools to manage workflows on Spark

. --- Original Message --- From: "Mayur Rustagi" Sent: February 28, 2015 7:07 PM To: "Qiang Cao" Cc: "Ted Yu" , "Ashish Nigam" , "user" Subject: Re: Tools to manage workflows on Spark Sorry not really. Spork is a way to migrate your existing p

Re: Tools to manage workflows on Spark

Sorry not really. Spork is a way to migrate your existing pig scripts to Spark or write new pig jobs then can execute on spark. For orchestration you are better off using Oozie especially if you are using other execution engines/systems besides spark. Regards, Mayur Rustagi Ph: +1 (760) 203 3257

Re: Tools to manage workflows on Spark

Thanks Mayur! I'm looking for something that would allow me to easily describe and manage a workflow on Spark. A workflow in my context is a composition of Spark applications that may depend on one another based on hdfs inputs/outputs. Is Spork a good fit? The orchestration I want is on app level.

Re: Tools to manage workflows on Spark

We do maintain it but in apache repo itself. However Pig cannot do orchestration for you. I am not sure what you are looking at from Pig in this context. Regards, Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoid.com @mayur_rustagi

Re: Tools to manage workflows on Spark

Here was latest modification in spork repo: Mon Dec 1 10:08:19 2014 Not sure if it is being actively maintained. On Sat, Feb 28, 2015 at 6:26 PM, Qiang Cao wrote: > Thanks for the pointer, Ashish! I was also looking at Spork > https://github.com/sigmoidanalytics/spork Pig-on-Spark), but wasn't

Re: Tools to manage workflows on Spark

Thanks for the pointer, Ashish! I was also looking at Spork https://github.com/sigmoidanalytics/spork Pig-on-Spark), but wasn't sure if that's the right direction. On Sat, Feb 28, 2015 at 6:36 PM, Ashish Nigam wrote: > You have to call spark-submit from oozie. > I used this link to get the idea

Re: Tools to manage workflows on Spark

You have to call spark-submit from oozie. I used this link to get the idea for my implementation - http://mail-archives.apache.org/mod_mbox/oozie-user/201404.mbox/%3CCAHCsPn-0Grq1rSXrAZu35yy_i4T=fvovdox2ugpcuhkwmjp...@mail.gmail.com%3E > On Feb 28, 2015, at 3:25 PM, Qiang Cao wrote: > > Than

Re: Tools to manage workflows on Spark

Thanks, Ashish! Is Oozie integrated with Spark? I knew it can accommodate some Hadoop jobs. On Sat, Feb 28, 2015 at 6:07 PM, Ashish Nigam wrote: > Qiang, > Did you look at Oozie? > We use oozie to run spark jobs in production. > > > On Feb 28, 2015, at 2:45 PM, Qiang Cao wrote: > > Hi Everyone

Re: Tools to manage workflows on Spark

Qiang, Did you look at Oozie? We use oozie to run spark jobs in production. > On Feb 28, 2015, at 2:45 PM, Qiang Cao wrote: > > Hi Everyone, > > We need to deal with workflows on Spark. In our scenario, each workflow > consists of multiple processing steps. Among different steps, there could

Tools to manage workflows on Spark

Hi Everyone, We need to deal with workflows on Spark. In our scenario, each workflow consists of multiple processing steps. Among different steps, there could be dependencies. I'm wondering if there are tools available that can help us schedule and manage workflows on Spark. I'm looking for somet