We've had success with Azkaban from LinkedIn over Oozie and Luigi. http://azkaban.github.io/
Azkaban has support for many different job types, a fleshed out web UI with decent log reporting, a decent failure / retry model, a REST api, and I think support for multiple executor slaves is coming in the future. We found Oozie's configuration and execution cumbersome, and Luigi immature. On Tue, Nov 11, 2014 at 2:50 AM, Adamantios Corais < adamantios.cor...@gmail.com> wrote: > Hi again, > > As Jimmy said, any thoughts about Luigi and/or any other tools? So far it > seems that Oozie is the best and only choice here. Is that right? > > On Mon, Nov 10, 2014 at 8:43 PM, Jimmy McErlain <jimmy.mcerl...@gmail.com> > wrote: > >> I have used Oozie for all our workflows with Spark apps but you will have >> to use a java event as the workflow element. I am interested in anyones >> experience with Luigi and/or any other tools. >> >> >> On Mon, Nov 10, 2014 at 10:34 AM, Adamantios Corais < >> adamantios.cor...@gmail.com> wrote: >> >>> I have some previous experience with Apache Oozie while I was developing >>> in Apache Pig. Now, I am working explicitly with Apache Spark and I am >>> looking for a tool with similar functionality. Is Oozie recommended? What >>> about Luigi? What do you use \ recommend? >>> >> >> >> >> -- >> >> >> "Nothing under the sun is greater than education. By educating one person >> and sending him/her into the society of his/her generation, we make a >> contribution extending a hundred generations to come." >> -Jigoro Kano, Founder of Judo- >> > >
