Hi JB, I notice the ticket of `decoupling` livy from spark. Which alternative engine do you consider to adopt ?
Jean-Baptiste Onofré <j...@nanthrax.net> 于2019年4月17日周三 上午12:06写道： > Hi Ryan, > > Thanks for the introduction e-mail and welcome ! > > While ago I already proposed some potential new features to Livy. > Especially I proposed to "decouple" Livy from Spark to be able to > interact with different execution engines. > > I'm planning to move forward on Livy PoCs. > > Looking forward to work together. > > Regards > JB > > On 16/04/2019 17:50, Ryan Skraba wrote: > > Hello! I haven't seen many "presentation" emails in the mailing list, > but > > I thought I'd introduce myself and share how we're using Livy in > production. > > > > I'm Ryan, a developer at Talend (not far from Alexey Romanenko and > > Jean-Baptiste Onofre), and we've just released Talend Pipeline > > Designer[1]. To simplify, we're a graphical front end in the cloud for > > designing data pipelines, and an "agent" on an edge node for running > them. > > For us, Livy is a great match for running pipelines in that agent, and > > we've been using it for quite some time now with success! > > > > I believe we've pushed a few changes back to Livy (thanks Alexey!), but > for > > the most part we're living in a forked repo at some arbitrary commit, > with > > lots of modifications that are *mostly* only interesting to our jobs. > > > > I'm pretty excited to hear about the release (belated congratulations!) > and > > I'm currently taking a look at rebasing our work and (1) identifying > > features that might be useful (e.g. around docker deployment, job log > > enhancements, some optional job argument preprocessing) and (2) getting > off > > of our fork by isolating our very specific code elsewhere. > > > > I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly > > interested in aligning back with the community! > > > > All my best, Ryan > > > > [1] For more info: > > > https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/ > > > -- Best Regards Jeff Zhang
