Seems like one of the proposals (channels) isn't really tied to the other (supporting more than Spark). I don't really have an opinion on the latter (other than nobody has shown enough interest to even try to prototype anything).
For the former, why do you need these separate "channels"? What different functionality would they support that you could not achieve via REST? Any new transport brings a whole lot of baggage with it - you need to make sure things like encryption and authentication work properly over it, at the very least. Then there's issue of clients. Then you have to support all apps over all transport types or things get confusing. So I'd like to understand exactly what's the advantage here before opening up that Pandora's box. On Mon, Nov 13, 2017 at 6:17 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi guys, > > Currently, Livy provides a great "façade" as REST service to interact and > deploy with a Spark cluster. > > I wonder if we could not extend this to other "channels" than REST. I'm > thinking about leveraging Apache Camel and others for that (why not > interacting with MQTT/AMQP, ...). > In terms of architecture, the Livy server could "expose" the cluster via > different channels (in term of wording). > > On the other hand, we are mostly focus on Spark cluster today. It could be > interesting to extend this to other execution engines (Flink, Gearpump, > Apex, ...) and backend (Yarn, Kubernetes, ...). > > The reason for those ideas is that basically I'm thinking about a Apache > Beam pipelines dispenser powered partly by Livy ;) > > Thoughts ? > > Regards > JB > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com -- Marcelo
