Hi Marcelo,
You are right the discussion is around two separated & isolated topics:
1. The backend. Today, we support Spark, but I think it could be interesting to
have some kind of execution engine plugins allowing us to support other backends
(Flink, Gearpump, ...).
2. About the channels. Now, we expose REST services and I think it's fine for
90% of the use cases. The idea of additional channels is to be able to interact
with Livy server using different clients (MQTT, AMQP, ...) in kind of async mode.
Regards
JB
On 11/14/2017 02:45 AM, Marcelo Vanzin wrote:
Seems like one of the proposals (channels) isn't really tied to the
other (supporting more than Spark). I don't really have an opinion on
the latter (other than nobody has shown enough interest to even try to
prototype anything).
For the former, why do you need these separate "channels"? What
different functionality would they support that you could not achieve
via REST? Any new transport brings a whole lot of baggage with it -
you need to make sure things like encryption and authentication work
properly over it, at the very least. Then there's issue of clients.
Then you have to support all apps over all transport types or things
get confusing. So I'd like to understand exactly what's the advantage
here before opening up that Pandora's box.
On Mon, Nov 13, 2017 at 6:17 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote:
Hi guys,
Currently, Livy provides a great "façade" as REST service to interact and
deploy with a Spark cluster.
I wonder if we could not extend this to other "channels" than REST. I'm
thinking about leveraging Apache Camel and others for that (why not
interacting with MQTT/AMQP, ...).
In terms of architecture, the Livy server could "expose" the cluster via
different channels (in term of wording).
On the other hand, we are mostly focus on Spark cluster today. It could be
interesting to extend this to other execution engines (Flink, Gearpump,
Apex, ...) and backend (Yarn, Kubernetes, ...).
The reason for those ideas is that basically I'm thinking about a Apache
Beam pipelines dispenser powered partly by Livy ;)
Thoughts ?
Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
