Hi guys,
Currently, Livy provides a great "façade" as REST service to interact and deploy
with a Spark cluster.
I wonder if we could not extend this to other "channels" than REST. I'm thinking
about leveraging Apache Camel and others for that (why not interacting with
MQTT/AMQP, ...).
In terms of architecture, the Livy server could "expose" the cluster via
different channels (in term of wording).
On the other hand, we are mostly focus on Spark cluster today. It could be
interesting to extend this to other execution engines (Flink, Gearpump, Apex,
...) and backend (Yarn, Kubernetes, ...).
The reason for those ideas is that basically I'm thinking about a Apache Beam
pipelines dispenser powered partly by Livy ;)
Thoughts ?
Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
