Re: Spark Server - How to implement

2014-12-12 Thread Patrick Wendell
Hey Manoj, One proposal potentially of interest is the Spark Kernel project from IBM - you should look for their. The interface in that project is more of a "remote REPL" interface, i.e. you submit commands (as strings) and get back results (as strings), but you don't have direct programmatic acce

Re: Spark Server - How to implement

2014-12-12 Thread Manoj Samel
Thanks Marcelo. Spark Gurus/Databricks team - do you have something in roadmap for such a spark server ? Thanks, On Thu, Dec 11, 2014 at 5:43 PM, Marcelo Vanzin wrote: > > Oops, sorry, fat fingers. > > We've been playing with something like that inside Hive: > https://github.com/apache/hive/tre

Re: Spark Server - How to implement

2014-12-11 Thread Marcelo Vanzin
Hi Manoj, I'm not aware of any public projects that do something like that, except for the Ooyala server which you say doesn't cover your needs. We've been playing with something like that inside Hive, though: On Thu, Dec 11, 2014 at 5:33 PM, Manoj Samel wrote: > Hi, > > If spark based services

Re: Spark Server - How to implement

2014-12-11 Thread Marcelo Vanzin
Oops, sorry, fat fingers. We've been playing with something like that inside Hive: https://github.com/apache/hive/tree/spark/spark-client That seems to have at least a few of the characteristics you're looking for; but it's a very young project, and at this moment we're not developing it as a pub

Spark Server - How to implement

2014-12-11 Thread Manoj Samel
Hi, If spark based services are to be exposed as a continuously available server, what are the options? * The API exposed to client will be proprietary and fine grained (RPC style ..), not a Job level API * The client API need not be SQL so the Thrift JDBC server does not seem to be option .. but