Re: .NET for Apache Spark interactive interpreter support

2020-01-14 Thread Steve Suh
Jeff you are correct. .NET for Spark behaves similarly to how pyspark and sparkR interacts with spark. Much like its counterparts, .NET for Spark acts as a wrapper to the underlying spark apis. Besides adding a SparkDotnetInterpreter to the livy codebase, a user will need to install a REPL on

Re: [VOTE] Release Livy 0.7.0 based on RC4

2020-01-14 Thread Saisai Shao
Thanks all, We have 3 +1 (binding), 3 +1 (non-binding), vote is passed. I will start another vote on incubating general. Thanks Saisai Jeff Zhang 于2020年1月14日周二 下午12:07写道: > +1 > > Bikas Saha 于2020年1月11日周六 下午9:40写道: > > > +1 > > > > Bikas > > > > From:

Re: Livy on Kubernetes support

2020-01-14 Thread Pat Ferrel
+1 from another user fwiw. We also have livy containers and helm charts. The real problem is deploying a Spark Cluster in k8s. We know of no working images for this. The Spark team seems focused on deploying Jobs with k8s, which is fine but is not enough. We need to deploy Spark itself.  We

Re: Livy on Kubernetes support

2020-01-14 Thread Marco Gaido
Hi Aliaksandr, thanks for your email and you work on this feature. As I mentioned to you in the PR, I agree with you on the usefulness of this feature and you have a big +1 from me having it in Livy. Unfortunately, it is not my area of expertise, so I don't feel confident merging it without other

Livy on Kubernetes support

2020-01-14 Thread Aliaksandr Sasnouskikh
Hi community, About a year ago I've started to work on the patch to Apache Livy for Spark on Kubernetes support in the scope of the project I've been working on. Since that time I've created a PR https://github.com/apache/incubator-livy/pull/167 which have already been discussed and reviewed a