Re: .NET for Apache Spark interactive interpreter support

@Saisai , Is there a design doc for the python/r/sql interpreter that I can use as a template? It will be very similar (most similar to R interpreter in fact) in design. Regards, -Steve On Tue, Jan 14, 2020 at 6:13 PM Steve Suh wrote: > Jeff you are correct. .NET for Spark behaves similarly

Re: .NET for Apache Spark interactive interpreter support

Jeff you are correct. .NET for Spark behaves similarly to how pyspark and sparkR interacts with spark. Much like its counterparts, .NET for Spark acts as a wrapper to the underlying spark apis. Besides adding a SparkDotnetInterpreter to the livy codebase, a user will need to install a REPL on

Re: .NET for Apache Spark interactive interpreter support

IIUC, donet spark is just a wrapper of spark. Besides adding DotNetSparkInterpreter, what kind of other things user need to do ? For now, user also need to specify SPARK_HOME at least, but to support donet, what other things user need to do. Saisai Shao 于2020年1月14日周二 上午11:01写道： > I see your

Re: .NET for Apache Spark interactive interpreter support

I see your point. Personally I don't have a strong opinion on this, I'm not sure what others think about it. Why don't you start a design doc and call for a vote about this feature. Thanks Saisai Steve Suh 于2020年1月14日周二 上午10:12写道： > @Saisai > > This will not be a wrapper around the REST API.

Re: .NET for Apache Spark interactive interpreter support

@Saisai This will not be a wrapper around the REST API. The plan is to support a Livy POST /sessions request where kind == "sparkdotnet". Livy will eventually push this request down to the ReplDriver, and in turn the ReplDriver will create a new SparkDotnet Interpreter class. This will be

Re: .NET for Apache Spark interactive interpreter support

Is this just a wrapper of Livy REST API, or it could support Livy Job API? >From my opinion, if it is just a wrapper of REST API, then it would be better to maintain out of Livy, since REST API is language independent, if we're going to have all the languages support in Livy, then it is hard to

.NET for Apache Spark interactive interpreter support

Hi, I contribute to the .NET for Apache Spark project and we have seen *a lot* of *user interest* in providing first class notebook support for *.NET for Apache Spark*. Livy currently supports *Scala*, *Python *and *R **interactive sessions*. We have a working