Re: Spark Core and ways of talking to it for enhancing application language support

available / features supported in each language. So for Haskell, one would need to see what is the best way to call the underlying Java API functions from Haskell and get results back. Thanks Shivaram On Mon, Jul 13, 2015 at 8:51 PM, Vasili I. Galchin vigalc...@gmail.com javascript:_e(%7B%7D,'cvml

Spark Core and ways of talking to it for enhancing application language support

Hello, So far I think there are at two ways (maybe more) to interact from various programming languages with the Spark Core: PySpark API and R API. From reading code it seems that PySpark approach and R approach are very disparate ... with the latter using the R-Java bridge.

Spark application examples

Hello, Reading slides entitled DATABRICKS written by Holden Karau, et. al. I am also reading Spark application examples under ../spark/examples/src/main/*. Let's assume examples Driver data-manipulation.R and dataframe.R. Question: where in these Drivers are the worker bees

language-independent RDD Spark core code?

I am looking at R side, but curious what the RDD core side looks like. Not sure which directory to look inside. ?? Thanks, Vasili - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail:

Re: language-independent RDD Spark core code?

think I found this RDD code On Fri, Jul 10, 2015 at 7:00 PM, Vasili I. Galchin vigalc...@gmail.com wrote: I am looking at R side, but curious what the RDD core side looks like. Not sure which directory to look inside. ?? Thanks, Vasili

Re: callJMethod?

them over a socket to the JVM. This is the socket-based R to JVM bridge described in the Spark Summit talk https://spark-summit.org/2015/events/sparkr-the-past-the-present-and-the-future/ Thanks Shivaram On Thu, Jul 9, 2015 at 4:28 PM, Vasili I. Galchin vigalc...@gmail.com wrote: Hello

PySpark vs R

Hello, Just trying to get up to speed ( a week .. pls be patient with me). I have been reading several docs .. plus ... reading PySpark vs R code. I don't see an invariant between the Python and R implementations. ?? Probably I should read native Scala code, yes? Kind thx, Vasili

Re: callJMethod?

the function names, arguments and sends them over a socket to the JVM. This is the socket-based R to JVM bridge described in the Spark Summit talk https://spark-summit.org/2015/events/sparkr-the-past-the-present-and-the-future/ Thanks Shivaram On Thu, Jul 9, 2015 at 4:28 PM, Vasili I

Spark and Haskell support

Hello, 1) I have been rereading kind email responses to my Spark queries. Thx. 2) I have also been reading R code: 1) RDD.R 2) DataFrame.R 3) All following API's = https://cwiki.apache.org/confluence/display/SPARK/Spark+Internals 4) Python ...

callJMethod?

Hello, I am reading R code, e.g. RDD.R, DataFrame.R, etc. I see that callJMethod is repeatedly call. Is callJMethod part of the Spark Java API? Thx. Vasili

Re: how can I write a language wrapper?

written by Josh Rosen -- It covers some of the issues like closure capture, serialization, JVM communication that you'll need to handle for a new language. Thanks Shivaram On Tue, Jun 23, 2015 at 1:35 PM, Vasili I. Galchin vigalc...@gmail.com wrote: Hello, I want to add language support

Re: R - Scala interface used in Spark?

for Haskell?? On Fri, Jun 26, 2015 at 5:21 PM, Vasili I. Galchin vigalc...@gmail.com wrote: How about Python?? On Friday, June 26, 2015, Shivaram Venkataraman shiva...@eecs.berkeley.edu wrote: We don't use the rscala package in SparkR -- We have an in built R-JVM bridge that is customized

Re: R - Scala interface used in Spark?

Spark Summit 2015 talk. Thanks Shivaram On Fri, Jun 26, 2015 at 3:19 PM, Vasili I. Galchin vigalc...@gmail.com javascript:_e(%7B%7D,'cvml','vigalc...@gmail.com'); wrote: A friend sent the below: http://cran.r-project.org/web/packages/rscala/index.html Is this the glue between R

Re: R - Scala interface used in Spark?

. Thanks Shivaram On Fri, Jun 26, 2015 at 3:19 PM, Vasili I. Galchin vigalc...@gmail.com javascript:_e(%7B%7D,'cvml','vigalc...@gmail.com'); wrote: A friend sent the below: http://cran.r-project.org/web/packages/rscala/index.html Is this the glue between R and Scala that is used

R - Scala interface used in Spark?

A friend sent the below: http://cran.r-project.org/web/packages/rscala/index.html Is this the glue between R and Scala that is used in Spark? Vasili

Re: how can I write a language wrapper?

. Thanks Shivaram On Tue, Jun 23, 2015 at 1:35 PM, Vasili I. Galchin vigalc...@gmail.com javascript:_e(%7B%7D,'cvml','vigalc...@gmail.com'); wrote: Hello, I want to add language support for another language(other than Scala, Java et. al.). Where is documentation that explains to provide

how can I write a language wrapper?

Hello, I want to add language support for another language(other than Scala, Java et. al.). Where is documentation that explains to provide support for a new language? Thank you, Vasili