Re: Creating a python port for a Scala Spark Projeect

Thank you Holden, I look forward to watching your talk! On Wed, Jun 22, 2016 at 7:12 PM Holden Karau wrote: > PySpark RDDs are (on the Java side) are essentially RDD of pickled objects > and mostly (but not entirely) opaque to the JVM. It is possible (by using > some

Re: Creating a python port for a Scala Spark Projeect

PySpark RDDs are (on the Java side) are essentially RDD of pickled objects and mostly (but not entirely) opaque to the JVM. It is possible (by using some internals) to pass a PySpark DataFrame to a Scala library (you may or may not find the talk I gave at Spark Summit useful

Creating a python port for a Scala Spark Projeect

Hi All, I've developed a spark module in scala that I would like to add a python port for. I want to be able to allow users to create a pyspark RDD and send it to my system. I've been looking into the pyspark source code as well as py4J and was wondering if there has been anything like this