Re: Strings not converted when calling Scala code from a PySpark app

Makes sense. Thanks Holden. Alexis On Mon, Sep 12, 2016 at 5:28 PM, Holden Karau wrote: > Ah yes so the Py4J conversions only apply on the driver program - your > DStream however is RDDs of pickled objects. If you want to with a transform > function use Spark SQL

Re: Strings not converted when calling Scala code from a PySpark app

Ah yes so the Py4J conversions only apply on the driver program - your DStream however is RDDs of pickled objects. If you want to with a transform function use Spark SQL transferring DataFrames back and forth between Python and Scala spark can be much easier. On Monday, September 12, 2016, Alexis

Strings not converted when calling Scala code from a PySpark app

Hi, *TL;DR - I have what looks like a DStream of Strings in a PySpark application. I want to send it as a DStream[String] to a Scala library. Strings are not converted by Py4j, though.* I'm working on a PySpark application that pulls data from Kafka using Spark Streaming. My messages are