Hi, I'm trying to connect to Cassandra through PySpark using the spark-cassandra-connector from datastax based on the work of Mike Sukmanowsky.
I can use Spark and Cassandra through the datastax connector in Scala just fine. Where things fail in PySpark is that an exception is raised in org.apache.spark.api.python.PythonRDD.writeIteratorToStream(...) with the message 'Unexpected element type com.datastax.spark.connector.japi.CassandraRow'. So just to be sure: is it only possible to communicate between a Python Spark program and the rest of the Spark ecosystem through binary or UTF-8 strings? Is there no way to communicate a richer object with at least types like a float, etc.? Cheers, Frens
