[
https://issues.apache.org/jira/browse/SPARK-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576739#comment-15576739
]
Michael Armbrust commented on SPARK-12776:
------------------------------------------
I would love to see better support here, but I don't think anyone has taken the
time to flesh out the API. Some suggestions I've heard are too support
arbitrary objects in Datasets that are pickled and stored as
{{ArrayType(BytesType)}}. If you want schema, you could also tell us that,
either with a schema string (where we use the schema to extract columns out of
the object) or by using something like a named tuple. This is all very rough
though and I knew when we did a prototype we ran into issues caused by the
batching we use when talking to the JVM (essentially things like {{df.count()}}
broke). If someone wants to flesh out these proposals that would be great.
> Implement Python API for Datasets
> ---------------------------------
>
> Key: SPARK-12776
> URL: https://issues.apache.org/jira/browse/SPARK-12776
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Reporter: Kevin Cox
> Priority: Minor
>
> Now that the Dataset API is in Scala and Java it would be awesome to see it
> show up in PySpark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
