[
https://issues.apache.org/jira/browse/SPARK-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199516#comment-16199516
]
Li Jin commented on SPARK-22216:
--------------------------------
[~hyukjin.kwon],
My intention is to keep this open to track all the features that we want to do
related to PySpark/Pandas interoperability and keep adding sub tasks as they
come up.
For existing Jiras, there are speed up to '"createDataFrame" and support for
complex types.
For new features, there are more functions that I think we can add pandas udf
support to, for instance, window functions. I haven't created Jiras for them
yet.
> Improving PySpark/Pandas interoperability
> -----------------------------------------
>
> Key: SPARK-22216
> URL: https://issues.apache.org/jira/browse/SPARK-22216
> Project: Spark
> Issue Type: Epic
> Components: PySpark
> Affects Versions: 2.2.0
> Reporter: Li Jin
>
> This is an umbrella ticket tracking the general effect of improving
> performance and interoperability between PySpark and Pandas. The core idea is
> to Apache Arrow as serialization format to reduce the overhead between
> PySpark and Pandas.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
