[
https://issues.apache.org/jira/browse/SPARK-32846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-32846:
-----------------------------------
Labels: arrow pandas pull-request-available sql (was: arrow pandas sql)
> Support createDataFrame from an RDD of pd.DataFrames
> ----------------------------------------------------
>
> Key: SPARK-32846
> URL: https://issues.apache.org/jira/browse/SPARK-32846
> Project: Spark
> Issue Type: New Feature
> Components: PySpark, SQL
> Affects Versions: 3.0.1
> Reporter: Linar Savion
> Priority: Minor
> Labels: arrow, pandas, pull-request-available, sql
>
> Add support to createDataFrame from a distributed collection of
> pandas.DataFrames by converting the RDD of pd.DFs to an RDD of arrow records
> batches, then directly creating the spark DataFrame from it.
>
> Performance is significantly better (vectorized) than creating a spark DF by
> converting each df to a list of rows, similar to the improvement of
> SPARK-20791.
>
> Initial example & benchmark for older spark versions:
> [https://gist.github.com/linar-jether/7dd61ed6fa89098ab9c58a1ab428b2b5|https://gist.github.com/linar-jether/7dd61ed6fa89098ab9c58a1ab428b2b5,]
>
> I'm currently working on a PR and will post it soon.
>
> Extends the work done in:
> https://issues.apache.org/jira/browse/SPARK-20791
> https://issues.apache.org/jira/browse/SPARK-23030
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
