[
https://issues.apache.org/jira/browse/SPARK-54639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-54639.
----------------------------------
Fix Version/s: 4.2.0
Resolution: Fixed
Issue resolved by pull request 53387
[https://github.com/apache/spark/pull/53387]
> Optimize Arrow serializers by avoiding unnecessary Table creation
> -----------------------------------------------------------------
>
> Key: SPARK-54639
> URL: https://issues.apache.org/jira/browse/SPARK-54639
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Yicong Huang
> Assignee: Yicong Huang
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.0
>
>
> Several serializers in pyspark.sql.pandas.serializers unnecessarily create
> pa.Table objects when processing single RecordBatch instances. When
> converting Arrow RecordBatches to pandas Series, the code creates a pa.Table
> wrapper for each batch just to iterate over columns, which introduces
> unnecessary object creation, extra function call overhead, and increases GC
> pressure.
> The issue appears in multiple serializers:
> {code:python}
> # ArrowStreamPandasSerializer.load_stream()
> # ArrowStreamAggPandasUDFSerializer.load_stream()
> # GroupPandasUDFSerializer.load_stream()
> for batch in batches:
> pandas_batches = [
> self.arrow_to_pandas(c, i)
> for i, c in enumerate(pa.Table.from_batches([batch]).itercolumns())
> ]
> {code}
> We can optimize this by directly accessing columns from RecordBatch instead:
> {code:python}
> for batch in batches:
> pandas_batches = [
> self.arrow_to_pandas(batch.column(i), i)
> for i in range(batch.num_columns)
> ]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
