[
https://issues.apache.org/jira/browse/SPARK-60763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60763:
-----------------------------------
Labels: pull-request-available (was: )
> Add migration note for the V2 scan grouping change in Spark 4.2
> ---------------------------------------------------------------
>
> Key: SPARK-60763
> URL: https://issues.apache.org/jira/browse/SPARK-60763
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
> Affects Versions: 4.2.0
> Reporter: Rakesh Kumar
> Priority: Major
> Labels: pull-request-available
>
> SPARK-55092 changed Spark 4.2 so that V2 scans no longer group input splits
> that share a reported partition key into one task unless an operator needs
> that grouping, such as a storage-partitioned join. Scans below a shuffle, or
> with no consumer that uses their partitioning, now run one task per split,
> which can add task overhead for connectors that report many small splits per
> key. Before 4.2, grouping multiple splits into one task also dropped their
> reported ordering.
> This change had no migration note. Add one to the "Upgrading from Spark SQL
> 4.1 to 4.2" section, and merge it to branch-4.2 as well as master. Raised in
> the review of SPARK-60265 (https://github.com/apache/spark/pull/59522).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
