[ 
https://issues.apache.org/jira/browse/SPARK-61296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Utkarsh Bajaj updated SPARK-61296:
----------------------------------
    Description: CollectLimitExec moves all sorted partitions into one 
partition/task through a shuffle, and then keeps the rows between OFFSET and 
LIMIT. The shuffle hands back the partitions in the order they arrive, not in 
partition order. So the rows can come back out of order, and OFFSET can skip 
the wrong rows for order by queries.  (was: CollectLimitExec moves all sorted 
partitions into one partition/task through a shuffle, and then keeps the rows 
between OFFSET and LIMIT. The shuffle hands back the partitions in the order 
they arrive, not in partition order. So the rows can come back out of order, 
and OFFSET can skip the wrong rows.)

> Return CollectLimitExec gather results in partition order
> ---------------------------------------------------------
>
>                 Key: SPARK-61296
>                 URL: https://issues.apache.org/jira/browse/SPARK-61296
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle, Spark Core
>    Affects Versions: 4.0.5
>            Reporter: Utkarsh Bajaj
>            Priority: Major
>
> CollectLimitExec moves all sorted partitions into one partition/task through 
> a shuffle, and then keeps the rows between OFFSET and LIMIT. The shuffle 
> hands back the partitions in the order they arrive, not in partition order. 
> So the rows can come back out of order, and OFFSET can skip the wrong rows 
> for order by queries.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to