[
https://issues.apache.org/jira/browse/SPARK-61561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61561:
-----------------------------------
Labels: pull-request-available (was: )
> TABLESAMPLE REPEATABLE (and seeded Dataset.sample) over ORDER BY returns
> different rows on every execution because RangePartitioner seeds bounds with
> the RDD id
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61561
> URL: https://issues.apache.org/jira/browse/SPARK-61561
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Linhong Liu
> Priority: Major
> Labels: pull-request-available
>
> RangePartitioner derives the seeds for the samples that pick its range bounds
> from the RDD id (RangePartitioner.sketch uses shift = rdd.id, and the
> imbalanced re-sample uses byteswap32(-rdd.id - 1)). A global sort builds a
> new RDD, with a new id, on every execution, so the range bounds of the same
> data move between runs. A seeded TABLESAMPLE ... REPEATABLE (or a seeded
> Dataset.sample) over an ORDER BY therefore returns different rows each time.
> Fix: derive the sketch and re-sample seeds from the partition index only, not
> the RDD id. A new conf spark.rdd.rangePartitioner.legacyRddIdSeed (default
> false) restores the previous behavior. Found via correctness analysis.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]