[ https://issues.apache.org/jira/browse/SPARK-40660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-40660: -------------------------------- Issue Type: Bug (was: Improvement) > Switch to XORShiftRandom to distribute elements > ----------------------------------------------- > > Key: SPARK-40660 > URL: https://issues.apache.org/jira/browse/SPARK-40660 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.4.0 > Reporter: Yuming Wang > Assignee: Yuming Wang > Priority: Major > Fix For: 3.4.0, 3.3.1, 3.2.3 > > > {code:scala} > import java.util.Random > import org.apache.spark.util.random.XORShiftRandom > import scala.util.hashing > def distribution(count: Int, partition: Int) = { > println((1 to count).map(partitionId => new > Random(partitionId).nextInt(partition)) > .groupBy(f => f) > .map(_._2.size).mkString(". ")) > println((1 to count).map(partitionId => new > Random(hashing.byteswap32(partitionId)).nextInt(partition)) > .groupBy(f => f) > .map(_._2.size).mkString(". ")) > println((1 to count).map(partitionId => new > XORShiftRandom(partitionId).nextInt(partition)) > .groupBy(f => f) > .map(_._2.size).mkString(". ")) > } > distribution(200, 4) > {code} > {noformat} > 200 > 50. 60. 46. 44 > 55. 48. 43. 54 > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org