[GitHub] [spark] zhengruifeng commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

[GitBox]

zhengruifeng commented on pull request #31480:
URL: https://github.com/apache/spark/pull/31480#issuecomment-773752136


   this previous pr https://github.com/apache/spark/pull/29185 is stale, so I 
open this one.
   
   
   testCode:
   ```
   import org.apache.spark.HashPartitioner
   
   val data = sc.parallelize(Seq((0, 5), (3, 8), (2, 6), (0, 8), (3, 8), (1, 
3)), 2)
   val partitioner = new HashPartitioner(2)
   val agg = data.reduceByKey(partitioner, _ + _)
   agg.persist()
   agg.count
   val sorted = agg.repartitionAndSortWithinPartitions(partitioner)
   sorted.count
   ```
   
   master:
   
![repart-master-2021-02-05-10-58-46](https://user-images.githubusercontent.com/7322292/106984081-dff98780-67a1-11eb-801e-1c2a205b5bd0.png)
   
   this pr:
   
![repart-pr-2021-02-05-10-57-00](https://user-images.githubusercontent.com/7322292/106984096-e5ef6880-67a1-11eb-8ef6-1a0dea8bd0b1.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org