GitHub user zhuoliu opened a pull request: https://github.com/apache/spark/pull/11060
[SPARK-11316] setupGroups in coalescedRDD causes super long delay. In coalescedRDD, the setupGroups causes super long delay due to the O(n^2) loop in the second while. That while is used to make sure that each PartitionGroup contains at least one partition. In some cases, this while will take O(n^2) to complete. If number of partitions is very large, this would take tens of minutes or even hours to complete. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhuoliu/spark 11316 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11060.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11060 ---- commit e202f579c994be49b255edd4014c0e7cfebaad78 Author: zhuol <zh...@yahoo-inc.com> Date: 2016-02-03T21:21:50Z [SPARK-11316] setupGroups in coalesceRDD causes super long delay due to the O(n^2) loop. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org