GitHub user aarondav opened a pull request: https://github.com/apache/spark/pull/1678
SPARK-2532: Minimal shuffle consolidation fixes All changes from this PR are by @mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle file consolidation that @mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1. This patch is **not** intended as a replacement for #1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests. If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable option. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aarondav/spark consol Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1678.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1678 ---- commit f298ac92f7df30f53636fca874c8bc6ed764af19 Author: Aaron Davidson <aa...@databricks.com> Date: 2014-07-28T19:02:24Z SPARK-2532: Minimal shuffle consolidation fixes All changes from this PR are by @mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle file consolidation that @mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1. This patch is **not** intended as a replacement for #1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests. If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable option. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---