[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-30 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53950022 Merging this now. I will take care of some minor things myself. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1241 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-29 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53841329 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53841355 Jenkins, ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53841508 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19454/consoleFull) for PR 1241 at commit

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53844784 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19454/consoleFull) for PR 1241 at commit

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16928153 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockId.scala --- @@ -59,6 +59,11 @@ case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int)

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-28 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16825748 --- Diff: core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala --- @@ -181,17 +171,30 @@ class ShuffleBlockManager(blockManager:

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-28 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53686699 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759047 --- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockManager.scala --- @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759034 --- Diff: core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala --- @@ -15,21 +15,24 @@ * limitations under the License. */

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759141 --- Diff: core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala --- @@ -17,14 +17,18 @@ package org.apache.spark.shuffle.sort

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759235 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -34,8 +34,11 @@ import org.apache.spark.io.CompressionCodec import

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759256 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -85,18 +87,62 @@ private[spark] class BlockManager( val

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759266 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -107,6 +153,11 @@ private[spark] class BlockManager( } }

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759276 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -573,7 +634,7 @@ private[spark] class BlockManager( serializer:

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759291 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -573,7 +634,7 @@ private[spark] class BlockManager( serializer:

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759331 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -89,25 +89,33 @@ private[spark] class DiskStore(blockManager: BlockManager,

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53531388 Thanks for doing this. To help with the review, can you write a short design doc discussing the interfaces between different components, similar to the one attached here

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759577 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -89,25 +89,33 @@ private[spark] class DiskStore(blockManager: BlockManager,

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759644 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -121,20 +129,16 @@ private[spark] class DiskStore(blockManager: BlockManager,

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759699 --- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockManager.scala --- @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759777 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759788 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759807 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759871 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759951 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16759963 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16760004 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16760032 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16760213 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16760248 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleManager.scala --- @@ -25,6 +25,9 @@ import org.apache.spark.shuffle._ * mapper

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16760326 --- Diff: core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala --- @@ -181,17 +171,30 @@ class ShuffleBlockManager(blockManager:

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16760826 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -107,6 +153,11 @@ private[spark] class BlockManager( } }

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16760900 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -107,6 +153,11 @@ private[spark] class BlockManager( } }

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16762489 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -121,20 +129,16 @@ private[spark] class DiskStore(blockManager: BlockManager,

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16762641 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -121,20 +129,16 @@ private[spark] class DiskStore(blockManager: BlockManager,

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-27 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r16763705 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -121,20 +129,16 @@ private[spark] class DiskStore(blockManager: BlockManager,

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-26 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53519186 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-26 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53522043 Thanks for keeping this up to date @colorant -- going to take a look soon. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-25 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-53368876 @mateiz @rxin , rebased upon netty shuffle changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52014668 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52014901 QA tests have started for PR 1241. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18425/consoleFull ---

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52016533 QA results for PR 1241:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52018301 QA tests have started for PR 1241. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18435/consoleFull ---

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52018311 QA results for PR 1241:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52139906 QA tests have started for PR 1241. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18515/consoleFull ---

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52146432 @mateiz rebased to latest master code. Have time to take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52010865 QA tests have started for PR 1241. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18417/consoleFull ---

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52010877 QA results for PR 1241:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-12 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52011647 hmm, a local RAT check did pass. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-31 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-50725957 @colorant thanks for doing this, but we should probably wait to merge it until after 1.1, since it doesn't add critical fixes for 1.1. I'll take a look at it after I'm

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-50732503 QA tests have started for PR 1241. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17572/consoleFull ---

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-50737111 QA results for PR 1241:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48569640 @rxin you mean by add ProblemFilters.exclude[MissingMethodProblem](org.apache.spark.storage.DiskStore.getValues)? Then, this will not be needed for later usage

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48569956 Actually it's still necessary once the PR gets merged. It basically compares all the public interfaces at bytecode level against 1.0, and see if any public interfaces

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48571858 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48571873 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48571955 QA tests have started for PR 1241. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16493/consoleFull ---

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48574743 QA results for PR 1241:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48576862 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-10 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48683259 @rxin Any thing else I can do? ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14695968 --- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala --- @@ -49,8 +49,12 @@ private[spark] trait ShuffleManager {

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14695993 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleBlockManager.scala --- @@ -59,8 +64,10 @@ private[spark] trait ShuffleWriterGroup {

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696031 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleBlockManager.scala --- @@ -59,8 +64,10 @@ private[spark] trait ShuffleWriterGroup {

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696062 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleBlockManager.scala --- @@ -156,17 +162,25 @@ class ShuffleBlockManager(blockManager:

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696089 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleManager.scala --- @@ -24,7 +24,10 @@ import org.apache.spark.shuffle._ * A

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696085 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleManager.scala --- @@ -49,12 +52,21 @@ class HashShuffleManager(conf: SparkConf) extends

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696083 --- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala --- @@ -49,8 +49,12 @@ private[spark] trait ShuffleManager {

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696114 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala --- @@ -25,6 +25,7 @@ import org.apache.spark.executor.ShuffleWriteMetrics

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696189 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -72,23 +74,48 @@ private[spark] class BlockManager( // Actual storage of

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696261 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -72,23 +74,48 @@ private[spark] class BlockManager( // Actual storage of

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696298 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -72,23 +74,48 @@ private[spark] class BlockManager( // Actual storage of

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696400 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -72,23 +74,48 @@ private[spark] class BlockManager( // Actual

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/1241#discussion_r14696621 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -72,23 +74,48 @@ private[spark] class BlockManager( // Actual

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48558907 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48558914 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48558966 QA tests have started for PR 1241. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16486/consoleFull ---

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48559404 @rxin addressed comments. Among them: 1. Rename HashShuffleBlockManager to FileShuffleBlockManager since it mainly handle block in File way. Though say sort

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48561260 QA results for PR 1241:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48561265 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16486/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48561263 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-09 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48563750 Test fail is due to API changes in this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48280584 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48280591 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48281236 Sorry will take a look tomorrow! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48284014 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16396/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-07-06 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-48135413 ping @rxin ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47491606 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47493566 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47493571 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47493840 @rxin Moved getBlockLocation method from shuffleBlockManager to HashShuffleBlockMananger to make the interface more general. Does current interface looks reasonable for

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495499 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16256/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495497 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16253/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495494 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495493 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495607 Thanks - we are all super busy with Spark Summit this week so probably will get to this later in the week... feel free to send a reminder if I don't revisit this towards

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47311880 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47311892 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-06-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47314420 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16199/ --- If your project is set up for it, you can

  1   2   >