[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160271599 **[Test build #46839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46839/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-28 Thread lianhuiwang
GitHub user lianhuiwang opened a pull request: https://github.com/apache/spark/pull/10024 [SPARK-4452][Core]Shuffle data structures can starve others on the same thread for memory In #9241 It implemented a mechanism to call spill() on those SQL operators that support spilling if

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160284311 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160284297 **[Test build #46839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46839/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-08-31 Thread lianhuiwang
Github user lianhuiwang closed the pull request at: https://github.com/apache/spark/pull/7130 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-125782798 /bump @andrewor14 @rxin, I think that we should look at this before 1.5. I share @lianhuiwang's concerns that our new usages of ShuffleMemoryManager will make the

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-121071219 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117744398 [Test build #36281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36281/consoleFull) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117793286 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117793176 [Test build #36281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36281/console) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117708565 [Test build #36270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36270/consoleFull) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117709747 [Test build #36270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36270/console) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117708291 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117708317 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117709756 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117744187 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-07-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117744164 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread lianhuiwang
Github user lianhuiwang commented on a diff in the pull request: https://github.com/apache/spark/pull/7130#discussion_r33641851 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -603,8 +539,165 @@ private[spark] class ExternalSorter[K, V,

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117250461 [Test build #36155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36155/consoleFull) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117284489 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread lianhuiwang
GitHub user lianhuiwang opened a pull request: https://github.com/apache/spark/pull/7130 [SPARK-4452][Core]Shuffle data structures can starve others on the same thread for memory Currently when there are 2 spillable objects in a thread, ShuffleMemoryManager does not work well.

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117247122 [Test build #36152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36152/consoleFull) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7130#discussion_r33594302 --- Diff: core/src/main/scala/org/apache/spark/util/collection/CollectionSpillable.scala --- @@ -17,14 +17,15 @@ package

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117250168 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117250932 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7130#discussion_r33594663 --- Diff: core/src/test/scala/org/apache/spark/shuffle/ShuffleMemoryManagerSuite.scala --- @@ -307,4 +321,20 @@ class ShuffleMemoryManagerSuite extends

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7130#discussion_r33594631 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -603,8 +539,165 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117283344 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117284587 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117284901 [Test build #36171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36171/consoleFull) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117284520 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117250927 [Test build #36155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36155/console) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117258030 [Test build #36157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36157/consoleFull) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117264562 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117264635 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117275891 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117275920 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117258895 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117258885 [Test build #36157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36157/console) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117257475 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117257410 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117246949 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117246917 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117250197 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117266992 [Test build #36162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36162/consoleFull) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117267854 [Test build #36162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36162/console) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117267874 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117247666 [Test build #36152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36152/console) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117247671 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117278669 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117278750 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117329702 [Test build #36171 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36171/console) for PR 7130 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-06-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7130#issuecomment-117329826 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

<    1   2