Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-128431151
I'm closing this in favor of the leveldb-based approach
https://github.com/apache/spark/pull/7943 -- but I'll make the relevant
suggestions from here
---
If your
Github user squito closed the pull request at:
https://github.com/apache/spark/pull/7839
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36436044
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +121,34 @@ private boolean
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-128027939
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36313470
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala
---
@@ -0,0 +1,101 @@
+/*
+* Licensed to the Apache
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-128068054
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-128068556
[Test build #39871 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39871/consoleFull)
for PR 7839 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-128117869
[Test build #39871 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39871/console)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-128118007
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36353966
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -252,4 +273,19 @@ public String toString()
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36354315
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -181,4 +249,44 @@ public ByteBuffer getMetaData() {
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36354984
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +121,34 @@ private boolean
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127814308
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127804134
[Test build #215 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/215/console)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127804164
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127787010
[Test build #39773 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39773/consoleFull)
for PR 7839 at commit
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36252212
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -252,4 +265,19 @@ public String toString()
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127785335
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127786058
[Test build #215 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/215/consoleFull)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127785427
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127783165
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127783359
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36252740
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -181,4 +249,38 @@ public ByteBuffer getMetaData() {
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36252347
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +121,34 @@ private boolean
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36252832
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +121,34 @@ private boolean
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127780335
[Test build #39748 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39748/console)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127780472
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36253637
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala ---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36253736
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala
---
@@ -0,0 +1,101 @@
+/*
+* Licensed to the Apache
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127749514
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127750539
[Test build #39748 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39748/consoleFull)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127749492
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127814291
[Test build #39773 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39773/console)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127444094
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127444117
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127417173
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127417190
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127419102
[Test build #39597 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39597/consoleFull)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127420210
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127439382
[Test build #39599 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39599/console)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127439506
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127419390
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127418728
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127418714
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127419386
[Test build #39597 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39597/console)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127417278
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127420483
[Test build #39599 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39599/consoleFull)
for PR 7839 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127444268
[Test build #39616 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39616/consoleFull)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127414327
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127414237
[Test build #39575 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39575/console)
for PR 7839 at commit
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36137193
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +119,33 @@ private boolean
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127420227
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127470714
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127470688
**[Test build #39616 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39616/console)**
for PR 7839 at commit
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36106227
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -181,4 +246,28 @@ public ByteBuffer getMetaData() {
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36106591
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala
---
@@ -0,0 +1,95 @@
+/*
+* Licensed to the Apache
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36106699
--- Diff:
yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceSuite.scala
---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36106780
--- Diff:
yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceSuite.scala
---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36120783
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -252,4 +264,20 @@ public String toString()
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36121140
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +119,33 @@ private boolean
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36120767
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -252,4 +264,20 @@ public String toString()
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36105847
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +119,33 @@ private boolean
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127334953
Approach looks ok to me, just left a few minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36105758
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +119,33 @@ private boolean
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36105507
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -252,4 +264,20 @@ public String toString()
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36106381
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala
---
@@ -0,0 +1,95 @@
+/*
+* Licensed to the Apache
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36107092
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -252,4 +264,20 @@ public String toString()
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127379677
[Test build #39575 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39575/consoleFull)
for PR 7839 at commit
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/7839#discussion_r36129114
--- Diff:
network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -100,11 +119,33 @@ private boolean
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127401757
I haven't looked at the code in detail, but did you consider using levelDB
which is what NM and MapReduce use? Perhaps that makes it more complicated for
non-yarn
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127378411
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-127378389
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126957369
[Test build #39379 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39379/console)
for PR 7839 at commit
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126948995
I think this is ready for review -- there are reasonable comments tests
in place.
Also I fixed the issue w/ a potential leak of applications which terminate
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126949670
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126957852
[Test build #39380 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39380/console)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126957866
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126948817
[Test build #39379 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39379/consoleFull)
for PR 7839 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126949664
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126957394
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7839#issuecomment-126949723
[Test build #39380 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39380/consoleFull)
for PR 7839 at commit
81 matches
Mail list logo
