GitHub user corruptmemory opened a pull request:
https://github.com/apache/spark/pull/13279
[SPARK-12583][MESOS] Mesos shuffle service: Don't delete shuffle fileâ¦
## What changes were proposed in this pull request?
This is a backport of https://github.com/apache/spark/pull
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11272
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-196484633
OK, I'm going to merge this into master. This *could* potentially go into
1.6 but it's a little scary to merge it there. I'm inclined to leave it out of
1.6 to be co
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195952284
I tested the rebased version manually. It is still working after the rebase.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195760735
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195760731
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195760285
**[Test build #53004 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53004/consoleFull)**
for PR 11272 at commit
[`3a9914b`](https://g
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195739569
Sorry about the long silence. Rebased the PR. I haven't had the time yet to
test it after the rebase. I should be able to do this until the beginning of
next week.
---
Github user bbossy commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r55921243
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark.dep
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195737081
**[Test build #53004 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53004/consoleFull)**
for PR 11272 at commit
[`3a9914b`](https://gi
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195727884
AFAIK this could go in. I did test it manually and things worked well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on Gi
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195603737
end of month I think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have t
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195576917
@andrewor14 I'll try to create a test to verify this, when is the 2.0
closing date?
---
If your project is set up for it, you can reply to this email and have your
rep
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-195506145
What's the status on this patch? Have we at least tested it manually? At
the very least we should merge it into 2.0.
---
If your project is set up for it, you can r
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-190833991
@dragos Yes it shouldn't block the PR, just mentioning it as I see the need
for it. I have two comments on this, otherwise it LGTM too.
---
If your project is set up f
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-190709339
Yes, we could have an integration test for this, shouldn't be too hard to
add. The basic idea is to decrease the network timeout and have a job that
exceeds that but sti
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-190475702
This looks like a great candidate to add integration tests for in the
mesos-spark-integration-tests suite. Ideally we have something long running
that we can run. @drag
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r54507553
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/mesos/MesosExternalShuffleClient.java
---
@@ -53,21 +65,58 @@ public MesosExternalS
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r54506572
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -93,7 +113,8 @@ private[mesos] class MesosExternalShuff
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-190353858
Also cc @tnachen and @mgummelt for another pair of eyes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as wel
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-190353767
@dragos any other comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53827666
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53806998
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark.dep
Github user bbossy commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53806198
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark.dep
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53802880
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/BlockTransferMessage.java
---
@@ -40,7 +41,8 @@
/** Preceding
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53802611
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark.dep
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53798220
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark.dep
Github user bbossy commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53796593
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark.dep
Github user bbossy commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53795749
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/BlockTransferMessage.java
---
@@ -40,7 +41,8 @@
/** Preceding
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53795420
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/BlockTransferMessage.java
---
@@ -40,7 +41,8 @@
/** Preceding
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187610813
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187610809
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187610448
**[Test build #51739 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51739/consoleFull)**
for PR 11272 at commit
[`0a2d4cb`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187570175
**[Test build #51739 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51739/consoleFull)**
for PR 11272 at commit
[`0a2d4cb`](https://gi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187447458
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187447456
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187447206
**[Test build #51670 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51670/consoleFull)**
for PR 11272 at commit
[`d4d1ad7`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187401085
**[Test build #51670 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51670/consoleFull)**
for PR 11272 at commit
[`d4d1ad7`](https://gi
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187329496
(Update: this will have to go into 1.6.2 since the 1.6.1 RC is being cut
today already)
---
If your project is set up for it, you can reply to this email and have y
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53674382
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53674006
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/BlockTransferMessage.java
---
@@ -40,7 +41,8 @@
/** Preced
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53673482
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,88 @@
package org.apache.spark
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187320271
I would like to backport a variant of this patch into 1.6.1. This was
clearly working in 1.5 but then became unusable in 1.6. This means people who
upgrade to 1.6 wi
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187306670
@dragos Thanks for the review! Let me know if you find anything else.
---
If your project is set up for it, you can reply to this email and have your
reply appear on Git
Github user bbossy commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53665765
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,92 @@
package org.apache.spark.dep
Github user bbossy commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53665784
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,92 @@
package org.apache.spark.dep
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-187217805
I'm done reviewing, I only have a couple of small observations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53634517
--- Diff:
network/shuffle/src/main/java/org/apache/spark/network/shuffle/mesos/MesosExternalShuffleClient.java
---
@@ -53,21 +65,55 @@ public MesosExternalSh
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53629422
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,92 @@
package org.apache.spark.dep
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11272#discussion_r53629124
--- Diff:
core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala
---
@@ -17,69 +17,92 @@
package org.apache.spark.dep
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186394697
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186394701
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186394414
**[Test build #51567 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51567/consoleFull)**
for PR 11272 at commit
[`9a3d625`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186354052
**[Test build #51567 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51567/consoleFull)**
for PR 11272 at commit
[`9a3d625`](https://gi
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186344308
Oops. Sorry about that..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have th
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186335134
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186335131
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186335125
**[Test build #51562 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51562/consoleFull)**
for PR 11272 at commit
[`bd30655`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186334787
**[Test build #51562 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51562/consoleFull)**
for PR 11272 at commit
[`bd30655`](https://gi
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186328158
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186283689
I'll have to come back to this on Monday. Judging by the description, it
looks good.
---
If your project is set up for it, you can reply to this email and have your
rep
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186234486
@dragos @tnachen : Let me know if you had something else in mind (or if I
should wire it up differently)
---
If your project is set up for it, you can reply to this ema
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11272#issuecomment-186234197
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your p
GitHub user bbossy opened a pull request:
https://github.com/apache/spark/pull/11272
[SPARK-12583][Mesos] Mesos shuffle service: Don't delete shuffle files
before application has stopped
## Problem description:
Mesos shuffle service is completely unusable since Spark 1.6.0
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/11207#issuecomment-184623985
I also agree with @dragos, and I think we should keep the same semantics by
having a heartbeat instead.
---
If your project is set up for it, you can reply to this ema
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11207#issuecomment-184622520
@dragos You're right. I'll have a look at your proposals.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user bbossy closed the pull request at:
https://github.com/apache/spark/pull/11207
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/11207#issuecomment-184619429
@bbossy thanks for picking this up!
I have a problem with the bandwidth this design implies. For instance, my
state.json is 200KB (a cluster of 1 master and 2 no
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11207#issuecomment-184568596
@dragos Take a look, please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user bbossy commented on the pull request:
https://github.com/apache/spark/pull/11207#issuecomment-184563434
@JoshRosen changed to a more descriptive title and added a more detailed
problem description.
---
If your project is set up for it, you can reply to this email and have
70 matches
Mail list logo
