Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3765
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70869778
looks good. Thanks @sryza
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23250764
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -115,26 +107,35 @@ private[yarn] class YarnAllocator(
new
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23251269
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70731891
@sryza did you run this through a bunch of manual tests as well? Try
cases like container dies/killed before all of the first ones allocated.
---
If your project is
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23263022
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70762709
@tgravescs, uploaded a new patch that addresses your review comments. I
just ran a bunch of manual tests on a 6-node, including
* request more resources than the
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23256149
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -43,18 +43,18 @@ object AllocationType extends Enumeration {
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23258368
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70749832
[Test build #25850 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25850/consoleFull)
for PR 3765 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70759190
[Test build #25850 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25850/consoleFull)
for PR 3765 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70759198
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23258867
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70762837
[Test build #25857 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25857/consoleFull)
for PR 3765 at commit
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23261539
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23258515
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23261733
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70769671
[Test build #25857 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25857/consoleFull)
for PR 3765 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-70769678
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23256094
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -28,14 +29,13 @@ import
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23253417
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23253526
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23254205
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r23254554
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -85,7 +85,7 @@ class YarnSparkHadoopUtil extends
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22984790
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22984854
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22950105
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -62,50 +62,42 @@ private[yarn] class YarnAllocator(
amClient:
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22950419
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-69882642
@tgravescs are you able to take a look at this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22933548
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -115,26 +107,35 @@ private[yarn] class YarnAllocator(
new
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22933978
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -153,498 +154,241 @@ private[yarn] class YarnAllocator(
}
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22934346
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -62,50 +62,42 @@ private[yarn] class YarnAllocator(
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22362922
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -115,26 +107,35 @@ private[yarn] class YarnAllocator(
new
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3765#discussion_r22363002
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -115,26 +107,35 @@ private[yarn] class YarnAllocator(
new
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68197234
[Test build #24847 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24847/consoleFull)
for PR 3765 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68198430
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68198429
[Test build #24847 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24847/consoleFull)
for PR 3765 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68078341
[Test build #24794 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24794/consoleFull)
for PR 3765 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68078366
[Test build #24794 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24794/consoleFull)
for PR 3765 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68078368
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68078397
Updated patch fixes the broken test, adds more comments, and simplifies
even further.
It also removes support for requesting containers based on locality, as
this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68080111
[Test build #24798 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24798/consoleFull)
for PR 3765 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68082537
[Test build #24798 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24798/consoleFull)
for PR 3765 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-68082539
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/3765
SPARK-1714. Take advantage of AMRMClient APIs to simplify logic in YarnA...
...llocator
The goal of this PR is to simplify YarnAllocator as much as possible and
get it up to the level of
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-67892552
[Test build #24715 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24715/consoleFull)
for PR 3765 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-67900217
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3765#issuecomment-67900210
[Test build #24715 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24715/consoleFull)
for PR 3765 at commit
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-61993923
@sryza can we just close this out until further work is done?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user sryza closed the pull request at:
https://github.com/apache/spark/pull/655
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-54780393
Unfortunately the cleanup refactored a bunch of common code between
yarn-alpha and yarn-stable that no longer would have been common after this
patch (because, after 2.2,
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-54877812
Hopefully not much longer but we need to have official proposal and make
sure others aren't using it. So I'm fine with waiting.
---
If your project is set up for
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-54307385
Hey @sryza. We put in some cleanup jira, would you be able to upmerge this?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-45571300
I tried testing this manually today and found that the locality stuff
doesn't work even without this patch -
https://issues.apache.org/jira/browse/SPARK-2089.
---
If your
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-45254365
cc @mridul as he wrote most of this code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/655#discussion_r13227671
--- Diff:
yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
---
@@ -105,278 +96,222 @@ private[yarn] class
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-44567226
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-44567239
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-44572188
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15292/
---
If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-44572185
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/655
SPARK-1714. Take advantage of AMRMClient APIs to simplify logic in YarnA...
...llocationHandler
This patch does a few things:
* Uses AMRMClient APIs for matching containers to requests.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-42255465
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-42255453
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-42257629
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-42257632
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14688/
---
If your project
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/655#discussion_r12307994
--- Diff:
yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
---
@@ -105,278 +96,222 @@ private[yarn] class
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/655#discussion_r12308167
--- Diff:
yarn/stable/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocationHandlerSuite.scala
---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/655#issuecomment-42259212
Looks OK to me; I'll test this out separately (too see if it has any effect
on the tests I'm currently running).
---
If your project is set up for it, you can reply to
68 matches
Mail list logo
