Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-163069474
@steveloughran , by default Yarn will preempt the container 15 seconds
after the warning. Yarn RM gives the warning to AM depends on the cluster usage
at that time, it
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-162965678
Jerry: do you know when YARN visits this decisions about having things
pre-emptible? That is: if you are given a warning dos that mean the container
will go any ti
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-153984512
Hi @mccheah sorry for late response, looks like this message is out of my
radar.
From Yarn's side, Yarn makes this preemption decision depends on status of
th
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-147785192
Quick question - Is it guaranteed that if the AM is told "Container X may
be preempted" it necessitates that "Container X will definitely be preempted"?
We should link d
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-144157425
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-144157243
[Test build #43091 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43091/console)
for PR 7786 at commit
[`57427f7`](https://github.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-144157421
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-144103185
[Test build #43091 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43091/consoleFull)
for PR 7786 at commit
[`57427f7`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-144101960
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-144101993
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-143934782
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40485255
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40484168
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -449,6 +458,13 @@ class CoarseGrainedSchedulerBack
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40484126
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to th
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40484088
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to th
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40484064
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to th
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40483975
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -449,6 +458,13 @@ class CoarseGrainedSchedulerBack
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142786412
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142786410
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142786314
[Test build #42940 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42940/console)
for PR 7786 at commit
[`41bebec`](https://github.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142764190
Hi @vanzin , I just updated the code and added unit test. Would you please
help to review again? Thanks a lot.
Also I discussed with Yarn guys about how to use
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142763867
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142763869
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142763459
[Test build #42940 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42940/consoleFull)
for PR 7786 at commit
[`41bebec`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142762310
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142762300
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142761244
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142761286
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40236632
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -346,7 +347,16 @@ private[spark] class ApplicationMaster(
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142463462
Yes, releasing some idle executors to alleviate the preemption in RM is a
good advise, basically I think there're two aspect of things we can handle
about preemption:
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142333162
Thanks @vanzin for your review, I will update the code according to your
comments.
---
If your project is set up for it, you can reply to this email and have your
rep
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-142122398
@jerryshao could you rebase the PR and address the feedback?
One thing that would be nice is to actually take action on the driver to
get rid of executors that ar
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40031826
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -238,6 +257,33 @@ private[yarn] class YarnAllocator(
}
}
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40031705
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -238,6 +257,33 @@ private[yarn] class YarnAllocator(
}
}
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40031211
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -346,7 +347,16 @@ private[spark] class ApplicationMaster(
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40030528
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -191,7 +194,7 @@ class CoarseGrainedSchedulerBacke
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r40030138
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
---
@@ -94,4 +94,5 @@ private[spark] object CoarseGrainedCl
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-139620952
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-139620947
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-139620720
[Test build #42340 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42340/console)
for PR 7786 at commit
[`90dd2f8`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-139586478
[Test build #42340 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42340/consoleFull)
for PR 7786 at commit
[`90dd2f8`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-139583724
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have thi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-139583764
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this f
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-127141123
Thanks a lot @CodingCat for your review, I will keep consistent about
variable names. Besides I'm not familiar with Mesos, if Mesos supports resource
preemption, I thi
Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126915231
is the same problem existing in Mesos?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r36033500
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -338,6 +338,7 @@ private[spark] class ApplicationMaster(
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r36033498
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -111,6 +124,16 @@ private[spark] abstract class YarnSche
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/7786#discussion_r36033488
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
---
@@ -94,4 +94,5 @@ private[spark] object CoarseGraine
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126611487
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126611454
[Test build #39178 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39178/console)
for PR 7786 at commit
[`90dd2f8`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126585493
[Test build #39178 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39178/consoleFull)
for PR 7786 at commit
[`90dd2f8`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126585064
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126585019
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126239171
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126238937
[Test build #39019 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39019/console)
for PR 7786 at commit
[`6d030ec`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126207375
[Test build #39019 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39019/consoleFull)
for PR 7786 at commit
[`6d030ec`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126206739
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7786#issuecomment-126206756
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/7786
[SPARK-9468][Yarn][Core] Avoid scheduling tasks on preemption executors
For Spark running on Yarn, if preemption is enabled by Yarn, containers
will be preempted by Yarn, this will be notified to
59 matches
Mail list logo
