Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60475508
The latest set of commits replaced all the new logic in the scheduler
package with a listener. This isolates the new logic very nicely within the new
file as @kayouste
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60475431
[Test build #22207 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22207/consoleFull)
for PR 2746 at commit
[`19c6c4b`](https://githu
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60470225
Hi all, I have implemented the relevant changes in #2840. The interfaces
there are very similar to what we have discussed. A major difference is that
the scheduler bac
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19351838
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Soft
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19327200
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60353109
Hey Andrew - these heuristics look good to me at this point. I think the
remaining things are doing this via a listener and using the SparkContext
API's. Also, we should
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60343190
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22105/consoleFull)
for PR 2746 at commit
[`092d1fd`](https://github.com/a
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60343192
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19321383
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Soft
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60339555
Alright, as of the latest commit the scheduler uses the suggested interface
to request executors. In particular, now it sets an absolute number of pending
executors de
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60339499
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22105/consoleFull)
for PR 2746 at commit
[`092d1fd`](https://github.com/ap
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60314429
@vanzin Yes, apparently so. I suppose it's fine to let `YarnAllocator` take
care of this for us for now since we're only targeting Yarn at the moment.
---
If your pro
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60279252
BTW I'm pretty sure the Spark allocator handles those over-allocation
scenarios (see ~L258 in YarnAllocator.scala, `if (numExecutorsRunningNow >
maxExecutors) {`).
---
I
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60198472
Yep - that's exactly what I was wondering about. If YARN doesn't expose the
internal counter it seems like there is a race (actually even if it does
expose it, the
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60197171
So yeah it internally decrements the pending number to 8. The app can and
is expected to infer YARN has decremented the counter. Maybe TMI, but for
getting a grasp on it,
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60196383
@sryza just so I understand. I tell YARN I want 10 executors to be pending.
Then say YARN grants me two executors. Does it internally decrement the pending
number to 8 (
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19260365
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19230787
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Soft
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60120863
In that case it might make sense for the interface to the schedulers to be
`setRequestedExecutors(num)` rather than `addExecutors(num)`. This makes more
sense for both Y
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60119850
@sryza - ah okay awesome. So think that case we can just "update" YARN with
the new number of executors we'd like to be pending.
---
If your project is set up for it, y
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60041418
YARN provides (2). After an app makes a request for X executors, YARN
keeps track of them until the app goes away. In fact, the API to YARN isn't "I
want X more executors
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60040691
Hey Andrew - I looked at this and I had some small suggestions around
naming.
However, there is a big open question here relating to the semantics of
requesting
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60040484
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60040481
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22020/consoleFull)
for PR 2746 at commit
[`8bc0e9d`](https://github.com/a
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19195340
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19195273
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19195258
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19195109
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19195056
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19194672
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19194607
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19194567
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r19194532
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Softwa
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60036614
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22020/consoleFull)
for PR 2746 at commit
[`8bc0e9d`](https://github.com/ap
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60036315
The latest changes reflect most of the design I have outlined in my bullet
point post above. I intend to refactor it a little more by accessing the task
information th
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59956545
> Even for longer-running tasks, we still have a problem. If the add timer
expires multiple times while these tasks are being run, we may end up double
counting the number
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59870868
With regard to the configuration barrier, I actually think the exposed
configs are pretty straightforward. I think even the inexperienced user can
reason about the num
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59869868
I think the main objection @pwendell and I have against the policy is not
that it is complicated in terms of implementation, but that it is complicated
in terms of its
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59863719
> By "timer" I was referring to the logical timer, not java.util.Timer as
an implementation detail.
@andrewor14 , my bad, should have looked closer at the patch.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59857927
I see that's a good point @andrewor14
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your proje
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59857732
@kayousterhout @sryza Correct me if I'm wrong, but I believe grabbing as
many executors as needed in MR/Tez comes for free in these two frameworks
because a container
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59856737
> What's the reasoning behind using Timers vs. a polling approach? I think
the latter is still a fair bit easier to understand if all else is equal.
I think yo
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59854796
One approach worth considering that I think would require less user
involvement would be to set a limit on the number of outstanding requests
instead of on the total number
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59854068
@sryza What's the policy used by MR/Tez? When talking to Patrick/Andrew
offline, I'd argued for allocating executors such that the total executors
given to the dri
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59853416
Everything sounds good except for a couple specific callouts below:
> we shouldn't wait for the new ones to register before asking for more.
This is still worryi
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59592148
Hi all. I have discussed the design offline with @kayousterhout and
@pwendell and we have come to the following high level consensus:
- We should treat add as
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59562692
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21859/consoleFull)
for PR 2746 at commit
[`9d516cc`](https://github.com/a
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59562701
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59556002
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21859/consoleFull)
for PR 2746 at commit
[`9d516cc`](https://github.com/ap
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59308640
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59308632
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21787/consoleFull)
for PR 2746 at commit
[`44f1832`](https://github.com/a
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59304965
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59304443
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21787/consoleFull)
for PR 2746 at commit
[`44f1832`](https://github.com/ap
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18934791
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Soft
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18933408
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager(
// the
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18932455
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Soft
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18932436
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Soft
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59295974
I agree with @pwendell that the retry thing looks a little convoluted.
Especially since I don't see any way for the backend to report back the status
of requests (e.g. has
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18931377
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18931227
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18931177
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18931117
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59281945
> This still uses the exponential increase policy where each round stops
and waits until the executors are actually granted by the cluster manager
before continuing.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59280725
@andrewor14 this looks cool! Two high level comments:
(1) Right now, there are a bunch of different kinds of timers that can be
set and then triggered late
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18924277
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager(
// t
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18923728
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache S
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18923301
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager(
// the
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922896
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Soft
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922754
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Soft
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922657
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Soft
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922597
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager(
// t
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922413
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache S
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922282
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -110,19 +110,26 @@ private[spark] class TaskSetManager(
// t
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922277
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Soft
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922212
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -101,7 +101,7 @@ private[spark] class TaskSetManager(
// TOD
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18922176
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18921891
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18921725
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920962
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache S
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920907
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache S
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920780
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache S
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920718
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache S
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920574
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache S
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18912692
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Softwa
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18908858
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Softwa
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59143491
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59143487
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21753/consoleFull)
for PR 2746 at commit
[`baaa403`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59143124
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21750/consoleFull)
for PR 2746 at commit
[`baaa403`](https://github.com/a
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59143131
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59138015
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59138007
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21747/consoleFull)
for PR 2746 at commit
[`9bcc0bc`](https://github.com/a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59137052
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21753/consoleFull)
for PR 2746 at commit
[`baaa403`](https://github.com/ap
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59136744
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21750/consoleFull)
for PR 2746 at commit
[`baaa403`](https://github.com/ap
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59136585
Hey Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59136518
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59133473
I have updated the code to use polling rather than Java `Timers`. The new
approach uses only 1 extra thread instead of 1 for each executor, and I find
the logic simple
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-59132886
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21747/consoleFull)
for PR 2746 at commit
[`9bcc0bc`](https://github.com/ap
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18865383
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorScalingManager.scala ---
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18743600
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorScalingManager.scala ---
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foun
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-58759844
Awesome, sounds good, will hold off.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does no
1 - 100 of 116 matches
Mail list logo
