Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86608908
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86607935
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86603422
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -72,8 +79,8 @@ private[scheduler] class TaskSetBlacklist(val
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86611783
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86611231
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86621152
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86609139
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86599669
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -51,13 +50,27 @@ import org.apache.spark.util.{AccumulatorV2
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86621369
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86468669
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -17,10 +17,254 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86649317
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -217,18 +219,35 @@ private[yarn] class YarnAllocator
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86606779
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86603980
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -84,7 +85,7 @@ private[spark] class TaskSetManager(
var
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86607623
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,10 +17,299 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86649685
--- Diff:
yarn/src/test/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackendSuite.scala
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86473290
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala
---
@@ -25,26 +25,30 @@ import
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86649835
--- Diff:
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
---
@@ -121,13 +121,20 @@ private[spark] abstract class
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86606181
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
---
@@ -94,7 +94,8 @@ private[spark] object
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86473130
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -17,10 +17,254 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86602962
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -39,9 +43,12 @@ private[scheduler] class TaskSetBlacklist(val
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86648949
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -408,4 +662,39 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86467265
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -17,10 +17,254 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86466456
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -17,10 +17,254 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86472481
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -17,10 +17,254 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r86467793
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -17,10 +17,254 @@
package org.apache.spark.scheduler
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86439529
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +317,154 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86464218
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +317,154 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86464030
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +317,154 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86439780
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +317,154 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86432873
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -17,11 +17,17 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86464194
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +317,154 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86464766
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86440067
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +317,154 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86464451
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +317,154 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85187855
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -17,11 +17,17 @@
package
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85192795
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -66,6 +77,29 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85248527
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85248542
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85248691
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85248998
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85248395
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85246863
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85248890
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -66,6 +77,29 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85247687
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r85247077
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15617
Thanks @zsxwing -- sorry about this! Looks like the test failure was
unrelated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
GitHub user kayousterhout opened a pull request:
https://github.com/apache/spark/pull/15617
[HOTFIX] Fix broken build from SPARK-17894
The named parameter in an overridden class isn't supported in Scala 2.10 so
was breaking the build.
cc @zsxwing
You can merge this
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/14079
Can you update the PR description and title? (to eliminate the per-task
stuff / make it obvious what's new here)
---
If your project is set up for it, you can reply to this email and
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r84799861
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +331,211 @@ class TaskSchedulerImplSuite extends
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15463
LGTM; merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15463
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15463
Jenkins, this is OK to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15463
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15463#discussion_r84139258
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -974,6 +974,22 @@ class TaskSetManagerSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15463#discussion_r84139585
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -974,6 +974,22 @@ class TaskSetManagerSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15463#discussion_r84139432
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -974,6 +974,22 @@ class TaskSetManagerSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15463#discussion_r84139701
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -974,6 +974,22 @@ class TaskSetManagerSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15463#discussion_r84140344
--- Diff: core/src/test/scala/org/apache/spark/scheduler/FakeTask.scala ---
@@ -37,12 +37,17 @@ object FakeTask {
}
def
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/12951
@njwhite do you have time to work on this and implement maxShares? If not,
can you close the PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
Awesome nice work!! Exciting to see this in! Let me know when the other
component, which blacklists across different stages, is ready for review.
---
If your project is set up for it, you
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
@tgravescs @mridulm To avoid being stuck in analysis paralysis for this
feature, I'd propose the following:
(1) We merge this PR. I think we're mostly in agreement that th
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82475099
--- Diff: core/src/test/scala/org/apache/spark/DistributedSuite.scala ---
@@ -108,7 +109,9 @@ class DistributedSuite extends SparkFunSuite with
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82476218
--- Diff: docs/configuration.md ---
@@ -1246,6 +1246,57 @@ Apart from these, the following properties are also
available, and may be useful
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82475487
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -634,8 +635,14 @@ class
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
@mridulm re: job failures, can you elaborate on the job failure scenario
you're concerned about?
Jobs can only fail when some tasks are unschedulable, which can happen if a
ta
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
@tgravescs no decision here yet.
@mridulm the main question for (2), though, is are the consequences a
deal-breaker? It doesn't seem disastrous if a task needs to run on a non-
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82278747
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
@squito re: the config message, thanks for the long explanation. That
makes sense and I can't think of a better error message. The current one is
very clear in telling the user how t
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82261807
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -592,34 +589,54 @@ private[spark] class TaskSetManager
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
I thought about this a little more and had some offline discussion with
Imran. @mridulm, I re-read all of your comments and it sounds like there are
two issues that are addressed by the old
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
Sorry something weird seems to have happened yesterday where github
published half of my review! Anyway the rest is above.
---
If your project is set up for it, you can reply to this email
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82105712
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82105546
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82104084
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -22,13 +22,13 @@ import java.util.{Timer, TimerTask
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82105149
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82105053
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82105250
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82103864
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82105321
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82101524
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82101088
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82087158
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82101036
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82100668
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82101976
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
Re: executor blacklisting, one more reason I've heard for this (I think
from Imran) is that tasks can fail on an executor because of memory pressure --
in which case the task may succe
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/12436
Yeah @mridulm that also seems like an issue with this approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/12775#discussion_r81835387
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala ---
@@ -135,8 +135,9 @@ private[spark] class TaskResultGetter(sparkEnv
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/12775#discussion_r81837726
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala ---
@@ -135,8 +135,9 @@ private[spark] class TaskResultGetter(sparkEnv
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/12951
@njwhite sorry to let this idle for so long. I just read through the
comments here and on the
[JIRA](https://issues.apache.org/jira/browse/SPARK-15176) and it looks like the
consensus on
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15326
@erenavsarogullari @markhamstra I just looked at this further and I
actually think this could be an issue:
(1) The first attempt for a stage fails with a fetch failure. The
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15326
@markhamstra if we do change it, we should prob merge only into master and
not 2.0.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15326
+1 to Mark's suggestion about putting the behavior in Pool::addSchedulable,
so that it works across all Schedulables.
I'm less convinced about maintaining the existing beha
Github user kayousterhout commented on the issue:
https://github.com/apache/spark/pull/15249
This mostly looks good -- I made a bunch of cosmetic comments. Sorry for
the delay -- I'll be quicker on the next review so we can get this in!
---
If your project is set up for it, yo
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81399785
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -592,34 +589,54 @@ private[spark] class TaskSetManager
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81422682
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetBlacklistSuite.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81400344
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -592,34 +589,54 @@ private[spark] class TaskSetManager
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81423312
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -809,32 +816,65 @@ class TaskSetManagerSuite extends
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81410592
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -325,7 +326,11 @@ private[spark] abstract class
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81229894
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache
501 - 600 of 1867 matches
Mail list logo
