Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r74670495
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -281,15 +323,216 @@ class TaskSchedulerImplSuite extends SparkFunSuite
with LocalSparkContext with B
assert(!failedTaskSet)
}
+ test("scheduled tasks obey task and stage blacklists") {
+ val blacklist = mock[BlacklistTracker]
+ taskScheduler = setupSchedulerWithMockTsm(blacklist)
+ (0 to 2).foreach { stageId =>
+ val taskSet = FakeTask.createTaskSet(numTasks = 2, stageId =
stageId, stageAttemptId = 0)
+ taskScheduler.submitTasks(taskSet)
+ }
+
+ val offers = Seq(
+ new WorkerOffer("executor0", "host0", 1),
+ new WorkerOffer("executor1", "host1", 1),
+ new WorkerOffer("executor2", "host1", 1),
+ new WorkerOffer("executor3", "host2", 10)
+ )
+
+ // setup our mock blacklist:
+ // stage 0 is blacklisted on node "host1"
+ // stage 1 is blacklisted on executor "executor3"
+ // stage 0, part 0 is blacklisted on executor 0
+ // (later stubs take precedence over earlier ones)
+ when(blacklist.isNodeBlacklisted(anyString())).thenReturn(false)
+ when(blacklist.isExecutorBlacklisted(anyString())).thenReturn(false)
+ // setup some defaults, then override them with particulars
+ stageToMockTsm.values.foreach { tsm =>
+ when(tsm.isNodeBlacklistedForTaskSet(anyString())).thenReturn(false)
+
when(tsm.isExecutorBlacklistedForTaskSet(anyString())).thenReturn(false)
+ when(tsm.isExecutorBlacklistedForTask(anyString(),
anyInt())).thenReturn(false)
--- End diff --
It is used, because the TSM is partially being tested as well here. Its
not a total mock, its a real tsm with just the blacklisting methods replaced.
with some of the other changes you have suggested, this is actually mostly just
checking the TSM, so I could move these tests over there if it would help. I
am trying to have some tests which aren't super narrow unit tests, that test a
slightly larger chunk of scheduling to ensure the components interact
correctly, but maybe in this case it would be clearer in the other suite.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
