Github user caneGuy commented on a diff in the pull request: https://github.com/apache/spark/pull/18739#discussion_r129753610 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -665,10 +667,15 @@ private[spark] class TaskSetManager( } } if (blacklistedEverywhere) { - val partition = tasks(indexInTaskSet).partitionId - abort(s"Aborting $taskSet because task $indexInTaskSet (partition $partition) " + - s"cannot run anywhere due to node and executor blacklist. Blacklisting behavior " + - s"can be configured via spark.blacklist.*.") + val dynamicAllocationEnabled = conf.getBoolean("spark.dynamicAllocation.enabled", false) + val mayAllocateNewExecutor = + conf.getInt("spark.executor.instances", -1) > currentExecutorNumber + if (!dynamicAllocationEnabled && !mayAllocateNewExecutor) { --- End diff -- @squito yes,i agree with you.Since i can not think of any solution to handle all cases well too.Fail-fast may be the right way.So i closed this pr. What i want to discuss(may be confirm) is that should i report blacklist nodes to yarn whenever it is blacklisted, but not wait for maybeFinishTaskSet being called.Right now,yarn do not know some node which is blacklisted in TaskSet because it must wait maybeFinishTaskSet being called to update.When the node has more resource than other, yarn may launch task on the blacklisted node when dynamic allocation is enabled.And job may fail by abortIfCompletelyBlacklisted.If we report blacklist information real-time,things may be improved a little? Does this make sense?If do,i will open an other pr.Thanks for your time.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org