[
https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374778#comment-16374778
]
Stavros Kontopoulos commented on SPARK-23485:
---------------------------------------------
How about locality preferences + a hardware problem, like the disk problem? I
see code in Spark Kubernetes scheduler related to locality (not sure if it is
completed). Will that problem be detected and will kubernetes scheduler
consider the node as problematic? If so then I guess there is no need for
blacklisting.
> Kubernetes should support node blacklist
> ----------------------------------------
>
> Key: SPARK-23485
> URL: https://issues.apache.org/jira/browse/SPARK-23485
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes, Scheduler
> Affects Versions: 2.3.0
> Reporter: Imran Rashid
> Priority: Major
>
> Spark's BlacklistTracker maintains a list of "bad nodes" which it will not
> use for running tasks (eg., because of bad hardware). When running in yarn,
> this blacklist is used to avoid ever allocating resources on blacklisted
> nodes:
> https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128
> I'm just beginning to poke around the kubernetes code, so apologies if this
> is incorrect -- but I didn't see any references to
> {{scheduler.nodeBlacklist()}} in {{KubernetesClusterSchedulerBackend}} so it
> seems this is missing. Thought of this while looking at SPARK-19755, a
> similar issue on mesos.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
