[
https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751830#comment-16751830
]
Imran Rashid commented on SPARK-26688:
--------------------------------------
OK that is a reasonable request ... but to play devil's advocate, there is a
downside to having devs try to use this workaround instead of going through
Ops. They never end up sending the feedback to Ops, Ops doesn't ever get
metrics on the bad nodes, because nothing even tries to run there. Meanwhile
devs start to apply this willy-nilly, as these configs tend to just keep
getting built up over time (I've seen cases where configs included 50 GB
containers, just because once, 3 years ago, somebody's app needed that much
memory, and that config just keeps getting copied forward to everyone).
Eventually you've got a hodge-podge of blacklisting in place, and nobody really
knows which nodes are actually good or bad.
Ideally, blacklisting and speculation should be able to prevent that problem
from being so noticeable in the first place, but I can see how reality might be
far from that ideal.
Nonetheless, on the whole I think I lean slightly in favor of adding this, but
would like to hear more opinions.
> Provide configuration of initially blacklisted YARN nodes
> ---------------------------------------------------------
>
> Key: SPARK-26688
> URL: https://issues.apache.org/jira/browse/SPARK-26688
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.0.0
> Reporter: Attila Zsolt Piros
> Priority: Major
>
> Introducing new config for initially blacklisted YARN nodes.
> This came up in the apache spark user mailing list:
> [http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-is-it-possible-to-manually-blacklist-nodes-before-running-spark-job-td34395.html]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
