[
https://issues.apache.org/jira/browse/SPARK-26269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16727167#comment-16727167
]
ASF GitHub Bot commented on SPARK-26269:
----------------------------------------
Ngone51 opened a new pull request #23366:
[SPARK-26269][YARN][BRANCH-2.4]Yarnallocator should have same blacklist
behaviour with yarn to maxmize use of cluster resource
URL: https://github.com/apache/spark/pull/23366
## What changes were proposed in this pull request?
As I mentioned in jira
[SPARK-26269](https://issues.apache.org/jira/browse/SPARK-26269), in order to
maxmize the use of cluster resource, this pr try to make `YarnAllocator` have
the same blacklist behaviour with YARN.
## How was this patch tested?
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> YarnAllocator should have same blacklist behaviour with YARN to maxmize use
> of cluster resource
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-26269
> URL: https://issues.apache.org/jira/browse/SPARK-26269
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.3.1, 2.3.2, 2.4.0
> Reporter: wuyi
> Assignee: wuyi
> Priority: Minor
> Fix For: 3.0.0
>
>
> Currently, YarnAllocator may put a node with a completed container whose exit
> status is not one of SUCCESS, PREEMPTED, KILLED_EXCEEDED_VMEM,
> KILLED_EXCEEDED_PMEM into blacklist. Howerver, for other exit status, e.g.
> KILLED_BY_RESOURCEMANAGER, Yarn do not consider its related nodes shoule be
> added into blacklist(see YARN's explaination for detail
> https://github.com/apache/hadoop/blob/228156cfd1b474988bc4fedfbf7edddc87db41e3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java#L273).
> So, relaxing the current blacklist rule and having the same blacklist
> behaviour with YARN would maxmize use of cluster resources.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
