[
https://issues.apache.org/jira/browse/YARN-8256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17802681#comment-17802681
]
Shilun Fan commented on YARN-8256:
----------------------------------
Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a
blocker. Retarget 3.5.0.
> Pluggable provider for node membership management
> -------------------------------------------------
>
> Key: YARN-8256
> URL: https://issues.apache.org/jira/browse/YARN-8256
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
> Affects Versions: 2.8.3, 3.0.2
> Reporter: Dagang Wei
> Priority: Major
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> h1. Background
> HDFS-7541 introduced a pluggable provider framework for node membership
> management, which gives HDFS the flexibility to have different ways to manage
> node membership for different needs.
> [org.apache.hadoop.hdfs.server.blockmanagement.HostConfigManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostConfigManager.java]
> is the class which provides the abstraction. Currently, there are 2
> implementations in the HDFS codebase:
> 1)
> [org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostFileManager.java]
> which uses 2 config files which are defined by the properties dfs.hosts and
> dfs.hosts.exclude.
> 2)
> [org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CombinedHostFileManager.java]
> which uses a single JSON file defined by the property dfs.hosts.
> dfs.namenode.hosts.provider.classname is the property determining which
> implementation is used
> h1. Problem
> YARN should be consistent with HDFS in terms of pluggable provider for node
> membership management. The absence of it makes YARN impossible to have other
> config sources, e.g., ZooKeeper, database, other config file formats, etc.
> h1. Proposed solution
> [org.apache.hadoop.yarn.server.resourcemanager.NodesListManager|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java]
> is the class for managing YARN node membership today. It uses
> [HostsFileReader|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java]
> to read config files specified by the property
> yarn.resourcemanager.nodes.include-path for nodes to include and
> yarn.resourcemanager.nodes.nodes.exclude-path for nodes to exclude.
> The proposed solution is to
> 1) introduce a new interface {color:#008000}HostsConfigManager{color} which
> provides the abstraction for node membership management. Update
> {color:#008000}NodeListManager{color} to depend on
> {color:#008000}HostsConfigManager{color} instead of
> {color:#008000}HostsFileReader{color}. Then create a wrapper class for
> {color:#008000}HostsFileReader{color} which implements the interface.
> 2) introduce a new config property
> {color:#008000}yarn.resourcemanager.hosts-config.manager.class{color} for
> specifying the implementation class. Set the default value to the wrapper
> class of {color:#008000}HostsFileReader{color} for backward compatibility
> between new code and old config.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
