Dagang Wei created YARN-8256:
--------------------------------

             Summary: Pluggable provider for node membership management
                 Key: YARN-8256
                 URL: https://issues.apache.org/jira/browse/YARN-8256
             Project: Hadoop YARN
          Issue Type: New Feature
          Components: resourcemanager
    Affects Versions: 3.0.2, 2.8.3
            Reporter: Dagang Wei


h1. Background

[HDFS-7541|https://issues.apache.org/jira/browse/HDFS-7541] introduced a 
pluggable provider framework for node membership management, which gives HDFS 
the flexibility to have different ways to manage node membership for different 
needs.

[org.apache.hadoop.hdfs.server.blockmanagement.HostConfigManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostConfigManager.java]
 is the class which provides the abstraction. Currently, there are 2 
implementations in the HDFS codebase:

1) 
[org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostFileManager.java]
 which uses 2 config files which are defined by the properties dfs.hosts and 
dfs.hosts.exclude.

2) 
[org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CombinedHostFileManager.java]
 which uses a single JSON file defined by the property dfs.hosts.

dfs.namenode.hosts.provider.classname is the property determining which 
implementation is used

h1. Problem

YARN should be consistent with HDFS in terms of pluggable provider for node 
membership management. The absence of it makes YARM impossible to have other 
config sources, e.g., ZooKeeper, database, etc.

h1. Proposed solution

[org.apache.hadoop.yarn.server.resourcemanager.NodesListManager|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java]
 is the class for managing YARN node membership today. It uses 
[HostsFileReader|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java]
 to read config files specified by the property 
yarn.resourcemanager.nodes.include-path for nodes to include and 
yarn.resourcemanager.nodes.nodes.exclude-path for nodes to exclude.

The proposed solution is to

1) introduce a new interface {color:green}HostsConfigManager{color} which 
provides the abstraction for node membership management. Update 
{color:green}NodeListManager{color} to depend on 
{color:green}HostsConfigManager{color} instead of 
{color:green}HostsFileReader{color}. Then create a wrapper class for 
{color:green}HostsFileReader{color} which implements the interface.

2) introduce a new config property 
{color:green}yarn.resourcemanager.hosts.provider.classname{color} for 
specifying the implementation class. Set the default value to the wrapper class 
of {color:green}HostsFileReader{color} for backward compatibility between new 
code and old config.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Reply via email to