[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[ https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753992#comment-13753992 ] yeshavora commented on YARN-1057: - Hitesh, the usecase will be when invalid hosts are added in include/exclude file, Yarn should recognize them and does not add/remove node from cluster. Yarn should not print below message. INFO util.HostsFileReader (HostsFileReader.java:readFileToSet(68)) - Adding invalidhost.net to the list of included hosts from /tmp/yarn.include Ideally it should say something like java.net.UnknownHostException: invalidhost.net I believe RM shutdown is not needed as long as It can verify existence of a host. Add mechanism to check validity of a Node to be Added/Excluded -- Key: YARN-1057 URL: https://issues.apache.org/jira/browse/YARN-1057 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Xuan Gong Attachments: YARN-1057.1.patch Yarn does not complain while passing an invalid hostname like 'invalidhost.com' inside include/exclude node file. (specified by 'yarn.resourcemanager.nodes.include-path' or 'yarn.resourcemanager.nodes.exclude-path'). Need to add a mechanism to check the validity of the hostname before including or excluding from cluster. It should throw an error / exception while adding/removing an invalid node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[ https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753048#comment-13753048 ] Hitesh Shah commented on YARN-1057: --- [~yeshavora] Could you give more details on the use-case you are trying to address? I am inclined to mark this as an invalid/wontfix jira as there is no clear objective on how the RM should handle an invalid entry in the include/exclude files? Is the expectation that the RM shutdown when an invalid entry is added to the file and the refresh command invoked? The use-case that may make more sense is that a connection from a nodemanager, whose reverse dns lookup fails, should be rejected. [~xgong] Regarding the patch, what is the objective of filtering the in-memory list of included/excluded nodes? If there is a transient dns issue, would this mean that the NM will never be allowed to register with the RM? Add mechanism to check validity of a Node to be Added/Excluded -- Key: YARN-1057 URL: https://issues.apache.org/jira/browse/YARN-1057 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Xuan Gong Attachments: YARN-1057.1.patch Yarn does not complain while passing an invalid hostname like 'invalidhost.com' inside include/exclude node file. (specified by 'yarn.resourcemanager.nodes.include-path' or 'yarn.resourcemanager.nodes.exclude-path'). Need to add a mechanism to check the validity of the hostname before including or excluding from cluster. It should throw an error / exception while adding/removing an invalid node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[ https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753050#comment-13753050 ] Hitesh Shah commented on YARN-1057: --- [~xgong] Likewise, from the patch, it looks like if a hostname fails to resolve, it would not be added to the exclude list. If the include list is empty, it means the host in question can easily register later on as the exclude list would not have the entry? Add mechanism to check validity of a Node to be Added/Excluded -- Key: YARN-1057 URL: https://issues.apache.org/jira/browse/YARN-1057 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Xuan Gong Attachments: YARN-1057.1.patch Yarn does not complain while passing an invalid hostname like 'invalidhost.com' inside include/exclude node file. (specified by 'yarn.resourcemanager.nodes.include-path' or 'yarn.resourcemanager.nodes.exclude-path'). Need to add a mechanism to check the validity of the hostname before including or excluding from cluster. It should throw an error / exception while adding/removing an invalid node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[
https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753089#comment-13753089
]
Hadoop QA commented on YARN-1057:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600503/YARN-1057.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.util.TestHostsFileReader
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1788//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1788//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1788//console
This message is automatically generated.
Add mechanism to check validity of a Node to be Added/Excluded
--
Key: YARN-1057
URL: https://issues.apache.org/jira/browse/YARN-1057
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: yeshavora
Assignee: Xuan Gong
Attachments: YARN-1057.1.patch
Yarn does not complain while passing an invalid hostname like
'invalidhost.com' inside include/exclude node file. (specified by
'yarn.resourcemanager.nodes.include-path' or
'yarn.resourcemanager.nodes.exclude-path').
Need to add a mechanism to check the validity of the hostname before
including or excluding from cluster. It should throw an error / exception
while adding/removing an invalid node.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[ https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737009#comment-13737009 ] Hitesh Shah commented on YARN-1057: --- [~yeshavora] Could you clarify what you mean by an invalid hostname/node? Add mechanism to check validity of a Node to be Added/Excluded -- Key: YARN-1057 URL: https://issues.apache.org/jira/browse/YARN-1057 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: yeshavora Yarn does not complain while passing an invalid hostname like 'invalidhost.com' inside include/exclude node file. (specified by 'yarn.resourcemanager.nodes.include-path' or 'yarn.resourcemanager.nodes.exclude-path'). Need to add a mechanism to check the validity of the hostname before including or excluding from cluster. It should throw an error / exception while adding/removing an invalid node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[ https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737030#comment-13737030 ] yeshavora commented on YARN-1057: - By 'Invalide hostname/node', I mean the incorrect name of the host/node which does not exists. Currently, Yarn just adds the invalid hostname without checking its existence. Like below. INFO util.HostsFileReader (HostsFileReader.java:readFileToSet(68)) - Adding invalidhost.net to the list of included hosts from Include_Yarn_File Yarn should firstly confirms host's existence and then include/exclude them. Add mechanism to check validity of a Node to be Added/Excluded -- Key: YARN-1057 URL: https://issues.apache.org/jira/browse/YARN-1057 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: yeshavora Yarn does not complain while passing an invalid hostname like 'invalidhost.com' inside include/exclude node file. (specified by 'yarn.resourcemanager.nodes.include-path' or 'yarn.resourcemanager.nodes.exclude-path'). Need to add a mechanism to check the validity of the hostname before including or excluding from cluster. It should throw an error / exception while adding/removing an invalid node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[ https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737171#comment-13737171 ] Hitesh Shah commented on YARN-1057: --- Thanks for the clarification. I am not sure if it is easy to figure out whether a host exists or not. An operations team may setup an include file with host01 to host99 even though only 50 hosts exist with the expectation that other hosts will be added in later. It would help if you can provide a use-case for this or if this is a regression from 1.x behavior, please let us know. Add mechanism to check validity of a Node to be Added/Excluded -- Key: YARN-1057 URL: https://issues.apache.org/jira/browse/YARN-1057 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: yeshavora Yarn does not complain while passing an invalid hostname like 'invalidhost.com' inside include/exclude node file. (specified by 'yarn.resourcemanager.nodes.include-path' or 'yarn.resourcemanager.nodes.exclude-path'). Need to add a mechanism to check the validity of the hostname before including or excluding from cluster. It should throw an error / exception while adding/removing an invalid node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
