[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243752#comment-17243752 ] Ayush Saxena commented on HDFS-15645: - bq. The DatanodeManager#getNumOfDataNodes() is used when verifying whether the given ec policies are supported in the cluster setup. Does HDFS-15117 or HDFS-14188 won't help in that case? Or is there some limitation there as well. > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235494#comment-17235494 ] Jinglun commented on HDFS-15645: Hi [~hexiaoqiao], thanks your comments and sorry for my late response ! The DatanodeManager#getNumOfDataNodes() is used when verifying whether the given ec policies are supported in the cluster setup. The verify result is true only when the getNumOfDataNodes() is no less than the number needed by the ec policies. Now the case is like HDFS-6397: only the nodes(live and dead) reporting to NameNode are counted. The nodes that are in dfs.include but haven't reported are not counted. I think the nodes in dfs.include should be counted too because they also belong to the cluster. Please correct me if I'm wrong. Thanks ! > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232686#comment-17232686 ] Xiaoqiao He commented on HDFS-15645: [~LiJinglun], Thanks involve me here. Sorry I don't get why we should change to #getDatanodeListForReport here, any case do you meet? Thanks. > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232659#comment-17232659 ] Jinglun commented on HDFS-15645: Hi [~jianghuazhu] [~hexiaoqiao] [~ayushsaxena], could you help reviewing this ? Thanks ! > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231129#comment-17231129 ] Jinglun commented on HDFS-15645: Hi [~jianghuazhu], thanks your comments. Yes, there will be dead nodes. In my opinion the method getNumOfDataNodes() try to return all the datanodes including the dead ones. Before we change it also returns all dead nodes. The difference is in the patch the nodes from included file are counted too. > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231103#comment-17231103 ] JiangHua Zhu commented on HDFS-15645: - [~LiJinglun] If all the host sets are returned, will there be dead nodes? > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222889#comment-17222889 ] Jinglun commented on HDFS-15645: The failed tests are not related. Hi [~linyiqun] [~leosun08] [~hexiaoqiao], do you have time to help reviewing this ? Thanks !!! > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[
https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218379#comment-17218379
]
Hadoop QA commented on HDFS-15645:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
21s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
18s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 11s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
13s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 56s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 30s{color}
|
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt|https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/250/artifact/
[jira] [Commented] (HDFS-15645) DatanodeManager.getNumOfDataNodes should consider the include set of host config.
[ https://issues.apache.org/jira/browse/HDFS-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218274#comment-17218274 ] Jinglun commented on HDFS-15645: Hi [~knanasi] [~hexiaoqiao] [~leosun08] , do you have time to take a look at this, thanks ! > DatanodeManager.getNumOfDataNodes should consider the include set of host > config. > - > > Key: HDFS-15645 > URL: https://issues.apache.org/jira/browse/HDFS-15645 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15645.001.patch > > > Currently the DatanodeManager.getNumOfDataNodes() only count the size of > datanodeMap. The nodes from host file's include set should also be counted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
