[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602224#comment-13602224 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Hdfs-0.23-Build #553 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/553/]) svn merge -c 1325531 Merging from trunk to branch-0.23 to fix HDFS-3256. (Revision 1455991) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455991 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0-alpha, 0.23.7 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253387#comment-13253387 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Mapreduce-trunk #1048 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1048/]) HDFS-3256. HDFS considers blocks under-replicated if topology script is configured with only 1 rack. Contributed by Aaron T. Myers. (Revision 1325531) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325531 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253347#comment-13253347 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Hdfs-trunk #1013 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1013/]) HDFS-3256. HDFS considers blocks under-replicated if topology script is configured with only 1 rack. Contributed by Aaron T. Myers. (Revision 1325531) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325531 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252925#comment-13252925 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2081 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2081/]) HDFS-3256. HDFS considers blocks under-replicated if topology script is configured with only 1 rack. Contributed by Aaron T. Myers. (Revision 1325531) Result = ABORTED atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325531 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252870#comment-13252870 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Common-trunk-Commit #2067 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2067/]) HDFS-3256. HDFS considers blocks under-replicated if topology script is configured with only 1 rack. Contributed by Aaron T. Myers. (Revision 1325531) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325531 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252868#comment-13252868 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Hdfs-trunk-Commit #2140 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2140/]) HDFS-3256. HDFS considers blocks under-replicated if topology script is configured with only 1 rack. Contributed by Aaron T. Myers. (Revision 1325531) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325531 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252841#comment-13252841 ] Hadoop QA commented on HDFS-3256: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522462/HDFS-3256.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2266//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2266//console This message is automatically generated. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252788#comment-13252788 ] Todd Lipcon commented on HDFS-3256: --- +1 > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252787#comment-13252787 ] Aaron T. Myers commented on HDFS-3256: -- In investigating the TestBlocksWithNotEnoughRacks failure, I came across a bug in one of the test util methods, filed here: HDFS-3266 and Todd found another issue with TestBlocksWithNotEnoughRacks specifically, filed here: HDFS-3267 I've been looping TestBlocksWithNotEnoughRacks for an hour now, with no failures. I'm pretty confident that the TestLeaseRecovery2 failure is this JIRA: HDFS-3181 I've also been looping TestLeaseRecovery2 now for a few minutes and haven't had any failures. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252766#comment-13252766 ] Hadoop QA commented on HDFS-3256: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522462/HDFS-3256.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestLeaseRecovery2 +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2264//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2264//console This message is automatically generated. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252695#comment-13252695 ] Hadoop QA commented on HDFS-3256: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522445/HDFS-3256.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2261//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2261//console This message is automatically generated. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252677#comment-13252677 ] Todd Lipcon commented on HDFS-3256: --- bq. not yet processing processing repl queues Too many "processing"s. You've been hanging out with Eli too much. Also, I'd say we should just log it at DEBUG level for the "Not checking" case. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252614#comment-13252614 ] Todd Lipcon commented on HDFS-3256: --- patch looks good. One minor comment: can you please add an INFO log saying something like: Datanode joining cluster has expanded a formerly single-rack cluster to multi-rack. Re-checking all blocks for replication, since they should now be replicated cross-rack" > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252577#comment-13252577 ] Tom White commented on HDFS-3256: - Thanks for clarifying, Aaron. I don't think AbstractDNSToSwitchMapping.isSingleSwitch is used (yet). > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252560#comment-13252560 ] Todd Lipcon commented on HDFS-3256: --- Is this check actually correct? I think you need to check whether the repl queues are initialized, explicitly. For example, you could enter manual safe mode, in which case the repl queues are still being tracked, and it's incorrect to not call processMisReplicatedBlocks. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252343#comment-13252343 ] Hadoop QA commented on HDFS-3256: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522394/HDFS-3256.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2259//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2259//console This message is automatically generated. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252240#comment-13252240 ] Todd Lipcon commented on HDFS-3256: --- I think there's an issue here with safemode's delayed initialization of repl queues. As the DNs are checking in, when the cluster transitions from single-rack to multi-rack, it will call processMisReplicatedBlocks, even if the threshold (DFS_NAMENODE_REPL_QUEUE_THRESHOLD_PCT_KEY) hasn't been crossed. I think you need to somehow tie this back to the flag in safemode which determines whether misreplicated blocks have been processed yet. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252227#comment-13252227 ] Aaron T. Myers commented on HDFS-3256: -- bq. How does this relate to AbstractDNSToSwitchMapping.isSingleSwitch that Steve added recently? I think the two are unrelated. This change only determines rack cardinality using the NetworkTopology class, which itself is fed by the configured DNSToSwitchMapping. This is the same methodology that the BlockManager has thus far been using to determine if replicas for a given block exist on more than one rack or not. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252164#comment-13252164 ] Hadoop QA commented on HDFS-3256: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522367/HDFS-3256.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2256//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2256//console This message is automatically generated. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252160#comment-13252160 ] Tom White commented on HDFS-3256: - How does this relate to AbstractDNSToSwitchMapping.isSingleSwitch that Steve added recently? > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252046#comment-13252046 ] Suresh Srinivas commented on HDFS-3256: --- Aaron, could you please describe the solution along with the patch. This helps folks who do not look at the code understand the solution and the behavior of HDFS. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252014#comment-13252014 ] Hadoop QA commented on HDFS-3256: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522322/HDFS-3256.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2252//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2252//console This message is automatically generated. > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251958#comment-13251958 ] Todd Lipcon commented on HDFS-3256: --- Looks good. +1 pending jenkins > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
