[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323894#comment-14323894 ] Hudson commented on HDFS-7647: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314226#comment-14314226 ] Yi Liu commented on HDFS-7647: -- Sorry for coming late, I took a look at this patch again when committing HDFS-7756 this morning and find few nits. I make some cleanup for {{LocatedBlock}} in HDFS-7761 after the patch of this JIRA, and details are as following. # The usage of following two variables is unnecessary. We can remove them to make code a bit brief. {quote} private final boolean hasStorageIDs; private final boolean hasStorageTypes; {quote} # In this JIRA, no need to modify {{LocatedBlock#getStorageTypes}} and {{LocatedBlock#getStorageIDs}}, we just need to update the cached {{storageIDs}} and {{storageTypes}} after *sort*. # Another thing is we'd better setSoftwareVersion when constructing {{DatanodeInfoWithStorage}} from {{DatanodeInfo}} > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314049#comment-14314049 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Yarn-trunk #834 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/834/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313989#comment-14313989 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #100 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/100/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313760#comment-14313760 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #800 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/800/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev 433672a67b2589bc77d80007c219d03f6a6bf656) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313713#comment-14313713 ] Hudson commented on HDFS-7647: -- SUCCESS: Integrated in HBase-0.98 #842 (See [https://builds.apache.org/job/HBase-0.98/842/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev 433672a67b2589bc77d80007c219d03f6a6bf656) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313519#comment-14313519 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in HBase-1.0 #724 (See [https://builds.apache.org/job/HBase-1.0/724/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev ea431871db5a99eb64eb742558e2cccb867c355f) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313454#comment-14313454 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in HBase-TRUNK #6108 (See [https://builds.apache.org/job/HBase-TRUNK/6108/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev f97c00fd99609214830e68f52c1ec48c4e506c1c) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313428#comment-14313428 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in HBase-1.1 #159 (See [https://builds.apache.org/job/HBase-1.1/159/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev 66ccc22117beb9908057f941c9e3ea3fbd6880c0) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313142#comment-14313142 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2050 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2050/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313137#comment-14313137 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2031 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2031/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313092#comment-14313092 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #100 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/100/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313041#comment-14313041 ] Hudson commented on HDFS-7647: -- SUCCESS: Integrated in Hadoop-trunk-Commit #7053 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7053/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312984#comment-14312984 ] Ted Yu commented on HDFS-7647: -- Sure. I am testing a patch - will log JIRA momentarily. > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312976#comment-14312976 ] Arpit Agarwal commented on HDFS-7647: - Yes it is Private by default but we should tag it. [~ted_yu], Do you want to file a Jira to fix both points? Thanks. > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312969#comment-14312969 ] Ted Yu commented on HDFS-7647: -- DatanodeInfoWithStorage doesn't have audience annotation. Is it Private ? Can DatanodeInfoWithStorage reside in org.apache.hadoop.hdfs.protocol as DatanodeInfo does ? > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312916#comment-14312916 ] Milan Desai commented on HDFS-7647: --- Thanks [~arpitagarwal] for the help and review > DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs > > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Fix For: 2.7.0 > > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312782#comment-14312782 ] Arpit Agarwal commented on HDFS-7647: - Thanks for the explanation - I missed that. The warning looks overaggressive but +1 for the v7 patch to avoid future findbugs warnings. I will commit it today. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312771#comment-14312771 ] Milan Desai commented on HDFS-7647: --- Hi [~arpitagarwal], I added those in because Findbugs was complaining that a subclass (DatanodeInfoWithStorage) didn't override equals() and hashCode() and recommending that I call the super and add an explanatory comment if I didn't want to override. Revision 5 is the same exact as Revision 7 but without the equals and hashCode overridden, so we can commit that one, but there will be two additional FindBugs warnings. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312736#comment-14312736 ] Arpit Agarwal commented on HDFS-7647: - Hi [~milandesai], thank you again for your patience with the patch revisions! The latest version looks good. Just a nitpick, and I do feel bad having you upload another patch revision - - I think {{DatanodeInfoWithStorage#equals}} and {{#hashCode}} are unnecessary since they just forward to the superclass versions with no additional work. I removed them and verified the test case still passes. Do you think we could eliminate them? > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311875#comment-14311875 ] Milan Desai commented on HDFS-7647: --- Test failure and findbugs warning are unrelated; test passes locally on my machine. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311781#comment-14311781 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697285/HDFS-7647-7.patch against trunk revision 1382ae5. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9495//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9495//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9495//console This message is automatically generated. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310699#comment-14310699 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697237/HDFS-7647-6.patch against trunk revision cfb829e. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.cli.TestHDFSCLI Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9488//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9488//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9488//console This message is automatically generated. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310561#comment-14310561 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697191/HDFS-7647-5.patch against trunk revision 8de80ff. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion org.apache.hadoop.cli.TestHDFSCLI Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9479//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9479//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9479//console This message is automatically generated. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647-5.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310191#comment-14310191 ] Arpit Agarwal commented on HDFS-7647: - bq. This poses a larger problem in that the client may expect changes to the datanode state to be reflected in the output of LocatedBlock.getLocations(), but they won't be because we create a new DatanodeInfoWithStorage object during construction and return it in getLocations(). I think this is a test specific problem. Clients will never get access to the NameNode object, instead they will have a replica constructed from the wire message. The test case directly calls the FSNameSystem.getBlockLocations. The test should be fixed to not poll the cached object but query a new one periodically. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310123#comment-14310123 ] Milan Desai commented on HDFS-7647: --- Some of the tests were failing because the .equals() and .hashCode() method in DatanodeInfoWithStorage also checked for storage equality, and tests like TestShortCircuitCache relied on the ability to identify DatanodeInfos in a map using a DatanodeInfoWithStorage object. The fix was to remove the equals() and hashCode() overrides so that they rely on the super. TestDecomission.testDecomissionWithOpenfile is failing because the test is waiting for the DatanodeInfoWithStorage object to become decommissioned, but it won't ever be decommissioned because it is not the same reference as the DatanodeInfo in the datanode map. This poses a larger problem in that the client may expect changes to the datanode state to be reflected in the output of LocatedBlock.getLocations(), but they won't be because we create a new DatanodeInfoWithStorage object during construction and return it in getLocations(). Instead of having DatanodeInfoWithStorage be a subclass of DatanodeInfo, we could make it be a wrapper for DatanodeInfo, StorageID, and StorageType. Then to make NetworkTopology.sortByDistance work, we would have to make it implement Node and/or DatanodeInfo. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309643#comment-14309643 ] Arpit Agarwal commented on HDFS-7647: - Thanks for your patience [~milandesai], the latest patch look good. There are a few reproducible test failures flagged by Jenkins we'll need to address before committing. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309368#comment-14309368 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697012/HDFS-7647-4.patch against trunk revision 644548f. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality org.apache.hadoop.hdfs.TestBlockReaderFactory org.apache.hadoop.hdfs.TestDataTransferKeepalive org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.http.TestHttpCookieFlag Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9456//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9456//console This message is automatically generated. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, > HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299989#comment-14299989 ] Arpit Agarwal commented on HDFS-7647: - We can replace _(reader == node)_ with _(reader.equals(node))_ in {{NetworkTopology#getWeight}}. Also {{NetworkTopologyWithNodeGroup.getWeight}}. This defaults to object reference equality unless {{Object.equals}} is overriden. Although there are no other callers of this method so the change looks safe. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299973#comment-14299973 ] Arpit Agarwal commented on HDFS-7647: - Hi [~milandesai], Thank for incorporating the feedback. The patch looks great for the most part. However I think it breaks the topology-based sorting. I caught it by adding the following checks at the end of your test case: {code} // Ensure the local node is first. final DatanodeInfoWithStorage[] sortedLocs = block.getLocations(); assertThat(sortedLocs[0].getIpAddr(), is(targetIp)); // Ensure the two decommissioned DNs were moved to the end. assertThat(sortedLocs[sortedLocs.length-1].getAdminState(), is(DatanodeInfo.AdminStates.DECOMMISSIONED)); assertThat(sortedLocs[sortedLocs.length-2].getAdminState(), is(DatanodeInfo.AdminStates.DECOMMISSIONED)); {code} where targetIp was used as follows: {code} final String targetIp = locs[4].getIpAddr(); // sort block locations dm.sortLocatedBlocks(targetIp, blocks); {code} The first assert kept failing for me, so I spent some time debugging it. The root cause is the following check in {{NetworkTopology#getWeight}}. {code} if (reader == node) { weight = 0; {code} The check for local node is via a simple object ID comparison which worked before but breaks now because the LHS is a DatanodeDescriptor while the RHS is DatanodeInfoWithStorage. We'll need to fix that. Also a nitpick: {code} if(!hasStorageTypes) return null; if(storageTypes != null) return storageTypes; ... if(!hasStorageIDs) return null; if(storageIDs != null) return storageIDs; {code} Please add curly braces for the ifs. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647.patch > > > ,m.t, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The > method is called by FSNamesystem.getBlockLocations(), so the client will not > know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299665#comment-14299665 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695696/HDFS-7647-3.patch against trunk revision 09ad9a8. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache org.apache.hadoop.hdfs.TestDataTransferKeepalive org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.TestBlockReaderFactory Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9387//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9387//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9387//console This message is automatically generated. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289725#comment-14289725 ] Arpit Agarwal commented on HDFS-7647: - Thanks for the patch [~milandesai], this looks like a good change. Couple of comments: # {{LocatedBlocks.getStorageTypes}} and {{.getStorageIDs}} should cache the generated arrays on first invocation since existing callers expect these calls to be cheap. Except for the sorting code the content of {{locs}} is not modified once the object is initialized. # The sorting code must invalidate the cached arrays from 1. # We should add a unit test for sortLocatedBlocks specifically for the invalidation. # Also it would be good to add a comment to {{LocatedBlocks}} stating the assumption that {{locs}} must not be modified by the caller, with the exception of {{sortLocatedBlocks}}. In a separate Jira it would be good to make {{locs}} an unmodifiable list or a Guava {{ImmutableList}}. The source of the issue is that an external function reaches into the LocatedBlock object and modifies its private fields. It doesn't help that Java lacks support for C++-style const arrays. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288892#comment-14288892 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693798/HDFS-7647-2.patch against trunk revision 5f124ef. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.TestDataTransferKeepalive org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache org.apache.hadoop.hdfs.TestBlockReaderFactory org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9308//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9308//console This message is automatically generated. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647-2.patch, HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286883#comment-14286883 ] Yi Liu commented on HDFS-7647: -- {quote} We should fix LocatedBlock to group the three fields in a new class. Existing code (except sortLocatedBlocks) can be shielded from the change by keeping public the interface the same. What do you think? {quote} +1 for this approach. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286858#comment-14286858 ] Hadoop QA commented on HDFS-7647: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693747/HDFS-7647.patch against trunk revision 0742591. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9299//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9299//console This message is automatically generated. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286594#comment-14286594 ] Milan Desai commented on HDFS-7647: --- Hi [~arpitagarwal], sounds good - I will give it a shot. > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286567#comment-14286567 ] Arpit Agarwal commented on HDFS-7647: - Hi Milan, while this solution would work I'd prefer the other approach you suggested. We should fix LocatedBlock to group the three fields in a new class. Existing code (except sortLocatedBlocks) can be shielded from the change by keeping public the interface the same. What do you think? > DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs > -- > > Key: HDFS-7647 > URL: https://issues.apache.org/jira/browse/HDFS-7647 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Milan Desai >Assignee: Milan Desai > Attachments: HDFS-7647.patch > > > DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside > each LocatedBlock, but does not touch the array of StorageIDs and > StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are > mismatched. The method is called by FSNamesystem.getBlockLocations(), so the > client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)