[jira] [Commented] (HDFS-4351) Fix BlockPlacementPolicyDefault#chooseTarget when avoiding stale nodes
[ https://issues.apache.org/jira/browse/HDFS-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545764#comment-13545764 ] Hudson commented on HDFS-4351: -- Integrated in Hadoop-Yarn-trunk #89 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/89/]) HDFS-4351. In BlockPlacementPolicyDefault.chooseTarget(..), numOfReplicas needs to be updated when avoiding stale nodes. Contributed by Andrew Wang (Revision 1429653) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1429653 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Fix BlockPlacementPolicyDefault#chooseTarget when avoiding stale nodes -- Key: HDFS-4351 URL: https://issues.apache.org/jira/browse/HDFS-4351 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 1.2.0, 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 1.2.0, 2.0.3-alpha Attachments: hdfs-4351-2.patch, hdfs-4351-3.patch, hdfs-4351-4.patch, hdfs-4351-branch-1-1.patch, hdfs-4351.patch There's a bug in {{BlockPlacementPolicyDefault#chooseTarget}} with stale node avoidance enabled (HDFS-3912). If a NotEnoughReplicasException is thrown in the call to {{chooseRandom()}}, {{numOfReplicas}} is not updated together with the partial result in {{result}} since it is pass by value. The retry call to {{chooseTarget}} then uses this incorrect value. This can be seen if you enable stale node detection for {{TestReplicationPolicy#testChooseTargetWithMoreThanAvaiableNodes()}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4244) Support deleting snapshots
[ https://issues.apache.org/jira/browse/HDFS-4244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545772#comment-13545772 ] Tsz Wo (Nicholas), SZE commented on HDFS-4244: -- - update SafeModeException message for FSNamesystem.deleteSnapshot - use getMutableINodesInPath for SnapshotManager.deleteSnapshot. - why removing checkPathLength(..) in NameNodeRpcServer.createSnapshot? - combine SnapshotManager.deleteDiffsForSnapshot(..) to INodeDirectorySnapshottable.removeSnapshot(..). Note that INodeDirectorySnapshottable is extending INodeDirectoryWithSnapshot. In the meantime, let me change TestINodeDirectoryWithSnapshot to test deleteSnapshotDiff. Support deleting snapshots -- Key: HDFS-4244 URL: https://issues.apache.org/jira/browse/HDFS-4244 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4244.001.patch, HDFS-4244.002.patch, HDFS-4244.003.patch, HDFS-4244.004.patch, HDFS-4244.005.patch Provide functionality to delete a snapshot, given the name of the snapshot and the path to the directory where the snapshot was taken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4230) Listing all the current snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545862#comment-13545862 ] Hudson commented on HDFS-4230: -- Integrated in Hadoop-Hdfs-Snapshots-Branch-build #63 (See [https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/63/]) HDFS-4230. Support listing of all the snapshottable directories. Contributed by Jing Zhao (Revision 1429643) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1429643 Files : * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshottableDirectoryStatus.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshottableDirListing.java Listing all the current snapshottable directories - Key: HDFS-4230 URL: https://issues.apache.org/jira/browse/HDFS-4230 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4230.001.patch, HDFS-4230.001.patch, HDFS-4230.002.patch, HDFS-4230.003.patch, HDFS-4230.004.patch Provide functionality to provide user with metadata about all the snapshottable directories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4351) Fix BlockPlacementPolicyDefault#chooseTarget when avoiding stale nodes
[ https://issues.apache.org/jira/browse/HDFS-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545867#comment-13545867 ] Hudson commented on HDFS-4351: -- Integrated in Hadoop-Hdfs-trunk #1278 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1278/]) HDFS-4351. In BlockPlacementPolicyDefault.chooseTarget(..), numOfReplicas needs to be updated when avoiding stale nodes. Contributed by Andrew Wang (Revision 1429653) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1429653 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Fix BlockPlacementPolicyDefault#chooseTarget when avoiding stale nodes -- Key: HDFS-4351 URL: https://issues.apache.org/jira/browse/HDFS-4351 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 1.2.0, 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 1.2.0, 2.0.3-alpha Attachments: hdfs-4351-2.patch, hdfs-4351-3.patch, hdfs-4351-4.patch, hdfs-4351-branch-1-1.patch, hdfs-4351.patch There's a bug in {{BlockPlacementPolicyDefault#chooseTarget}} with stale node avoidance enabled (HDFS-3912). If a NotEnoughReplicasException is thrown in the call to {{chooseRandom()}}, {{numOfReplicas}} is not updated together with the partial result in {{result}} since it is pass by value. The retry call to {{chooseTarget}} then uses this incorrect value. This can be seen if you enable stale node detection for {{TestReplicationPolicy#testChooseTargetWithMoreThanAvaiableNodes()}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4098) Support append to original files which are snapshotted
[ https://issues.apache.org/jira/browse/HDFS-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4098: - Attachment: h4098_20130107.patch h4098_20130107.patch: adds FileWithLink, INodeFileUnderConstructionWithLink and INodeFileUnderConstructionSnapshot. Support append to original files which are snapshotted -- Key: HDFS-4098 URL: https://issues.apache.org/jira/browse/HDFS-4098 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h4098_20130107.patch When a regular file is reopened for append, the type is changed from INodeFile to INodeFileUnderConstruction. The type of snapshotted files (i.e. original files) is INodeFileWithLink. We have to support similar under construction INodeFileWithLink. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4352) Encapsulate arguments to BlockReaderFactory in a class
[ https://issues.apache.org/jira/browse/HDFS-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546016#comment-13546016 ] Tsz Wo (Nicholas), SZE commented on HDFS-4352: -- The committed patch actually generated some new javadoc warnings since it did not update the @param, e.g. {code} /** * Create a new BlockReader specifically to satisfy a read. * This method also sends the OP_READ_BLOCK request. * * @param sock An established Socket to the DN. The BlockReader will not close it normally. * This socket must have an associated Channel. * @param file File location * @param block The block object * @param blockToken The block token for security * @param startOffset The read offset, relative to block head * @param len The number of bytes to read * @param bufferSize The IO buffer size (not the client buffer size) * @param verifyChecksum Whether to verify checksum * @param clientName Client name * @return New BlockReader instance, or null on error. */ public static BlockReader newBlockReader(BlockReaderFactory.Params params) throws IOException { {code} However, test-patch failed to detect these warnings. Encapsulate arguments to BlockReaderFactory in a class -- Key: HDFS-4352 URL: https://issues.apache.org/jira/browse/HDFS-4352 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0 Attachments: 01b.patch, 01.patch Encapsulate the arguments to BlockReaderFactory in a class to avoid having to pass around 10+ arguments to a few different functions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4363) Combine PBHelper and HdfsProtoUtil and remove redundant methods
[ https://issues.apache.org/jira/browse/HDFS-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546021#comment-13546021 ] Tsz Wo (Nicholas), SZE commented on HDFS-4363: -- - Do you also want to clean up DataTransferProtoUtil? Some minor suggestions: {code} @@ -41,14 +41,12 @@ public abstract class DataTransferProtoUtil { static BlockConstructionStage fromProto( OpWriteBlockProto.BlockConstructionStage stage) { -return BlockConstructionStage.valueOf(BlockConstructionStage.class, -stage.name()); +return BlockConstructionStage.valueOf(stage.name()); } static OpWriteBlockProto.BlockConstructionStage toProto( BlockConstructionStage stage) { -return OpWriteBlockProto.BlockConstructionStage.valueOf( -stage.name()); +return OpWriteBlockProto.BlockConstructionStage.valueOf(stage.name()); } {code} - This is not directly related to the patch: use valueOf(..) instead of switch-case for converting enum types, e.g. {code} public static DatanodeInfoProto.AdminState convert( final DatanodeInfo.AdminStates inAs) { return DatanodeInfoProto.AdminState.valueOf(inAs.name()); } {code} - PBHelper.convert(DatanodeInfo[] dnInfos, int startIdx) and other related methods should return List instead of ArrayList. - Understood that you probably had used ecilpse for formatting. However, it might not be better than manual formatting in some cases. E.g. {code} // DataEncryptionKey public static DataEncryptionKey convert(DataEncryptionKeyProto bet) { String encryptionAlgorithm = bet.getEncryptionAlgorithm(); -return new DataEncryptionKey(bet.getKeyId(), -bet.getBlockPoolId(), -bet.getNonce().toByteArray(), -bet.getEncryptionKey().toByteArray(), -bet.getExpiryDate(), -encryptionAlgorithm.isEmpty() ? null : encryptionAlgorithm); +return new DataEncryptionKey(bet.getKeyId(), bet.getBlockPoolId(), bet +.getNonce().toByteArray(), bet.getEncryptionKey().toByteArray(), +bet.getExpiryDate(), encryptionAlgorithm.isEmpty() ? null +: encryptionAlgorithm); } {code} {code} @@ -1132,41 +1140,44 @@ if (fsStats.length = ClientProtocol.GET_STATS_REMAINING_IDX + 1) result.setRemaining(fsStats[ClientProtocol.GET_STATS_REMAINING_IDX]); if (fsStats.length = ClientProtocol.GET_STATS_UNDER_REPLICATED_IDX + 1) - result.setUnderReplicated( - fsStats[ClientProtocol.GET_STATS_UNDER_REPLICATED_IDX]); + result + .setUnderReplicated(fsStats[ClientProtocol.GET_STATS_UNDER_REPLICATED_IDX]); if (fsStats.length = ClientProtocol.GET_STATS_CORRUPT_BLOCKS_IDX + 1) - result.setCorruptBlocks( - fsStats[ClientProtocol.GET_STATS_CORRUPT_BLOCKS_IDX]); + result + .setCorruptBlocks(fsStats[ClientProtocol.GET_STATS_CORRUPT_BLOCKS_IDX]); if (fsStats.length = ClientProtocol.GET_STATS_MISSING_BLOCKS_IDX + 1) - result.setMissingBlocks( - fsStats[ClientProtocol.GET_STATS_MISSING_BLOCKS_IDX]); + result + .setMissingBlocks(fsStats[ClientProtocol.GET_STATS_MISSING_BLOCKS_IDX]); return result.build(); } {code} Combine PBHelper and HdfsProtoUtil and remove redundant methods --- Key: HDFS-4363 URL: https://issues.apache.org/jira/browse/HDFS-4363 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-4363.patch, HDFS-4363.patch There are many methods overlapping between PBHelper and HdfsProtoUtil. This jira combines these two helper classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4350: -- Status: Open (was: Patch Available) Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4350: -- Attachment: hdfs-4350-1.patch Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4350: -- Status: Patch Available (was: Open) Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4350: -- Attachment: (was: hdfs-4350-1.patch) Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546034#comment-13546034 ] Andrew Wang commented on HDFS-4350: --- Tried to bump Jenkins (cancel patch, upload, submit patch?). I think it's currently ready for review though. Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546036#comment-13546036 ] Hadoop QA commented on HDFS-4350: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563592/hdfs-4350-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3749//console This message is automatically generated. Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4350: -- Attachment: hdfs-4350-1.patch Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4350: -- Attachment: (was: hdfs-4350-1.patch) Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546042#comment-13546042 ] Hadoop QA commented on HDFS-4350: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563592/hdfs-4350-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3750//console This message is automatically generated. Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4350: -- Attachment: hdfs-4350-2.patch Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch, hdfs-4350-2.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-3970: -- Status: Open (was: Patch Available) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.0.2-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-3970: -- Attachment: (was: hdfs-3970-1.patch) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.0.2-alpha Reporter: Vinay Assignee: Vinay Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-3970: -- Attachment: hdfs-3970-1.patch BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.0.2-alpha Reporter: Vinay Assignee: Vinay Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-3970: -- Status: Patch Available (was: Open) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.0.2-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4352) Encapsulate arguments to BlockReaderFactory in a class
[ https://issues.apache.org/jira/browse/HDFS-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546130#comment-13546130 ] Colin Patrick McCabe commented on HDFS-4352: Hi Nicholas, HDFS-4353 adds asserts to {{BlockReaderFactory#newBlockReader}} that check that all essential parameters are set. That is how you can know that you have set all the essential parameters. Does that address your concerns? If not, we can revert this. It was done to improve readability (and reviewability) but it is not an essential part of the patch set. Encapsulate arguments to BlockReaderFactory in a class -- Key: HDFS-4352 URL: https://issues.apache.org/jira/browse/HDFS-4352 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0 Attachments: 01b.patch, 01.patch Encapsulate the arguments to BlockReaderFactory in a class to avoid having to pass around 10+ arguments to a few different functions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546171#comment-13546171 ] Hadoop QA commented on HDFS-4350: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563594/hdfs-4350-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3751//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3751//console This message is automatically generated. Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch, hdfs-4350-2.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4362) GetDelegationTokenResponseProto does not handle null token
[ https://issues.apache.org/jira/browse/HDFS-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546197#comment-13546197 ] Aaron T. Myers commented on HDFS-4362: -- +1, patch looks good to me. Thanks, Suresh. GetDelegationTokenResponseProto does not handle null token -- Key: HDFS-4362 URL: https://issues.apache.org/jira/browse/HDFS-4362 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Critical Attachments: HDFS-4362.patch While working on HADOOP-9173, I notice that the GetDelegationTokenResponseProto declares the token field as required. However return of null token is to be expected both as defined in FileSystem#getDelegationToken() and also based on HDFS implementation. This jira intends to make the field as optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4295) Using port 1023 should be valid when starting Secure DataNode
[ https://issues.apache.org/jira/browse/HDFS-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546206#comment-13546206 ] Aaron T. Myers commented on HDFS-4295: -- Hi liuyang, we should really continue this conversation on the u...@hadoop.apache.org mailing list, since it's not an issue with this bug/patch. The short answer to your question is: you have to start the DN as root, and make sure that the HADOOP_SECURE_DN_USER environment variable is set to 'hdfs' so that the DN knows which user to switch to. If you have any more questions about this, please email u...@hadoop.apache.org. Using port 1023 should be valid when starting Secure DataNode - Key: HDFS-4295 URL: https://issues.apache.org/jira/browse/HDFS-4295 Project: Hadoop HDFS Issue Type: Bug Components: security Affects Versions: 2.0.0-alpha Reporter: Stephen Chu Assignee: Stephen Chu Labels: trivial Fix For: 3.0.0, 2.0.3-alpha Attachments: HDFS-4295.patch In SecureDataNodeStarter: {code} if ((ss.getLocalPort() = 1023 || listener.getPort() = 1023) UserGroupInformation.isSecurityEnabled()) { throw new RuntimeException(Cannot start secure datanode with unprivileged ports); } {code} This prohibits using port 1023, but this should be okay because only root can listen to ports below 1024. We can change the = to . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546217#comment-13546217 ] Todd Lipcon commented on HDFS-4353: --- Looking almost ready. A few minor nits: {code} + * it has been configured-- like when we're reading from the last replica + * of a block. {code} This description is a little unclear to me. Better to say such as when the caller has explicitly asked for a file to be opened without checksum verification. {code} public static BlockReader newBlockReader(Params params) throws IOException { +assert(params.getPeer() != null); +assert(params.getBlock() != null); +assert(params.getDatanodeID() != null); {code} I think {{Preconditions.checkArgument}} is more appropriate here - the checks are cheap (likely free) null checks, so we may as well always verify them. {code} + // TODO: create a wrapper class that makes channel-less sockets look like + // they have a channel, so that we can finally remove the legacy + // RemoteBlockReader. {code} Can you reference the JIRA for removing RBR here? I know we have one filed somewhere. +1 after these are addressed. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-4353: --- Attachment: 02f.patch Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546246#comment-13546246 ] Todd Lipcon commented on HDFS-4353: --- Latest patch looks good. +1 pending Jenkins. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546271#comment-13546271 ] Hadoop QA commented on HDFS-3970: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563599/hdfs-3970-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3775//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3775//console This message is automatically generated. BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.0.2-alpha Reporter: Vinay Assignee: Vinay Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3970: - Target Version/s: 2.0.3-alpha BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.0.2-alpha Reporter: Vinay Assignee: Vinay Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546310#comment-13546310 ] Hadoop QA commented on HDFS-3970: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563599/hdfs-3970-1.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3784//console This message is automatically generated. BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.0.2-alpha Reporter: Vinay Assignee: Vinay Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3970: - Resolution: Fixed Fix Version/s: 2.0.3-alpha Assignee: Andrew Wang (was: Vinay) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2 based on Todd's +1. Thanks a lot for the contribution, Vinay and Andrew. BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.0.2-alpha Reporter: Vinay Assignee: Andrew Wang Fix For: 2.0.3-alpha Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[ https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546317#comment-13546317 ] Todd Lipcon commented on HDFS-4350: --- {code} + * Set the value of {@link DatanodeManager#isAvoidingStaleDataNodesForWrite}. + * The HeartbeatManager disables write avoidance when more than + * dfs.namenode.write.stale.datanode.ratio of DataNodes are marked as stale. {code} The double negative here is confusing. I think better to say: The HeartbeatManager will allow writes to stale datanodes when more than ... {code} + boolean getCheckForStaleDataNodes() { {code} I think for a boolean-type return value, something like {{shouldCheckForStale...} is better than {{getCheckForStale...}} {code} HeartbeatManager(final Namesystem namesystem, - final BlockManager blockManager, final Configuration conf) { + final BlockManager blockManager, final Configuration conf, + final boolean avoidStaleDataNodesForWrite, final long staleInterval) { {code} Not a fan of proliferating constructor parameters here. Since we already have the conf, and those two new parameters just come from the conf, I think the earlier approach was better (having both classes access the conf). {code} + this.heartbeatRecheckInterval = staleInterval; + LOG.info(Setting hearbeat interval to + staleInterval + + since dfs.namenode.stale.datanode.interval + + dfs.namenode.heartbeat.recheck-interval); {code} This info message is a little unclear -- the heartbeat interval isn't actually being changed, just the interval at which the HeartbeatManager wakes up to check for expired DNs. The message makes it sound like the datanodes will heartbeat more often, but in fact it's only an NN-side frequency that's being clamped down. Also, please interpolate the {{BLAH_BLAH_KEY}} constants instead of actually writing the configuration keys in the log message. {code} +Indicate whether or not to avoid reading from stale datanodes whose {code} Should use quot; here, for valid XML. Make enabling of stale marking on read and write paths independent -- Key: HDFS-4350 URL: https://issues.apache.org/jira/browse/HDFS-4350 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-4350-1.patch, hdfs-4350-2.patch Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking. I propose renaming the first key to {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3970) BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
[ https://issues.apache.org/jira/browse/HDFS-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546343#comment-13546343 ] Hudson commented on HDFS-3970: -- Integrated in Hadoop-trunk-Commit #3188 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3188/]) HDFS-3970. Fix bug causing rollback of HDFS upgrade to result in bad VERSION file. Contributed by Vinay and Andrew Wang. (Revision 1430037) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430037 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRollback.java BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. --- Key: HDFS-3970 URL: https://issues.apache.org/jira/browse/HDFS-3970 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.0.2-alpha Reporter: Vinay Assignee: Andrew Wang Fix For: 2.0.3-alpha Attachments: hdfs-3970-1.patch, HDFS-3970.patch {code}// read attributes out of the VERSION file of previous directory DataStorage prevInfo = new DataStorage(); prevInfo.readPreviousVersionProperties(bpSd);{code} In the above code snippet BlockPoolSliceStorage instance should be used. other wise rollback results in 'storageType' property missing which will not be there in initial VERSION file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546381#comment-13546381 ] Eli Collins commented on HDFS-4261: --- Any update Junping? TestBalancerWithNodeGroup currently fails 100% of the time on my local jenkins slave running trunk. We should annotate these test methods with timeouts ala HDFS-4061 and HDFS-4008 so we get clean test failures in case this regresses. TestBalancerWithNodeGroup times out --- Key: HDFS-4261 URL: https://issues.apache.org/jira/browse/HDFS-4261 Project: Hadoop HDFS Issue Type: Bug Components: balancer Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha Reporter: Tsz Wo (Nicholas), SZE Assignee: Junping Du Fix For: 3.0.0 Attachments: HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, jstack-mac-18567, jstack-win-5488, org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win When I manually ran TestBalancerWithNodeGroup, it always timed out in my machine. Looking at the Jerkins report [build #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], TestBalancerWithNodeGroup somehow was skipped so that the problem was not detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4035) LightWeightGSet and LightWeightHashSet increment a volatile without synchronization
[ https://issues.apache.org/jira/browse/HDFS-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546428#comment-13546428 ] Aaron T. Myers commented on HDFS-4035: -- +1, patch looks good. LightWeightGSet and LightWeightHashSet increment a volatile without synchronization --- Key: HDFS-4035 URL: https://issues.apache.org/jira/browse/HDFS-4035 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-4035.txt LightWeightGSet and LightWeightHashSet have a volatile modification field that they use to detect updates while iterating so they can throw a ConcurrentModificationException. Since these LightWeight classes are explicitly not thread safe (eg access to their members is not synchronized) then the current use is OK, we just need to update findbugsExcludeFile.xml to exclude them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4030) BlockManager excessBlocksCount and postponedMisreplicatedBlocksCount should be AtomicLongs
[ https://issues.apache.org/jira/browse/HDFS-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546431#comment-13546431 ] Aaron T. Myers commented on HDFS-4030: -- +1, patch looks good. BlockManager excessBlocksCount and postponedMisreplicatedBlocksCount should be AtomicLongs -- Key: HDFS-4030 URL: https://issues.apache.org/jira/browse/HDFS-4030 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-4030.txt, hdfs-4030.txt The BlockManager excessBlocksCount and postponedMisreplicatedBlocksCount fields are currently volatile longs which are incremented, which isn't thread safe. It looks like they're always incremented on paths that hold the NN write lock but it would be easier and less error prone for future changes if we made them AtomicLongs. The other volatile long members are just set in one thread and read in another so they're fine as is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4031) Update findbugsExcludeFile.xml to include findbugs 2 exclusions
[ https://issues.apache.org/jira/browse/HDFS-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546432#comment-13546432 ] Aaron T. Myers commented on HDFS-4031: -- +1, patch looks good. Update findbugsExcludeFile.xml to include findbugs 2 exclusions --- Key: HDFS-4031 URL: https://issues.apache.org/jira/browse/HDFS-4031 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-4031.txt Findbugs 2 warns about some volatile increments (VO_VOLATILE_INCREMENT) that unlike HDFS-4029 and HDFS-4030 are less problematic: - numFailedVolumes is only incremented in one thread and that access is synchronized - pendingReceivedRequests in BPServiceActor is clearly synchronized It would be reasonable to make these Atomics as well but I think they're uses are clearly correct so figured for these the warning was more obviously bogus and so could be ignored. There's also a SE_BAD_FIELD_INNER_CLASS warning (LocalDatanodeInfo's anonymous class is serializable but it is not) in BPServiceActor is OK to ignore since we don't serialize LocalDatanodeInfo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4034) Remove redundant null checks
[ https://issues.apache.org/jira/browse/HDFS-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546435#comment-13546435 ] Aaron T. Myers commented on HDFS-4034: -- +1, patch looks good to me. Remove redundant null checks Key: HDFS-4034 URL: https://issues.apache.org/jira/browse/HDFS-4034 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-4034.txt, hdfs-4034.txt Findbugs 2 catches a number of places where we're checking for null in cases where the value will never be null. We might need to wait until we switch to findbugs 2 to commit this as the current findbugs may not be so smart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4340) Update addBlock() to inculde inode id as additional argument
[ https://issues.apache.org/jira/browse/HDFS-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4340: - Attachment: HDFS-4340.patch Update addBlock() to inculde inode id as additional argument Key: HDFS-4340 URL: https://issues.apache.org/jira/browse/HDFS-4340 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-4340.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4340) Update addBlock() to inculde inode id as additional argument
[ https://issues.apache.org/jira/browse/HDFS-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4340: - Status: Patch Available (was: Open) Update addBlock() to inculde inode id as additional argument Key: HDFS-4340 URL: https://issues.apache.org/jira/browse/HDFS-4340 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-4340.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4244) Support deleting snapshots
[ https://issues.apache.org/jira/browse/HDFS-4244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4244: Attachment: HDFS-4244.006.patch Thanks for the comments Nicholas! Upload the new patch to address all the comments, also fix a bug with INodeFileWithLink (in certain scenarios when combining diffs an INodeFileWithLink should be removed from the circular list). A new test is also included to test the INodeFileWithLink case. Support deleting snapshots -- Key: HDFS-4244 URL: https://issues.apache.org/jira/browse/HDFS-4244 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4244.001.patch, HDFS-4244.002.patch, HDFS-4244.003.patch, HDFS-4244.004.patch, HDFS-4244.005.patch, HDFS-4244.006.patch Provide functionality to delete a snapshot, given the name of the snapshot and the path to the directory where the snapshot was taken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4033) Miscellaneous findbugs 2 fixes
[ https://issues.apache.org/jira/browse/HDFS-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546439#comment-13546439 ] Aaron T. Myers commented on HDFS-4033: -- +1, patch looks good. Miscellaneous findbugs 2 fixes -- Key: HDFS-4033 URL: https://issues.apache.org/jira/browse/HDFS-4033 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-4033.txt, hdfs-4033.txt Fix some miscellaneous findbugs 2 warnings: - Switch statements missing default cases - Using \n instead of %n in format methods - A socket close that should use IOUtils#closeSocket that we missed - A use of SimpleDateFormat that is not threadsafe - In ReplicaInputStreams it's not clear that we always close the streams we allocate, moving the stream creation into the class where we close them makes that more obvious - A couple missing null checks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4032) Specify the charset explicitly rather than rely on the default
[ https://issues.apache.org/jira/browse/HDFS-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546444#comment-13546444 ] Aaron T. Myers commented on HDFS-4032: -- Patch looks pretty good to me. Almost all the changes look like they shouldn't change any behavior at all, but I think this one will: {code} public static byte[] string2Bytes(String str) { -try { - return str.getBytes(UTF8); -} catch(UnsupportedEncodingException e) { - assert false : UTF8 encoding is not supported ; -} -return null; +return str.getBytes(Charsets.UTF_8); } {code} Since assertions are typically disabled when not running the tests, this will result in a behavior change when running in production. Before this change, the assert in the catch block would just be skipped and null would be returned. After this change, the UnsupportedEncodingException would be thrown from this method. It's not obvious to me whether this could be problematic or not. Thoughts? Specify the charset explicitly rather than rely on the default -- Key: HDFS-4032 URL: https://issues.apache.org/jira/browse/HDFS-4032 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-4032.txt Findbugs 2 warns about relying on the default Java charset instead of specifying it explicitly. Given that we're porting Hadoop to different platforms it's better to be explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546476#comment-13546476 ] Hadoop QA commented on HDFS-4353: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563623/02f.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3787//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3787//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[ https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-4356: --- Attachment: 04d-cumulative.patch This patch addresses a point that Todd brought up on reviewboard: that we don't want to cause a performance regression when doing a lot of seeks. It introduces a cache for the {{FileInputStream}} objects. This cache resides in {{FSInputStream}}, so it persists until we close the file we're reading, in keeping with HDFS' open-to-close consistency. BlockReaderLocal should use passed file descriptors rather than paths - Key: HDFS-4356 URL: https://issues.apache.org/jira/browse/HDFS-4356 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client, performance Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 04b-cumulative.patch, 04-cumulative.patch, 04d-cumulative.patch {{BlockReaderLocal}} should use file descriptors passed over UNIX domain sockets rather than paths. We also need some configuration options for these UNIX domain sockets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546524#comment-13546524 ] Todd Lipcon commented on HDFS-4353: --- I'll commit this later tonight or tomorrow morning unless there are any other comments. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4340) Update addBlock() to inculde inode id as additional argument
[ https://issues.apache.org/jira/browse/HDFS-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546538#comment-13546538 ] Hadoop QA commented on HDFS-4340: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563652/HDFS-4340.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3788//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3788//console This message is automatically generated. Update addBlock() to inculde inode id as additional argument Key: HDFS-4340 URL: https://issues.apache.org/jira/browse/HDFS-4340 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-4340.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4362) GetDelegationTokenResponseProto does not handle null token
[ https://issues.apache.org/jira/browse/HDFS-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4362: -- Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Status: Resolved (was: Patch Available) Aaron, thanks for the review. I have committed the patch. GetDelegationTokenResponseProto does not handle null token -- Key: HDFS-4362 URL: https://issues.apache.org/jira/browse/HDFS-4362 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Critical Fix For: 3.0.0 Attachments: HDFS-4362.patch While working on HADOOP-9173, I notice that the GetDelegationTokenResponseProto declares the token field as required. However return of null token is to be expected both as defined in FileSystem#getDelegationToken() and also based on HDFS implementation. This jira intends to make the field as optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[ https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546599#comment-13546599 ] Hadoop QA commented on HDFS-4356: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563669/04d-cumulative.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 19 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestParallelUnixDomainRead org.apache.hadoop.hdfs.TestParallelShortCircuitRead org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3789//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3789//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3789//console This message is automatically generated. BlockReaderLocal should use passed file descriptors rather than paths - Key: HDFS-4356 URL: https://issues.apache.org/jira/browse/HDFS-4356 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client, performance Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 04b-cumulative.patch, 04-cumulative.patch, 04d-cumulative.patch {{BlockReaderLocal}} should use file descriptors passed over UNIX domain sockets rather than paths. We also need some configuration options for these UNIX domain sockets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4362) GetDelegationTokenResponseProto does not handle null token
[ https://issues.apache.org/jira/browse/HDFS-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546601#comment-13546601 ] Suresh Srinivas commented on HDFS-4362: --- BTW this incompatible change needs to go into 2.0.3-alpha as well. GetDelegationTokenResponseProto does not handle null token -- Key: HDFS-4362 URL: https://issues.apache.org/jira/browse/HDFS-4362 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Critical Fix For: 3.0.0 Attachments: HDFS-4362.patch While working on HADOOP-9173, I notice that the GetDelegationTokenResponseProto declares the token field as required. However return of null token is to be expected both as defined in FileSystem#getDelegationToken() and also based on HDFS implementation. This jira intends to make the field as optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4364) GetLinkTargetResponseProto does not handle null path
[ https://issues.apache.org/jira/browse/HDFS-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4364: -- Attachment: HDFS-4364.patch Here is a patch thank makes the field optional. GetLinkTargetResponseProto does not handle null path Key: HDFS-4364 URL: https://issues.apache.org/jira/browse/HDFS-4364 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-4364.patch ClientProtocol#getLinkTarget() returns null targetPath. Hence protobuf definition GetLinkTargetResponseProto#targetPath should be optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4364) GetLinkTargetResponseProto does not handle null path
[ https://issues.apache.org/jira/browse/HDFS-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4364: -- Status: Patch Available (was: Open) GetLinkTargetResponseProto does not handle null path Key: HDFS-4364 URL: https://issues.apache.org/jira/browse/HDFS-4364 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-4364.patch ClientProtocol#getLinkTarget() returns null targetPath. Hence protobuf definition GetLinkTargetResponseProto#targetPath should be optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4362) GetDelegationTokenResponseProto does not handle null token
[ https://issues.apache.org/jira/browse/HDFS-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546605#comment-13546605 ] Hudson commented on HDFS-4362: -- Integrated in Hadoop-trunk-Commit #3189 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3189/]) HDFS-4362. GetDelegationTokenResponseProto does not handle null token. Contributed by Suresh Srinivas. (Revision 1430137) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430137 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto GetDelegationTokenResponseProto does not handle null token -- Key: HDFS-4362 URL: https://issues.apache.org/jira/browse/HDFS-4362 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Critical Fix For: 3.0.0 Attachments: HDFS-4362.patch While working on HADOOP-9173, I notice that the GetDelegationTokenResponseProto declares the token field as required. However return of null token is to be expected both as defined in FileSystem#getDelegationToken() and also based on HDFS implementation. This jira intends to make the field as optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4154) BKJM: Two namenodes usng bkjm can race to create the version znode
[ https://issues.apache.org/jira/browse/HDFS-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-4154: -- Status: Patch Available (was: Open) BKJM: Two namenodes usng bkjm can race to create the version znode -- Key: HDFS-4154 URL: https://issues.apache.org/jira/browse/HDFS-4154 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Ivan Kelly Assignee: Han Xiao Attachments: HDFS-4154.patch nd one will get the following error. 2012-11-06 10:04:00,200 INFO hidden.bkjournal.org.apache.zookeeper.ClientCnxn: Session establishment complete on server 109-231-69-172.flexiscale.com/109.231.69.172:2181, sessionid = 0x13ad528fcfe0005, negotiated timeout = 4000 2012-11-06 10:04:00,710 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join java.lang.IllegalArgumentException: Unable to construct journal, bookkeeper://109.231.69.172:2181;109.231.69.173:2181;109.231.69.174:2181/hdfsjournal at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1251) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:226) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:206) at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:657) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:590) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:259) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:544) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:423) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:385) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:611) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1135) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1201) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1249) ... 14 more Caused by: java.io.IOException: Error initializing zk at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.init(BookKeeperJournalManager.java:233) ... 19 more Caused by: hidden.bkjournal.org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /hdfsjournal/version at hidden.bkjournal.org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at hidden.bkjournal.org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at hidden.bkjournal.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.init(BookKeeperJournalManager.java:222) ... 19 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-4154) BKJM: Two namenodes usng bkjm can race to create the version znode
[ https://issues.apache.org/jira/browse/HDFS-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-4154: - Assignee: Han Xiao (was: Ivan Kelly) BKJM: Two namenodes usng bkjm can race to create the version znode -- Key: HDFS-4154 URL: https://issues.apache.org/jira/browse/HDFS-4154 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Ivan Kelly Assignee: Han Xiao Attachments: HDFS-4154.patch nd one will get the following error. 2012-11-06 10:04:00,200 INFO hidden.bkjournal.org.apache.zookeeper.ClientCnxn: Session establishment complete on server 109-231-69-172.flexiscale.com/109.231.69.172:2181, sessionid = 0x13ad528fcfe0005, negotiated timeout = 4000 2012-11-06 10:04:00,710 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join java.lang.IllegalArgumentException: Unable to construct journal, bookkeeper://109.231.69.172:2181;109.231.69.173:2181;109.231.69.174:2181/hdfsjournal at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1251) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:226) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:206) at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:657) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:590) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:259) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:544) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:423) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:385) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:611) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1135) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1201) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1249) ... 14 more Caused by: java.io.IOException: Error initializing zk at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.init(BookKeeperJournalManager.java:233) ... 19 more Caused by: hidden.bkjournal.org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /hdfsjournal/version at hidden.bkjournal.org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at hidden.bkjournal.org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at hidden.bkjournal.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.init(BookKeeperJournalManager.java:222) ... 19 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4154) BKJM: Two namenodes usng bkjm can race to create the version znode
[ https://issues.apache.org/jira/browse/HDFS-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546618#comment-13546618 ] Hadoop QA commented on HDFS-4154: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562773/HDFS-4154.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3791//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3791//console This message is automatically generated. BKJM: Two namenodes usng bkjm can race to create the version znode -- Key: HDFS-4154 URL: https://issues.apache.org/jira/browse/HDFS-4154 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Ivan Kelly Assignee: Han Xiao Attachments: HDFS-4154.patch nd one will get the following error. 2012-11-06 10:04:00,200 INFO hidden.bkjournal.org.apache.zookeeper.ClientCnxn: Session establishment complete on server 109-231-69-172.flexiscale.com/109.231.69.172:2181, sessionid = 0x13ad528fcfe0005, negotiated timeout = 4000 2012-11-06 10:04:00,710 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join java.lang.IllegalArgumentException: Unable to construct journal, bookkeeper://109.231.69.172:2181;109.231.69.173:2181;109.231.69.174:2181/hdfsjournal at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1251) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:226) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:206) at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:657) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:590) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:259) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:544) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:423) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:385) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:611) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1135) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1201) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1249) ... 14 more Caused by: java.io.IOException: Error initializing zk at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.init(BookKeeperJournalManager.java:233) ... 19 more Caused by:
[jira] [Commented] (HDFS-4364) GetLinkTargetResponseProto does not handle null path
[ https://issues.apache.org/jira/browse/HDFS-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546658#comment-13546658 ] Hadoop QA commented on HDFS-4364: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563694/HDFS-4364.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDecommission {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3790//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3790//console This message is automatically generated. GetLinkTargetResponseProto does not handle null path Key: HDFS-4364 URL: https://issues.apache.org/jira/browse/HDFS-4364 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-4364.patch ClientProtocol#getLinkTarget() returns null targetPath. Hence protobuf definition GetLinkTargetResponseProto#targetPath should be optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira