[ https://issues.apache.org/jira/browse/HDFS-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681087#comment-14681087 ]
Yi Liu commented on HDFS-8863: ------------------------------ The patch looks good, thanks Kihwal. Agree to check the maximum remaining of storage instead of summing all remaining spaces, since we can split a block to multiple storage. One question is: should we still keep {{getRemaining(StorageType t)}} there? No place to call it, so can remove it? Nits in the test: {code} DatanodeStorageInfo si = new DatanodeStorageInfo( storages[5].getDatanodeDescriptor(), extraStorage); {code} It's unused and can be removed. {code} // No available space in the extra storage of dn0 updateHeartbeatForExtraStorage(0L, 0L, 0L, 0L); {code} typo in comment: dn0 should be dn5 It's better to set no available space in the extra storage at the end of {{testChooseNodeWithMultipleStorages}}: {code} updateHeartbeatForExtraStorage(0L, 0L, 0L, 0L); {code} > The remiaing space check in BlockPlacementPolicyDefault is flawed > ----------------------------------------------------------------- > > Key: HDFS-8863 > URL: https://issues.apache.org/jira/browse/HDFS-8863 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.6.0 > Reporter: Kihwal Lee > Assignee: Kihwal Lee > Priority: Critical > Labels: 2.6.1-candidate > Attachments: HDFS-8863.patch > > > The block placement policy calls > {{DatanodeDescriptor#getRemaining(StorageType}}}} to check whether the block > is going to fit. Since the method is adding up all remaining spaces, namenode > can allocate a new block on a full node. This causes pipeline construction > failure and {{abandonBlock}}. If the cluster is nearly full, the client might > hit this multiple times and the write can fail permanently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)