[
https://issues.apache.org/jira/browse/HDFS-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681679#comment-14681679
]
Yi Liu commented on HDFS-8863:
------------------------------
I found an issue here, I think {{getRemaining}} should take parameter
{{DatanodeStorageInfo}} instead of {{StorageType}}, and it should just return
current storage remaining instead of get the maximum of all storages, since
when choosing datanode storage to place block, {{BlockPlacementPolicyDefault}}
will iterator all storages and try to find a suitable one. Both sum and maximum
of remaining is not correct.
I give an example here, suppose the datanode has two storages whose storage
type is DISK, and we are going to find the DISK type too, but the remaining
space of the first storage is not enough to place the block, and the second one
has enough space. According to current patch, the first datanode storage will
return, actually we want the second one.
> The remiaing space check in BlockPlacementPolicyDefault is flawed
> -----------------------------------------------------------------
>
> Key: HDFS-8863
> URL: https://issues.apache.org/jira/browse/HDFS-8863
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Priority: Critical
> Labels: 2.6.1-candidate
> Attachments: HDFS-8863.patch
>
>
> The block placement policy calls
> {{DatanodeDescriptor#getRemaining(StorageType}}}} to check whether the block
> is going to fit. Since the method is adding up all remaining spaces, namenode
> can allocate a new block on a full node. This causes pipeline construction
> failure and {{abandonBlock}}. If the cluster is nearly full, the client might
> hit this multiple times and the write can fail permanently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
