[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15150908#comment-15150908
]
Rakesh R commented on HDFS-8786:
--------------------------------
I’ve started analyzing this and attached a draft patch for discussion.
Currently, the striped blocks from the decommissioned node is considered as
insufficiently replicated and is sending EC_RECONSTRUCT command to the
datanode. But I’ve below observation and requires changes in this part. It
would be helpful to get input from others. Thanks!
# During reconstruction process, like contiguous block behavior the
decommissioning datanode also be added to the {{liveBlockIndices}} list, which
will trouble the reconstruction decoding algo and throws the following
exception. In the draft patch, I just skipped the source node for simplicity.
Can we consider decommission as a special case and skip the erased index checks
? It would be good if someone can give more details about this condition.
{code}
RSRawDecoder.java
//....
if (erasedIndexes[i] == erasedOrNotToReadIndexes[j]){
found = true;
}
//....
if (!found) {
throw new HadoopIllegalArgumentException(
"Inputs not fully corresponding to erasedIndexes in null places");
}
{code}
# Secondly, contiguous block have the sorting logic and is keeping the
decommissioned node at the end of the block location list. This is done using
{{DFSUtil.DECOM_COMPARATOR}} as shown below. But in the striped block, it
doesn’t have the sorting logic and is maintaining the index position. Whats
your opinion in replacing the decommissioned storage index with the newly
reconstructed node and keep the decommissioned node at the end in
{{BlockInfoStriped}} itself.
{code}
DatanodeManager.java
Comparator<DatanodeInfo> comparator = avoidStaleDataNodesForRead ?
new DFSUtil.DecomStaleComparator(staleInterval) :
DFSUtil.DECOM_COMPARATOR;
{code}
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> -----------------------------------------------------------------------------------
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Rakesh R
> Attachments: HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
