[
https://issues.apache.org/jira/browse/HDFS-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372191#comment-14372191
]
Colin Patrick McCabe commented on HDFS-7960:
--------------------------------------------
bq. EMPTY_STORAGE_INFO_LIST is a little dangerous since it can still be
modified, maybe use ImmutableLists instead? Alternatively we can just allocate
the empty LinkedList, it'll be young gen and cheap.
Yeah, let's use ImmutableList
bq. double space after equals here
fixed
bq. unnecessary whitespace change?
fixed
Other changes: I thought about this a little more and I think instead of just a
boolean to tell when the last rpc is happening, we should have 3 things: the
current unique (across same DN) 64-bit block report ID, the total number of
RPCs in the block report, and the current rpc in the block report.
Can get a unique ID on the datanode from the monotonic time in nanoseconds.
(If for some bizarre reason the clock doesn't advance between BRs, we can just
add one to the previous time). This should also ensure that if we restart the
DN, we don't re-use IDs. The ID should help match up DN and NN in the logs.
Having a total RPC count and a current RPC index is more flexible if we want to
chunk block report RPCs differently in the future.
> The full block report should prune zombie storages even if they're not empty
> ----------------------------------------------------------------------------
>
> Key: HDFS-7960
> URL: https://issues.apache.org/jira/browse/HDFS-7960
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Lei (Eddy) Xu
> Assignee: Colin Patrick McCabe
> Priority: Critical
> Attachments: HDFS-7960.002.patch, HDFS-7960.003.patch
>
>
> The full block report should prune zombie storages even if they're not empty.
> We have seen cases in production where zombie storages have not been pruned
> subsequent to HDFS-7575. This could arise any time the NameNode thinks there
> is a block in some old storage which is actually not there. In this case,
> the block will not show up in the "new" storage (once old is renamed to new)
> and the old storage will linger forever as a zombie, even with the HDFS-7596
> fix applied. This also happens with datanode hotplug, when a drive is
> removed. In this case, an entire storage (volume) goes away but the blocks
> do not show up in another storage on the same datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
