[
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404413#comment-15404413
]
Daryn Sharp commented on HDFS-10301:
------------------------------------
bq. According to Rolling upgrade documentation we first upgrade NameNodes, then
DataNodes. So in practice new DNs don't talk to old NNs.
Although the docs claim downgrading the NN requires full downtime, or rolling
downgrade DNs first, we should make an effort to ensure DNs are compatible when
possible. An emergency NN downgrade shouldn't require full downtime when a
failover to the prior release would suffice.
--
I don't like the idea of BRs triggering pruning of storages. That aside, the
patch doesn't appear to close the race. The lock is released after the storage
report is processing and re-acquired to find to find the "zombies". We're back
to out of order processing of heartbeats, which I think is the real problem,
causing false-positives.
How about something like this? {{DatanodeDescriptor}} descriptor tracks the
last {{BlockReportContext#reportId}}. The value is updated when processing a
BR - which has latest value if BR lease let it in. Heartbeat now includes the
last used {{reportId}}. On the NN, if the heartbeat contains this field, NN
will ignore heartbeart if not equal to DND. There's little details like DN
re-registration resetting the field, etc, but wouldn't something simple like
this work?
> BlockReport retransmissions may lead to storages falsely being declared
> zombie if storage report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.6.1
> Reporter: Konstantin Shvachko
> Assignee: Vinitha Reddy Gankidi
> Priority: Critical
> Fix For: 2.7.4
>
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch,
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch,
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch,
> HDFS-10301.012.patch, HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch,
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it
> sends the block report again. Then NameNode while process these two reports
> at the same time can interleave processing storages from different reports.
> This screws up the blockReportId field, which makes NameNode think that some
> storages are zombie. Replicas from zombie storages are immediately removed,
> causing missing blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
