[
https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576938#comment-15576938
]
Andrew Wang commented on HDFS-10999:
------------------------------------
Thanks for the insight Allen,
{quote}
M: "How long for recovery?"
A: "No idea. The NN doesn't tell me if these are EC blocks or regular blocks
that were lost and one is faster to recover than the other."
{quote}
That's what I was getting at with the pendingReconstructionBlocksCount. If we
fix it as I talked about above, it'd actually tell you how much work is
remaining, and how fast that work is progressing.
{quote}
...I've also used it during system recovery and migrations as a measurement of
how many more DNs I need to bring up such that I have more sources for block
replication.
{quote}
Would the "pending" queue metrics also work for this? We can also look at
improved DN-side metrics related to replication work.
{quote}
This number represents something that I as an admin have some semblance of
control over: I could always manually copy blocks from one node to another to
speed things up.
Under EC, I don't know of anything manual I can do if it is missing chunks of
blocks.
{quote}
I really, really hope that manually copying blocks around is not a normal part
of operating an HDFS cluster.
Point is still valid though, maybe we should take a harder look at the recovery
work throttles on the NN and DN, and make them dynamically reconfigurable if
they aren't. I recall seeing some customer issues where we temporarily bumped
up these values to more quickly recover from failures.
> Use more generic "low redundancy" blocks instead of "under replicated" blocks
> -----------------------------------------------------------------------------
>
> Key: HDFS-10999
> URL: https://issues.apache.org/jira/browse/HDFS-10999
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
> Affects Versions: 3.0.0-alpha1
> Reporter: Wei-Chiu Chuang
> Assignee: Yuanbo Liu
> Labels: supportability
>
> Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic
> term "low redundancy" to the old-fashioned "under replicated". But this term
> is still being used in messages in several places, such as web ui, dfsadmin
> and fsck. We should probably change them to avoid confusion.
> File this jira to discuss it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
