[
https://issues.apache.org/jira/browse/HDFS-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639330#comment-13639330
]
Varun Sharma commented on HDFS-4721:
------------------------------------
Yep, this patch does not fix that behaviour on the DataNode.
There is one setting for everyone for the connect retries and timeout and
AFAIK, that setting is not configurable on a case by case basis - I am talking
about the hadoop ipc client. The default is 45 retries and 20 seconds. The way
to get around it is to configure your HDFS cluster to retry maybe 3 times and
with timeout of 1 second. Also tune down your dfs.socket.timeout to 3 seconds.
With this patch, if the bad DN also stopped heart beating, you will end up
avoiding it during recovery at primary DN but if its still heartbeating but
bad, then you need to tune your timeouts.
But as I said, this is beyond the scope of this issue and I want to keep this
focused on the namenode side of things.
> Speed up lease/block recovery when DN fails and a block goes into recovery
> --------------------------------------------------------------------------
>
> Key: HDFS-4721
> URL: https://issues.apache.org/jira/browse/HDFS-4721
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.3-alpha
> Reporter: Varun Sharma
> Fix For: 2.0.4-alpha
>
> Attachments: 4721-hadoop2.patch, 4721-v2.patch, 4721-v3.patch,
> 4721-v4.patch, 4721-v5.patch
>
>
> This was observed while doing HBase WAL recovery. HBase uses append to write
> to its write ahead log. So initially the pipeline is setup as
> DN1 --> DN2 --> DN3
> This WAL needs to be read when DN1 fails since it houses the HBase
> regionserver for the WAL.
> HBase first recovers the lease on the WAL file. During recovery, we choose
> DN1 as the primary DN to do the recovery even though DN1 has failed and is
> not heartbeating any more.
> Avoiding the stale DN1 would speed up recovery and reduce hbase MTTR. There
> are two options.
> a) Ride on HDFS 3703 and if stale node detection is turned on, we do not
> choose stale datanodes (typically not heart beated for 20-30 seconds) as
> primary DN(s)
> b) We sort the replicas in order of last heart beat and always pick the ones
> which gave the most recent heart beat
> Going to the dead datanode increases lease + block recovery since the block
> goes into UNDER_RECOVERY state even though no one is recovering it actively.
> Please let me know if this makes sense. If yes, whether we should move
> forward with a) or b).
> Thanks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
