[
https://issues.apache.org/jira/browse/HDFS-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847744#comment-13847744
]
Jing Zhao commented on HDFS-5496:
---------------------------------
Thanks for the further explanation, Vinay. Now I see what you mean.
{code}
blockManager.clearQueues();
blockManager.processAllPendingDNMessages();
{code}
In my previous comment, I thought you also planned to remove the clearQueues()
call..
So if we clear all the queues, and continue with the ongoing initialization,
the blocks that were processed before the startActiveService call will not be
re-processed? I.e., we finally will generate the replication queues for only
part of the blocks.
> Make replication queue initialization asynchronous
> --------------------------------------------------
>
> Key: HDFS-5496
> URL: https://issues.apache.org/jira/browse/HDFS-5496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Kihwal Lee
> Assignee: Vinay
> Attachments: HDFS-5496.patch, HDFS-5496.patch
>
>
> Today, initialization of replication queues blocks safe mode exit and certain
> HA state transitions. For a big name space, this can take hundreds of seconds
> with the FSNamesystem write lock held. During this time, important requests
> (e.g. initial block reports, heartbeat, etc) are blocked.
> The effect of delaying the initialization would be not starting replication
> right away, but I think the benefit outweighs. If we make it asynchronous,
> the work per iteration should be limited, so that the lock duration is
> capped.
> If full/incremental block reports and any other requests that modifies block
> state properly performs replication checks while the blocks are scanned and
> the queues populated in background, every block will be processed. (Some may
> be done twice) The replication monitor should run even before all blocks are
> processed.
> This will allow namenode to exit safe mode and start serving immediately even
> with a big name space. It will also reduce the HA failover latency.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
