[
https://issues.apache.org/jira/browse/HDFS-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849670#comment-13849670
]
Jing Zhao commented on HDFS-5496:
---------------------------------
bq. So it can completely miss initialization of replication queues itself.
So here will SafeMode#leave finally call the initializeReplQueues?
As a summary, if we make the following change:
# in startActiveService(), change the condition of replication queue
initialization to
{code}
!isInSafeMode()
{code}
# in checkMode(), change the condition to
{code}
canInitializeReplQueues() && !isPopulatingReplQueues() && !haEnabled
{code}
# and in SafeMode#leave, we still have
{code}
if (!isPopulatingReplQueues() && shouldPopulateReplQueues()) {
initializeReplQueues();
}
{code}
Then in non-HA mode, we have:
# if the FSN enters safemode before starting active service,
checkMode()/leave() will initialize the replication queue
# if the FSN does not enter safemode, startActiveService will initialize the
replication queue
In HA mode, we have:
# checkMode will no longer initialize replication queue
# if the FSN enters safemode before starting active server, SafeMode#leave will
call initializeReplQueues
# if FSN does not enter safemode, startActiveService will initialize
replication queue
> Make replication queue initialization asynchronous
> --------------------------------------------------
>
> Key: HDFS-5496
> URL: https://issues.apache.org/jira/browse/HDFS-5496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Kihwal Lee
> Assignee: Vinay
> Attachments: HDFS-5496.patch, HDFS-5496.patch, HDFS-5496.patch,
> HDFS-5496.patch
>
>
> Today, initialization of replication queues blocks safe mode exit and certain
> HA state transitions. For a big name space, this can take hundreds of seconds
> with the FSNamesystem write lock held. During this time, important requests
> (e.g. initial block reports, heartbeat, etc) are blocked.
> The effect of delaying the initialization would be not starting replication
> right away, but I think the benefit outweighs. If we make it asynchronous,
> the work per iteration should be limited, so that the lock duration is
> capped.
> If full/incremental block reports and any other requests that modifies block
> state properly performs replication checks while the blocks are scanned and
> the queues populated in background, every block will be processed. (Some may
> be done twice) The replication monitor should run even before all blocks are
> processed.
> This will allow namenode to exit safe mode and start serving immediately even
> with a big name space. It will also reduce the HA failover latency.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
