Kihwal Lee created HDFS-5496:
--------------------------------

             Summary: Make replication queue initialization asynchronous
                 Key: HDFS-5496
                 URL: https://issues.apache.org/jira/browse/HDFS-5496
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
            Reporter: Kihwal Lee


Today, initialization of replication queues blocks safe mode exit and certain 
HA state transitions. For a big name space, this can take hundreds of seconds 
with the FSNamesystem write lock held.  During this time, important requests 
(e.g. initial block reports, heartbeat, etc) are blocked.

The effect of delaying the initialization would be not starting replication 
right away, but I think the benefit outweighs. If we make it asynchronous, the 
work per iteration should be limited, so that the lock duration is capped. 

If full/incremental block reports and any other requests that modifies block 
state properly performs replication checks while the blocks are scanned and the 
queues populated in background, every block will be processed. (Some may be 
done twice)  The replication monitor should run even before all blocks are 
processed.

This will allow namenode to exit safe mode and start serving immediately even 
with a big name space. It will also reduce the HA failover latency.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Reply via email to