Stephen O'Donnell created HDFS-15942:
----------------------------------------
Summary: Increase Quota initialization threads
Key: HDFS-15942
URL: https://issues.apache.org/jira/browse/HDFS-15942
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
On large namespaces, the quota initialization at started can take a long time
with the default 4 threads. Also on NN failover, often the quota needs to be
calculated before the failover can completed, delaying the failover.
I performed some benchmarks some time back on a large image (316M inodes 35GB
on disk), the quota load takes:
{code}
quota - 4 threads 39 seconds
quota - 8 threads 23 seconds
quota - 12 threads 20 seconds
quota - 16 threads 15 seconds
{code}
As the quota is calculated when the NN is starting up (and hence doing no other
work) or at failover time before the new standby becomes active, I think the
quota should use as many threads as possible.
I proposed we change the default to 8 or 12 on at least trunk and branch-3.3 so
we have a better default going forward.
Has anyone got any other thoughts?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
