[ https://issues.apache.org/jira/browse/HDFS-15942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ayush Saxena updated HDFS-15942: -------------------------------- Status: Patch Available (was: Open) > Increase Quota initialization threads > ------------------------------------- > > Key: HDFS-15942 > URL: https://issues.apache.org/jira/browse/HDFS-15942 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Stephen O'Donnell > Assignee: Stephen O'Donnell > Priority: Major > Attachments: HDFS-15942.001.patch > > > On large namespaces, the quota initialization at started can take a long time > with the default 4 threads. Also on NN failover, often the quota needs to be > calculated before the failover can completed, delaying the failover. > I performed some benchmarks some time back on a large image (316M inodes 35GB > on disk), the quota load takes: > {code} > quota - 4 threads 39 seconds > quota - 8 threads 23 seconds > quota - 12 threads 20 seconds > quota - 16 threads 15 seconds > {code} > As the quota is calculated when the NN is starting up (and hence doing no > other work) or at failover time before the new standby becomes active, I > think the quota should use as many threads as possible. > I proposed we change the default to 8 or 12 on at least trunk and branch-3.3 > so we have a better default going forward. > Has anyone got any other thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org