[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876500#comment-14876500
]
Nathan Roberts commented on HDFS-8873:
--------------------------------------
Thanks [~templedf] for the patch. A few comments.
hdfs-default.xml
- I personally would prefer the default to be 1000. In my mind 0 is a special
out-of-range condition that we're allowing to mean "full rate". Just reading
the default of 0 and then the first sentence of the description could easily
lead one to believe the report threads are effectively off by default.
Test
- Any way to avoid the sleep(5000) in the test? Our tests already take a really
long time, so anytime we can avoid sleeping it's better. Maybe wait at most 5
seconds for timeWaitingMs.get() to become > 0
directoryScannerThrottle
- Shouldn't stop call resume() instead of just notifyAll(). Will cycle() get
out if we try to shutdown while in that wait?
- Did we hit this problem with too big of hammer? Couldn't cycle() be
implemented with a simple sleep? For example, with a 75% duty cycle,
{noformat}
n = Time.monotonicNow() % 1000;
if (n > 1000 * 0.75) sleep(1000- n)
{noformat}
Seems like it could be as simple as a config and a couple of lines of code.
Maybe I'm missing something or there are grander plans for the throttle.
> throttle directoryScanner
> -------------------------
>
> Key: HDFS-8873
> URL: https://issues.apache.org/jira/browse/HDFS-8873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Nathan Roberts
> Assignee: Daniel Templeton
> Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch,
> HDFS-8873.003.patch, HDFS-8873.004.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms
> of disk seeks (see HDFS-8791) for details.
> It would be good if the directoryScanner() had a configurable duty cycle that
> would reduce its impact on disk performance (much like the approach in
> HDFS-8617).
> Without such a throttle, disks can go 100% busy for many minutes at a time
> (assuming the common case of all inodes in cache but no directory blocks
> cached, 64K seeks are required for full directory listing which translates to
> 655 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
