[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876537#comment-14876537
]
Daniel Templeton commented on HDFS-8873:
----------------------------------------
bq. I personally would prefer the default to be 1000.
I went with 0 because that's the general convention for "off", but if 1000
makes more sense to people it's fine with me.
bq. Any way to avoid the sleep(5000) in the test?
I hear ya. I'll see if I can reformulate the test a bit to get the sleep time
down to the bare minimum.
bq. Shouldn't stop call resume() instead of just notifyAll().
Sure. It won't matter since stop() is only called by shutdown(), which first
sets shouldRunCompile to false. But for correctness, you're right.
bq. Did we hit this problem with too big of hammer?
The majority of the patch is refactoring the report compilers so that they can
be throttled at all. The additional code to do the throttling isn't much.
It's more formal than just a sleep, but it's also more testable and extensible.
> throttle directoryScanner
> -------------------------
>
> Key: HDFS-8873
> URL: https://issues.apache.org/jira/browse/HDFS-8873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Nathan Roberts
> Assignee: Daniel Templeton
> Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch,
> HDFS-8873.003.patch, HDFS-8873.004.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms
> of disk seeks (see HDFS-8791) for details.
> It would be good if the directoryScanner() had a configurable duty cycle that
> would reduce its impact on disk performance (much like the approach in
> HDFS-8617).
> Without such a throttle, disks can go 100% busy for many minutes at a time
> (assuming the common case of all inodes in cache but no directory blocks
> cached, 64K seeks are required for full directory listing which translates to
> 655 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
