[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated HDFS-8873:
-----------------------------------
Attachment: HDFS-8873.002.patch
[~cmccabe], I implemented all of your suggestions except the throttle limit
property name.
bq. I would prefer to see per-type configuration keys that are more
descriptive. For example,
dfs.datanode.directoryscan.timeslice.throttle.ms.per.sec.
Encapsulating the throttle stuff requires a reasonable abstraction of what a
throttle is. The various kinds of throttles (time, file, iops, ...) are all
pretty different and aren't easy to overlay with a single abstraction. I
decided to give up on the idea of making the throttle type selectable. The
limit therefore always means the same thing, and so I think it's fair to leave
it's name as is.
bq. What's the purpose of rethrowing exceptions here?
I'm only rethrowing Errors, which is the right thing to do. It would be bad if
a thread caught an OOME and discarded it.
> throttle directoryScanner
> -------------------------
>
> Key: HDFS-8873
> URL: https://issues.apache.org/jira/browse/HDFS-8873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Nathan Roberts
> Assignee: Daniel Templeton
> Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms
> of disk seeks (see HDFS-8791) for details.
> It would be good if the directoryScanner() had a configurable duty cycle that
> would reduce its impact on disk performance (much like the approach in
> HDFS-8617).
> Without such a throttle, disks can go 100% busy for many minutes at a time
> (assuming the common case of all inodes in cache but no directory blocks
> cached, 64K seeks are required for full directory listing which translates to
> 655 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
