[
https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905122#comment-14905122
]
Colin Patrick McCabe edited comment on HDFS-8873 at 9/23/15 7:47 PM:
---------------------------------------------------------------------
I guess the short summary is that I'm not happy with the "modulo" solution to
the timekeeping problem since I think its inaccuracy (in the direction of not
running often enough, or at all) will be unacceptable at lower throttle rates.
We have a StopWatch class which should provide a pretty easy solution to this
problem that doesn't suffer from these issues. Just start a StopWatch when you
enable scanning. Check the elapsed duration on the StopWatch before scanning
each block-- if it is above
dfs.datanode.directoryscan.throttle.limit.ms.per.sec, then disable scanning and
go into the other loop, which calls Thread.sleep until 1000 -
dfs.datanode.directoryscan.throttle.limit.ms.per.sec ms have elapsed.
This solution is robust against slightly shorter or longer thread pauses. Like
the modulo solution, it also may run at a slightly lower rate than expected in
the case where we oversleep. Unlike the modulo solution, it does not have a
pathological case where we never get to run at all despite a non-zero throttle
rate.
{code}
399 public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT =
21600;
...
460 int defaultLimit =
461 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT;
462 String logMsg;
463
464 if (throttleLimitMsPerSec < defaultLimit) {
465 logMsg = String.format(START_MESSAGE_WITH_THROTTLE, firstScanTime,
466 scanPeriodMsecs, throttleLimitMsPerSec);
467 } else {
468 logMsg = String.format(START_MESSAGE, firstScanTime,
scanPeriodMsecs);
469 }
{code}
I'm confused about this code... why are we comparing the directory scan
interval (in seconds) with the throttle limit in milliseconds? It seems like
this was intended to test for the case where the throttle is disabled, but some
wires got crossed?
was (Author: cmccabe):
I guess the short summary is that I'm not happy with the "modulo" solution to
the timekeeping problem since I think its inaccuracy (in the direction of not
running often enough, or at all) will be unacceptable at lower throttle rates.
We have a StopWatch class which should provide a pretty easy solution to this
problem that doesn't suffer from these issues. Just start a StopWatch when you
enable scanning. Check the elapsed duration on the StopWatch before scanning
each block-- if it is above
dfs.datanode.directoryscan.throttle.limit.ms.per.sec, then disable scanning and
go into the other loop, which calls Thread.sleep until 1000 -
dfs.datanode.directoryscan.throttle.limit.ms.per.sec ms have elapsed.
This solution is robust against slightly shorter or longer thread pauses. Like
the module solution, it also may run at a slightly lower rate than expected in
the case where we oversleep. Unlike the modulo solution, it does not have a
pathological case where we never get to run at all despite a non-zero throttle
rate.
{code}
399 public static final int DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT =
21600;
...
460 int defaultLimit =
461 DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT;
462 String logMsg;
463
464 if (throttleLimitMsPerSec < defaultLimit) {
465 logMsg = String.format(START_MESSAGE_WITH_THROTTLE, firstScanTime,
466 scanPeriodMsecs, throttleLimitMsPerSec);
467 } else {
468 logMsg = String.format(START_MESSAGE, firstScanTime,
scanPeriodMsecs);
469 }
{code}
I'm confused about this code... why are we comparing the directory scan
interval (in seconds) with the throttle limit in milliseconds? It seems like
this was intended to test for the case where the throttle is disabled, but some
wires got crossed?
> throttle directoryScanner
> -------------------------
>
> Key: HDFS-8873
> URL: https://issues.apache.org/jira/browse/HDFS-8873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.7.1
> Reporter: Nathan Roberts
> Assignee: Daniel Templeton
> Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch,
> HDFS-8873.003.patch, HDFS-8873.004.patch, HDFS-8873.005.patch,
> HDFS-8873.006.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms
> of disk seeks (see HDFS-8791) for details.
> It would be good if the directoryScanner() had a configurable duty cycle that
> would reduce its impact on disk performance (much like the approach in
> HDFS-8617).
> Without such a throttle, disks can go 100% busy for many minutes at a time
> (assuming the common case of all inodes in cache but no directory blocks
> cached, 64K seeks are required for full directory listing which translates to
> 655 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
