Hey all,
I noticed that the maximum throttle for the datanode block scanner is
hardcoded at 8MB/s.
I think this is insufficient; on a fully loaded Sun Thumper, a full
scan at 8MB/s would take something like 70 days.
Is it possible to make this throttle a bit smarter? At the very
least, would anyone object to a patch which exposed this throttle as a
config option? Alternately, a smarter idea would be to throttle the
block scanner at (8MB/s) * (# of volumes), under the assumption that
there is at least 1 disk per volume.
Hm... on second thought, however trivial the resulting disk I/O would
be, on the Thumper example, the maximum throttle would be 3Gbps:
that's a nontrivial load on the bus.
How do other "big sites" handle this? We're currently at 110TB raw,
are considering converting ~240TB over from another file system, and
are planning to grow to 800TB during 2009. A quick calculation shows
that to do a weekly scan at that size, we're talking ~10Gbps of
sustained reads.
I still worry that the rate is too low; if we have a suspicious node,
or users report a problematic file, waiting a week for a full scan is
too long. I've asked a student to implement a tool which can trigger
a full block scan of a path (the idea would be able to do "hadoop
fsck /path/to/file -deep"). What would be the best approach for him
to take to initiate a high-rate "full volume" or "full datanode" scan?
Brian
- Datanode block scans Brian Bockelman
-