On Nov 13, 2008, at 11:32 AM, Raghu Angadi wrote:
Brian Bockelman wrote:
Hey all,
I noticed that the maximum throttle for the datanode block scanner
is hardcoded at 8MB/s.
I think this is insufficient; on a fully loaded Sun Thumper, a full
scan at 8MB/s would take something like 70 days.
Is it possible to make this throttle a bit smarter? At the very
least, would anyone object to a patch which exposed this throttle
as a config option? Alternately, a smarter idea would be to
throttle the block scanner at (8MB/s) * (# of volumes), under the
assumption that there is at least 1 disk per volume.
Making the max configurable seems useful. Either of the above
options is fine, though the first one might be simpler for
configuration.
8MB/s is calculated for around 4TB of data on a node. given 80k
seconds a day, it is around 6-7 days. 8-10 MB/s is not too bad a
load on 2-4 disk machine.
Hm... on second thought, however trivial the resulting disk I/O
would be, on the Thumper example, the maximum throttle would be
3Gbps: that's a nontrivial load on the bus.
How do other "big sites" handle this? We're currently at 110TB
raw, are considering converting ~240TB over from another file
system, and are planning to grow to 800TB during 2009. A quick
calculation shows that to do a weekly scan at that size, we're
talking ~10Gbps of sustained reads.
You have a 110 TB on single datanode and moving to 800TB nodes? Note
that this rate applies to amount of data on a single datanode.
Nah -110TB total in the system (200 datanodes), and will move to 800TB
total (probably 250 datanodes).
However, we do have some larger nodes (we range from 80GB to 48TB per
node); recent and planned purchases are in the 4-8TB per node range,
but I'd sure hate to throw away 48TB of disks :)
On the 48TB node, a scan at 8MB/s would take 70 days. I'd have to run
at a rate of 80MB/s to scan through in 7 days. While 80MB/s over 48
disks is not much, I was curious about how the rest of the system
would perform (the node is in production on a different file system
right now, so borrowing it is not easy...); 80MB/s sounds like an
awful lot for "background noise".
Do any other large sites run such large nodes? How long of a period
between block scans do sites use in order to feel "safe" ?
Brian
Raghu.
I still worry that the rate is too low; if we have a suspicious
node, or users report a problematic file, waiting a week for a full
scan is too long. I've asked a student to implement a tool which
can trigger a full block scan of a path (the idea would be able to
do "hadoop fsck /path/to/file -deep"). What would be the best
approach for him to take to initiate a high-rate "full volume" or
"full datanode" scan?