Peter A wrote:
On Wednesday, January 05, 2011 08:19:04 pm Spelic wrote:
I'd just make it always use the fs block size. No point in making it
variable.
Agreed. What is the reason for variable block size?
First post on this list - I mostly was just reading so far to learn more on fs
design but this is one topic I (unfortunately) have experience with...
You wouldn't believe the difference variable block size dedupe makes. For a
pure fileserver, its ok to dedupe on block level but for most other uses,
variable is king. One big example is backups. Netbackup and most others
produce one stream with all data even when backing up to disk. Imagine you
move a whole lot of data from one dir to another. Think a directory with huge
video files. As a filesystem it would be de-duped nicely. The backup stream
however may and may not have matching fs blocks. If the directory name before
and after has the same lengths and such - then yeah, dedupe works. Directory
name is a byte shorter? Everything in the stream will be offset by one byte -
and no dedupe will occur at all on the whole dataset. In real world just
compare the dedupe performance of an Oracle 7000 (zfs and therefore fs block
based) to a DataDomain (variable lenght) in this usage scenario. Among our
customers we see something like 3 to 17x dedupe ration on the DD, 1.02 - 1.05
in the 7000.
Can you elaborate what you're talking about here? How does the length of
a directory name affect alignment of file block contents? I don't see
how variability of length matters, other than to make things a lot more
complicated. Have you some real research/scientifically gathered data
(non-hearsay/non-anecdotal) on the underlying reasons for the
discrepancy in the deduping effectiveness you describe? 3-17x difference
doesn't plausibly come purely from fixed vs. variable length block sizes.
The only case where I'd bother consider variable length deduping is in
file deduping (rather than block), in this case we can just make a COW
hard-link and it's _really_ cheap and effective.
Gordan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
