On 6/9/2010 4:35 PM, Pawel Jakub Dawidek wrote:
Hi.
Currently zfs send has -D flag, which allows to deduplicate blocks
in single stream.
I'm wondering if it would be possible not to send blocks in incremental
stream if we know they are part of the given dataset already and were
sent to the remote site with some earlier snapshots.
I know deduplication is pool-wide mechanism and the block might be part
of many different datasets. In my case I'd need to know that the block
I'm about to send is part of this particular dataset.
With the current ZFS design, is something like this even possible to
implement in some clean way or would there be a need for heavy
modifcations of ZFS internals?
If it is doable, could you suggest a good starting point
First off, even with an incremental, you could dedup at the receiving
end easily, so really, the only thing you would be doing is cutting down
on the amount of data traffic being send over the wire (which, could be
significant).
You'd have to run some sort of process on the receiving system. There's
no other way to design this kind of thing - you can't rely on any
config/state/etc on the "sending" system; for consistency, you'd *have*
to query the receiver for its state.
The closest analog to what you're asking for is rsync.
I don't see any modifications to ZFS that has to be done to support
something like that - it's just a userland app. Note that there would
have to be a non-trivial amount of overhead data communication between
the two hosts. For each block being sent, the send would have to send
the checksum over to the receiving side, which would have to check it's
DDT to see if it's already there. It would then either send an ACK or
NAK back to tell the sender whether or not to send the actual data. So,
there's be a *lot* of small packet traffic between the two machines. I
suppose one could be smart and package up multiple blocks' checksums in
a single packet, but the fact remains that such a system would be
non-trivially chatty. Much chattier than rsync.
--
Erik Trimble
Java System Support
Mailstop: usca22-123
Phone: x17195
Santa Clara, CA
_______________________________________________
zfs-code mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/zfs-code
