we use rsync to a zfsonlinux fs with compression + dedup and snapshots for incrementals.
works well :) means your most recent copy is a live fs people can access easily. On Mon, Jun 17, 2019 at 9:44 PM Bill Wichser <[email protected]> wrote: > We have moved to a rsync disk backup system, from TSM tape, in order to > have a DR for our 10 PB GPFS filesystem. We looked at a lot of options > but here we are. > > md5 checksums take a lot of compute time with huge files and even with > millions of smaller ones. The bulk of the time for running rsync is > spent in computing the source and destination checksums and we'd like to > alleviate that pain of a cryptographic algorithm. > > Googling around, I found no mention of using a technique like this to > improve rsync performance. I did find reference to a few hashing > algorithms though which could certainly work here (xxhash, murmurhash, > sbox, cityhash64). > > Rsync has certainly been around for a few years! We are going to pursue > changing the current checksum algorithm and using something much faster. > If anyone has done this already and would like to share their > experiences that would be wonderful. Ideally this could be some optional > plugin for rsync where users could choose which checksummer to use. > > Bill > _______________________________________________ > Beowulf mailing list, [email protected] sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf > -- Dr Stuart Midgley [email protected]
_______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
