On 6/18/19 11:00 AM, Bill Wichser wrote:
Well thanks for THAT pointer! Using --checksum-choice=none results in
speedup of somewhere between 2-3 times. That's my validation of the
checksum theory things have been pointing towards. Now to get xxhash
into rsync and I think we are all set.
Glad to hear that improved things for you. I will caution against using
a non-crypto hash for this though. Unless you can verify xxhash offers
at least as much protection against collisions as MD5, you may not be
protecting your transfers sufficiently. I haven't looked into xxhash
enough to offer advice on this.
If you're already using rsync as just a piece of a larger framework, it
really might be worth cooking up a script to md5sum the list in
files-from on both sides in parallel to use all of your cores on both
sides and then compare the results after the transfer. Then you can
just use --checksum-choice=none but still know your data made it safely
to the other side.
Best,
ellis
--
Ellis H. Wilson III, Ph.D.
www.ellisv3.com
_______________________________________________
Beowulf mailing list, [email protected] sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
