Ok thanks Matt, Ah. I think I've realised my mistake. I saw this webpage http://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/ which showed the following command. GZIP="--rsyncable" tar zcvf toto.tar.gz /toto
I've just realised that gzip must be picking up the environment variable when it's invoked by tar. Cool. So if I understand you correctly you are suggesting that I modify my backup line to be something like this: tar czf - --files-from $FILE_LIST | gzip --rsyncable > ${BACKUP_FILE} or: GZIP="--rsyncable" tar cvf - --files-from $FILE_LIST > ${BACKUP_FILE} then rsync ${BACKUP_FILE} $HOST:${BACKUP_FILE} I see that Debian's gzip has --rsycnable available. Hoorah. I just tried some tests but didn't get the results I expected. I seem to be getting quite different files: $ GZIP="--rsyncable" tar zcf m5.tgz ./newdir $ GZIP="--rsyncable" tar zcf m6.tgz ./newdir $ sum -r m[56]* 25302 2929 m5.tgz 60054 2929 m6.tgz $ cmp m[56]* m5.tgz m6.tgz differ: byte 5, line 1 I also tried this with the other format: $ tar cf - ./newdir | gzip --rsyncable > m7.tgz $ tar cf - ./newdir | gzip --rsyncable > m8.tgz $ sum -r m[78]* 36275 2929 m7.tgz 15148 2929 m8.tgz $ cmp m[78]* m7.tgz m8.tgz differ: byte 5, line 1 I tried this: $ tar cf m9.tar ./newdir $ tar cf m10.tar ./newdir $ sum -r m*tar 54744 3340 m10.tar 54744 3340 m9.tar $ gzip --rsyncable m9.tar $ gzip --rsyncable m10.tar $ sum -r m*tar 21783 2929 m9.tar.gz 30175 2929 m10.tar.gz It seems that gzip produces different output even given the same input. Not very rsync friendly. I did these tests on Debian Lenny v5.0.3 with gzip v1.3.12. Have I missed the point? I'm not sure I can see what I'd use the rdiff command for since rsync will do the delta-transfer cleverness for me. Cheers, Mark > Subject: RE: rsync of STDIN to a file. > From: m...@mattmccutchen.net > To: mark_yo...@hotmail.com > CC: rsync@lists.samba.org > Date: Thu, 19 Nov 2009 11:53:30 -0500 > > On Thu, 2009-11-19 at 16:28 +0000, Mark Young wrote: > > I tried an experiment to see how rsync coped with the tar compressed > > files versus the uncompressed files. I took a .tgz from last week and > > rsync'd it with last nights version. rsync achieved the transfer with > > a 1.61 speed up. Specifically 172MB of the 262MB file was transferred. > > I then tried the same thing with the uncompressed tar files. rsync > > achieved the transfer with a 690 speed up. Specifically 5MB for the > > 3.6GB file was transferred. Clearly if diskspace were not an issue > > this would be by far the superior option. Obviously I could increase > > the complexity of the overall solution with compress & uncompress jobs > > in sync with the backup strategy, but I believe in keeping it as > > simple as possible when it comes to backups. > > Right. The gzip --rsyncable option is designed for this case: files > compressed with --rsyncable should delta-transfer almost as well as > uncompressed files, at a minor penalty in the compression ratio. > > > rdiff is a new one on me. Very interesting and something I'll > > experiment with. For our strategy of 7 daily backups and 12 monthly > > backups I guess I'd need to use two rdiff destinations so I could > > apply the --remove-older-than 1W and --remove-older-than 365D options > > respectively. It might not suit my current situation as explained > > above regarding storing a tarball versus a directory hierachy, but > > it's definitely worth knowing about. > > I was talking about rdiff (a command-line interface to the > delta-transfer algorithm), not rdiff-backup (the incremental backup > tool). > > > Also I'd not heard of the --rsyncable option for tar. It's not there > > on my Debian Etch system, nor in my cygwin tar. So I guess it's not in > > common distributions by default. Still it sounds useful and is > > something I will bear in mind as something to consider. > > The option applies to gzip, not tar. In fact, it looks like Fedora > applies an updated version of the gzip-rsyncable patch to their gzip > package. You can get their patch from: > > http://cvs.fedoraproject.org/viewvc/rpms/gzip/F-11/gzip-1.3.9-rsync.patch?view=co > > -- > Matt >
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html