--append and then --append-verify will not work.
Hi all, I do some testings on rsync by using the following method: 1- rsyning the following file with rsync: rsync rsync://ftp.cn.debian.org/debian/dists/Debian7.8/main/binary-all/ Packages.bz2 . 2- Then I split the file into little files with the file size equaling to 128K, which is the maximum of the block-size accepted by rsync. split -b 128K Packages.bz2 And at this point, I obtained the following 20 files: werner@debian:~/http_resume_test$ ls x* xaa xac xae xag xai xak xam xao xaq xas xab xad xaf xah xaj xal xan xap xar xat Then I delete the some files, say, xab and xae, from the above little files, and then use the cat tool to regenerate the Packages.bz2 by only using the remaining little files: werner@debian:~/http_resume_test$ for i in `ls x*`; do cat $i Packages.bz2; done 3- Then, I re-run rsync with the following commands to syncing on the above re-generated Packages.bz2 file for testing: $ rsync -v -B131072 --append rsync://ftp.cn.debian.org/debian/dists/ Debian7.8/main/binary-all/Packages.bz2 . [snip] Packages.bz2 sent 47 bytes received 577,028 bytes 230,830.00 bytes/sec total size is 2,541,616 speedup is 4.40 $ rsync -v -B131072 --append-verify rsync://ftp.cn.debian.org/debian/ dists/Debian7.8/main/binary-all/Packages.bz2 . [snip] sent 28 bytes received 1,381 bytes 563.60 bytes/sec total size is 2,541,616 speedup is 1,803.84 4- Finally, I use diff to see the differences between the original Packages.bz2 file the the one generated by the above two rsyncing runnings with --append and --append-verify options. Note, in the following command, the ~/Packages.bz2 is the original Packages.bz2 file directly obtained from http://ftp.cn.debian.org/debian/ dists/Debian7.8/main/binary-all/Packages.bz2. werner@debian:~/http_resume_test$ diff Packages.bz2 ~/Packages.bz2 Binary files Packages.bz2 and /home/werner/Packages.bz2 differ As you can see, the finally file obtained by --append then --append- verify is a wrong file which is differ from the original one. 5- In the above test, if I only the running the following command to re- rsyncing on the re-gererated Packages.bz2: $ rsync -v -B131072 --append-verify rsync://ftp.cn.debian.org/debian/ dists/Debian7.8/main/binary-all/Packages.bz2 . The result file will be correct, see following for detail: werner@debian:~/http_resume_test$ diff Packages.bz2 ~/Packages.bz2 werner@debian:~/http_resume_test$ Any hints on this issue? Regards -- .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Downloading specific files with rsync and make them keeping the original directories structures.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 See --relative though it will need a little bit of massaging to avoid the debian dir. On 04/06/2015 12:29 AM, Hongyi Zhao wrote: Hi all, See the following command: $ rsync -av ftp.cn.debian.org::debian/dists/Debian7.8/Release . Which will download the file Release under the directory from which the rsync command is isssued. If I want to keep the original directories structures, say, for this case, put the the Release in the following location: ./dists/Debian7.8/Release If the directory tree doesn't exist, let the rsync create the directory structure firstly and then put the file into it. Is this possible for this case? Regards - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlUiDN0ACgkQVKC1jlbQAQfMrwCdH5vYHTsUFcetEgd1I9jrNpN8 clwAoIEUYPJZuq7Cr6wfEln1vYxvhFGn =iMUS -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--append and then --append-verify will not work.
Hi all, I do some testings on rsync by using the following method: 1- rsyning the following file with rsync: rsync rsync://ftp.cn.debian.org/debian/dists/Debian7.8/main/binary-all/ Packages.bz2 . 2- Then I split the file into little files with the file size equaling to 128K, which is the maximum of the block-size accepted by rsync. split -b 128K Packages.bz2 And at this point, I obtained the following 20 files: werner@debian:~/http_resume_test$ ls x* xaa xac xae xag xai xak xam xao xaq xas xab xad xaf xah xaj xal xan xap xar xat Then I delete some files, say, xab and xae, from the above little files, and then use the cat tool to regenerate the Packages.bz2 by only using the remaining little files: werner@debian:~/http_resume_test$ for i in `ls x*`; do cat $i Packages.bz2; done 3- Then, I re-run rsync with the following commands to syncing on the above re-generated Packages.bz2 file for testing: $ rsync -v -B131072 --append rsync://ftp.cn.debian.org/debian/dists/ Debian7.8/main/binary-all/Packages.bz2 . [snip] Packages.bz2 sent 47 bytes received 577,028 bytes 230,830.00 bytes/sec total size is 2,541,616 speedup is 4.40 $ rsync -v -B131072 --append-verify rsync://ftp.cn.debian.org/debian/ dists/Debian7.8/main/binary-all/Packages.bz2 . [snip] sent 28 bytes received 1,381 bytes 563.60 bytes/sec total size is 2,541,616 speedup is 1,803.84 4- Finally, I use diff to see the differences between the original Packages.bz2 file the the one generated by the above two rsyncing runnings with --append and --append-verify options. Note, in the following command, the ~/Packages.bz2 is the original Packages.bz2 file directly obtained from http://ftp.cn.debian.org/debian/ dists/Debian7.8/main/binary-all/Packages.bz2. werner@debian:~/http_resume_test$ diff Packages.bz2 ~/Packages.bz2 Binary files Packages.bz2 and /home/werner/Packages.bz2 differ As you can see, the finally file obtained by --append then --append- verify is a wrong file which is differ from the original one. 5- In the above test, if I only running the following command to re- rsyncing on the re-gererated Packages.bz2: $ rsync -v -B131072 --append-verify rsync://ftp.cn.debian.org/debian/ dists/Debian7.8/main/binary-all/Packages.bz2 . The result file will be correct, see following for detail: werner@debian:~/http_resume_test$ diff Packages.bz2 ~/Packages.bz2 werner@debian:~/http_resume_test$ Any hints on this issue? Regards -- Hongyi Zhao hongyi.z...@gmail.com Xinjiang Technical Institute of Physics and Chemistry Chinese Academy of Sciences GnuPG DSA: 0xD108493 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Downloading specific files with rsync and make them keeping the original directories structures.
Hi all, See the following command: $ rsync -av ftp.cn.debian.org::debian/dists/Debian7.8/Release . Which will download the file Release under the directory from which the rsync command is isssued. If I want to keep the original directories structures, say, for this case, put the the Release in the following location: ./dists/Debian7.8/Release If the directory tree doesn't exist, let the rsync create the directory structure firstly and then put the file into it. Is this possible for this case? Regards -- .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --link-dest won't link even if existing file is out of date
Feature request: allow --link-dest dir to be linked to even if file exists in target. This statement from the man page is adhered to too strongly IMHO: This option works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so it never looks in the link-dest dirs when a destination file already exists). I was suprised by this behaviour as generally the scheme is to be efficient/save space with rsync. When the file is out of date but exists in the --l-d target, it would be great if it could be removed and linked. If an option was supplied to request this behaviour, I'd actually throw some money at making it happen. (And a further option to retain a copy if inode permissions/ownership would otherwise be changed.) Reasoning: I backup many servers with --link-dest that have filesystems of 10+M files on them. I do not delete old backups - which take 60min per tree or more just so rsync can recreate them all in an empty target dir when 1% of files change per day (takes 3-5 hrs per backup!). Instead, I cycle them in with mv $olddate $today then rsync --del --link-dest over them - takes 30-60 min depending. (Yes, some malleability of permissions risk there, mostly interested in contents tho). Problem is, if a file exists AT ALL, even out of date, a new copy is put overtop of it per the above man page decree. Thus much more disk space is used. Running this scheme with moving old backups to be written overtop of accumulates many copies of the exact same file over time. Running pax -rpl over the copies before rsyncing to them works (and saves much space!), but takes a very long time as it traverses and compares 2 large backup trees thrashing the same device (in the order of 3-5x the rsync's time, 3-5 hrs for pax - hardlink(1) is far worse, I suspect a some non-linear algorithm therein - it ran 3-5x slower than pax again). I have detailed an example of this scenario at http://unix.stackexchange.com/questions/193308/rsyncs-link-dest-option-does-not-link-identical-files-if-an-old-file-exists which also indicates --delete-before and --whole-file do not help at all. /kc -- Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
