[Bug 8201] rsync 3.0.8 destroys SELinux security context of symbolic links
https://bugzilla.samba.org/show_bug.cgi?id=8201 --- Comment #4 from Martin Wilck martin.wi...@ts.fujitsu.com 2011-06-06 09:49:00 UTC --- (In reply to comment #3) Wayne, your change regressed bug 7109. Linux needs NO_SYMLINK_XATTRS only for the user namespace. IMHO NO_SYMLINK_XATTRS doesn't have the right semantics. Under Linux, trying to read or set a user attribute on a symlink will raise EPERM. The Right Thing to do for rsync would be to ignore this specific type of error. That's not what NO_SYMLINK_XATTRS does, AFAICS. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync and many files
Hello together, I have a question about using rsync with many files. We are using rsync via rsnapshot, but this is not elementary. It is used to backup many (above 100 servers) and works very well. Now there is one server with many (several millions) files. The files are not very big, so the complete backup is about 500 GB. Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. My rsync-command is: /usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative --delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh --link-dest=/dest.path/daily.1/ root@192.x.x.x:/path.to.backup/ Do you have an idea to reduce the backup time? Btw: The bwlimit should not be the problem, because generating the filelist is the most time. Thank you very much! Cliff Simon -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
The --max-delete option and also question about depth first deletions
Hello, I'm trying to build a bullet-proof rsync deployment, using rsync version 3.0.8 protocol version 30, and I'll be using the --delete option. As a way of limiting any catastrophic losses I'm thinking about using --max-delete. I have a couple of questions: - If the maximum number of deletions occurs, will rsync keep chugging away, copying over files to my destination, but just not delete any further? In that way, perfectly legitimate rync commands will work to completion, only they will just stop deleting excess files. - Does the deletion occur in a depth-first fashion? So if we have top level folders top/a/ top/b/ top/c/ and we have a runaway deletion, begining at top/, will it first delete everything in top/a, then top/b. etc. until it hits the max-delete number? I have other measures in place to make sure none of this happens. I just wanted to understand the worst-case scenario. -- Speech, not just for humans http://www.google.com/profiles/egilchri about.me/ted.gilchrist -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync each given file to a given destination ?
Hello everybody, this question has already been asked (but not answered) by Matt McCutchen (at the end of his last answer) http://lists.samba.org/archive/rsync/2006-January/014426.html. Is there a way, given a list of unrelated (source, destination) pairs, to copy each source to the corresponding destination in a single run of rsync ? Sorry if this question has already been answered, i couldn't find it :) Regards, Jean-Baptiste -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 8188] Mechanism for taking an rsync server down for maintenance
https://bugzilla.samba.org/show_bug.cgi?id=8188 --- Comment #8 from Brian K. White br...@aljex.com 2011-06-06 15:36:52 UTC --- (In reply to comment #6) I've committed a change to 3.1.0dev git that allows a pre-xfer exec script to send an error message to go with its non-zero exit status. This will allow someone to tweak their rsyncd.conf file to add a global option like the following (assuming that they don't override pre-xfer exec elsewhere): pre-xfer exec = /usr/local/bin/output-rsyncd-downtime-message That script may output a message to stdout and then exit with a non-zero status. You could also choose to leave the pre-xfer exec configured, and make the script do your file-check: #!/bin/sh if [ -f /rsyncd.downtime ]; then cat /rsyncd.downtime exit 1 fi Sometimes, you ask for an inch, they give you a mile. ;) -- bkw -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
On Mon 06 Jun 2011, Cliff Simon wrote: Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. Are you using a recent version of rsync? One that does incremental recursion? Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
AW: rsync and many files
Hi Paul, Thank you for your reply! Hm...I´m using 3.0.3 at the Dest-Server, but now I saw that the Source-Server has 2.6.9 Do I have to enable incremental recursion and from which version is incremental supported? Cliff Simon -Ursprüngliche Nachricht- Von: Paul Slootman [mailto:paul+rs...@wurtel.net] Gesendet: Montag, 6. Juni 2011 18:00 An: rsync@lists.samba.org Betreff: Re: rsync and many files On Mon 06 Jun 2011, Cliff Simon wrote: Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. Are you using a recent version of rsync? One that does incremental recursion? Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
On Mon 06 Jun 2011, Cliff Simon wrote: Hm...I´m using 3.0.3 at the Dest-Server, but now I saw that the Source-Server has 2.6.9 Do I have to enable incremental recursion and from which version is incremental supported? Both ends have to be at least 3.0.0 to enable the incremental recursion. It should kick in automatically, unless you enable an option that prevents it; search the man page for 'incremental' and the second paragraph you find explains all. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
On 6/6/2011 12:54 PM, Paul Slootman wrote: On Mon 06 Jun 2011, Cliff Simon wrote: Hm...I´m using 3.0.3 at the Dest-Server, but now I saw that the Source-Server has 2.6.9 Do I have to enable incremental recursion and from which version is incremental supported? Both ends have to be at least 3.0.0 to enable the incremental recursion. It should kick in automatically, unless you enable an option that prevents it; search the man page for 'incremental' and the second paragraph you find explains all. Paul That will make it begin to transfer files sooner, and require less ram while working, but how will it make it take less total time? -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 3444] Deal with case-insensitive file-systems better (perhaps by adding an option)
https://bugzilla.samba.org/show_bug.cgi?id=3444 Jamie Zawinski j...@jwz.org changed: What|Removed |Added CC||j...@jwz.org --- Comment #10 from Jamie Zawinski j...@jwz.org 2011-06-06 20:42:30 UTC --- I hate to just post a me too, but, me too. This bug has been idle for four years. I use rsync between MacOS HFS+ systems all the time, and having files be deleted and re-transferred just because the capitalization on an intervening directory name has changed is really annoying and wasteful. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
In f992406d6e81b54dbb33210217fe7afd07f80...@exchange1.mtb.netclusive.de, on 06/06/11 at 12:04 PM, Cliff Simon cliff.si...@netclusive.com said: Hi, We are using rsync via rsnapshot, but this is not elementary. It is used to backup many (above 100 servers) and works very well. Now there is one server with many (several millions) files. The files are not very big, so the complete backup is about 500 GB. Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. My rsync-command is: /usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative --delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh --link-dest=/dest.path/daily.1/ root@192.x.x.x:/path.to.backup/ Do you have an idea to reduce the backup time? A bit of math says 2*10^6 / 14 hours is about 40 files/second. How fast do you think rsync should be and how does this compare to backups on your other servers? Are you sure the it is not the hardware that is limiting the rsync's performance? Based on my knowledge of the rsync sources, I believe the file list generation algorithms are pretty efficient. There is quite a bit of code in the code path, but it's hard to avoid this given the number of options available to control the sync process. Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync is coring
We are running the following version of rsync: -rwxr-xr-x1 root system 235216 Sep 27 2004 rsync I ran dbx which gave me this: [using memory image in core] reading symbolic information ...warning: no source compiled with -g Segmentation fault in . at 0x100236e0 0x100236e0 (???) 8c040001lbzu r0,0x1(r4) (dbx) It looks like since the program was not compiled with the -g option it is not providing adequate debug info. Would appreciate any thoughts. Vince m...@mattmccutchen.net 06/02/2011 10:18 AM To vincent.soo...@daimler.com cc rsync@lists.samba.org Subject Re: rsync is coring On Thu, 2011-06-02 at 09:27 -0700, vincent.soo...@daimler.com wrote: We are running an rsync process every hour and it is producing a core file. We thought initially there was a corrupt file but that is gone and the core file is still being produced. Would appreciate any help in analysing the core file. Two things you can do: - Open the core file in a debugger and get a stack trace. - Try to reproduce the problem on a smaller set of files or with simpler options, etc., until you find a minimal case that fails. This may shed light on what is causing the problem. If the same minimal case fails on another machine, that is probably grounds to file a bug. -- Matt If you are not the intended addressee, please inform us immediately that you have received this e-mail in error, and delete it. We thank you for your cooperation. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
40 files a second seems very slow. Are you sure the majority of the time is generating the file list and determine what's changed? How many of the millions of files are changed? On modern hardware I see 1000's of files per second when scanning for changed files. On Jun 6, 2011, at 12:39 PM, Steven Levine stev...@earthlink.net wrote: In f992406d6e81b54dbb33210217fe7afd07f80...@exchange1.mtb.netclusive.de, on 06/06/11 at 12:04 PM, Cliff Simon cliff.si...@netclusive.com said: Hi, We are using rsync via rsnapshot, but this is not elementary. It is used to backup many (above 100 servers) and works very well. Now there is one server with many (several millions) files. The files are not very big, so the complete backup is about 500 GB. Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. My rsync-command is: /usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative --delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh --link-dest=/dest.path/daily.1/ root@192.x.x.x:/path.to.backup/ Do you have an idea to reduce the backup time? A bit of math says 2*10^6 / 14 hours is about 40 files/second. How fast do you think rsync should be and how does this compare to backups on your other servers? Are you sure the it is not the hardware that is limiting the rsync's performance? Based on my knowledge of the rsync sources, I believe the file list generation algorithms are pretty efficient. There is quite a bit of code in the code path, but it's hard to avoid this given the number of options available to control the sync process. Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 A lot of this has to do with the filesystems and operating systems involved. Since you didn't specify I will guess Linux with ext3. If that is the case run don't walk to ext4. Also, mount the filesystems with the noatime and nodiratime options. This will prevent every stat() call from also writing to the filesystem which can be a huge performance benefit. On 06/06/11 22:48, Greg Siekas wrote: 40 files a second seems very slow. Are you sure the majority of the time is generating the file list and determine what's changed? How many of the millions of files are changed? On modern hardware I see 1000's of files per second when scanning for changed files. On Jun 6, 2011, at 12:39 PM, Steven Levine stev...@earthlink.net wrote: In f992406d6e81b54dbb33210217fe7afd07f80...@exchange1.mtb.netclusive.de, on 06/06/11 at 12:04 PM, Cliff Simon cliff.si...@netclusive.com said: Hi, We are using rsync via rsnapshot, but this is not elementary. It is used to backup many (above 100 servers) and works very well. Now there is one server with many (several millions) files. The files are not very big, so the complete backup is about 500 GB. Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. My rsync-command is: /usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative --delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh --link-dest=/dest.path/daily.1/ root@192.x.x.x:/path.to.backup/ Do you have an idea to reduce the backup time? A bit of math says 2*10^6 / 14 hours is about 40 files/second. How fast do you think rsync should be and how does this compare to backups on your other servers? Are you sure the it is not the hardware that is limiting the rsync's performance? Based on my knowledge of the rsync sources, I believe the file list generation algorithms are pretty efficient. There is quite a bit of code in the code path, but it's hard to avoid this given the number of options available to control the sync process. Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3tkkIACgkQVKC1jlbQAQdFZwCgxycYzAP98QFZX/2GMUllYcug skwAoKO5VIOhq/ttIYAua7lHHD24LXIM =2fSh -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
