RE: Dealing with an unreliable remote
(until a better answer comes along) The killed rsync process should leave a temporary file .file-name.random If you rename the temporary to the real file name, rsync should continue from about where it left off. -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of net.rs...@io7m.com Sent: Tuesday, November 25, 2014 9:03 AM To: rsync@lists.samba.org Subject: Dealing with an unreliable remote 'Lo. I've run into a frustrating issue when trying to synchronize a directory hierarchy over a reliable (but slow) connection to an unreliable remote. Basically, I have the following: http://mvn-repository.io7m.com/com/io7m/ This is a set of nested directories containing binaries and sources for projects I develop/maintain. Every time a new release is made, I deploy the binaries and sources to an exact copy of the above hierarchy on my local machine, and then rsync that (over SSH) to mvn-repository.io7m.com. $ rsync -avz --delete --progress local/ io7m.com:/home/io7m/mvn-repository.io7m.com/ The problem: The latest project produces one .jar file that's about 80mb. Evidently, the hosting provider I use for io7m.com is using some sort of process tracking system that kills processes that have been running for too long (I think it just measures CPU time). The result of this is that I get about 50% of the way through copying that (comparatively) large file, and then the remote rsync process is suddenly killed because it has been running for too long. This would be fine, except that it seems that rsync is utterly refusing all efforts to continue copying that file from wherever it left off. It always restarts copying of the file from nothing and tries to copy the full 80mb, resulting it being killed halfway through and causing much grinding of teeth. The documentation for --partial states that Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.. Well, for whatever reason, it doesn't (or it at least fails to continue using it). I've tried --partial-dir, passing it an absolute path to the temporary directory in my home directory. It created a file in there the first time, but after being killed by the remote side and restarting, it ignored that file and instead created a new temporary file (with a random suffix) in the destination directory! Am I doing something wrong? $ rsync -avz --delete --progress --partial-dir=/home/io7m/tmp/rsync io7m.com:/home/io7m/mvn-repository.io7m.com/ I'm at a loss. How can I reliably get this directory hierarchy up onto the server? I don't care if I have to retry the command multiple times until the copy has fully succeeded, but I obviously can't do that if rsync keeps restarting the failed file from scratch every time. M -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: speedup is always 0.99
Check the TIMESTAMPS on both source and target. (you probably want -av or such (instead of -vr) to include ...) -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Grant Sent: Friday, February 22, 2013 3:24 PM To: rsync@lists.samba.org Subject: speedup is always 0.99 I'm syncing from a USB disk to my hard disk like this: rsync -vr /path/to/usb/disk/dir/ /path/to/hard/disk/dir/ But the speedup is always 0.99 which I think means it is just copying the files each time instead of syncing them. What could be wrong? - Grant -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links?
This may help: (man ln) A hard link to a file is indistinguishable from the original directory entry; any changes to a file are effectively independent of the name used to reference the file. Hard links may not normally refer to directories and may not span file systems. Assuming you do many backups and many of the files do not change, hard links are your friend. Backing up soft links: Do you back up the link or what the link points to? (Even that simple thing has interesting ways to get complicated.) -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Joe Sent: Tuesday, January 22, 2013 1:32 AM To: rsync@lists.samba.org Subject: Is there a howto/tutorial on backups/rsync that covers the use of hard and soft links? There have been a lot of posts on the list lately about issues with hard links. It has been very interesting, but I don't understand it very thoroughly. I haven't used hard links for anything yet. I've used symlinks - not for backups, of course - and have seen them get broken or deleted in backups. Is there a tutorial anywhere that will explain how this works (assuming that the reader understands the basic concepts of backups and knows how to program, but doesn't really understand how to use links to create things like incremental or differential backups)? It seems like there are a lot of fine points to consider, some of which can really bite you if you don't take them into account. I'm working on my own personal backup system using bash and rsync. When it's done it will be pretty good, but it would be *much* better if I rewrote it to have more backup versions using hard links to save space. I am writing my own because (aside from learning a lot) I have only seen two types of backup utilities - those that are very simplistic and won't let me do what I want and those which are enterprise level and I can't figure out how to get them to do anything without extensive study. (I did experiment with areca (I think it uses rsync libraries under the hood) which would probably do everything I want, but I got stuck too many times and couldn't get enough support on their forum to keep going.) TIA Joe -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync in daemon mode
From man rsyncd.conf max connections This parameter allows you to specify the maximum number of simultaneous connections you will allow. Any clients connecting when the maximum has been reached will receive a message telling them to try later. From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Dariusz Dolecki Sent: Wednesday, September 05, 2012 10:25 AM To: rsync@lists.samba.org Subject: rsync in daemon mode Hi, We use one server from which many other clients download files. This operates in daemon mode over ssh. Is it possible that there is a max number of connections the rsyncd can have on the server? Where is this value set? This is what I see on the client side: rsync: read error: Connection reset by peer (131) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [receiver=3.0.6] rsync: connection unexpectedly closed (88 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6] This was while rsync was operating on a sparse file, I have since added the --sparse flag to it. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsyncing huge file, timeout on rsyncd
--timeout=SECONDS set I/O timeout in seconds I think this timeout must be set big enough so that data actually gets transferred during the window. Having the target verify that nothing has changed yet seems to not qualify as resetting the timeout. Figure on enough time so that the target can read the entire target file. (Most likely there is a wee bit of change at the very end of the files) From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Libor Klepác Sent: Thursday, August 02, 2012 5:47 AM To: rsync@lists.samba.org Subject: Re: Rsyncing huge file, timeout on rsyncd Just to clarify When using rsyncd script, there is also no progress from beginning, it just timeouts Libor Dne Thursday 02 August 2012 12:26:13, Libor Klepáč napsal(a): Hello, let me describe my setup. Source server is debian wheezy, rsync 3.0.9 Destination is qnap TS-410, with rsyncd enabled, rsync 3.0.7 I'm trying to rsync two files, which are exported from source server as iscsi targets (windows iscsi backups). Files are on btrfs, so I use snapshots during rsync, so files don't change. One file is 50GB, second is 550GB in size. Initial sync over rsyncd daemon went fine. Now, i'm trying to sync data, after change. When i try to sync over rsyncd, it ends after 20 minutes with: rsync: connection unexpectedly closed (2438836 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] When i try to rsync over ssh, it takes ages to show progress status (around 1 hour for smaller file, 4-5 hours for big file). Whats going out there? Source server is idle during this no progress period, destination Qnap has one rsync process eating cpu. My commands are (example with one file) for rsyncd rsync --inplace --progress --numeric-ids --times --password-file .passwd iscsi/iscsiDisk1 rsync://backup-rsync@192.168.5.250:/iSCSI-backup/iscsiDisk1 for ssh rsync --inplace --progress --numeric-ids --times iscsi/iscsiDisk1 admin@192.168.5.250:/share/MD0_DATA/iSCSI-backup/iscsiDisk1mailto:admin@192.168.5.250:/share/MD0_DATA/iSCSI-backup/iscsiDisk1 With regards Libor Klepac -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: disable interpretation of trailing slash
If the directories are large enough, /path/* becomes too long. My own opinion is that that syntax is a very small price to pay for the flexibility and power. -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync- boun...@lists.samba.org] On Behalf Of Kevin Korb Sent: Saturday, January 07, 2012 8:48 PM To: rsync@lists.samba.org Subject: Re: disable interpretation of trailing slash -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I don't speak for anyone else but I would be opposed to this. Primarily because there isn't an alternate syntax that performs the exact same function /path/* doesn't work as expected if using --delete (and for good reason). On 01/07/12 21:40, John J Foerch wrote: Hello, This is a feature request for a command-line option to disable special interpretation of trailing slashes on source directories. I have been using rsync for a couple of years now (what an awesome program!) and the meaning of the trailing slash on a source is always the one aspect of its syntax that slows me down and makes me check and re-check the manual to ensure that I have remembered its behavior correctly. The likelihood of making a mistake with a trailing slash is compounded by the fact that shell completion (e.g. in bash) puts it there by default, and it was this very behavior that caused me to be a little too carefree last week and mess up a large backup. While the trailing slash is a powerful and useful feature, it also strikes me as risky to have so much power built into a single character of syntax, especially a character which usually has no special meaning in other programs. Others I have chatted with on this topic shared the same view. My feature request then, is for a command line switch to disable the special interpretation of the trailing slash on sources. A short option would be preferred. My reasoning is that it is easier to remember to always use a particular option, except in situations requiring more nuance, than it is to remember the special rule and to always check the syntax of the sources. Thank you for your time. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*- ,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*- ,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8JA90ACgkQVKC1jlbQAQcecwCcCCIUg19hL/gxyDKJh2OTBBO4 N5QAnA95PRm1pDwn2MPeCmy7o2zldc+I =osqT -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: restoring system
Until you get responses from people who actually know what they are talking about. ?1 what is the file system on the usb drive. Fat32 cannot preserve owner/group or their permissions. ?2 preserving hard links end-to-end a3 rsync will use whatever targets are currently existing and their underlying file systems. -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Robert Holtzman Sent: Sunday, April 24, 2011 12:51 PM To: rsync@lists.samba.org Subject: restoring system I have restored my home directory in the past using rsync but never the entire system. If I rsync / (on my usb drive containing the backup) to / on the computer with a fresh install of the OS, what problems will I run into? What about links? I am presently triple booting 3 linux distros and want to delete one to install something else. The distro I'm installing will be on 3 partitions (/, /boot, and /home). Will what I'm proposing populate the directories in these partitions or will it dump everything into /? I realize that I probably be better off creating a snapshot and restoring from that but I have never had good luck with MondoRescue or Clonezilla. Thought I'd better ask now before I stuck my foot in it. -- Bob Holtzman Key ID: 8D549279 If you think you're getting free lunch, check the price of the beer -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: files not moved immediately to final destination from temp location after rsync returns with success
OK, I'll bite. With all file system designs, there is a tradeoff between speed and safety. This tradeoff occurs at all levels where there might be something that buffers information. Writing into the middle of a structure can be incredibly slow if everything is done safely. Enter disk caches (Operating System, Disk controller, Disk itself) Much much faster, BUT if mother nature pulls the plug, very weird things can happen. Expect everything to be very non-informative about the strategy used. With an SD card I would expect everything in the chain to NOT wait for the SD card to be finished. Might even take a few minutes for the disk cache to be flushed to the SD card. Makes no difference unless you pull the SD card prematurely (or possibly in some cases actually look at what is actually on the card) -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Henri Shustak Sent: Wednesday, April 20, 2011 6:17 PM To: rsync Subject: Re: files not moved immediately to final destination from temp location after rsync returns with success I am using rsync version 3.0.7 on an arm linux based embedded device. The device pulls data periodically from a rsync server and stores the files on an SD card. The partial, temp and final rsync destinations all reside on the SD card. I came across an issue where it seems that the rsync call returned with a success but the files that it pulled are not moved immediately to its final destination. You could try issuing the 'sync' command? However, I do not think believe that this should be required. Perhaps someone else on this list will have some other ideas? This email is protected by LBackup http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: weird result when using --link-dest
Thanks for the detail. Try du -s /tmp/full_20110329_122743 /tmp/20110329_122743 du -s /tmp/20110329_122743 /tmp/full_20110329_122743 du -s /tmp/full_20110329_122743 du -s /tmp/20110329_122743 Technically, the directories have entries which point to the inodes which are in fact the files. When you delete one of these, either of these, you will not save much space. -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Janne H Sent: Tuesday, March 29, 2011 7:01 AM To: rsync@lists.samba.org Subject: weird result when using --link-dest Hello. I create one full backup, then another backup based on that, using --link-dest to reduce the size of the following backups, but the result is very suprising. the full backup seems to be reduzed in size!! I've reduced the commands to the bare minimum that still procduces the str. ange result. mkdir -p /tmp/full_20110329_122743 mkdir -p /tmp/20110329_125251 rsync --archive -e ssh server:/var/webdir/ /tmp/full_20110329_122743 du -s /tmp/*20110329* 18272 /tmp/full_20110329_122743 12 /tmp/20110329_125251 rsync --archive -e ssh --link-dest /tmp/full_20110329_122743 server:/var/webdir /tmp/20110329_125251/ du -s /tmp/*20110329* 18272 /tmp/20110329_125251 532 /tmp/full_20110329_122743 I thought /tmp/full_20110329_122743 would be of size 18272 and not the other way? Can someone help me spot the error? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync errors
Robert Holtzman wrote: Running Ubuntu 10.04.1 and rsync 3.0.7-1. Used rsync successfully for over a year for backups. Recently there was a kernel upgrade to 2.6.32-25. Don't know if that has any bearing. When I tried to run a backup today rsync failed with the error message: sending incremental file list rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: mkdir /media/disk/10.04laptop failed: No such file or directory What does ls -l /media/disk on the target give? rsync must be able to create (ie write) in that directory. (2) rsync error: error in file IO (code 11) at main.c(595) [Receiver=3.0.7] rsync: connection unexpectedly closed (9 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7] /var/log/syslog just showed the above message. Searched on the message and got one hit that said it was because a FAT32 partition had a size limit. I doubt that I've come close to that. FAT32 has a 2G file size limit. Some other old file systems have the same limit (32-bit signed) Couldn't find anything in this list's archives that matched the error message. Did I miss something? The command that has been working until now is: sudo rsync -vaHz --exclude '/proc' --exclude '/sys' --exclude '*.iso' --exclude '/home/holtzm/Documents/*.iso' --exclude '/media' /. /media/disk/10.04laptop I'm at a loss as to how to proceed. any pointers appreciated. Also check SELinux. It tends to get in the way of anything it does not know about. -- Bob Holtzman Key ID: 8D549279 If you think you're getting free lunch, check the price of the beer -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: strange behavier after finished backup
balearenin...@gmx.net wrote: Hola, new to the list and maybe having problems with english I ask for a friendly and a not too difficult answer. Using Windows 7 Pro, D-link DNS-323 NAS-Storage rSync: actual Version (just downloaded) Here is the stange behavier: I use rsync.exe -a -v --progress --delete --ignore-errors --force /cygdrive/N/ /cygdrive/ //SPEICHER/Volume_1/Backup/10-er/D$/ Assuming the above was all on one line: Rsync.exe -a -v --progress --delete --ignore errors --force /cygdrive/N/##-- this is the N drive /cygdrive/ ##-- this is ALL the drives that cygwin can find //SPEICHER/... ##-- this is the TARGET. The above TWO parameters are sources. So I want to backup the drive N from the Windows-Computer to //SPEICHER/Volume_1/Backup/10-er/D$/ on the NAS-Drive That works fine until all files are copied from N to //SPEICHER/Volume_1/Backup/10-er/D$/ After finishing that rSync starts to coppy ALL Drives (c:,D:,E:,...) from the Windows system to the NAS-Drive //SPEICHER/Volume_1/Backup/10- er/D$/ which is, of course not, what I want :( This comes from the SECOND SOURCE PARAMETER /cygdrive/ Well knowing, that the mistakeis mostly sitting in front of the screen, I ask your experianced eyes, if you see my mistake or have a helpfull idea. Thanks and - sorry - for making mistakes with my english Horst --- Horst Peters C./Ladera 22-2 E-07750 Cala Galdana Menorca / Spanien Fon:+34-971- 15 45 62 Mobil: +34 971-15 45 62 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync --link-dest, --delete and hard-link count
Grarpamp wrote: Yes, hardlinks save data block duplication... yet on filesystems with millions of files / long pathnames, just the directory entries alone can take up gigs per image. Multiply that out by frequency and it can quickly add up. Huh? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Ignoring parts of stat(2)
Matthias Schniedermeyer wrote: On 05.01.2010 05:36, grarpamp wrote: That option can be easily missed it's: --size-only That seems like at least part of what would be useful :) Certainly it covers the most common case of modtimes. However, it doesn't seem to work :( cp -p /etc/passwd a cp -p /etc/passwd b ls -liT a b 5 -rw-r--r-- 1 user wheel 1706 Dec 25 03:08:02 2009 a 6 -rw-r--r-- 1 user wheel 1706 Dec 25 03:08:02 2009 b touch a ls -liT a b 5 -rw-r--r-- 1 user wheel 1706 Jan 5 05:24:17 2010 a 6 -rw-r--r-- 1 user wheel 1706 Dec 25 03:08:02 2009 b rsync -Haxi --delete --size-only ./a ./b .f..t.. a ls -liT a b 5 -rw-r--r-- 1 user wheel 1706 Jan 5 05:24:17 2010 a 6 -rw-r--r-- 1 user wheel 1706 Jan 5 05:24:17 2010 b rsync always updates the meta-data, even for files that it doesn't transfer. See: chmod 777 a rsync -Haxi --delete --size-only ./a ./b AFAICT from the man-page there is nothing you can do about that without other drawbacks like don't synchronizing the mtime at all. (See the replacement-list of options for -a and skip -t) You might get the desired effect with riciculously large values of --modify-window -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync exclude
Gary Montalbine wrote: I am trying to backup my /home directory. A friend helped me with this script: #!/bin/sh #backup friday #Spinning up backup drive and mounting it .. cd / mount /mnt/hd2 #Starting backup procedures rsync -avx --exclude=/home/gary/.thumbnails/ --exclude=/home/gary/tmp/ --delete --ignore-errors /home/gary /mnt/hd2/2010 umount /mnt/hd2 #END It backs up /home OK. However it does not exclude .thumbnails or tmp. I have read the manual and am confused on the use of --exclude. I am strictly a user. I am using rsync supplied by ML2010. Help would be greatly appreciated. Thanks, Gary from man page --exclude=PATTERN exclude files matching PATTERN Seems like you are excluding /home/gary/home/gary/.thumbnails/ /home/gary/home/gary/tmp/ you probably want something like --exclude=/.thumbnails --exclude=/.tmp to match where /home/gary/ leaves off. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Option to create ancestors of destination, like mkdir -p
henri wrote: I agree with everyone else that it would be incredibly useful for rsync to either adopt such behaviour by default (assuming always doing 'mkdir -p' isn't harmful in any way), or have a tunable to enable it. I also agree the suggested behavior makes a lot of sense and I think it is a good idea. However, some deployments may rely upon the current behavior. As an example, a setup may rely on rsync failing if a file system is not mounted. In such a situation, it is worth contemplating whether you would you want rsync to generate the mount point and any subdirectories, specified for this rsync. I personally think that such checks should be external from rsync. However, changing the current default behavior has the potential to create problems on setups which may rely on the current default behavior. I recommend the consideration of potential problems which could occur if the deployed setups are dependent on the current rsync behavior and that default behavior changes. Then based upon the severity of these potential problems move ahead or delay the change to a major revision change? #!/bin/sh #backup friday #Spinning up backup drive and mounting it .. cd / mount /mnt/hd2 #Starting backup procedures rsync -avx --exclude=/home/gary/.thumbnails/ --exclude=/home/gary/tmp/ --delete --ignore-errors /home/gary /mnt/hd2/2010 umount /mnt/hd2 #END The quoted script above is from another thread which was just posted to the rsync mailing list : http://www.mail- archive.com/rsync@lists.samba.org/msg25132.html This script above is a good example of the potential problems with making 'mkdir -p' the default. I agree that this feature is the way to move forward. However, changing the default behavior may cause potential issues for various scripts. I have no idea how many scripts are there like this, which could potentially start start backing up to the wrong file system if the drive is not successfully mounted. I suggest considering these potential issues and suggest that leaving such a change to the next major release is seriously considered. my 4ç - This email is protected by LBackup, an open source backup solution. Free as in freedom; LBackup is licensed under the GNU GPL Download LBackup now : http://www.lbackup.org - Allow me to chime in with my own not-so-expert experience. With large masses of data to keep updated, with bad internet connection at time between locations, with the masses of data sometimes being moved around, I experience the following: If I mistype something in the path to the destination, the rsync errors out and does so very quickly. Making new subdirectories to conform to my mistyping looks friendlier ... at least until I suffer the consequences. The existing default has saved my bacon several times. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: retransfer fail of large files with inplace and broken pipe
Tom wrote: to make things more clear 1.) first transfer is done either a initial setup or with a usb hdd to get sender and receiver in sync. 2.) transfer does not stop because rsync had a timeout, it stops because the dsl line is broken (which i could see at dyndns) 3) if dsl line is stable the transfer is successfull (which works furtunately most of the time) 4.) i am searching for a way to reduce the time to retransfer the file or in other words to resume the filetransfer after a broken pipe (e.g. if you download a 4.4 GB Centos Image it is comfortable to resume the transfer of a 99 % transfered file instead to download all from scratch) Tom But you already have 100% of the image, only it is an older version of the image. The only thing I've found that works (and this is ONLY on something UNIXy) is to monitor the temporary file on the target and if it is big enough, rename it to the intended target file before the target rsync destroys it. For real disasters, you can attempt to automate this process. First: Transfer or re-transfer. I think, particularly with bad connections, you need to treat those VERY differently. For the initial transfer, --partial should help. For retransfers, where stuff in the middle has changed, I would expect the necessary state information to exist ONLY in the two running processes, and that information is lost if the connection goes down. This includes the connection dying because both sides are going through the file and have nothing worthwhile to say to each other. As usual, flames invited if I've got any of this wrong. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: retransfer fail of large files with inplace and broken pipe
tom raschel wrote: Hi, i have to tranfer large files each 5-100 GB (mo-fri) over dsl line. unfortunately dsl lines are often not very stable and i got a broken pipe error. (dsl lines are getting a new ip if they are broken or at least after a reconnect every 24 hours) i had a script which detect the rsync error and restart the transmission. this means that if a file has transfered e.g. 80 % i start again from beginning. using partial and partial-dir was no solution to resync because rsync cut the original file (e.g. from 20 GB to 15 GB) which means that i have to transfer the whole rest of 5 GB. so i had a look at --inplace which I thougt could do the trick, but inplace is updating the timestamp and if the script start a retransfer after a broken pipe it fails because the --inplace file is newer than the original file of the sender. using ignore-times could be a solution but slow down the whole process to much. is there a option to tell rsync not to change the time of a --inplace transfered file, or maybe preserve the mtime and do a comparison of mtime instead of ctime. Thx Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html First: Transfer or re-transfer. I think, particularly with bad connections, you need to treat those VERY differently. For the initial transfer, --partial should help. For retransfers, where stuff in the middle has changed, I would expect the necessary state information to exist ONLY in the two running processes, and that information is lost if the connection goes down. This includes the connection dying because both sides are going through the file and have nothing worthwhile to say to each other. As usual, flames invited if I've got any of this wrong. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Handling spaces in filenames
FILE NAME WITH SPACES -- this is 4 different space-separated parameters fed from the shell to the program FILE NAME WITH SPACES -- this is one parameter fed from the shell to the program FOLDER-NAME/ -- this is one parameter and means all the files in the directory FOLDER-NAME/ FOLDER-NAME -- this is one parameter and means the directory FOLDER-NAME/ (and presumably taking all the contents with the directory) Unless the program does something special to parse out the spaces in the parameters, what you are seeing is normal behavior. I assume that the directory names with spaces are actually subdirectories of whatever you are backing up, ie those name you specify on the command line. You can also use wild-cards in addition to the names with spaces to give the illusion that it understands. -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Larry Alkoff Sent: Tuesday, August 11, 2009 10:04 PM To: Rsync Subject: Handling spaces in filenames Man rsync says that If you need to transfer a filename that contains whitespace, you'll need to either escape the whitespace in a way that the remote shell will understand, or use wildcards in place of the spaces.. I am regularly doing backups with rsync and notice that files names with a space in them copy properly, without any special escape or other characters. Is this a change in rsync? Is the warning in the man page no longer needed? I will say that my 'filenames' with spaces are mostly (if not all) directory names so perhaps rsync knows to handle directory names better than file names. Can anyone enlighten me? Larry -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)
Carlos Carvalho wrote: Ryan Malayter (malay...@gmail.com) wrote on 14 July 2009 17:00: On Mon, Jul 13, 2009 at 4:54 PM, Jamie Lokierja...@shareable.org wrote: Remembering hashes doesn't make any difference to speed, if the bottleneck is the sending side. Except that in the rsync pipeline, the reading the destination file to get hashes happens BEFORE the sender reads its file. And the sender calculates hashes and finds matches on-the-fly. So, when transferring a large file, it goes something like this from the sender's perspective: 1) sending file list 2) receving file list 3) file is different! Recevier, please give me some hashes 4) wait 20+ minutes for receiver to compute hashes got hashes 5) begin transfer, calculating my hashes and compressing on the fly as I transfer 6) file complete By caching hashes on the receiving side, the transfer can begin almost instantaneously if the file on the receiver is unchanged since the last run of rsync. This is, in fact, almost always true for the way most people use rsync (backups, file distribution, etc.) Most of my rsync scripts stall for minutes doing no effective work, because they are waiting for the destination to read and calculate hashes of a large file that was already hashed yesterday. Hash calculation is very fast; rsync has a negligible cpu consumption. What limits it is reading the disk. If you run a hash check you'll see the process stalled in io and not cpu. Maybe your machine has a particularly different IO/cpu ratio? This, and the fact that the maintainer(s?) want to keep rsync stateless, makes me think that a change to remember hashes is unlikely. You also have the consequences of several of these being run on top of each other. Life is much better when that does not destroy everything. ( There are people (like me) who do things like that ;-) Just guessing, but the target probably has to read a lot of disk before it finds something different, and the sending process stalls until it gets something that it can send. Seems like the -P may be informative. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files)
Silly question, but are you doing something like compacting, removing air, or optimizing the database in any form? If the blobs keep moving around, that makes finding common stuff much much harder. It will still have to read both sides even if almost everything is the same -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Krzysztof Nosek Sent: Friday, July 10, 2009 7:48 AM To: malay...@gmail.com; rsync@lists.samba.org Subject: Re: How to make big MySQL database more diffable/rsyncable? (aka rsyncing big files) Hello, So I do not think the basic data structure is the problem, unless mysql hotcopy does something really strange like inserting a timestamp or other changing data info every few KB in the output stream. No, really, mysqlhotcopy performs just a raw file system copy of /var/lib/mysql taken from the locked database. If nothing particular happens meanwhile in the running database, the copy is 1:1 with the original. Easy to check with any smaller database. I would suggest trying a tool like xdelta (on the same machine) against two consecutive backup files, just to see if it can extract similarities. If xdelta can find significant matched data, rsync should be able to as well. I'd love to do that but I can't make it actually working: xdelta: open ../mantis_game_20090707/mantis_bug_file_table.MYD failed: Value too large for defined data type Same for dumps. I think it's running out of memory just like diff does with files that large, isn't it? Also, is the transfer CPU bound or network bound? Can you send the output of rsync with the --stats and -v options? I'm pretty sure it's the network. The rsync jobs on both machines use no more than 30-50% of the CPU. I may be wrong - please find the log attached. Perhaps I am memory bound, could it be? Regards, nosek -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync Windows to opensolaris or linux
I think you need to look at the permissions you will need on Windows if (ie when) you need to recover. You can do a little better than distinguish read-only from writable, but not much and not easily. The simpler you can stand it, the happier you will be. With -a parameter, rsync will complain going to anything that cannot handle users and groups (including SMB shares) Keep in mind that the easiest (only?) way to ensure consistency is to destroy all conflicting/inconsistent information. -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Harry Putnam Sent: Wednesday, July 08, 2009 11:59 AM To: rsync@lists.samba.org Subject: rsync Windows to opensolaris or linux I know this subject has been covered a fair bit here but I'm getting confused with the vast array of possible options I've seen in various cmdlines posted here, and the vast (and sometimes confusing) rsync manual. Can someone show me the simplest case to allow rsync to push data from a windows XP or vista machine onto an opensolaris host using rsync on cygwin? What options are commonly required? is a simple command line like this: (NOTE: The --inplace part has to do with the zfs filesytem on opensolaris and its snapshot function) rsync -avv --inplace /cygdrive/c/ u...@opensol:/ZHOSTS/someHost/ I'm guessin the -a might get me introuble trying to make windows ntfs match a unix permissions setup. So what is better? My concern here is when it comes time to restore something I don't hit some unexpected snag involving permissions on windows. Also I seem to recall something on windows having to do with ntfs that can fool rsync into thinking the data is all new and different than the data on the remote .. so invoking a massive transfer where a differences only transfer should have occurred. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to get first level folder names with rsync commands?
Something like ls -d NetBackup/* | wc should be informative. (assuming something unixy, of course -- cygwin stuff might work) -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Daniel.Li Sent: Wednesday, July 01, 2009 12:17 AM To: Rsync Subject: How to get first level folder names with rsync commands? Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: what kind attrs are going to be transfered by rsync?
Each different OS has a different idea of what attributes exist for whatever they call files. The exact combination of attributes will not map exactly from any OS to any other OS, although it is possible to preserve some crude distinctions in many (most?) cases. You look at the error messages to see what did not go through. You read the man pages to get some idea of what to tell it to attempt. Critical information is what attributes exist in what files system on which system. That information is really outside the scope of documentation about rsync. You might have some luck with some kind of universal file system, but I'd bet on snake oil. If you can limit yourself to what is in common, you can make it look like it works. -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Daniel.Li Sent: Thursday, June 04, 2009 2:53 AM To: Rsync Subject: Re: what kind attrs are going to be transfered by rsync? It seems there is a discussion long time ago. But little info. http://fixunix.com/tools/540757-do-not-reply-bug-3245-improve-rsyncs-error-m essages-make-themclearer.html I noticed that it's status is assigned, Has it been improved? https://bugzilla.samba.org/show_bug.cgi?id=3245 On Thu, 2009-06-04 at 14:44 +0800, Daniel.Li wrote: Dear list, I have met this error. I wanna know what attrs are transfer by rsync program? sent 5045 bytes received 262853 bytes 107159.20 bytes/sec total size is 130560 speedup is 0.49 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1535) [generator=3.0.5] Any help is appreciated. Thanks. -- Daniel Li -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Difference between rsync -r S1 S2 T and cp -r S1 S2 T
Dunno for sure, but I'd expect differences if S1 is /dev Simple case, a copy is just a copy. Edge cases, there's lot of room for differences in exactly what happens when things are not exactly perfect. Many times I use rsync because I know what it does and I'm not exactly sure what cp will do. These differences increase in importance with distance between the source and destination. -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Nobuko Three Sent: Tuesday, June 02, 2009 2:29 AM To: rsync@lists.samba.org Subject: Difference between rsync -r S1 S2 T and cp -r S1 S2 T The manual says rsync can be used as an improved cp command. If S1 and S2 are (local) source files or directories and T is an (local) existing directory, is it safe to think that $ rsync -r S1 S2 T does the same thing as $ cp -r S1 S2 T , assuming no trailing slashes in S1, S2? -- View this message in context: http://www.nabble.com/Difference-between-rsync--r-S1-S2-T-and- cp--r-S1-S2-T-tp23827717p23827717.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync to SAMBA mount or NTFS tutorial?
There are three areas of fundamental incompatibility (that I am aware of) Basic File Permissions: Unix has rwx for owner/group/world, Windows has Hidden/System/Read-Only/Archive. NTFS has ACLs (The above is an oversimplification, and things can be done so it works well enough for what is necessary, but ...) Timestamps: Unix has one-second resolution and base is UTC regardless of the current timezone of whoever is asking Windows has the computer in the local timezone --- lots of fun when crossing time zones or even daylight savings time. (Think of two users, each in different time zones... ;-) Windows creates/updates the directory entry and then eventually (most of the time) fills in the actual file contents. This applies also to windows accessing SAMBA shares. (That's why I backup my SAMBA servers at 3:33 AM) Good luck. _ From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Dale E. Moore Sent: Thursday, May 14, 2009 10:21 AM To: rsync@lists.samba.org Subject: rsync to SAMBA mount or NTFS tutorial? Will someone please tell me where to go, and how to get there to get rsync running to mounted windows shares or mounted NTFS volumes? I'm doing OK with rsync-ing to EXT3 mounted volumes but the others still perplex me. I look forward to hearing from you, Dale -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Local disk rsync
listserv.traf...@sloop.net wrote: I've done quite a bit of looking, but I haven't found an answer that answers this question. Environment: cygwin on Windows rsync 3.0.4 I know that rsync isn't optimzed for speed on local copies - that's clear in my testing. I'm attemting to sync a large volume of files. (In this case, I'm syncing a rdiff-backup set...) An initial sync will be about one fourth as fast a a cp of the same files. Ok, that's not a huge deal. Subsequent syncs should be a lot faster I though. No. Using a native windows sync tool, a sync of the source to the destination takes perhaps 5 minutes. (~24G, 80K files) However using rsync takes a very long time. (I've never let it finish, but it's still running an hour later or more...) Here's how I run it: rsync --recursive --delete --verbose --stats --perms --super /cygdrive/e/somedir/ /cygdrive/f/somedir/ Thoughts? Any copying, local, from and to the same spindle looks like; read some stuff seek to the other place write some stuff seek to the other place Timing should be dominated by how much stuff in each place before it seeks to the new place. For Windows, ROBOCOPY may do what you want (Resource kit XP and below, native on Vista) You will probably want the switches /R:1 and /W:1 or similare. I'm mainly using rsync 'cause I know it, and use it. If there's some alternative I'd be glad for a pointer. I don't know of any other good sync tools like rsync. (Unison I guess...though I only need one way sync.) [Whole file copy is fine if needed, though some of the files are large 1 GB+ and a delta copy would speed these up...] Why do you think I use Linux ;=) Cygwin is an incredible program. It almost makes Windows useable. Kinda like a very bad unix. Opinions are my own and not necessarily representative of anyone sane. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Backing up from fat32 to ext3
Thomas Ebert wrote: Hi, I have a problem backing up my music collection, that is stored on a fat32 formated hard drive to my ext3 formated backup drive (ext3 to ext3 works like a charm :) ). The problem is, that rsync always transfers all of the files and not only the one that were changed/added. I know of the limitations fat32 has but I can't find the correct options to run the backup. Thanks Thomas -- Probably need something like (from man) --modify-window=NUM compare mod-times with reduced accuracy To store time-of-day in 16 bits, Windows can only handle even seconds. (or is it odd?) Maybe any choice is wrong ;) --modify-window=1 is probably adequate personally, I' use something like --modify-window=7 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Wrong uptodate
Matt McCutchen wrote: Sent: Sunday, March 02, 2008 3:14 PM To: [EMAIL PROTECTED] Cc: rsync@lists.samba.org Subject: Re: Wrong uptodate On Sun, 2008-03-02 at 17:43 +0100, [EMAIL PROTECTED] wrote: Rsync claims files to be uptudate, but they are not ... From the log: export/opt/bup/status/1 is uptodate export/opt/bup/status/2 is uptodate . . Source directory (locally on server): -rw-r--r-- 1 root root 28 2008-03-01 22:44 1 -rw-r--r-- 1 root root 28 2008-03-01 22:37 2 . . Destination directory (nfs share): -rw-r--r-- 1 root root 28 2008-02-29 22:45 1 -rw-r--r-- 1 root root 28 2008-02-29 22:42 2 . . Weird. Those files clearly differ in mtime, so rsync should be transferring them. Perhaps the source and destination being passed to rsync are not what they are supposed to be. Please put set -x and a pwd call at the beginning of the script, run the cron job, and post the current directory and actual rsync command line that result. Also, what are the absolute paths of the directories that you listed above? If this information does not reveal the problem, the next step would be to strace rsync (strace -f -o /tmp/rsync.strace rsync ...) to see what files it is accessing and what mtimes it is seeing. Matt Also worth checking, particularly when the script works when run directly and does not work when run from a cron job: On some systems, the defaults for parameters are empty. Specifically rsync does not run because it is not in the path that isn't there. I think it SUCCESSFULLY does not run. Not sure. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync-3.0.0pre10 and iconv
Thankee! Thankee! Thankee! I am definitely switching to 3.0 or the next pre. (currently anything like that that actually gets to the other side is gravy) (not all that bad since anything that actually matters is supposed to be in english not chinese ;-) This is on/to/through computers which are NOT native UTF-8 If the backups are acutally ever needed, it will be a long time before anybody actually cares if some Chinese words got mangled in the file names. However it does look much better if the names are NOT mangled. And sometime, sooner or later, it will actually matter. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, February 27, 2008 5:19 AM To: rsync@lists.samba.org Subject: RE: rsync-3.0.0pre10 and iconv I think that UTF8 is simply used as the transport encoding. The sending side will ensure that the filenames on the wire are UTF8, and the receiving side will convert that UTF8 into whatever is required. I checked with tcpdump what is being transmitted between the hosts and found that filenames are not in UTF8 during the transfer, unless sending side is using UTF8 natively. Here's partial packet export from utf-8 to iso8859-1 transfer: 0060 04 18 2f 65 78 61 6d 70 6c 65 2d 66 69 6c 65 2d ../example-file- 0070 c3 b6 c3 a4 c3 a5 2e 74 78 74 00 05 00 47 2f 99 ...txt...G/. Here's partial packet export from iso8859-1 to utf-8 transfer: 0060 04 15 2f 65 78 61 6d 70 6c 65 2d 66 69 6c 65 2d ../example-file- 0070 f6 e4 e5 2e 74 78 74 00 05 00 47 bd f1 c3 a4 81 txt...G. Transferred / converted filename is still example-file-öäå.txt. Regards, Sami -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync 2.6.9 does not skip any files based on modification time
Paul Slootman wrote: I think you mean file owner? Of file times? cannot set file names seems unlikely :-) File OWNER File GROUP and DOS/Windows complains Me, I let it complain --- much more complaining with cp than with rsync DOS and Windows have different ideas of what file attributes are worth preserving. NTFS has some elaborate ACLs, but far too much trouble to do what is trivial on *nix with users and groups. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync 2.6.9 does not skip any files based on modification time
Are you preserving the times? There are many combinations. Me, I always use -a and then add stuff. Copying or Rsyncing windows stuff always seems to have stuff (essential to Linux/Unix) missing from the windows side. But nothing in that parameter list looks like it would preserve the modification time (or any other time). -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Paul Slootman Sent: Friday, February 08, 2008 3:03 AM To: rsync@lists.samba.org Subject: Re: Rsync 2.6.9 does not skip any files based on modification time On Fri 08 Feb 2008, Stefan Malte Schumacher wrote: I am trying to rsync some ghost images from a windows client running Windows XP to my Linux server. The problem is that rsync sends the complete files again even if nothing changed on the client side. The only way to avoid this is to use the -c-option but this takes nearly as long as uploading the files would. The server is running rsync-2.6.9, /etc/rsyncd.conf looks as follows. /mnt/D/ is a FAT32 partition. [...] used to call rsync from cwrsync.cmd rsync -r --del --modify-window=3 /cygdrive/E/HDD-Abbilder/ [EMAIL PROTECTED]::HDD-Abbilder You're correctly using --modify-window, which is otherwise the probable solution to your problem. However, somehow rsync still considers an update necessary. You can use -i (--itemize-changes) to let rsync show what attributes are incorrect, triggering the update. Once you know that, you can try to fix it. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Windows Trouble with --link-dest set: file not found whenrsync tries to create hard link
Thanks. Paul Johnson wrote: soft link is totally different from hard link. that junction on directory is not the same meaning of hard link in Unix concept. Windows can (maybe) come close to doing a soft link, directories/drives at least somewhat (but try following the links with Explorer on Vista;) (At least if you don't push too hard) If Windows had any hard link ability, they would do many things differently. With Cygwin, you can do things that LOOK unixy. This is not the same as that they ARE unixy. On unix it is completely normal and reaasonable to delete a file that someone else is writing to. On unix it is reasonable to do the configure;make;make install bit and only then remember that you really need to restart the program or you are still running the old program. I don't mean that those are good ideas, but on anything unix you can; on windows you cannot. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsyncing files that might change
Flames/Cluestick invited if I've got this wrong. I would expect: rsync checks blocks on source to see if they are the same. blocks which seemed to be the same (past tense) are not sent. blocks which seemed to be different will be sent with whatever the current content of the block happens to be. there is no check at the end to see that nothing changed in the interim. There MIGHT be something about file changed during transfer --- but some things big long messy --- you do what you can with what you've got. Hi, I hope I have not been google-incompetant, but I have been unable to find an explicit answer a case I am concerned about. If am rsyncing a file and I have the the following sequence of events happen in the same second 1. rsync starts 2. rsync sends some chunk of data to the other end 3 a local process modifies the chunk that has just been sent I have seen statements that rsync does a whole file checksum once the transfer is complete - but I couldn't find an explicit statement of whether this checksum is computed after the transfer has completed. So, my question - is this case a race condition in which a subsequent run of rsync may miss that the file has been modified and hence skip it ? thanks -- Franc -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync and opened files
The short answer is It depends Egoitz Aurrekoetxea wrote: Hello, I'm trying to determine if rsync is a sure method of backing up servers (Linux and Windows) whose files are constantly being accesed and are not able to be stoped they're services for backing up purposes... I would use it over ssh for making incremental backups... in my tests seem to always have worked backing up from a debian server to the copy server that runs debian too... I'm using the next : OPTS=--force --ignore-errors --delete --backup --backup-dir=/home/ramattack/pruebas-rsync/$BACKUPDIRMES/$dia -avz rsync $OPTS $BDIR /home/ramattack/pruebas-rsync/$BACKUPDIRMES/imagen-copia BDIR is source I want to backup and /home/ramattack/pruebas-rsync... is the destination... could this copy correctly opened files? Normally I will use it for backing up linux machines normally... and the backup server will be of course a linux machine (debian machine). but how does it behave with linux machines? P.D. I have googled and searched over there but all posts I've find are old... and I wanted to have a recent answer. Thanks a lot mates!!! __ I can speak only of my own experience. The standard caveats apply. It depends and YMMV. Backing up live (and in-use) MySQL databases seems to work OK. The file being open is itself somewhat irrelevant. You can, with a bit of difficulty, do a backup of only closed files which is very unusable. Difficult, but not very. What matters is that the backup you copy is usable for your intended/needed purposes. Poor and buggy can be much much better than nothing. In my own case, it's really more a case of needing a local access to recent info from remote databases than backup per se. This means that if I do run into a problem, I just run another rsync and put things back consistent. Almost no problems, particularly where it matters. (even without CHECK TABLES). To further decrease the window of opportunity for mother nature to do its thing, I run back-to-back rsyncs to that the window when stuff is changing is quite small. Note: This style would NEVER be recommended practice. This depends in large part on the nature of MyISAM database tables and the fact that on standard MySQL, readers and writers DO INTERFERE WITH EACH OTHER. (never thought that was an advantage, did you? ;-) This also depends on Unix (or Linux) file locking semantics. Basically, Unix file locks work only with other things that happen to use the same locking machinery. Anything can totally ignore what other things call locked. In particular, LOCK TABLES is ignored by rsync. Windows has different semantics. Very different. In particular, best I can tell, the directory entry is written with timestamp and file size. ... ... ... sometime later, the contents of the file might actually come dribbling in. I back up Samba home directories at 1:11 AM when reasonable users Should be home asleep. I play some fungames with multiple backups at 2:22 and 3:33 (gotta love hard links --- as well as usable soft links) Assuming a short break in accessibility is tolerable, I'd 1) run rsync to the backup 2) stop the server 3) run rsync to the backup (should be much much faster now) 4) restart the server -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Data corruption check
Fabian Cenedese wrote: At 15:15 18.09.2007 -0400, Matt McCutchen wrote: On 9/18/07, Fabian Cenedese [EMAIL PROTECTED] wrote: I was wondering what happens if a file that is regularly synched but seldom changes gets corrupted in the copy. Are you referring to rsync writing corrupted data to the destination file or a problem with the destination filesystem or disk causing the file to read as data different from what was written? I was thinking of any problem, even a transport error. There's also the -I, --ignore-times switch. If I use this but without -c what method is used for checking then? Or does -I imply -c? -I rewrites the destination file no matter what, while -c computes its MD4 or MD5 checksum first and then rewrites it only if its checksum differs from that of the source file. Either option gives the same end result for the destination file. However, they may have different performance (-c uses more disk reading but potentially less disk writing and slightly less network traffic), and -I logs more transfers than -c and interferes with --link-dest. Thanks for the explanations. That means that -l and -c are not usable together as they contradict themselves, right? I was asking because I'm responsible for our backups. The current solution with rsync works nicely. While the RAID storage also monitor the HD's SMART state I was still wondering about a way to detect otherwise unknown data corruption. I guess if I first made a normal rsync and then a rsync --dry-run -c I could find file differences that shouldn't be (provided there wasn't any real change otherwise, like in the middle of the night). Of course that wouldn't tell me what side had changed, but still something worth considering doing once a month or so... Thanks for your help bye Fabi Seems like an old sailors rule: Have one chronograph or three, never two. With two, you know something is wrong, but no idea what to do. Disk is cheap. Thanks for the idea of the rsync --dry-run -c Methinks it will help a lot of Windows users. (even with only two comparands) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: File bit synchronization?
Stephen Zemlicka wrote: No idea why it's not. I usually use the -v -rlt -z --delete options. You could try the -c switch. I believe that will ignore the mod time and size and skip based on the checksum only. Though I would think you need a switch that does the opposite of that. Maybe someone else has an idea. Over sometimes very bad trans-pacific internet connections rsync -avPz --timeout= --exclude '.?*.*' --exclude temp* Many files where nothing changes goes very fast. Large files with a few scattered changes throughout the middle will go rather slowly. Usually instructive just to watch it and see what happens. Just a guess, but take a look at whatever the contents of pg_dump actually look like. In particular, compare one dump to another taken later. You might spot something obvious. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: Robert Fitzpatrick [mailto:[EMAIL PROTECTED] Sent: Monday, September 17, 2007 4:38 PM To: Stephen Zemlicka Cc: 'rsync' Subject: RE: File bit synchronization? On Mon, 2007-09-17 at 16:02 -0500, Stephen Zemlicka wrote: Rsync performs delta copies. This is by far the best feature of rsync IMHO. I have many backup solutions setup for a number of clients that perform offsite synchronization of multi GB sql, exchange, etc. databases in minutes over standard DSL connections. Great! Thanks. Why do you think my pgsql backup from pg_dump in postgresql 8.2.4 does not seem to be doing this? Do you mind if I ask what options you are using with rsync? -- Robert -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: File changed during save....
Matt McCutchen wrote: On 9/15/07, roland [EMAIL PROTECTED] wrote: what`s the rsync equivalent to this? how can i see which files changed while rsync was transferring them ? Handling of concurrent changes to source files is one of rsync's weaknesses. The rsync sender naively reads each source file from beginning to end and sends what it sees; it doesn't detect if source files change while being transferred. In many cases, the concurrent modification will leave the source file with an mtime different from the mtime rsync saw when it statted the file during file-list building (which gets set on the destination file), so a subsequent rsync run will fix the corrupted destination file. See this thread for more information: http://lists.samba.org/archive/rsync/2006-January/014534.html Matt -- Note that back-to-back rsyncs make the window of opportunity much much smaller for things to change during transit. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: File changed during save....
[EMAIL PROTECTED] wrote: Note that back-to-back rsyncs make the window of opportunity much much smaller for things to change during transit. yes, but it still leaves room for corrupted transfers nobody would probably know about !? With UNIX file semantics, Loop until nothing is transferred -- might be infinite. With WINDOWS file semantics, all bets are off. Seems that mtime and filesize are stored long before the new contents are actually written. What you can actually get away with depends on the exact nature of the contents of a file as it gets changed. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Looking inside device files
Andre Majorel wrote: On 2007-08-21 05:55 -0400, Matt McCutchen wrote: On 8/20/07, Andre Majorel [EMAIL PROTECTED] wrote: Is there a way to make rsync look inside device files ? The goal is to copy the contents of a block device to a regular file incrementally. Short of that, even just getting it to dump the block checksums of the content a block device would help (if I know which blocks are out of sync, I can fix them with dd). Are the block device and the target regular file on the same machine? They could be. Or not. Doesn't matter. Your remark about dd suggests that they are. Not necessarily. dd if=myfile skip=100 | ssh [EMAIL PROTECTED] dd of=myfile seek=100 conv=notrunc If so, why do you want an incremental copy? It does at least as much disk I/O as an ordinary copy (since it reads the entire basis file), and the reduction in data passed from the sender process to the receiver process is irrelevant. Sometimes, you need to copy a whole block device and you can't just cat it to wherever you want, go out to lunch and expect it to work. The destination could be unreliable. The network could be unreliable. The source could trigger DMA timeouts and lock up the bus, forcing you to reboot. Many things can and do go wrong. In those situations, you need to be able to verify that the copy is identical to the original and where's it's not, fix it (NOT start over, because if it failed the first time, it may fail the second time). rsync already has most of the mechanisms to do that easily and efficiently. It would be even better if it could complete the copy *first* and check the beginning *second* but that's probably more involved. Anyway, the attached patch for the current CVS rsync adds an option --copy-devices that makes rsync copy the contents of source device files as if they were regular files. I tested the option on my computer to copy a block device to a regular file locally, and it seemed to work. Wayne, you might like to include this patch in the patches/ dir of the source distribution. Thanks for the patch. I hope something like that goes in. I understand rsync was not meant as a block-device-level copy tool, but since it already has most of what it takes, it's tempting. Could be extremely useful (in some cases) but beware of rsync of /dev/zero and friends One very useful stunt is an rsync that takes a reasonable amount of time followed immediately by an rsync that takes a very short amount of time ??an offsite full-disk-backup of a moving target almost feasible?? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Option --numeric-ids not the default?
Darryl Dixon wrote: Hi All, I've browsed the history of the list, but can't seem to find an answer to something that I find quite surprising - why isn't --numeric-ids the default when rsync is told to preserve permissions? It seems to me that the current behaviour runs against the grain of more than a quarter century of unix tradition, and certainly caught me by surprise the first time I encountered it. Can anyone enlighten me? regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com If you have (mostly) the same set of users (names) on two different systems, what are the odds that the numeric ids are identical? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rewritten rsync man page
Wayne Davison wrote: On Sat, Aug 04, 2007 at 07:26:19PM -0400, Matt McCutchen wrote: What do you all think so far? Am I going in the right direction? I'll hopefully have some time to check into it more later on, but I did give your pages a look and they look pretty good. One thing I had been trying to do was to keep the option section from wrapping on an 80- column screen (which is why none of the lines were longer than 72 characters). What do you think? Is that constraint needed these days? (I've used 128 columns for quite a while...) I didn't get a chance to delve into the content and categorizations yet, but if anyone has taken a read and has a reaction, it would be nice to hear what folks think. ..wayne.. Reaction. I did a fast look-at initially --- I LIKE! Critical information -- and extremely hard to convey: Design level. What is in scope and what is outside. If I want to copy stuff from one place to another, where is the boundary between what works and what doesn't? Good metrics do not exist. Generally, you document what you can do. I suspect that rsync may be good and flexible enough that it is productive to document what is outside the scope of what rsync does. When things get complicated, WHY becomes very useful to know. Very short description of rsync When cp (or scp) CANNOT do what you want/need. A one-paragraph cheat sheet if possible would at least be interesting. Lines that fit on an 80-column display Fair-weather friend Foul-weather friend 128-column is almost always useable, BUT Staying within the 80 column or so limit advertises that you intend to stay usable under duress. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Might hanging bugs remain in rsync 3.0.0?
Matt McCutchen wrote: On 7/16/07, Wayne Davison [EMAIL PROTECTED] wrote: On Sun, Jul 15, 2007 at 11:09:57PM -0400, Matt McCutchen wrote: Furthermore, from April 27 to July 10, about 2.5 months passed without any hanging bugs being found; then, on July 11, Warren Oates reported one. That was a brand new one that I introduced into the code when I tweaked the index numbering in incremental mode and failed to update a second section of code that interpreted the index values. The hang bug cropped up right away in both my testing and in the testing of at least one other person, and was quickly fixed. OK, I didn't realize that. This makes me worry that [...] it may nevertheless have a hanging bug that will be found after it is released and create a serious problem for some users. Such a thing is possible, but I don't think it is going to be any more likely than a hang bug happening in prior versions. It seems to me that the incremental recursion is a much more dramatic change to rsync than anything else I've seen while I've been working with rsync and has the potential to introduce lots of obscure bugs. They may still all be found, but at least there are more potential bugs than in previous versions. And the user will always have the option of specifying --no-ir if they need to. I don't like this logic; rsync should work by default. Instead of making users scratch their heads when something goes wrong, I think it would be prudent to make --no-ir the default in rsync 3.0.0. (I'm afraid that if you don't, some distributions might.) Users who care about the improved performance and are aware of the possible consequence of instability could still pass --ir. Other users would enjoy the numerous other enhancements in rsync 3.0.0 without the chance of a bug in incremental recursion stopping the show. Once incremental recursion has gone through a release or two without any problems, I would make it the default. Matt I follow this list, mostly as an outsider, but a point or so maybe worth considering. Version jumps from 2.6.9 to 3.0.0 (not to 2.6.10 or 2.7) You're probably ahead to start with the world-view you will want to have after whatever dust settles. Rsync is not just remote copy, I sometimes use it locally Because it is better behaved in odd places/situations And if it's slower, it's not enough slower to matter. Somehow I would expect several subtle changes in expectations Of what is considered a normal rsync transfer. Wild guess, but seems more likely to have a hang bug in prior versions if there is a significant shift in expectations. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Local disk to disk Rsync taking an hour longer than disk to remote
From an old old old-timer The first disks that IBM came out with were effectively the same speed as card readers and line printers. For unblocked records. Disks are NOT asynchronous. They spin at a very predictable Rate and timing are extremely different based on whether the Head is in the right place just before or just after the point In time when the data comes by. The exact parameters of disk caches (all of them) can do all Sorts of strange things to the timings. Even the where on the disk the files are can matter. With a bit of experimenting, you should be able to get some Very real and violently counter-intuitive results. Optimized for known contention can mess up badly faced with unknown contention (two disks on same anything) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of W Smith Sent: Monday, July 16, 2007 3:36 PM To: rsync@lists.samba.org Subject: Local disk to disk Rsync taking an hour longer than disk to remote Back in June I posted about the trouble I've been having backing up some local directories and I'm no further ahead than back then. Link for that discussion: http://lists.samba.org/archive/rsync/2007-June/017882.html In summary: I'm copying nearly a million small files from the main disk in a server to another disk in the same machine. Still on this server, but at a different time, I am backing up the same million files to a remote server which is completed massively quicker and I just can't get my head around why this should be. Here are some new stats I got whilst using time to measure the difference between the two processes. Local backup --- Number of files: 1060320 Number of files transferred: 2233 Total file size: 26814206753 bytes Total transferred file size: 58711290 bytes Literal data: 58711290 bytes Matched data: 0 bytes File list size: 47383393 Total bytes sent: 106196771 Total bytes received: 44680 sent 106196771 bytes received 44680 bytes 12914.54 bytes/sec total size is 26814206753 speedup is 252.39 real 137m5.932s user 0m16.843s sys 125m35.697s Remote backup --- Number of files: 1060823 Number of files transferred: 1758 Total file size: 28255934663 bytes Total transferred file size: 63027228 bytes Literal data: 63027228 bytes Matched data: 0 bytes File list size: 47404505 Total bytes sent: 110514244 Total bytes received: 35180 sent 110514244 bytes received 35180 bytes 89045.05 bytes/sec total size is 28255934663 speedup is 255.60 real 20m40.908s user 0m11.344s sys 0m22.644s Sure there are a few less files on the remote backup, due to the scripts running at different times but this can't explain the hour difference in runtime can it? I've studied the rsync man page for hours, trying to find some elusive option to speed up local performance to no avail. I've also tried Unison and rdiff-backup, hoping that they might be better for local copying, but nothing comes close to rsync remote backup speeds. At this stage I'm close to giving up and am considering forgetting local backups and getting another server to have two remote backups instead, which wouldn't be a bad thing. My last hope is in the wisdom of this mailing list... :) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Archive parameter doesn't preserve owner:group property!
On BOTH sides do a ls -n whatever This will show the numeric IDs on both sides. Most likely the SAME numeric id will be one thing on one host and something else on the other host. Most likely --numeric-ids is doing exactly what it is supposed to do ls -l will show names if it can find a name that corresponds to the (numeric) id. Most every tool will store the file with the owner and group of what runs on the target. The alternative is probably to tar the files on the source, transfer the tarball, untar the files on the target. _ From: David (Spartoo) [mailto:[EMAIL PROTECTED] Sent: Friday, June 01, 2007 6:41 AM To: [EMAIL PROTECTED] Cc: rsync@lists.samba.org Subject: Re: Archive parameter doesn't preserve owner:group property! As a convenience and courtesy, rsync attempts to preserve names across the transfer I don't know if my request is clear...sorry for my bad english... I just need to preserve ID...I don't want rsync to convert id to name group because it takes the name group of the server instead of my local machine. I run rsync command in local machine in root, the deamon is run in root too, in rsyncd.conf in my module I set gid=root, uid=root...I have tried --numeric-ids parameter...and nothing... So if you know another tool which do that... 2007/5/31, Tony Abernethy [EMAIL PROTECTED]: The uid and gid are numbers. These numbers **MIGHT** correspond to one or more names. As a convenience and courtesy, rsync attempts to preserve names across the transfer Who ever is running the rsync daemon must have the rights to make the changes or there is nothing possible for rsync to do. To preserve the groups, I think the owner can change groups to/from groups that the owner is in. You will want to test on you system exactly what is tolerated --- I'd expect that the owner needs to belong to both groups. If it were me, I'd use chroot and uid=root and gid=root --- but then I've found that OPEN-NESS seems to be the only effective and cost-effective security mechanism. (Even to ridiculous extents;) (Everything else seems like turning out the street lights so the bad guys can't see you) _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David (Spartoo) Sent: Thursday, May 31, 2007 9:08 AM To: rsync@lists.samba.org Subject: Re: Archive parameter doesn't preserve owner:group property! someone?? Does I have to set gid and uid to root? or can I preserve initial owner:group setting uid to david and gid to rsync-users? Can I give to rsync-users the right to preserve owner:group? How? Is there a conflict between --numeric-ids and others parameters? because rsync doesn't preserve even in root owner:group if gid is a number instead of string. I have post to many Linux forums...rsync mailing-list is my last chance. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Archive parameter doesn't preserve owner:group property!
The uid and gid are numbers. These numbers **MIGHT** correspond to one or more names. As a convenience and courtesy, rsync attempts to preserve names across the transfer Who ever is running the rsync daemon must have the rights to make the changes or there is nothing possible for rsync to do. To preserve the groups, I think the owner can change groups to/from groups that the owner is in. You will want to test on you system exactly what is tolerated --- I'd expect that the owner needs to belong to both groups. If it were me, I'd use chroot and uid=root and gid=root --- but then I've found that OPEN-NESS seems to be the only effective and cost-effective security mechanism. (Even to ridiculous extents;) (Everything else seems like turning out the street lights so the bad guys can't see you) _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David (Spartoo) Sent: Thursday, May 31, 2007 9:08 AM To: rsync@lists.samba.org Subject: Re: Archive parameter doesn't preserve owner:group property! someone?? Does I have to set gid and uid to root? or can I preserve initial owner:group setting uid to david and gid to rsync-users? Can I give to rsync-users the right to preserve owner:group? How? Is there a conflict between --numeric-ids and others parameters? because rsync doesn't preserve even in root owner:group if gid is a number instead of string. I have post to many Linux forums...rsync mailing-list is my last chance. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Problem with shared xls file. Could it be blamed on rsync?
Before you trust this always-up-to-date, try it on some relatively large files. I suspect you get some nasty surprises. The basic problem is that Windows writes the directory entry and then sometime later writes the file contents. If you only rsync files that are say at least 10 minutes Old, you will probably get the correct contents. Chances are high that you notice the problem on shared files. Chances are high that the problem exists on all the files. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of syncro Sent: Friday, March 16, 2007 3:31 AM To: rsync@lists.samba.org Subject: RE: Problem with shared xls file. Could it be blamed on rsync? Thanks alot! That's what I wanted to hear ;) We want to have an always-up-to-date-copy thus rsync every minute and not just at night. However my preventive measure will be a forbiddance of sharing xls files or the like. Cheers, syncro -- View this message in context: http://www.nabble.com/Problem-with-shared-xls-file.-Could-it-b e-blamed-on-rsync--tf3408546.html#a9511198 Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: owner and group
-a gets several options you probably want (in particular -o and -g) (from man rsync) -a, --archive archive mode, equivalent to -rlptgoD -r, --recursive recurse into directories -l, --links copy symlinks as symlinks -p, --perms preserve permissions -o, --owner preserve owner (root only) -g, --group preserve group -D, --devices preserve devices (root only) -t, --times preserve times -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Hans Mignon Sent: Thursday, February 08, 2007 3:11 AM To: rsync@lists.samba.org Subject: owner and group Hello, I am trying to copy, on the same server, a set of files. I need to copy them every day with a cron job. This is working fine the only problem is that he always changes the owner and group to the root user. Is there a possibility that he uses the user of the remote files and not the root user ? rsync --archive --exclude .htaccess /home/virtual/site5/fst/var/www/ html/ /home/virtual/site10/fst/var/www/html/aaa/ Thanks, Hans Mignon -- View this message in context: http://www.nabble.com/owner-and-group-tf3192124.html#a8861757 Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: bug with --link-dest ?
[EMAIL PROTECTED] wrote: Hi! I am running rsync 2.6.4 (on Debian sarge) and I am experiencing a strange behaviour. I am trying to create an identical filetree on the same filesystems with the single files being hardlinks to the source like this: rsync -vaH --progress --delete --stats --numeric-ids -x --link-dest=/path/to/filetree/ /path/to/filetree/ /path/to/current/ /path/to is one filesystem. This creates hardlinks for the files in /path/to/filetree/ in /path/to/current/ When calling rsync with relative paths like this: cd /path/to; rsync -vaH --progress --delete --stats --numeric-ids -x --link-dest=./filetree/ ./filetree/ ./current/ rsync does not create hardlinks but copies the files instead. Is this intentional behaviour? If my understanding is right, that would attempt to link files from ./current/./filetree You maybe want a --link-dest=../filetree (Flames invited if I've got this wrong) Greetings -- Robert Sander -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync over smb share
The modify-window switch (actually 1 is supposed to be enough) is because DOS cannot store odd seconds and an even number and an odd number are never the same number. If I am understanding you right, the ssh should NOT be there. (flames invited if I'm wrong) You are rsyncing from a local source to a local source (the smb share is remote, but your access to it via samba is local) The times of interest The timestamp on the source file The implicit timestamp based on when the source was copied to the destination Which is what is left on the target? (With windows, there is a confusion between when the copy started and when the copy ended) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Robert Fitzpatrick Sent: Wednesday, January 10, 2007 4:24 PM To: rsync@lists.samba.org Subject: Re: rsync over smb share I have a FreeBSD unix server that I want to backup to a USB drive attached to a Windows 2000 server. I mount the smb share on the Unix server and then try to rsync from the source directory to the mounted directory, but it syncs all files every time. Can someone suggest if this should work and what my problem may be? This email is the property of ECS Technology Ltd. This company is registered in Scotland with company number 212513. VAT registered GB 761 7456 12 http://www.ecs-tech.com/ On Tue, 2007-01-09 at 10:39 +, Stuart Halliday wrote: You need to make allowances for the 2 different time systems. Put this in the Rsync line --modify-window=3 Thanks, but no help, I even set it as high as 100 and still syncs all files every time. I compared the time on both server, the Unix box where rsync is running the following command is 17:20:38 while the Windows 2000 server is 5:17:30 PM. I'm sure I can get them in sync, but wouldn't the --modify-window suggestion take care of this? rsync -aze ssh --progress --stats --modify-window=100 /path/to/source /path/to/mounted/smb/share/ -- Robert -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync not updating files even with checksum flag
Cheap shot that might be effective. Something like this might work. On the samba share, after the rsync has finished, run a script that touches any file that was last modified in the last few minutes. This marks files that should be retransmitted because they might have their contents changed without the directory entrybeing changed. Basically you want to run rsync on the file while windows is writing to the file and then (once) run rsync on the file after window has finished. It's a fundamental difference in approach :: windows : unix -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Mark OsborneSent: Monday, November 06, 2006 2:37 PMTo: rsync@lists.samba.orgSubject: rsync not updating files even with checksum flagHello everyone, I am having a problem with rsync that I hope someone can help me figure out. We are using rsync to sync up files between our staging and production ftp servers. Basically internal users are allowed to upload files via a samba share to a staging server. Those files are then synced out every 15 minutes via cron to our production ftp servers. The problem occurs when a large file is being upload from a windows machine via the samba share. If a rsync is instantiated during the time that the file is being uploaded the destination machines get the file with the correct filesize and timestamp. Unfortunately, even though the file shows the correct size it is not a good copy of the file. An md5sum of the files on both the source and destination machine returns different results. I believe this occurs because windows automatically "reserves" the full size of the file and fills it out with 0s and then overwrites this as it goes along copying. This wouldn't be a big deal except, subsequent runs of rsync (even with -c) fail to overwrite the file. I have tested running an rsync while I was copying the file over to the source directory locally (not via samba) and the file was corrupted. However after running an rsync again, the file was updated. Basically the problem only occurs when uploading the file via the samba share. I know I could use the -I flag to ignore times and filesizes, but from my understanding this would result in resyncing every file every time, and this is not what we want. We are currently using rsync 2.5.5 on the source machine, and 2.5.7 on the 2 destination machines, but I have also tested using 2.6.8 on both source and destination with the same results. Has anyone else experienced this problem before, or have any ideas for a fix? Let me know if I'm not being clear enough, or any other information I can provide. ~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~Mark OsborneWeb Systems Engineer[EMAIL PROTECTED](512) 683-5019~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~^~ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync over samba mounts (bad file descriptor)
Some things fast off the top of my head (flames invited where I'm wrong;) I am assuming: Files live on the Window server. smb mounted onto the BSD server (It does what it can, but it's a few cards shy of a full deck) There is a buch of whatever in Windows ACLs, but there is enough UNIX permission stuff MISSING that you can expect trouble. (Neither is an extension of the other. THey are different) You will probably want a window on the timestamps since DOS cannot represent odd seconds. You will likely to an all or nothing gizmo regarding ownerships. You will likely get the single-user attitude of DOS in order to make things work. Likely, the owner and group of everything in the mount is whoever mounted it. (Which naturally has nothing to do with who owns it on the Windows machine) If it were me, I would set multiple [names] in the target /etc/rsyncd.conf where the distinguishing difference was the uid= and gid= Good Luck ;) With luck, you should even get some actually helpful answers from the list. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Wayne Swart Sent: Friday, July 07, 2006 3:41 AM To: rsync@lists.samba.org Subject: Rsync over samba mounts (bad file descriptor) Hello everyone I am trying to do incremental backups using rsync and SSH. I have a windows server with two shares on it that I want to sync remotely to another machine. [] | windows server | [] [] | FreeBSD server | [] [] | SuSe server| [] I have two samba mounted shares on the BSD server, and then sync those samba mounts from the BSD machine to the Linux machine. Now the problem: When my rsync starts building its file list it gets a lot of Bad File Descriptor Errors, and does not do incremental syncs. Is there a way around this? (Without running cygwin on the Windows server). I am running my rsync with the following flags: -avuz Your help is appreciated. Wayne Swart Systems Administrator ZA Internet Solutions http://www.za24.co.za +2712 663 1800 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync over samba mounts (bad file descriptor)
rsync -avuz The -a wants most all of the UNIX permission (which do NOT exist on a samba mount) Easiest way is to try copying from the samba mount to a local filesystem and see which attributes you can actually copy. (man cp to see how to spell things) (You have my congrats if you get more than just one timestamp (to even seconds)) Repeat, you will almost certainly want a window on the timestamps. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Wayne Swart Sent: Friday, July 07, 2006 4:34 AM To: rsync@lists.samba.org; [EMAIL PROTECTED] Subject: RE: Rsync over samba mounts (bad file descriptor) Hi Tony Thanks for the speedy reply. Yes, the windows server is mounted on the BSD machine like you explained. I don't really mind to preserve date / time stamps on the files, or permissions. Surely there must be a way to get this to work. Copying the whole folder (folder at a time) to the BSD machine, and then sync it to the Linux box works, but its very time consuming, and definitely not the way to do it. Wayne -Original Message- From: Tony Abernethy [mailto:[EMAIL PROTECTED] Sent: Friday, July 07, 2006 11:17 AM To: Wayne Swart; rsync@lists.samba.org Subject: RE: Rsync over samba mounts (bad file descriptor) Some things fast off the top of my head (flames invited where I'm wrong;) I am assuming: Files live on the Window server. smb mounted onto the BSD server (It does what it can, but it's a few cards shy of a full deck) There is a buch of whatever in Windows ACLs, but there is enough UNIX permission stuff MISSING that you can expect trouble. (Neither is an extension of the other. THey are different) You will probably want a window on the timestamps since DOS cannot represent odd seconds. You will likely to an all or nothing gizmo regarding ownerships. You will likely get the single-user attitude of DOS in order to make things work. Likely, the owner and group of everything in the mount is whoever mounted it. (Which naturally has nothing to do with who owns it on the Windows machine) If it were me, I would set multiple [names] in the target /etc/rsyncd.conf where the distinguishing difference was the uid= and gid= Good Luck ;) With luck, you should even get some actually helpful answers from the list. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Wayne Swart Sent: Friday, July 07, 2006 3:41 AM To: rsync@lists.samba.org Subject: Rsync over samba mounts (bad file descriptor) Hello everyone I am trying to do incremental backups using rsync and SSH. I have a windows server with two shares on it that I want to sync remotely to another machine. [] | windows server | [] [] | FreeBSD server | [] [] | SuSe server| [] I have two samba mounted shares on the BSD server, and then sync those samba mounts from the BSD machine to the Linux machine. Now the problem: When my rsync starts building its file list it gets a lot of Bad File Descriptor Errors, and does not do incremental syncs. Is there a way around this? (Without running cygwin on the Windows server). I am running my rsync with the following flags: -avuz Your help is appreciated. Wayne Swart Systems Administrator ZA Internet Solutions http://www.za24.co.za +2712 663 1800 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: some errors
/sys ( also /proc , probably others ) some things that looks like directories and files, do not make sense to copy to another computer. generally, these give some kind of access to running kernel-type things. I think some of these "files" are write-only or would give information about some hardware (if you had the hardware) also some things like /dev/zero or /dev/null (anything in /dev, probably) needs special handling. /mnt/ tends to catch anything mounted (like CDs and stuff) # mount/dev/sda3 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Manuel KissoyanSent: Monday, June 26, 2006 11:13 AMTo: rsync listSubject: some errors Just wondering what could cause the following errors: rsync: read errors mapping "/sys/block/loop4/dev": No data available (61) rsync: send_files failed to open "/sys/bus/pci/drivers/Promise_Old_IDE/new_id": Permission denied (13) ERROR: sys/block/ram2/stat failed verification -- update retained.rsync: read errors mapping "/sys/block/ram2/stat": No data available (61) Thank you in advance! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: no true incrementals with rsync?
(Flames invited if I've got this wrong or misleading;) Traditional backup. (call this DOS-style backup) Base: (full backup) (and turn all the archive bits off) Incremental: copy files with archive bit on (and turn them off) Backup Files. Add something (suffix) to indicate backup copies. variations include suffixing (~) or changing extension to .bak or such. It looks like --backup plays this game. Everything in the same directory (or a separate backup directory) Call this one TRUE INCREMENTALS (This works on UNIX-type system, not DOS-type) The key to the gizmo is that cp -l (or cp --link) does a real-fast copy by just copying hard links. Each TRUE INCREMENTAL has its own target directory. The rsync is (I think) a straight rsync (you do NOT want the --backup) into a freshly prepared copy of the previous target. The parameters to rsync need to be such that the anything that matters is preserved on the OTHER copies of the files. (When there are several files hard-linked, you need rsync to unlink from the existing file and link to a newly-created file) (instead of changing the contents of the file everything is linked to) There are parameters to make rsync do EXACTLY what you want/need. (I do not even pretend to know what they are or what they do) (If everything is nice and simple, copy is nice and simple. If.) (Rsync plays well in spaces where copy is (very) complicated) (Just changing Permissions cause ...) (If the changed file is very similar, ...) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of tim594 Sent: Monday, June 26, 2006 12:16 PM To: rsync@lists.samba.org Subject: no true incrementals with rsync? for example's sake: With traditional backup systems, you keep a base (full backup, let's say every 30 days), then build incrementals on top of that, eg. (what has changed since the base). So, to restore, you copy over your base, then copy each incremental over the base to rebuild up to the latest snapshot. (*copying new incrementals files over older base files*) With rsync, (using --backup, --backup-dir) it's opposite, you sync nightly, so your base is your most up to date, and your incrementals are files that have changed x days ago... EG. if you want to restore a snap shot from 5 days ago, you copy over your base, then 1 days ago, 2 days ago, up to 5 days ago. (*copying older incrementals over newer base files*). am i right? Actually i think this method is better, because on a full uptodate restore you only need to copy over your base, as opposed to your base + all incrementals. THE QUESTION IS: anyone know a way to keep true incrementals including deletions? With rsync 2.6.6, when i use --delete or --del with the --backup command, the files it deletes are not backed up. It would seem logical to me, if you're running the --backup command, all changes should be backed up, updates AND deletions. Thanks -- View this message in context: http://www.nabble.com/no-true-incrementals-with-rsync--t1850225.html#a505110 9 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Can rsync monitor a file system?
To monitor the file system, you have to have something down inside the file system. Unless you know what you are doing, you don't really want to mess around with any such. Any slipup copy each others data Now if this means an update to one implies an update to the other that should be doable. If it means a delete from one imples a delete from both (me, I'm brave and daring but I wouldn't even try.) I can see two reasonable ways. 1) have an rsync daemon running on each server and a constantly running script on each server going either to or from the other server. 2) have an rsync daemon running on one server and a constantly running script on the other, alternately pushing and pulling. Regardless, -u (--update) is probably what you want. You may want to exclude rsync temporaries somehow (--exclude='.*' might be what you want) (Copying a temporary of a temporary after ... if (when) something goes down) You can get a wee bit of a mess if cron jobs start stepping on each other. No real damage other than producing multiple copies of dead temporaries. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of [EMAIL PROTECTED] Sent: Monday, June 26, 2006 2:47 PM To: rsync@lists.samba.org Subject: Can rsync monitor a file system? I am trying to get two servers to copy each others data to the other server. I need it to be done real time a not use a cron. Can rsync running as a daemon monitor the files system to trigger a transmission? If so how do I configure it? I am running Solaris 9. Thank you. __ Charles Berman Senior Unix Administrator Think Globally The contents of this email are the property of the sender. If it was not addressed to you, you have no legal right to read it. If you think you received it in error, please notify the sender. Do not forward or copy without permission of the sender. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync fails to rename on Windoze2003
This is mostly wild guesses (with luck you get some more knowledgeable answers) From what I have observed ... file of the form .RICHED~1.cab.IEKJmo is an rsync temporary of file RICHED~1.CAB Trying to rename the temporary file to its permanent name, it ran into trouble on the name RICHED~1.cab The idea is to NOT destroy the older target if the transfer goes south. Do a DIR cdrom/riched32/RICHED*.cab and see if it turns up anything. If you have more than one of them, Windows probably gets it wrong as to which is RICHED~1.cab and which is RICHED~2.cab Windows does have a kinda-sorta soft-link with the DOS 8.3 naming. Anything complicated in the naming and Windows manages to get it wrong at least half the time. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Corey Wirun - ListsSent: Saturday, June 24, 2006 11:05 PMTo: rsync@lists.samba.orgSubject: Re: Rsync fails to rename on Windoze2003 Some further information: I went back to 2.6.6 of rsync on the client and server and at least the entire sync completes. I still get the 'Permission Denied' problem on the rename though. I had a look at the logsand it appears the rename error is _always_ on a file name with a '.' as the starting character. e.g. ... rsync: rename "MEDIA_2006_LATEST/cdrom/riched32/.RICHED~1.cab.lEKJmo" (in MEDIA_2006) - "cdrom/riched32/RICHED~1.cab": Permission denied (13) ... total: matches=278531 tag_hits=1689129 false_alarms=123 data=""> sent 80988322 bytes received 2116535 bytes 72233.69 bytes/sectotal size is 1470681718 speedup is 17.70rsync error: some files could not be transferred (code 23) at /home/lapo/packaging/tmp/rsync-2.6.6/main.c(791) My source tree has no files with a '.' at the beginning. There was a file 'RICHED~1.cab' in the source - don't know why rsync thinks there should be a '.' in front of it on the destination. So it appears that file permissions to not have a bearing on the problem, but something in rsync is looking for files with a '.' at the beginning. But why does this only occur at random, not for every file? Note the 'false alarms' above - does that mean anything? Does anyone have any further notion why I get these rename errors? Thanks in Advance! Corey. - Original Message - From: Corey Wirun - Lists To: rsync@lists.samba.org Sent: Saturday, 24 June, 2006 14:22 Subject: Rsync fails to rename on Windoze2003 Hi All, I've seen several variations on this topic, but nothing exactly the same: I have two Windows 2003 servers that I want to use rsync (2.6.8) to mirror. These machines are separated by a WAN. Initial attempts to get rsync working between them have not been successful. The first rsync went fine, which seeded the files, but the second sync always fails (at a ramdom time) with a rsync: rename "AP16A7~1.cab.V55PQE" (in MEDIA_2006) - "AP16A7~1.cab": Permission denied (13) On the server, the rsyncd (from cwRsyncd), I initially ran the service as user SvcwRsync (from the installer), then added the user to the Administrators group, then changed the server to run as Administrator, all to no avail. If it was a perms problem, I wouldn't get _any_ files updated, but this fails on the rename at a random time. So I changed the rsync client to --inplace and I see (again random): unexpected tag 3 [sender]rsync error: error in rsync protocol data stream (code 12) at io.c(828) [sender=2.6.8] I really would prefer to not have to use --inplace to optimize the bandwidth taken up. Any thoughts on why the rename fails? I would have thought the owner of the service as local Administrator would be fine! Thanks in Advance! Corey. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsyncBefore posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync fails to rename on Windoze2003
TARGET The rename is on the target side, not the source. If the file is OPEN on the target side, Windows will not let you rename or otherwise mess with it. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Corey Wirun - personalSent: Sunday, June 25, 2006 10:16 AMTo: rsync@lists.samba.orgSubject: Re: Rsync fails to rename on Windoze2003 Thanks Tony, The source only has one file: RICHED~1.cab, so my guess is that the short name business isn't what is biting me here. Corey. - Original Message - From: Tony Abernethy To: Corey Wirun - Lists ; rsync@lists.samba.org Sent: Sunday, 25 June, 2006 07:31 Subject: RE: Rsync fails to rename on Windoze2003 This is mostly wild guesses (with luck you get some more knowledgeable answers) From what I have observed ... file of the form .RICHED~1.cab.IEKJmo is an rsync temporary of file RICHED~1.CAB Trying to rename the temporary file to its permanent name, it ran into trouble on the name RICHED~1.cab The idea is to NOT destroy the older target if the transfer goes south. Do a DIR cdrom/riched32/RICHED*.cab and see if it turns up anything. If you have more than one of them, Windows probably gets it wrong as to which is RICHED~1.cab and which is RICHED~2.cab Windows does have a kinda-sorta soft-link with the DOS 8.3 naming. Anything complicated in the naming and Windows manages to get it wrong at least half the time. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Corey Wirun - ListsSent: Saturday, June 24, 2006 11:05 PMTo: rsync@lists.samba.orgSubject: Re: Rsync fails to rename on Windoze2003 Some further information: I went back to 2.6.6 of rsync on the client and server and at least the entire sync completes. I still get the 'Permission Denied' problem on the rename though. I had a look at the logsand it appears the rename error is _always_ on a file name with a '.' as the starting character. e.g. ... rsync: rename "MEDIA_2006_LATEST/cdrom/riched32/.RICHED~1.cab.lEKJmo" (in MEDIA_2006) - "cdrom/riched32/RICHED~1.cab": Permission denied (13) ... total: matches=278531 tag_hits=1689129 false_alarms=123 data=""> sent 80988322 bytes received 2116535 bytes 72233.69 bytes/sectotal size is 1470681718 speedup is 17.70rsync error: some files could not be transferred (code 23) at /home/lapo/packaging/tmp/rsync-2.6.6/main.c(791) My source tree has no files with a '.' at the beginning. There was a file 'RICHED~1.cab' in the source - don't know why rsync thinks there should be a '.' in front of it on the destination. So it appears that file permissions to not have a bearing on the problem, but something in rsync is looking for files with a '.' at the beginning. But why does this only occur at random, not for every file? Note the 'false alarms' above - does that mean anything? Does anyone have any further notion why I get these rename errors? Thanks in Advance! Corey. - Original Message - From: Corey Wirun - Lists To: rsync@lists.samba.org Sent: Saturday, 24 June, 2006 14:22 Subject: Rsync fails to rename on Windoze2003 Hi All, I've seen several variations on this topic, but nothing exactly the same: I have two Windows 2003 servers that I want to use rsync (2.6.8) to mirror. These machines are separated by a WAN. Initial attempts to get rsync working between them have not been successful. The first rsync went fine, which seeded the files, but the second sync always fails (at a ramdom time) with a rsync: rename "AP16A7~1.cab.V55PQE" (in MEDIA_2006) - "AP16A7~1.cab": Permission denied (13) On the server, the rsyncd (from cwRsyncd), I initially ran the service as user SvcwRsync (from the installer), then added the user to the Administrators group, then changed the server to run as Administrator, all to no avail. If it was a perms problem, I wouldn't get _any_ files updated, but this fails on the rename at a random time. So I changed the rsync client to --inplace and I see (again random): unexpected tag 3 [sender]rsync error: error in rsync protoco
RE: Rsync fails to rename on Windoze2003
The place for confusion is on the target. Nothing is renamed on the source. If the target has more than one file matching RICHED*.cab, however it got there, expect trouble. For whatever reason, the temp file .RICHED~1.cab.X was renamed to RICHED~1.cab (on the target), is exactly where you are running into trouble. Why is pretty much a guessing game, but is almost certainly outside of any version of rsync. What would easily explain the symptoms is if you have a file RICHEDIT-THI-STUFF.cab or some such and that file is open or something decides that that file should not be clobbered. Out of curiosity, why would you have a file named RICHED~1.cab? That looks like the DOS 8.3 alias for a long file name It is possible that the first time creates a new RICHED~1.cab -- this goes successfully It is possible that any later RICHED~1.cab file is the WRONG RICHED~1.cab file (Windows (possibly Cygwin)idea of wrong) (Renaming long files based on Russian Roulette with the DOS 8.3 filenames has to be chancy at best) -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Corey Wirun - personalSent: Sunday, June 25, 2006 12:40 PMTo: rsync@lists.samba.orgSubject: Re: Rsync fails to rename on Windoze2003 Hi Tony, I understood. I thought you were referring to possible confusion between two files on the source - like RICHED~2.cab and RICHED~1.cab getting copied to the target and the 'wrong' file being renamed. In this case, there's only one file on the source getting copied to the target. You would think that if the temp file .RICHED~1.cab.X was renamed to RICHED~1.cab (on the target), that shouldn't cause a problem. Unless, as you say, it's not yet closed when the rename occurs. In that case, this appears to be a problem with rsync, is it not? Especially considering how different 2.6.8 works compared to 2.6.6 (which works 'better'). In any case, I'm going to checksum the source and target. It does appear that the sync did do something, and that the errors might be noise. Thanks for the reply. Corey. - Original Message - From: Tony Abernethy To: Corey Wirun - personal ; rsync@lists.samba.org Sent: Sunday, 25 June, 2006 10:17 Subject: RE: Rsync fails to rename on Windoze2003 TARGET The rename is on the target side, not the source. If the file is OPEN on the target side, Windows will not let you rename or otherwise mess with it. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Corey Wirun - personalSent: Sunday, June 25, 2006 10:16 AMTo: rsync@lists.samba.orgSubject: Re: Rsync fails to rename on Windoze2003 Thanks Tony, The source only has one file: RICHED~1.cab, so my guess is that the short name business isn't what is biting me here. Corey. - Original Message - From: Tony Abernethy To: Corey Wirun - Lists ; rsync@lists.samba.org Sent: Sunday, 25 June, 2006 07:31 Subject: RE: Rsync fails to rename on Windoze2003 This is mostly wild guesses (with luck you get some more knowledgeable answers) From what I have observed ... file of the form .RICHED~1.cab.IEKJmo is an rsync temporary of file RICHED~1.CAB Trying to rename the temporary file to its permanent name, it ran into trouble on the name RICHED~1.cab The idea is to NOT destroy the older target if the transfer goes south. Do a DIR cdrom/riched32/RICHED*.cab and see if it turns up anything. If you have more than one of them, Windows probably gets it wrong as to which is RICHED~1.cab and which is RICHED~2.cab Windows does have a kinda-sorta soft-link with the DOS 8.3 naming. Anything complicated in the naming and Windows manages to get it wrong at least half the time. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Corey Wirun - ListsSent: Saturday, June 24, 2006 11:05 PMTo: rsync@lists.samba.orgSubject: Re: Rsync fails to rename on Windoze2003 Some further information: I went back to 2.6.6 of rsync on the client and server and at least the entire sync completes. I still get the 'Permission Denied' problem on the rename though. I had a look at the logsand it appears the rename error is _always_ on a file name with a '.' as the starting character. e.g. ... rsync: rename "MEDIA_2006_LATEST/cdrom/riched32/.RICHED~1.cab.l
RE: Rsync fails to rename on Windoze2003
A
"simple" test to try (this assumes LOTS of free disk
space)
1)
Rsync from source to a fresh empty target.
2) Use
Windows (explorer) to copy (or move) from that (now not-empty) target to the
original target.
3) If
your luck is like mine, the copy will bomb out on one of the DOS 8.3
filenames.
-Original Message-From:
[EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]On Behalf Of
Corey Wirun - personalSent: Sunday, June 25, 2006 12:40
PMTo: rsync@lists.samba.orgSubject: Re: Rsync fails to
rename on Windoze2003
Hi Tony,
I understood. I thought you were
referring to possible confusion between two files on the source - like
RICHED~2.cab and RICHED~1.cab getting copied to the target and the 'wrong'
file being renamed. In this case, there's only one file on the source
getting copied to the target.
You would think that if the temp file
.RICHED~1.cab.X was renamed to RICHED~1.cab (on the target), that
shouldn't cause a problem. Unless, as you say, it's not yet closed when
the rename occurs. In that case, this appears to be a problem with
rsync, is it not? Especially considering how different 2.6.8 works
compared to 2.6.6 (which works 'better').
In any case, I'm going to checksum the source and
target. It does appear that the sync did do something, and that the
errors might be noise.
Thanks for the reply.
Corey.
- Original Message -
From:
Tony
Abernethy
To: Corey Wirun - personal ; rsync@lists.samba.org
Sent: Sunday, 25 June, 2006 10:17
Subject: RE: Rsync fails to rename on
Windoze2003
TARGET
The rename is on the target side, not the source.
If
the file is OPEN on the target side, Windows will not let you rename or
otherwise mess with it.
-Original Message-From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]On Behalf Of
Corey Wirun - personalSent: Sunday, June 25, 2006 10:16
AMTo: rsync@lists.samba.orgSubject:
Re: Rsync fails to rename on Windoze2003
Thanks Tony,
The source only has one file:
RICHED~1.cab, so my guess is that the short name business isn't what is
biting me here.
Corey.
- Original Message -
From:
Tony
Abernethy
To: Corey Wirun - Lists ;
rsync@lists.samba.org
Sent: Sunday, 25 June, 2006
07:31
Subject: RE: Rsync fails to rename
on Windoze2003
This is mostly wild guesses (with luck you get some more
knowledgeable answers)
From what I have observed ...
file of the form .RICHED~1.cab.IEKJmo is an rsync temporary
of file RICHED~1.CAB
Trying to rename the temporary file to its permanent name, it ran
into trouble on the name RICHED~1.cab
The idea is to NOT destroy the older target if the transfer goes
south.
Do a DIR cdrom/riched32/RICHED*.cab and see if it
turns up anything.
If you have more than one of them, Windows probably gets it wrong
as to which
is RICHED~1.cab and which is RICHED~2.cab
Windows does have a kinda-sorta soft-link with the DOS 8.3
naming.
Anything complicated in the naming and Windows manages to get it
wrong at least half the time.
-Original Message-From:
[EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]On Behalf
Of Corey Wirun - ListsSent: Saturday, June 24, 2006
11:05 PMTo: rsync@lists.samba.orgSubject: Re:
Rsync fails to rename on Windoze2003
Some further information:
I went back to 2.6.6 of rsync on the
client and server and at least the entire sync completes. I
still get the 'Permission Denied' problem on the rename
though. I had a look at the
logsand it appears the rename error is _always_ on a file name
with a '.' as the starting character. e.g.
...
rsync: rename
"MEDIA_2006_LATEST/cdrom/riched32/.RICHED~1.cab.lEKJmo" (in
MEDIA_2006) - "cdrom/riched32/RICHED~1.cab": Permission denied
(13)
...
total: matches=278531
tag_hits=1689129 false_alarms=123 data="">
sent 80988322 bytes received
2116535 bytes 72233.69 bytes/sectotal size is
1470681718 speedup is 17.70rsync error: some files could not
be transferred (code 23) at
/home/lapo/packaging/tmp/rsync-2.6.6/main.c(791)
My source tree has no files with a '.' at
the beginning. There was a file 'RICHED~1.cab' in the source -
don't know why rsync thinks there should be a '.' in front of it on
the dest
RE: rsync and mysql with users connected
With the usual caveats of "Your Mileage May Vary", this may give you an idea of how far you can push things and get away with it. In practice, I back up production MySQL databases when they are running. Not that big a deal. Some notes on improving the odds (of getting away with it): One logical rsync is two rsyncs back-to-back. The first one to "do all the work" The second one to catch anything that changed during the first. This still leaves a window of expsure, but it is fairly small. If the rsync is over internet (or WAN) links, a local rsync to a staging area and the long slow rsync from something "stable". This is with straightMyISAM tables. I would expect trouble from anything with "transactional integrity". With MyISAM, things are simple enough so that even if you catch somehting in mid-something, its actually pretty hard to trash the table. That's hard, not impossible. Actually with staged backups, it's pretty well impossible to actually do yourself in. The only time I've gotten bit (that I'm aware of) is the .MYD and the .MYI out of sync and adding a new record and getting bit because the autoincrement record already existed. Actually more funny than serious. If you have only the primary and one backup (just one backup seems a recipe for disaster, anyway), you will need to stop the database and do the rsync. (And what do you do when the primary fails DURING the rsync?) Backing up from Windows, my guess is that you will have better luck doing an smb mount and doing the rsync from something unixy to something unixy. With windows, a file opened normally will have EVERYTHING ELSE locked out of doing anything with that file. With things unixy, it is considered kosher to delete a file while someone else is writing it, the standard sequence of an update is to ./configure; make; make install. Then to shut down the old running program and bring up the new program. The smb mount and unix rsync will NOT overcome the Windows semantics, but should at least do as much as can be done. Windows likes to go part way in a random order and the die at the first sign of trouble. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Grignon MickaëlSent: Friday, June 23, 2006 11:02 AMTo: rsyncSubject: rsync and mysql with users connected Hi, I want to apologise for my bad english (I'm french). So I need to Back-up a lot of datas from a place to an another by the way of rsync with ssh naturally. A good question from my boss was : What happened when the mysql database is used by a user during the Backup ? Is the system stop the Backup or is it transparency for rsync ? Sorry if my question is stupid but I haven't already finish the reading of all the documents treating rsync. Thanks in advance to your answers and comprehension. Oh yes, I forgot an important fact : The Backup is donebetween SUSE 10.0 and Mandrake 10.0 so Linux envirronement. Some datas froms Windows will be Backuped too. I think we will use a ntfs partition. Some advises about that or all is OK and though that don't cause any problems ? Mickaël. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Windows rsync
This is the kind of place where EVERY piece of punctuation matters. -delete is very unlikely. You probably mean --delete (two dashes before a "meaningful" parameter) (-delete would be something wierd following a -d) 192.168.1.152 might be some kind of anonymous I would expect to see something like [EMAIL PROTECTED] The :: (two colons) means that it is speaking rsync protocol I think there is expected to be a "module" name after the :: (where module is something in /etc/rsyncd.conf on the server side) (and generally there is enough stuff that you know (and nobody else knows) so that it works for you and some passerby like me;) You should get some better advice from people on this list who actually know what the're talking about. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of corySent: Tuesday, June 06, 2006 8:51 PMTo: rsync@lists.samba.orgSubject: Windows rsync Hi there I am new to rsync so be gentle Ok i install cwRsync and i used the script that came with it but just added my filename and remote computer.right now it says rsync -r -delete /c/backup/ 192.168.1.152::/c/backupi also tried it with rsync -r -delete /cygdrive/c/backup/ 192.168.1.152::/c/backupandrsync -r -delete /cygdrive/c/backup/ 192.168.1.152::/cygdrive/c/backupthe error message i get is Environment Variable rsync -r -delete /cygdrive/c/backup/ not defined Thanks Cojast --No virus found in this outgoing message.Checked by AVG Anti-Virus.Version: 7.1.394 / Virus Database: 268.8.2/356 - Release Date: 6/5/2006 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync shows poor throughput vs. scp
Marty Mulligan wrote: Thanks for your suggestions. I'll try to put together an rsync call with a more explicit set of options, although I was under the impressions that by having the "dont compress" option set in the conf file on the server, the -z option in the call from the host was ignored (which begs the question, "why the -z in the call from the host?" :) ) (I suspect that here is a case where you use rsync because it's easier to figure out rsync that to use something (actually quite different) that is more "appropriate". hmm... ok, I am open to suggestions! What do you recommend that would be more appropriate? One thing I am currently investigating is "File Alteration Monitor" or fam, which was alluded to somewhere in this lists archives. The "something more appropriate" is something you design and write yourself. And you get to debug it yourself. Including all the little suble thingees which are there and do matter. They are at least theoretically possible. I wouldn't say I recommend them. If everything goes perfectly, a lot of things are equivalent. It's when things mess up in the middle, what is the resultant state? What should it be? This is really a question of which errors do you commit (to try to avoid other errors) Question: how long does it take to write a file to disk? Answer: It depends ... actually on EVERYTHING that is going on DURING the write. Generally disk is written much faster if it is written sequentially. One way to slow it way way down, it to interrupt the stream of writes with a bunch of (any) disk IO elsewhere on the disk. You should realize that disk is called random access, but is in fact anything but random. Like scheduled trains or busses or airplanes, the place under the heads that is to be read or written comes up very predictably. This means that tiny changescan have big consequences. Miss a flight by a few seconds and wait for the next day (or week or ...). Most likely you have the drive reading in one place and writing in a different place and the seek time has a significant impact on what happens. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync shows poor throughput vs. scp
You should get some better answers, but a couple of points jump out at me. If the files are already compressed, "small" changes result in very different files, so the business of reading both the target and the source to find common stuff is kinda counterproductive. Also the -z (compressing something already compressed) can't really do much good, particularly with LAN speeds Methinks in this case, you want to disable a bunch of stuff that makes rsync great. Seriously. Basically, you want rsync to copy the entire file if the timestamp is different. You do not want thingees that are extremely useful for transferring big files over bad connections. (I suspect that here is a case where you use rsync because it's easier to figure out rsync that to use something (actually quite different) that is more "appropriate". You discover that "copy stuff from here to there (or vice-versa) is actually incredibly complicated as to exactly what you (should) mean. -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Marty MulliganSent: Wednesday, May 24, 2006 2:29 PMTo: rsync@lists.samba.orgSubject: rsync shows poor throughput vs. scphi all- been reading through the archives but I still can't seem to find a solution to my problem. I am using rsync to keep mirror copies of content which is being served (via http) on both the sender and receiver.The files on average are 20-50mb each, (mostly already in a compressed format... mp3, etc) and both sender and receiver pushes around 30mbps on average.the sender is a Celeron 1.3GHz with 1GB RAMreceiver is an AMD 1.2GHz with 1GB RAMBoth have dual IDE drives configured for software RAID0When I initiate an rsync from the receiver, the file list is built in around / under 1 minute, and the transfer proceeds at around 500K/sThis seems very slow since the machines are on the same subnet and sitting right next to each other in the same datacenter. While this rsync is running, I open a second terminal and do an scp, again initiated from the receiver copying from the sender, and the transfer proceeds at around 5MB/s!Why is scp so much faster than rsync here? Is there anything I can do to improve the speed of these transfers?Fwiw, this is the rsync command I'm issuing:rsync -azL --whole-file --stats --progress --delete sender::my_files /test_destination and rsyncd.conf on the sender looks like:use chroot = nomax connections = 10pid file = /var/run/rsyncd.pidmotd file = /etc/rsync/rsyncd.motdtimeout = 300transfer logging = yeslog file = /var/log/rsyncd.loglog format = %t %h (%a) %m %l %b %o %f[my_files]uid = rootpath = /my_filesmax connections = 5read >hosts allow = 192.168.0.0/24dont compress = *list = falseMUCH respect and appreciation to anyone who can help!Thanks,Marty -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: ignoring file times - but still examining content
Assuming the problem is Windows and time zones and daylight savings etc. man rsync --modify-window=NUM compare mod times with reduced accuracy. Windows still uses the old DOS method of storing times, which can't put the value of the seconds into a 32-bin number, so can only represent even seconds. (That's the theoretical best it can do) I think going on or coming off Daylight Savings cause times to go beserk, generally by an hour (3600 seconds or so) Guessing, but --modify-window=4000 should make your life a bit more pleasant. If your problem is time-zones, laugh, but do the math and ... -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of tyko brown Sent: Tuesday, May 23, 2006 3:52 AM To: rsync@lists.samba.org Subject: ignoring file times - but still examining content Hello. I would like to transfer files which have changed on another box. The rsync man page says: --size-only ..This is useful when starting to use rsync after using another mirroring system which may not preserve timestamps exactly.. Which is my situation. My problem is that the size-only switch (clearly) ignores files with different content, but that have the same size. I'd like to ignore timestamps but still examine content. How can I do that? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync speed
Excuse the "humor", but it sounds like you have a virus. One of the virus that is called "anti". ( Case of the "cure" being worse than the disease? ;) -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of Julian Pace RossSent: Tuesday, May 23, 2006 4:22 AMTo: Wayne DavisonCc: rsync@lists.samba.orgSubject: Re: rsync speed Pushing the file from Windows to Linux over ssh takes around 15 min, with an average speed of ~400Kbps (using --progress). Pulling the same file using the same arguments takes around 45 mins, with an average speed of ~150Kbps. Do you mean pulling it back again from Linux to Windows? Or switching machines and pulling it in the same direction from Windows to Linux? I'm going to assume the former. Yep its the former... pushing and pulling from the same Windows PC which is a P4. The linux server is a PIIIi delete the 350MB file from the source/dest in turn in order to try it... It seems slow to me (ssh overhead??) This depends on how slow (or loaded) your CPU is. For instance, on an old Pentium III 733mhz, the encrypting that ssh does cuts down the top transfer speed compared to an unencrypted daemon connection (I think the ssh connection was about 33% of the daemon connection, IIRC). Both CPUs reach a max of 2-3%, so the bottleneck is somewhere else... If you're backing-up over a local network, it may well be safe enough to dropp the ssh transport of your daemon-style transfers and just run a password-protected daemon on your Linux box. I know, but I'm trying to simulate transfer over an internet conncetion on my lan first.I'll experiment further and let you know what I come up with... but I'm quite sure its something particular to my setup.Thanks wayne!Julian -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Issue with hard links, please help!
Max Kipness wrote: You could of course (right after an rsync run) do a cd newdir; find . -type f -links 1 -print and then randomly check a couple and compare all their attributes such as mtime, permissions to the previous dir. (I still recommend using the --link-dest thing over using cp -al first.) Ok, I think I've figured out the problem with this one, although I'm not exactly sure of the reason. I have now started using --link-dest and this works great. Here again is the stat screen: Number of files: 50285 Number of files transferred: 38 Total file size: 16193254538 bytes Total transferred file size: 4077908049 bytes Literal data: 86201342 bytes Matched data: 3989904700 bytes File list size: 945440 File list generation time: 6.615 seconds File list transfer time: 0.000 seconds Total bytes sent: 87436048 Total bytes received: 539014 sent 87436048 bytes received 539014 bytes 97913.26 bytes/sec total size is 16193254538 speedup is 184.07 Well, it ends up that there is a Microsoft backup file (a .bkf file) that is around 4GB in size that is being changed daily. Now my question (I think the final one) is why the entire file seems to be transferred even though rsync obviously detects that only a fraction of the file has changed. Evidently the Literal Data shows 86201342 of changes which appears correct. Also, since I'm using option --log-format=%f %l %b, I see on the file in question, the following results: SERVER/E$/exchange.bkf 4076087296 86454659 Isn't this stating that the file size is 4076087296, and the changes to the file are 86454659? So why is the entire file transferring each day. I'm using the --no-whole-files option. Here is the rsync command options I used for the latest test: Rsync has NO guarantee that the only changes are to the END. Rsync has to work when the changes are to the beginning or scattered throughout. Rsync goes to a lot of trouble to find and transmit only the changes. This is extremely useful over slow and/or erratic network connections. This is probably significantly slower over gigabit ethernet. Also, be aware that of the times that are representable in Unix, DOS and derivatives are only capable of represententing half of them. Depending on whatever, you may have DOS files that are always seen as being different because the times do not and cannot match. rsync /share/ /backup/05-13-2006/ -v --link-dest=/backup/05-12-2006/ --stats --recursive --archive --times --modify-window=1 --delete --ignore-errors --files-from=/var/www/html/backup/adlist.txt --exclude-from=/scripts/file-exclude --no-whole-file --log-format=%f %l %b 2 errors.log 1 stats.log\ In the previous posts I stated that du showed every incremental directory to be around 4-5gb in size. This is because each day the exchange.bkf has some change associated with it, so I guess the file cannot be linked. So in reality if you have very large files that have very small changes applied, hard-links really serve no purpose, correct? And I assume there is nothing else that can be done with these large files to conserve space? Hard links are how unix names files (the file itself) Hard links allow one file to have more than one name. Any change to the file (by any name) is done to the file and shows up in all the other names. When the last name (actually reference) is deleted, the file is deleted. There is no yes, but associated with hard links. Hard links will not help save space on similar but not exactly the same files. Thanks Max -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Issue with hard links, please help!
Recheck the statistics: 4GB file something like 4,000,000,000 bytes Total bytes sent:87,436,048 -- MUCH LESS than 4 GB Total bytes received:539,014 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Max Kipness Sent: Sunday, May 14, 2006 6:56 PM To: Tony Abernethy; rsync@lists.samba.org Subject: RE: Issue with hard links, please help! Number of files: 50285 Number of files transferred: 38 Total file size: 16193254538 bytes Total transferred file size: 4077908049 bytes Literal data: 86201342 bytes Matched data: 3989904700 bytes File list size: 945440 File list generation time: 6.615 seconds File list transfer time: 0.000 seconds Total bytes sent: 87436048 Total bytes received: 539014 sent 87436048 bytes received 539014 bytes 97913.26 bytes/sec total size is 16193254538 speedup is 184.07 Well, it ends up that there is a Microsoft backup file (a .bkf file) that is around 4GB in size that is being changed daily. Now my question (I think the final one) is why the entire file seems to be transferred even though rsync obviously detects that only a fraction of the file has changed. Evidently the Literal Data shows 86201342 of changes which appears correct. Also, since I'm using option --log-format=%f %l %b, I see on the file in question, the following results: SERVER/E$/exchange.bkf 4076087296 86454659 Isn't this stating that the file size is 4076087296, and the changes to the file are 86454659? So why is the entire file transferring each day. I'm using the --no-whole-files option. Here is the rsync command options I used for the latest test: Rsync has NO guarantee that the only changes are to the END. Rsync has to work when the changes are to the beginning or scattered throughout. Rsync goes to a lot of trouble to find and transmit only the changes. This is extremely useful over slow and/or erratic network connections. This is probably significantly slower over gigabit ethernet. Also, be aware that of the times that are representable in Unix, DOS and derivatives are only capable of represententing half of them. Depending on whatever, you may have DOS files that are always seen as being different because the times do not and cannot match. Rsync seems to be detecting what the changes on this large file. Based on what you are saying, rsync in this case knows what the changes are in the file, roughly 86mb, but cannot transmit only the changes and therefore transmits the entire 4Gb file? If there is no way around this, I guess I'll have to live with it. Recheck the statistics. It did in fact transmit only the changes. (plus some traffic to know where the changes are) And a fair amount of work on both sides to find the pieces of the 4GB file that are the same. Total bytes sent: 87436048 Total bytes received: 539014 97913.26 bytes/sec-- This is a combination of CPU work and transmission. Dunno if your connection is about 1Mbps, but if it is, 4GB will take over 11 hours. The 4GB file will appear to be transferred in slow motion as both sides cooperate in finding which parts are the same and which parts are different. There is no instant knowledge of which 87MB is different and needs to be transferred. 4GB must be read from disk on BOTH sides to determine exactly which 87MB rsync /share/ /backup/05-13-2006/ -v --link-dest=/backup/05-12-2006/ --stats --recursive --archive --times --modify-window=1 --delete --ignore-errors --files-from=/var/www/html/backup/adlist.txt --exclude-from=/scripts/file-exclude --no-whole-file --log-format=%f %l %b 2 errors.log 1 stats.log\ In the previous posts I stated that du showed every incremental directory to be around 4-5gb in size. This is because each day the exchange.bkf has some change associated with it, so I guess the file cannot be linked. So in reality if you have very large files that have very small changes applied, hard-links really serve no purpose, correct? And I assume there is nothing else that can be done with these large files to conserve space? Hard links are how unix names files (the file itself) Hard links allow one file to have more than one name. Any change to the file (by any name) is done to the file and shows up in all the other names. When the last name (actually reference) is deleted, the file is deleted. There is no yes, but associated with hard links. Hard links will not help save space on similar but not exactly the same files. That's what I figured, just wanted to clarify. So if you had a directory with 10 1GB files, and each day you made a 10k change to each, all your incremental directories would have 10GB total, nothing saved from hard-linking. I think you misunderstand. Without hard links you have NO access to ANY files. It's not an additional way to access files. It is THE way
RE: Issue with hard links, please help!
100Mbps is about 10MBps so 4GB would take about 400 seconds or a bit under 7 minutes. BOTH sides will have to read the 4GB file and compare results. Comparing results will be almost instantaneous over 100Mbps LAN. Getting the results to compare will depend on disk speed and CPU speed. It doesn't have to be extremely old for the disks to be slower than the LAN. I've had local rsyncs (disk-to-disk) that took longer than trans-pacific internet. Also quite often very painfully the reverse;) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Max Kipness Sent: Sunday, May 14, 2006 8:03 PM To: rsync@lists.samba.org Subject: RE: Issue with hard links, please help! Recheck the statistics: 4GB file something like 4,000,000,000 bytes Total bytes sent:87,436,048 -- MUCH LESS than 4 GB Total bytes received:539,014 Total transferred file size: 4077908049 bytes Sorry, got it now. I missed the 'Total bytes sent' stat and was assuming the 'Total transferred file size' meant that is what was transferred. For this test I'm on a 100mb local connection, so it seemed like it was taking long enough for this to be true. Thanks for all the help. Max -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: random file corruption on NTFS
I wouldn't say that FAT32 is "lenient", HOWEVER it is almost certain that NTFS is extremely dependent on there not being ANY errors anywhere else, or VERY bad things will be done. Cheap shot, if you can identify files (or clusters of files) is to rename the stuff something like BAD-DATA and (very important this) NEVER EVEN THINK ABOUT "DELETING" the stuff (What happens is that deleting puts the clusters or whatever on the free-track/whatever list) sometimes they will be the first to get "re-used". FAT32 is well-defined and well-known. NTFS has some very strange goings on Sufficiently strange that you always wanted to be able to boot early NT4 servers from FLOPPY! -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of darrin hodgesSent: Friday, May 12, 2006 12:49 AMTo: rsync@lists.samba.orgSubject: Re: random file corruption on NTFSHi Tony,the volume was previously a FAT32 before it was reformatted to NTFS. I've wonderedabout the drive being dodgy, is it that FAT32 is more leanient in terms of file errors?thanks.Darrin. On 5/12/06, Tony Abernethy [EMAIL PROTECTED] wrote: Wild guess, but that sounds a lot like a disk drive going bad. An error message like The file or directory is corrupted and unreadable. There is essentially no way that an application (including rsync) can cause that kind of thing. You probably can cause that by messing with the controller registers or sticking strange stuff into the control or data lines. A bad place on the disk (and it has to be pretty bad or you will not notice it) or cables that aren't really plugged in or a failing or very weak power supply are all potential causes. Probably no way to know, but does it happen on the same sectors every time? FAT32 is probably on a different drive, but (essentially they are race conditions or timing errors) working on one and failing on the other doesn't mean much. Although,FAT32 is probably much less likely to lose you data (I've seen NTFS "fixed" by wiping out a directory that was unreadable) Curiously, backups have to be the standard place to discover you need the backup. (while in the process of destroying the backup you need;) -Original Message-From: rsync-bounces+tony=[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of darrin hodgesSent: Friday, May 12, 2006 12:06 AMTo: rsync@lists.samba.orgSubject: random file corruption on NTFSHi,We are using Rsync version 2.6.8 protocol version 29 on a winNT box to backup a linux (RedHat 9.0) box (same version of rsync) and everynight a different file on the NT server is reported as being corrupt, there are no errors in the rsync logs on either side. NT Event log records: Event Type: Error Event Source: Ntfs Event Category: Disk Event ID: 55 Date: 12/05/2006 Time: 3:05:19 AM User: N/A Computer: [ Deleted ] Description: The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume TESTDATA. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp . Data: : 00 00 04 00 02 00 52 we use BrightStor ARCserve Backup as backup software on the NT server, it reports: Unable to open file. (FILE=F:\database\base_currency\country_list.frm, EC=The file or directory is corrupted and unreadable.) The file reported is different every day, we have tried turning off the 'z' option, as that was causing an error when attempting to transfer large already compressed files but did not resolve this particular problem. We have noticed that the error does not occur on a FAT32 volume, is there a problem with rsync interacting with the NTFS?. ThanksDarrin. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: random file corruption on NTFS
Wild guess, but that sounds a lot like a disk drive going bad. An error message like The file or directory is corrupted and unreadable. There is essentially no way that an application (including rsync) can cause that kind of thing. You probably can cause that by messing with the controller registers or sticking strange stuff into the control or data lines. A bad place on the disk (and it has to be pretty bad or you will not notice it) or cables that aren't really plugged in or a failing or very weak power supply are all potential causes. Probably no way to know, but does it happen on the same sectors every time? FAT32 is probably on a different drive, but (essentially they are race conditions or timing errors) working on one and failing on the other doesn't mean much. Although,FAT32 is probably much less likely to lose you data (I've seen NTFS "fixed" by wiping out a directory that was unreadable) Curiously, backups have to be the standard place to discover you need the backup. (while in the process of destroying the backup you need;) -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]On Behalf Of darrin hodgesSent: Friday, May 12, 2006 12:06 AMTo: rsync@lists.samba.orgSubject: random file corruption on NTFSHi,We are using Rsync version 2.6.8 protocol version 29 on a winNT box to backup a linux (RedHat 9.0) box (same version of rsync) and everynight a different file on the NT server is reported as being corrupt, there are no errors in the rsync logs on either side. NT Event log records: Event Type: Error Event Source: Ntfs Event Category: Disk Event ID: 55 Date: 12/05/2006 Time: 3:05:19 AM User: N/A Computer: [ Deleted ] Description: The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume TESTDATA. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: : 00 00 04 00 02 00 52 we use BrightStor ARCserve Backup as backup software on the NT server, it reports: Unable to open file. (FILE=F:\database\base_currency\country_list.frm, EC=The file or directory is corrupted and unreadable.) The file reported is different every day, we have tried turning off the 'z' option, as that was causing an error when attempting to transfer large already compressed files but did not resolve this particular problem. We have noticed that the error does not occur on a FAT32 volume, is there a problem with rsync interacting with the NTFS?. ThanksDarrin. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync ok with open files?
Fundamental difference between unix and windows. Windows locks against simultaneous whatever. Unix assumes you know what you're doing. Unix allows deleting a file while someone is writing to it. Also, you will likely need to stop BOTH outlooks. There's usually one running that you do not see. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Tevfik Karagülle Sent: Wednesday, May 10, 2006 2:34 PM To: 'Peter'; 'rsync' Subject: RE: rsync ok with open files? AFAIK, rsync cannot copy open files. Outlook must be stopped in your case. Rgrds Tev -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Sent: Wednesday, May 10, 2006 6:06 PM To: rsync Subject: rsync ok with open files? Hi gang. I am synchronizing Windows machines (cwRsync/cygwin) with FreeBSD server. A batch file is used to initiate the process (shell invoked rsync daemon). My question is whether I need to tell my users to close any affected applications (ex: Outlook; I am synchronizing the outlook.pst file) during the procedure? Peter __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync via ssh hangs on same file repeatedly
Flames invitied if I'm wrong, but I think you're looking at the last file successfully transferred as opposed to the first file unsuccessfully transferred. I think I saw a /var in there You get something ungodly long trying to rsync the file that is logging the rsync. something like rsync -av --exclude log/ --exclude work/ $name /sdb1/2006-04-01-hostname/ may be in order (the above is just a local copy to a USB drive. You probably have some timing issues as to whether or not rsyncing the log file actually works or is an infinite loop. I'd put watching the grass grow at a higher priority than trying to fix the problem. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Gawain Sent: Tuesday, May 09, 2006 8:53 PM To: rsync@lists.samba.org Subject: Rsync via ssh hangs on same file repeatedly Hi all, I'm having problems with an rsync via ssh process hanging during transfer. The basic problem, (gory details to follow): I've set up a machine on my LAN to act as the receiver and have two (soon to be more) remote servers transferring files via a script. It seems as if certain files are causing the transfer to hang. These are different files on each of the sending machines, and there doesn't seem to be any correlation between them. The plan is to back up several directories (/usr /var /home /etc) so I thought at first there could be some sort of memory issue as the transfers stalled partway through. Then I realized that even if I only transferred one directory the process would still hang on this particular file. In one case, the file was a 57kb executable and in another it was a 238kb JPEG. Everything works fine up to that point and then bam! It just sits there on that file and never finishes. I've turned on some of the extended reporting options, and here's some output from one of the transfers from a /usr/bin directory: - 1205 files to consider appletproxy 4867 100%0.00kB/s0:00:00 (xfer#1, to-check=1185/1205) apropos 2378 100%2.27MB/s0:00:00 (xfer#2, to-check=1184/1205) ar 35480 100%2.82MB/s0:00:00 (xfer#3, to-check=1183/1205) ark 4831 100% 314.52kB/s0:00:00 (xfer#4, to-check=1182/1205) artsbuilder 344201 100% 517.13kB/s0:00:00 (xfer#5, to-check=1181/1205) artscat Killed by signal 2. 48.71kB/s0:00:00 - In the case above the 'artscat' file hangs at precisely 55% each time, at exactly 32768 bytes. H. A clue, perhaps? One of the sending systems is a RedHat box, the other is FC2. The Fedora system is running rsync 2.6.2, the RH box is now running 2.6.8 after hoping that an upgrade from 2.5.5 would cure the problem. No change. That leaves the receiving system as the common link, so I suspect the problem may lie there or with the HD. It's a G4 Mac running OS X 10.4.6 with the standard install of rsync 2.6.3. It's a fresh install of the OS and the machine will be used solely as a backup server. I'm backing up to a LaCie 250GB Ethernet disk, formatted with the Ext3 filesystem. I don't have enough free space on the receiver's internal HD to test the entire backup, but the backup of just the /usr/bin directory completes successfully if I back up to the internal HD instead of the Ethernet disk. As you can see above, the transfer hangs only six files into a 1205 file run when backing up to the LaCie. However it works fine on other large directories. I'm out of ideas on this one, except for possibly installing a larger internal HD and temporarily forgetting about the LaCie. Any ideas? I'd be grateful for any insights you can provide. Gawain -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
