Re: Advanced rsync includes and excludes
Am Mittwoch, 7. August 2019, 17:36:01 CEST schrieb Kevin Korb via rsync: > I believe you can shorten that to: > > + /some/very/ > + /some/very/deep/ > + /some/very/deep/path/ > + /some/very/deep/path/to/ > + /some/very/deep/path/to/save/*** > - /some/* Unfortunately, this doesn't work, since it picks up everything in /some/very/, /some/very/deep/, and so on. > You could also exclude /some and then use /some/very/deep/path/to/save > as an additional source. I don't know if rsnapshot can handle multiple > sources in 1 rsync but rsync itself can. No, it would need different backup jobs then... Well, looks like my scheme from below is still the best way to handle such cases. Anyway, thanks for your feedback, Kevin. Cheers, Pete > On 8/7/19 9:54 AM, Hans-Peter Jansen via rsync wrote: > > Hi, > > > > I'm a happy camper @ rsync (and rsnapshot) since years. Thanks for this > > major piece of software. > > > > In an attempt to reorganize my rsnapshot backups, I stumbled across an > > issue, that I'm trying to seeking a more sophisticated solution here. > > > > Given, I have a deeply branched tree, where I would like to include a > > specific directory deep under, while excluding anything else on that > > path, I find myself doing: > > > > + /some/ > > + /some/very/ > > + /some/very/deep/ > > + /some/very/deep/path/ > > + /some/very/deep/path/to/ > > + /some/very/deep/path/to/save/ > > - /some/very/deep/path/to/* > > - /some/very/deep/path/* > > - /some/very/deep/* > > - /some/very/* > > - /some/* > > > > While it works, it feels rather awkward and gets very complicated, if you > > have more of such items to deal with. Sure, I could run this separately, > > but this isn't the real McCoy either with a complicated tree structure. > > > > What are your favorite ways of doing such tasks with rsync? > > > > Thanks in advance, > > Pete -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Advanced rsync includes and excludes
Hi, I'm a happy camper @ rsync (and rsnapshot) since years. Thanks for this major piece of software. In an attempt to reorganize my rsnapshot backups, I stumbled across an issue, that I'm trying to seeking a more sophisticated solution here. Given, I have a deeply branched tree, where I would like to include a specific directory deep under, while excluding anything else on that path, I find myself doing: + /some/ + /some/very/ + /some/very/deep/ + /some/very/deep/path/ + /some/very/deep/path/to/ + /some/very/deep/path/to/save/ - /some/very/deep/path/to/* - /some/very/deep/path/* - /some/very/deep/* - /some/very/* - /some/* While it works, it feels rather awkward and gets very complicated, if you have more of such items to deal with. Sure, I could run this separately, but this isn't the real McCoy either with a complicated tree structure. What are your favorite ways of doing such tasks with rsync? Thanks in advance, Pete -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
A small addition to the manpage and "clone mode" suggestion
Dear developers, In the rsync's manpage, there is a line describing the "archive mode" option: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) Archive mode means that all data is backed up sensibly while the same doesn't apply for all metadata (hence the mentioned -H, -A and -X options). There is another similar "special file system option" as -H, -A and -X are, and that is the -S option for handling sparse files. While the negated options in the brackets are not strictly required to be mentioned here in the manpage, they provide an useful information to the user who actually wants to perform as synced copy as possible, meaning including all of the special file properties (for example while trying to migrate to another filesystem). I simply propose that the -S option is to be added for the line to read: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X,-S) And my second suggestion is an appeal to discuss a possibility of adding something such as "clone mode", which is actually the archive mode plus the family of special file properties, so: "rsync -aHAXS ..." rsync's name itself implies that it is capable of doing any file sync so why wouldn't there be such option? Please don't take the report as too picky. It can only be a good thing that rsync is so mature piece of software that we can discuss such minor issues. And thank you for all the great work! Best regards, Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
syntax of a local Windows path
Hi! We want to migrate from a cygwin based rsync to a MingW based version. 'rsync' is used here to install/update a complete TeX tree (see http://wiki.contextgarden.net/ConTeXt_Standalone) from a remote server to a local PC. On Windows based machines the destination is currently something like /cygdrive/c/path_to_directory What is the correct way to do the same with the MingW version? I've tried some options, but I'm not quite sure, what is the correct syntax here. 1./c/path_to_directory 2./c:/path_to_directory 3.'--files-from' should work too, but is a bit cumbersome My preferred solution would be the first (without colon). Is this ok to use? I couldn't find a satisfying answer in the documentation or the archive and I have to be sure about this. Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cache file list in daemon mode?
On 8/2/2012 7:30 PM, Jason Haar wrote: So what you're really saying is gluster is quite slow at doing recursive directory listings, so how about just using find on the real backend bricks to find the files that have changed since last run, merge those listings together (to get rid of dupes) and then get rsync to just update those against the gluster service? Yes, that's pretty much the solution we're converging on. It's probably 20 lines max*, most of which is anticipating race conditions, but I'm congenitally lazy* and thought I might not be the first person with these requirements - not getting as specific as Gluster, for sure, but still with enough files and slowness on the target to raise the question of why not operate in some sort of incremental mode. I dislike having to think of every possible race condition. * = Perl -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Cache file list in daemon mode?
Hello. I suspect that what I want to do is not possible with rsync, but this is the best place to double-check. We are pushing files to a remote target that stores them on a very slow network file system. There are also over a million files on the target. Consequently, running rsync to push an update takes hours while the remote side enumerates and stats all those files. I thought, that wouldn't be necessary if the remote side was running rsync in daemon mode, and that it only built its internal map of the files there once after startup, thereafter updating that in-memory list with every push it receives. For that to work, there would have to be some flag I could set to promise rsyncd that no files in the target would be updated through any means other than rsyncd. I looked for such an option, did some experimenting with write-only targets, straced the daemon and saw I wasn't getting anywhere. Our alternatives are options like building a list of what has changed and copying only those things across. There's some housekeeping and race condition avoidance in there that makes it more than a couple of simple commands and replicates some of what rsync knows how to do already. By any chance is what I want to do possible with rsyncd, or some other tool? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cache file list in daemon mode?
We're considering that, but it's Gluster, not NFS, and it's peer-to-peer, not client-server. Options in that direction start getting more complicated than the 'find -mtime... scp' approach pretty fast. On 8/2/2012 3:43 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 is it possible to talk directly to the NFS server via rsyncd or rsync over ssh? Eliminating the extra hop through a network mount should make a big difference. On 08/02/12 18:30, Peter Scott wrote: Hello. I suspect that what I want to do is not possible with rsync, but this is the best place to double-check. We are pushing files to a remote target that stores them on a very slow network file system. There are also over a million files on the target. Consequently, running rsync to push an update takes hours while the remote side enumerates and stats all those files. I thought, that wouldn't be necessary if the remote side was running rsync in daemon mode, and that it only built its internal map of the files there once after startup, thereafter updating that in-memory list with every push it receives. For that to work, there would have to be some flag I could set to promise rsyncd that no files in the target would be updated through any means other than rsyncd. I looked for such an option, did some experimenting with write-only targets, straced the daemon and saw I wasn't getting anywhere. Our alternatives are options like building a list of what has changed and copying only those things across. There's some housekeeping and race condition avoidance in there that makes it more than a couple of simple commands and replicates some of what rsync knows how to do already. By any chance is what I want to do possible with rsyncd, or some other tool? - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlAbApcACgkQVKC1jlbQAQcYHgCg6HYJATQn4kf/duAC+owxPq5w TNAAoNFXN8MAcKbBEbXk2cKgF/4MNgE5 =D9DM -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
handling of final partial block in rsync
Greetings, I have been using the rsync C source as a guide for porting the rsync algorithm to Java. This has been successful, but there is one low level detail I am having some issues with. Assuming the default block size of 700 bytes, when stepping through the file and generating checksums for each block it is very likely that there will be a partial block at the end of the file. Right now I am just generating a checksum for that block and including it in the signature. There will end up being a match for this block if the source file has the same partial block at the very end of the file, but it won't match anywhere else in the file. If I don't generate a checksum for the partial block then it will never match even when present at the end of the source file, which increases the size of the patch file. I've read the thesis and the source again, and it looks like rsync is generating a signature on the final partial block. Would someone be able to confirm this, and is rsync handling this as I describe? Many thanks in advance, Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
show how much bytes transfered per rsynced file?
We use rsync a lot to backup various stuff over bigger distances and its my project to look deeper into this backups to optimize backup. While its easy to make rsync print the size of the files it is transferring I didnt figure out how to determine how much bytes are actually transferred per file which is even more interesting. If I sync a whole data-structure with 100.000 files lets assume 1000 files are actually changed and they have 10M - 1G each. The Delta-algorithm only transfers the amount of data that has really changed (at least it tries and it does a good job) and I would want to know how much bytes this are per file. rsync only tells me the total amount of transfered bytes but not per file. The reason for this is that I need to find out which files causes most of our backuptraffic and then probably look for different and more efficient ways on syncing this files. thnx a lot, peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete won't delete certain files
Hi, I still couldn't find a really simple test case but at least I can provide some more details regarding this obscure issue: - The problem only occurs when there is more than 1 file system to sync. When I only backup /, everything is fine, when I add another source file system, nothing below /var will be deleted (I still don't have a clue, why this occurs on /var but not on other paths) - The issue has been around for quite a while: It has been introduced by some change between rsync 2.6.4 and 2.6.5 Regards, Peter Daum Peter Daum wrote: as I just discovered, in recent versions of rsync the --delete option seems to not work in certain cases. I couldn't come up with a simple scenario to reproduce the problem, so I'll have to tell the whole story: I use rsync to regularly backup a system onto other disks. The filesystems I want to save are mounted on /, /usr/ and /data, the target is below /backup (and has /backup/usr and /backup/data on separate file systems). the rsync command line looks like this: rsync [other options ...] -a --delete / /usr /data /backup/ For some reason, it seems like old data below /backup/var will never be deleted again I don't see any reason why /var would be treated somehow special, but I couldn't find another directory where this phenomenon also occurs; the other options don't include anything referring to /var. New data below /var will be synced, but when files below /var are deleted, they will stay on /backup/var. I don't know when this started (fortunately, I didn't need the backup data recently ;-), I just discovered it when the partition on /backup/ ran out of space. This happened with rsync 3.03; I also tried 3.05 with the same result. When I tried the same command line using rsync 2.6.3, the leftover data on /backup/var/ was finally deleted. However, I would rather use a recent version. Any idea what is going on? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete won't delete certain files
Peter Daum wrote: - The problem only occurs when there is more than 1 file system to sync. When I only backup /, everything is fine, when I add another source file system, nothing below /var will be deleted (I still don't have a clue, why this occurs on /var but not on other paths) ... a little detail, that I forgot to explicitly mention: the other options used include of course --one-file-system; I need a full backup and find it very error-prone to specify all exceptions (like /proc /sys the directory where the backup disk is mounted ...) with --exclude -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete won't delete certain files
Hi, Izidor Jerebic wrote: Are there any errors while doing the rsync? I have recently discovered that to create identical copy on mac os x (in the presence of any errors), you need to additionally specify --ignore-errors and --force-delete, otherwise nothing will be deleted (it seems that as soon as any error is encountered, delete is switched off). There is no indication that anything might be wrong - until the last backup failed because the space was exhausted and I investigated, how this could happen, I was confident to have an identical copy of my data on the 2nd drive (A pretty unsettling thought ...) For some reason, rsync 3.x just does not seem to think it should delete those files ... Regards, Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete won't delete certain files
Hi, as I just discovered, in recent versions of rsync the --delete option seems to not work in certain cases. I couldn't come up with a simple scenario to reproduce the problem, so I'll have to tell the whole story: I use rsync to regularly backup a system onto other disks. The filesystems I want to save are mounted on /, /usr/ and /data, the target is below /backup (and has /backup/usr and /backup/data on separate file systems). the rsync command line looks like this: rsync [other options ...] -a --delete / /usr /data /backup/ For some reason, it seems like old data below /backup/var will never be deleted again I don't see any reason why /var would be treated somehow special, but I couldn't find another directory where this phenomenon also occurs; the other options don't include anything referring to /var. New data below /var will be synced, but when files below /var are deleted, they will stay on /backup/var. I don't know when this started (fortunately, I didn't need the backup data recently ;-), I just discovered it when the partition on /backup/ ran out of space. This happened with rsync 3.03; I also tried 3.05 with the same result. When I tried the same command line using rsync 2.6.3, the leftover data on /backup/var/ was finally deleted. However, I would rather use a recent version. Any idea what is going on? Regards, Peter Daum -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
proposal to speed rsync with lots of files
Hello, I have followed the discussion of speeding up rsync when there are lots of files, and I have a proposal which I think would greatly speed rsync when doing routine mirroring of large filesystems. One of the speed-limiting issues with rsync is having to send huge file lists when mirroring large file systems, even for incremental updates where only a small part of the file system might have changed. My proposal is to first send a checksum of the file list for each directory. If is found to be identical to the same checksum on the remote side then the list need not be sent for that directory! That would reduce the size of the file list greatly when there are directories containing many files which do not change from on rsync to the next. Here's an example: remotelocal dir1 dir1 - file list checksum same as on remote - don't send file list for dir1 dir2 dir2 - file list checksum same as on remote - don't send file list for dir2 dir3 dir3 - file list checksum different from remote - send file list for dir3 It might even be possible to use the rsync checksum algorithm on the directory lists themselves to determine which portion of the directory lists to send, in the case of directories which nearly identical. I would appreciate hearing from rsync developers if this feasible with the current implementation and if they think it would help. Thanks, Peter Salameh -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: proposal to speed rsync with lots of files
Kyle Lanclos wrote: Peter Salameh wrote: One of the speed-limiting issues with rsync is having to send huge file lists when mirroring large file systems, even for incremental updates where only a small part of the file system might have changed. Personally, I find that the sending of the file list, whether incremental or otherwise, takes orders of magnitude less time than the construction of the file list in the first place. The act of stat'ing millions of files takes an enormous amount of time in comparison to just about anything else, assuming that you are not on a low-bandwidth link. Many of the locations I mirror do have low-bandwidth links, and I imagine that a sizable fraction of rsync users have issues with bandwidth at least some of the time. I have not studied rsync internals either, but I imagine using a checksum to avoid sending directory file lists unnecessarily would not hurt anything and would help folks with lower bandwidth. Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync 3.0.4 ACL corruption
Hi, I use rsync 3.0.4 on two opensuse 11 machines. Every night, a big tree of machine A is synced to machine B. These machines are samba PDC and BDC, users and groups are ldap-based, and ACLs are heavily used. There are about 2.8 million files and dirs, 2.2 terabytes of data, and a complete ACL list produced with getfacl has some 600 megabytes. I've just noticed that ACLs get partially corrupted on the receiving side. This is my command line, running as root: /usr/bin/rsync --quiet --links --numeric-ids --acls --perms --times --recursive --owner --group --delete-during --ignore-errors --backup --backup-dir=/wzb/backup/wzb= --password-file=/etc/wzb/rsync/password --filter '- /backup/' rsync://selene.wzb.eu/wzb /wzb /wzb has subtrees user, group, software. Wrong ACLs show up in several places. /wzb/user (home directories) seems ok. Both /wzb/group and /wzb/software have wrong subentries. In /wzb/software, all subentry ACLs, both files and dirs, a partially (but not completely) wrong, and all in the same way. Example: directory /wzb/software/aida This is as should be: # file: aida # owner: root # group: root user::rwx group::r-x group:users:r-x mask::rwx other::--- default:user::rwx default:group::r-x default:group:users:r-x default:mask::rwx default:other::--- This is what happens: # file: aida # owner: root # group: root user::rwx user:spura:rwx group::--- mask::rwx other::--- default:user::rwx default:group::r-x default:group:users:r-x default:mask::rwx default:other::--- It looks like the lines group::r-x group:users:r-x are replaced by user:spura:rwx group::--- The latter is a (partial) ACL from /wzb/user/spura, the spura person's home dir. The very same thing happens to all files and dirs below /wzb/software. User spura is completely wrong here. Further observations: - At the next rsync run, same thing occurs but with a different(!) user. - The number of ACL entries remains unchanged. - Only existing users show up in the wrong ACLs. - rsync happens to backup the /wzb/user and /wzb/group subtrees first, making it understandable that the wrong ACLs parts are somewhere in rsync's memory. - If I rsync the /wzb/software branch alone, everything is ok. - It does not matter whether or not I use --numeric-ids. Peter Rindfuss -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
tricky rsync setup quit working
Hi, some time ago I had started working on a rsync-based backup system. After I longer break; I now tried to continue this project but can't get it to work anymore. Maybe somebody here has any idea. The basic setup looks like this: client: $rsync $rsync_opts --rsh='ssh -i $ssh_key' $filesystems $bak_host::backup_module/path on the server side, the ssh key triggered a shell script that generated a configuration like [backup_module] path=/some/path use chroot=1 ... and eventually invoked rsync with the command line: rsync --daemon --no-detach --config $cfg_file Back then, this setup actually worked: rsync read the configuration file and started its work using the ssh-tunneled network connection. In the syslog, this looked like this: rsyncd: connect from UNKNOWN (localhost) rsyncd: rsync to backup_module/hostname from UNKNOWN (localhost) ... Now the same setup just causes rsync to listen on a new network socket After looking through the documentation and digging in the rsync sources, it doesn't look like this could ever work, but thrust me - it did! (It looks like the only way to get rsync to honor the configuration file is to specify --daemon, which causes it to listen on the network instead of doing its work and the only way to do it's job on the existing network connection is to specify --server, in which case it doesn't honor the configuration file. Unfortunately, some options like chroot can only be specified via configuration file) Unfortunately, I can't figure out what changed in the meantime that broke this setup, Does anybody have an idea? Any help would be greatly appreciated! Peter Daum -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync of LVM Snapshots copies whole file
Hello does nobody has a clue about this? Best regards Peter Peter P GMX schrieb: Hello Paul, here is the status of the transfer. If I understand it right, about 3% of the data was changed (literal data against matched data). It took about 1h40min to transfer ~250MB of data. But to be honest, I do not get a real clue out of this status report. E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file transfer) I end up at 40 minutes. So almost half of the complete data transfer time of this volume is needed, although this drive is rather passive and there were almost no changes on the drive between the last sync. (The machines are a AMD X2 3800 on one originating side and a Atom 1,6GHz on the destination side, Atom pulls from AMDs side). Best regards Peter receiving file list ... 13 files to consider mit_backup/Debian40.vdi 1395660800 100%5.66MB/s0:03:55 (xfer#1, to-check=11/13) mit_backup/Laptop_D_10GB.vdi 10734313984 100%6.74MB/s0:25:19 (xfer#2, to-check=10/13) mit_backup/RubyServer_1_9GB.vdi 2039489024 100%8.15MB/s0:03:58 (xfer#3, to-check=8/13) mit_backup/_web.vdi 2874171904 100%6.17MB/s0:07:24 (xfer#4, to-check=7/13) mit_backup/Ubuntu_8_04_Development.vdi 4367340032 100%6.80MB/s0:10:12 (xfer#5, to-check=6/13) mit_backup/Ubuntu__3GB.vdi 3221238272 100%5.61MB/s0:09:07 (xfer#6, to-check=5/13) mit_backup/W2000_6GB.vdi 6442476032 100%7.49MB/s0:13:39 (xfer#7, to-check=4/13) mit_backup/.vdi 4294984192 100%8.05MB/s0:08:28 (xfer#8, to-check=3/13) Number of files: 13 Number of files transferred: 8 Total file size: 47.710.416.177 bytes Total transferred file size: 35.369.674.240 bytes Literal data: 1.181.132.240 bytes Matched data: 34.188.542.000 bytes File list size: 530 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 3.984.972 Total bytes received: 249.776.322 sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec total size is 47.710.416.177 speedup is 188.01 Paul Slootman schrieb: On Wed 20 Aug 2008, Peter P GMX wrote: Hallo Paul, please keep replies to list messages on the list, others may also be able to help. here is the call and the protocol for the transfer of one file. For me it seems that the whole file is transferred, right? Nothing in the output indicates that all the file data is being transferred; only that the file is indeed being transferred. The output of --stats will help in showing how much file data is actually transferred. Paul Slootman sudo -u rsyncbackup rsync -avzP --exclude-from /home/rsyncbackup/excludefile -e ssh 192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup /mnt/crypteddevice/linuxX2/mit_backup receiving file list ... 14 files to consider mit_backup/ mit_backup/Debian40_.vdi 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1% 10.02MB/s 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s 0:02:52 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43 50797900 3% 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523 4% 5.27MB/s 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5% 5.60MB/s 0:03:50 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s 0:03:15 99630301 7% 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44 113938846 8% 7.07MB/s 0:02:56 125802777 9% 8.53MB/s 0:02:25 132129910 9% 7.57MB/s 0:02:42 139575721 10% 7.58MB/s 0:02:41 144926021 10% 7.23MB/s 0:02:49 149718979 10% 5.61MB/s 0:03:36 154452981 11% 5.32MB/s 0:03:47 160614584 11% 5.00MB/s 0:04:01 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s 0:03:58 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22 202404006 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02 216901117 15% 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44 233657909 16% 7.19MB/s 0:02:37 239352121 17% 6.67MB/s 0:02:49 244896643 17% 6.44MB/s 0:02:54 249979132 17% 5.75MB/s 0:03:14 255669742 18% 5.23MB/s 0:03:32 262874393 18% 5.58MB/s 0:03:18 275524040 19% 7.27MB/s 0:02:30 294537718 21% 10.58MB/s 0:01:41 306682448 21% 12.06MB/s 0:01:28 316879483 22% 12.74MB/s 0:01:22 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s 0:02:03 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24 394514192 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03 409033762 29% 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58 429225330 30% 7.96MB/s 0:01:58 443550036 31% 10.15MB/s 0:01:31 452116589 32% 9.86MB/s 0:01:33 457430449 32% 8.85MB/s 0:01:43 465762610 33% 8.68MB/s 0:01:44 471632053 33% 6.66MB/s 0:02:15 477048104 34% 5.91MB/s 0:02:31 483430930 34% 5.93MB/s 0:02:30 490941562 35% 5.74MB/s 0:02:33
Re: Rsync of LVM Snapshots copies whole file
Hello Paul, here is the status of the transfer. If I understand it right, about 3% of the data was changed (literal data against matched data). It took about 1h40min to transfer ~250MB of data. But to be honest, I do not get a real clue out of this status report. E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file transfer) I end up at 40 minutes. So almost half of the complete data transfer time of this volume is needed, although this drive is rather passive and there were almost no changes on the drive between the last sync. (The machines are a AMD X2 3800 on one originating side and a Atom 1,6GHz on the destination side, Atom pulls from AMDs side). Best regards Peter receiving file list ... 13 files to consider mit_backup/Debian40.vdi 1395660800 100%5.66MB/s0:03:55 (xfer#1, to-check=11/13) mit_backup/Laptop_D_10GB.vdi 10734313984 100%6.74MB/s0:25:19 (xfer#2, to-check=10/13) mit_backup/RubyServer_1_9GB.vdi 2039489024 100%8.15MB/s0:03:58 (xfer#3, to-check=8/13) mit_backup/_web.vdi 2874171904 100%6.17MB/s0:07:24 (xfer#4, to-check=7/13) mit_backup/Ubuntu_8_04_Development.vdi 4367340032 100%6.80MB/s0:10:12 (xfer#5, to-check=6/13) mit_backup/Ubuntu__3GB.vdi 3221238272 100%5.61MB/s0:09:07 (xfer#6, to-check=5/13) mit_backup/W2000_6GB.vdi 6442476032 100%7.49MB/s0:13:39 (xfer#7, to-check=4/13) mit_backup/.vdi 4294984192 100%8.05MB/s0:08:28 (xfer#8, to-check=3/13) Number of files: 13 Number of files transferred: 8 Total file size: 47.710.416.177 bytes Total transferred file size: 35.369.674.240 bytes Literal data: 1.181.132.240 bytes Matched data: 34.188.542.000 bytes File list size: 530 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 3.984.972 Total bytes received: 249.776.322 sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec total size is 47.710.416.177 speedup is 188.01 Paul Slootman schrieb: On Wed 20 Aug 2008, Peter P GMX wrote: Hallo Paul, please keep replies to list messages on the list, others may also be able to help. here is the call and the protocol for the transfer of one file. For me it seems that the whole file is transferred, right? Nothing in the output indicates that all the file data is being transferred; only that the file is indeed being transferred. The output of --stats will help in showing how much file data is actually transferred. Paul Slootman sudo -u rsyncbackup rsync -avzP --exclude-from /home/rsyncbackup/excludefile -e ssh 192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup /mnt/crypteddevice/linuxX2/mit_backup receiving file list ... 14 files to consider mit_backup/ mit_backup/Debian40_.vdi 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1% 10.02MB/s 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s 0:02:52 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43 50797900 3% 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523 4% 5.27MB/s 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5% 5.60MB/s 0:03:50 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s 0:03:15 99630301 7% 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44 113938846 8% 7.07MB/s 0:02:56 125802777 9% 8.53MB/s 0:02:25 132129910 9% 7.57MB/s 0:02:42 139575721 10% 7.58MB/s 0:02:41 144926021 10% 7.23MB/s 0:02:49 149718979 10% 5.61MB/s 0:03:36 154452981 11% 5.32MB/s 0:03:47 160614584 11% 5.00MB/s 0:04:01 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s 0:03:58 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22 202404006 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02 216901117 15% 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44 233657909 16% 7.19MB/s 0:02:37 239352121 17% 6.67MB/s 0:02:49 244896643 17% 6.44MB/s 0:02:54 249979132 17% 5.75MB/s 0:03:14 255669742 18% 5.23MB/s 0:03:32 262874393 18% 5.58MB/s 0:03:18 275524040 19% 7.27MB/s 0:02:30 294537718 21% 10.58MB/s 0:01:41 306682448 21% 12.06MB/s 0:01:28 316879483 22% 12.74MB/s 0:01:22 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s 0:02:03 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24 394514192 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03 409033762 29% 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58 429225330 30% 7.96MB/s 0:01:58 443550036 31% 10.15MB/s 0:01:31 452116589 32% 9.86MB/s 0:01:33 457430449 32% 8.85MB/s 0:01:43 465762610 33% 8.68MB/s 0:01:44 471632053 33% 6.66MB/s 0:02:15 477048104 34% 5.91MB/s 0:02:31 483430930 34% 5.93MB/s 0:02:30 490941562 35% 5.74MB/s 0:02:33 496540662 35% 5.70MB/s 0:02:34 501713859 35% 5.64MB/s 0:02:34 508037197 36% 5.83MB/s 0
Rsync of LVM Snapshots copies whole file
Hello, anybody has a clue why this happens? The scenario: Each night I create LVM snapshot of my volume which contains some virtual disk images (VirtualBox VDIs). Then I rsync the content of one of the snapshot directories to a backup server. Each file size is between 1.3 GB and 6GB. I run the following commmand: sudo -u rsyncbackup rsync -avzP --exclude-from /home/rsyncbackup/excludefile -e ssh 192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup /mnt/crypteddevice/linuxX2/mit_backup The source directory is part of the LVM snapshot. The destination directory is a truecrypt device which is already mounted. What I would expect: Only the differences of the virtual disk files should be transfered. What happens: I discover that always the whole file is transfered. Even if the original virtual machine is idle (so only some log files should have been enhaned in the virtual disk images). I do the following: - take LVM snapshot - Rsync the desired VDI file (2GB are transferred), it takes 10min - destroy LVM snapshot - take new LVM snapshot with same parameters as before - Rsync the same file (2GB are transferred), it takes 10min Am I doing something wrong? Best regards Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync roadmap and version numbering (3.1.0 vs 3.0.4)
I'd like to see the create times and fileflags patches included so all metadata tests pass with backup bouncer 'out of the box' on Mac OS X. Those patches are Mac-specific, so I'm pretty sure they won't go into the main rsync, but it would make sense to include them in packagings of rsync for Mac OS X. But can't they be automatically included by the configure and make steps of the compilation even if its too messy to have them in the mainline code using #ifdefs? After all, there are a lot more Mac boxes out there than Linux, so maybe the patches ought to be the other way around :) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync-3.0.2 -- two build problems
On Tue, 17 Jun 2008, Wayne Davison wrote: On Thu, Jun 05, 2008 at 08:22:09AM +0200, Peter Breitenlohner wrote: However, cp -p and touch -r round to the nearest second. Can you try out the attached patch and let me know if it fixes the rounding issues? I have just rebuilt rsync-3.0.2 with your patch (and have removed proto.h-tstamp from the source tree in order to prevent continuous recompilation). Your patch works nicely: make check used to have a failure rate of about 50%; now I had twelve successful make check runs. Thanks a lot Peter Breitenlohner [EMAIL PROTECTED] -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync-3.0.2 -- two build problems
On Wed, 4 Jun 2008, Wayne Davison wrote: On Mon, Jun 02, 2008 at 07:31:02PM +0200, Peter Breitenlohner wrote: (1) With the Linux kernel =2.6.20, make check occasionally fails, due to subsecond timestamps sometimes being truncated and sometimes being rounded upwards (both on i686 and x86_64). Rsync doesn't do any rounding, so I don't know where the inconsistency would be. We just set the desination time based on the mtime value, and the tls program outputs an interpretation of the mtime value. Yesterday I did some tests (on jfs with subsecond timestamps, ext2 doesn't have them): True, rsync just truncates, i.e. ignores the fractional seconds, as do freerdist-0.92 and (GNU) tar. However, cp -p and touch -r round to the nearest second. So it seems the failures are caused by comparing a directory created by cp with one created by rsync. Since spurious failures from make check defeat the purpose of a test suite, it might be a good idea to avoid such comparisions. regards Peter Breitenlohner [EMAIL PROTECTED] -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync-3.0.2 -- two build problems
Hi, when I tried to build rsync-3.0.2 on i686-linux/gnu, I noticed 2 problems: (1) With the Linux kernel =2.6.20, make check occasionally fails, due to subsecond timestamps sometimes being truncated and sometimes being rounded upwards (both on i686 and x86_64). Attached are two files demonstrating the problem: rsync-3.0.2-check-out with the relevant output from make check, and rsync-3.0.2-check-ls-full-time with the output from ls -l --full-time `find exclude -name file4.junk` (in the testtmp directory, coreutils-6.9). This same problem occured already for 3.0.0 and 3.0.1 as well as when rebuilding 2.6.9 with the newer Linux kernel. All this somehow defeats the purpose of a test suite, and it would be extremely nice if such discrepancies could be avoided. (2) I first run ../rsync-3.0.2/configure ; make then make check and finally (as root) make install For rsync-3.0.2, make check recompiles everything (not just the special tools not needed otherwise), and make install again compiles everything. The later is really bad, because root may not have write access to the build directory (e.g., on a root-squashed nfs filesystem). This problem was not present for rsync-3.0.1 or earlier. It would be really great if both problem could be fixed for 3.0.3. please reply directly to me, since I am not subscribed to this list. Regards Peter Breitenlohner [EMAIL PROTECTED]check how the directory listings compare with diff: --- /.th/thsrc/archive/rsync/rsync-3.0.2-build-linux-gnu/testtmp/exclude/ls-from 2008-06-02 18:48:09.0 +0200 +++ /.th/thsrc/archive/rsync/rsync-3.0.2-build-linux-gnu/testtmp/exclude/ls-to 2008-06-02 18:48:09.0 +0200 @@ -14,9 +14,9 @@ -rw-r--r--9 9000.90001 2008-06-02 16:48:07 ./bar/down/to/foo/.filt2 -rw-r--r--7 9000.90001 2008-06-02 16:48:07 ./bar/down/to/foo/file1 -rw-r--r--5 9000.90001 2008-06-02 16:48:07 ./bar/down/to/foo/file4 --rw-r--r-- 18 9000.90001 2008-06-02 16:48:08 ./bar/down/to/foo/file4.junk +-rw-r--r-- 18 9000.90001 2008-06-02 16:48:07 ./bar/down/to/foo/file4.junk l-0 0.0 1 ./bar/down/to/foo/sym - too --rw-r--r--8 9000.90001 2008-06-02 16:48:08 ./bar/down/to/foo/to +-rw-r--r--8 9000.90001 2008-06-02 16:48:07 ./bar/down/to/foo/to drwxr-xr-x0 9000.90002 2008-06-02 16:48:07 ./bar/down/to/foo/too drwxr-xr-x0 9000.90003 2008-06-02 16:48:07 ./foo -rw-r--r-- 23 9000.90001 2008-06-02 16:48:07 ./foo/.filt - check how the files compare with diff: - - exclude log ends FAILexclude -rw-r--r-- 1 peb THEORY 18 2008-06-02 18:48:08.0 +0200 exclude/chk/bar/down/to/foo/file4.junk -rw-r--r-- 1 peb THEORY 18 2008-06-02 18:48:07.880843508 +0200 exclude/from/bar/down/to/foo/file4.junk -rw-r--r-- 1 peb THEORY 18 2008-06-02 18:48:07.0 +0200 exclude/to/bar/down/to/foo/file4.junk -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync bandwidth usage
How do you call rsync? rsync -av --delete --perms --acls ... You left out the rest. Specifically, what protocol are you using over the ethernet? Are you using an rsync server, ssh, NFS, SMB, or what? A straight rsync server is by far the fastest, I've found. So you'd do something like so: rsync -av ... /local/path/ hostname::modulename/remote/path/ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync bandwidth usage
As an example, I transfered 1930420 KBytes with rsync between two 2.8 GHz Intel P4 machines over gigabit ethernet and got this: transfer method:ssh rshnfs rsync wall time in sec 130.01 79.77 176.03 74.92 MBit/sec116 189 86201 Mind you, this is no benchmark as I only did each on once and wasn't particularly careful that nothing else was running on the two machines or on the network. Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
using rsync with scripts (cronjobs) and automated backups
Hello all, I am wondering if it would be possible to write a script or a cronjob in linux using Rsync to run an automated backup of a server, or serveral servers if possible. I am very new with writing scripts and such, so any help or suggestions with how to get started would be great!!! Thanks ahead of time for the help!!! - Computers are like air conditioners. They both dont work, if you open windows. -- View this message in context: http://www.nabble.com/using-rsync-with-scripts-%28cronjobs%29-and-automated-backups-tp16824035p16824035.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Suggestions for basic rsync configuration
Wayne Davison-2 wrote: Search for the string admins in the config file. You presumably set the gid in more than one spot, such as in the module's settings. I have checked the config file and have not found any duplicates. Here is my config file: uid = user gid = users read only = true use chroot = true transfer logging = true log format = %h %o %f %l %b log file = /var/log/rsyncd.log hosts allow = * slp refresh = 300 [remote] path = /rsync/backup_out comment = Verzeichnis fuer rsync auth users = root secrets file = /etc/rsyncd.secrets I am also having problems again with the daemon, it is not responding to repests... instead it just times out. It works fine without any problems when I transfer files without the daemon. I am starting to loose myself here I think with the daemon thing. With everything configured on the Firewall correctly, and with all ports opened, no idea what I am still missing. I think I will just go without the daemon and just use the regular thing without the daemon. Is there anything that you can suggest from my config file? Everthing looks ok to me. - Computers are like air conditioners. They both dont work, if you open windows. -- View this message in context: http://www.nabble.com/Suggestions-for-basic-rsync-configuration-tp16025015p16417871.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Suggestions for basic rsync configuration
Matt McCutchen-7 wrote: I don't know, but here are a few things you can try. First, confirm that the daemon is accepting connections by running nc localhost 873 on the remote machine. You should see the daemon's greeting, beginning with @RSYNCD. Then, attempt to the daemon by running nc remote-server-ip 873 on the local machine. If the first command works but the second times out, then there is something blocking the connection. The most common cause would be a firewall, but you say you've opened the port in the firewall, so I don't know what it could be. I got the daemon back up, but I am getting a new error after going through the authorizing process. After putting in the password for authentification, I get the following error: # rsync -zav --progress realperson@remote-server-ip::realperson /random/file-or-directory/ Password: @ERROR: invalid gid admins rsync error: error starting client-server protocol (code 5) at main.c(1383) [receiver=2.6.9] I have users set as the gid in my rsync config file. I have also ran rsync --daemon --config=rsyncd.conf to make sure that rsync was lookin at the right config file. Do you have an idea why this returning this error, even though is it otherwise in my config file? - Computers are like air conditioners. They both dont work, if you open windows. -- View this message in context: http://www.nabble.com/Suggestions-for-basic-rsync-configuration-tp16025015p16301526.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Suggestions for basic rsync configuration
Paul Slootman-5 wrote: # rsync -zav --progress root@remote-server-ip::realperson /random/file-or-directory rsync: failed to connect to remote-server-ip: Connection timed out (110) A timeout would indicate a firewall problem, the rsync port (873) is probably not allowed. Yes I understand that ssh is not involved anymore here. I opened up the port on the firewall, so that problem is out of the way. The weird thing is that when I use the following command I dont get an error, I only get the timeout when i use the daemon to connect the remote server: # rsync -zav remote-server-ip:/random/file-or-directory/ /random/file-or-directory/ Password: With this command I can transfer files without any errors, it works perfectly. But when I use call the daemon on the remote server, it just sits there and does nothing until the connection times out. I haved double checked the firewall and everything is set up the way it should be. here my config files just in case there is something wrong with them (they look right ok to me): uid = root gid = roots read only = true use chroot = true transfer logging = true log format = %h %o %f %l %b log file = /var/log/rsyncd.log hosts allow = * slp refresh = 300 [realperson] path = /srv/www/vhosts/rsync/backup_out comment = Verzeichnis fuer rsync auth users = root secrets file = /etc/rsyncd.secrets Is there anything else that I can do here? [EDIT: I forgot to say that I will not be back until next tuesday. I will get back to this then... Happy Easter!!!] - Computers are like air conditioners. They both dont work, if you open windows. -- View this message in context: http://www.nabble.com/Suggestions-for-basic-rsync-configuration-tp16025015p16181496.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Suggestions for basic rsync configuration
Now I have a new situation, the linux OS was reinstalled (it crashed after a bad restart) on the remote linux box where I was working with rsync. I have reconfigured everything back to the way tit was before and I am now getting a new error with the following command: # rsync -zav --progress remote-server-ip:/random/file-or-directory /random/file-or-directory @@@ @WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is e4:00:85:b2:5b (I deleted a section from this value before posting). Please contact your system administrator. Add correct host key in /root/.ssh/known_hosts to get rid of this message. Offending key in /root/.ssh/known_hosts:1 RSA host key for remote-server-ip has changed and you have requested strict checking. Host key verification failed. rsync: connection unexpectedly closed (0 bytes received so far) [receiver] rsync error: unexplained error (code 255) at io.c(453) [receiver=2.6.9] And when I run the command to access my daemon on the remote server it just sits there and does nothing for about 2 minutes, and then it drops the connection with this error. I have enabled rsync on the network services again through yast and all that. I have checked /root/.ssh/known_hosts, and the folder doesnt exist under the root directory. I also installed ssl and the ssl-devel again on the server. Is there anything that I am missing so far? How can I go about creating/editing this file so I can get rsync up and running again? - Computers are like air conditioners. They both dont work, if you open windows. -- View this message in context: http://www.nabble.com/Suggestions-for-basic-rsync-configuration-tp16025015p16142991.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Suggestions for basic rsync configuration
Matt McCutchen-7 wrote: hosts allow = trusted.hosts BTW, the hosts allow field needs to contain the actual list of trusted hosts, not the name of a file holding the list. So now I have tried the daemon again, and got a password prompt, which is better. I edited the host allow = * so that it would allow all hosts. But I tried at first adding the IP Address of my local router, and it didnt work. How can I configure the hosts allow field correctly to allow any requests from my network. Here is the error that I got: # rsync -zav remote-server-ip::realperson2 /random/file-or-directory Password: @ERROR: auth failed on module realperson2 rsync error: error starting client-server protocol (code 5) at main.c(1383) [receiver=2.6.9] I know that I had the password right, I even checked the rsync.secret file to double check. Do I have to have an actual domain name in order to give access to the daemon, or would putting in a IP Address be ok? - Computers are like air conditioners. They both dont work, if you open windows. -- View this message in context: http://www.nabble.com/Suggestions-for-basic-rsync-configuration-tp16025015p16118166.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Suggestions for basic rsync configuration
Matt McCutchen-7 wrote: OK, let's be clear here. If you want to start a daemon to accept connections, the command is rsync --daemon; pass a --config=FILE option if you want to use a configuration file other than the default /etc/rsyncd.conf . If you want to access an rsync daemon, put a *double* colon after the remote hostname: remote-server-ip-address::/random/file-or-directory I am now able to backup from the remote server, I have tested it several times with random file backups. Although, I am still having problems with the daemon on the remote server. I ran the command rsync --daemon on the remote server, no error there. But then I ran the following error to do a test backup and received this error: # rsync server-ip::/random/file-or-directory/ /random/file-or-directory/ ERROR: The remote path must start with a module name not a / rsync error: error starting client-server protocol (code 5) at main.c(1383) [receiver=2.6.9] What does the error mean, do I need to make a new configuration file as a module? I will post the config that I have on the remote server and on my local machine: Remote server config: uid = admin gid = admins read only = true use chroot = true transfer logging = true log format = %h %o %f %l %b log file = /var/log/rsyncd.log hosts allow = trusted.hosts slp refresh = 300 [Realperson Backups] path = /random/file-or-directory comment = for running backups auth users = admin secrets file = /etc/rsyncd.secrets [Realperson Backup Daemon] path = /random/file-or-directory/daemon comment = for running Daemon backups auth users = admin secrets file = /etc/rsyncd.secrets And the local config from my local linux machine: uid = rsync-admin gid = admins read only = true use chroot = true transfer logging = true log format = %h %o %f %l %b log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid hosts allow = trusted.hosts slp refresh = 300 [Backup Directory] comment = rsync Datensicherung Verzeichnis path = /random/file-or-directory readonly = false transfer logging = yes list = yes # auth users = root secrets file = /etc/rsyncd.secrets Is there anything that I missed in my configuration files? I was not sure whether or not I needed the second configuration, can I delete it? - Computers are like air conditioners. They both dont work, if you open windows. -- View this message in context: http://www.nabble.com/Suggestions-for-basic-rsync-configuration-tp16025015p16092572.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync man pages, rsync faqs, problem with RSYNC_PASSWORD and --password-file
Hi all, This isn't a bug, its just a lapse in the documentation. I just got a script going that repeats every 5 seconds, that uses rsync to send data from A, running Ubuntu V6.06.1, to B, running cygwin. I got stuck on the issue of preventing the password prompt from interrupting the script. The problem is that the man pages are misleading, they suggest that RSYNC_PASSWORD and --password-file are the answer. That might have been true in the past, but now they just don't work, as far as I can tell. The documentation needs to say up front that the right way to do it is to use ssh-keygen on A, to create a public key file id_dsa.pub. Then to install that key's content in a file on B, in ~/.ssh/authorized_keys. This information, in an easily found place, would save many people some time and frustration. Other than that gripe, it's great to have rsync as a stable tool. Thanks Sincerely, Peter H. -- Peter Halverson Promethean Devices [EMAIL PROTECTED] 626-287-3225 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
nfs4 acl support to be implemented
Hi List, I'm interested in implementing nfs4 acl support for rsync. Are there any pointers to start? Is anybody else working on this? -- Peter Somogyi Gamax Kft Bartok Bela ut 15/D H-1114, Budapest, Hungary e-mail: [EMAIL PROTECTED] phone: +36 1 382 5469 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem with rsync over ssh
Hello, I have the following problem: On our internet host I have running rsync-static-2.5.7-1 (I have to use the static one as our web hoster had installed a limited Suse 9.1 vserver system) On our local server which I would like to rsync I have running rsync version 2.6.8 protocol version 29. I can remotely execute applications throungh ssh without a password. But rsync won't run. On the server's side rsync is running as a deamon. On both servers I have a user copy50 who is allowed to execute /usr/bin/rsync-static (which is linked to /usr/bin/rsync). I get rsync error: unexplained error (code 255) at io.c(463) [sender=2.6.8] on the clients side. It may have to do with the diffente rsync versions but I have no other solution as our web hoster didn't provide neither an updated rsync version nor the kernel headers for this kind of Suse Linux vserver) I've googled around and found a number of similar problems but all solutions there did't fix my problem. Any clue? Best regards Peter Here's our rsyncd.conf === read only = false use chroot = false transfer logging = yes log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid hosts allow = 217.24.xx.xxx/255.255.255.248 [backup] path = /home/copy50/backup read only = false list = false I run the following command on the clients's side as user copy50 === rsync -avzP --exclude-from rsyncexclude -e ssh -l copy50 /home/copy50/backup 62.75.xx.xxx:/home/copy50/backup This is the output === $ ./rsync50.sh building file list ... 4 files to consider rsync: connection unexpectedly closed (8 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(463) [sender=2.6.8] The ssh logs says: === *Dec 11 19:59:44 vsxxx sshd[19558]: debug1: monitor_child_preauth: copy50 has been authenticated by privileged process* Dec 11 19:59:44 vsxxx sshd[19560]: debug1: PAM: reinitializing credentials Dec 11 19:59:44 vsxxx sshd[19560]: debug1: permanently_set_uid: 65007/100 *Dec 11 19:59:44 vsxxx sshd[19560]: debug1: Entering interactive session for SSH2.* Dec 11 19:59:44 vsxxx sshd[19560]: debug1: server_init_dispatch_20 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: server_input_channel_open: ctype session rchan 0 win 131072 max 32768 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: input_session_request Dec 11 19:59:44 vsxxx sshd[19560]: debug1: channel 0: new [server-session] Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_new: init Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_new: session 0 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_open: channel 0 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_open: session 0: link with channel 0 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: server_input_channel_open: confirm session Dec 11 19:59:44 vsxxx sshd[19560]: debug1: server_input_channel_req: channel 0 request exec reply 0 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_by_channel: session 0 channel 0 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_input_channel_req: session 0 req exec *Dec 11 19:59:44 vsxxx sshd[19560]: debug1: Received SIGCHLD.* Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_by_pid: pid 19561 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_exit_message: session 0 channel 0 pid 19561 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_exit_message: release channel 0 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: session_close: session 0 pid 19561 Dec 11 19:59:44 vsxxx sshd[19560]: debug1: channel 0: free: server-session, nchannels 1 Dec 11 19:59:44 vsxxx sshd[19560]: Connection closed by 217.24.xx.xxx Dec 11 19:59:44 vsxxx sshd[19560]: debug1: do_cleanup Dec 11 19:59:44 vsxxx sshd[19560]: debug1: PAM: cleanup Dec 11 19:59:44 vsxxx sshd[19560]: Closing connection to 217.24.xx.xxx Dec 11 19:59:44 vsxxx sshd[19560]: debug1: PAM: cleanup More detailed around *Received SIGCHLD *with debug level 3: === Dec 11 20:11:34 vsxxx sshd[3444]: debug1: server_input_channel_open: ctype session rchan 0 win 131072 max 32768 Dec 11 20:11:34 vsxxx sshd[3444]: debug1: input_session_request Dec 11 20:11:34 vsxxx sshd[3444]: debug1: channel 0: new [server-session] Dec 11 20:11:34 vsxxx sshd[3444]: debug1: session_new: init Dec 11 20:11:34 vsxxx sshd[3444]: debug1: session_new: session 0 Dec 11 20:11:34 vsxxx sshd[3444]: debug1: session_open: channel 0 Dec 11 20:11:34 vsxxx sshd[3444]: debug1: session_open: session 0: link with channel 0 Dec 11 20:11:34 vsxxx sshd[3444]: debug1: server_input_channel_open: confirm session Dec 11 20:11:34 vsxxx sshd[3444]: debug1: server_input_channel_req: channel 0
Unexplained error (code 24)
I am running windows xp and I suddenly have no sound, maybe there is a simply explanation? Can you give a long distance diagnosis? Thanks in advance peter -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: reducing file list bytes transferred
Thanks for that. Doesn't the delta-transfer algorithm compare the files on sender and receiver? For the file list, we would only need compare the new file list with the last one on the sender (it would be a simple matter to check that the receiver's file list is still valid with a checksum). If the same rsync command is done over and over on a huge number of files, then a command like --save_list=path/filename could be used specify where to store the list (on both ends). The delta-transfer algorithm (or some difference code specific to the file lists) could be used on the sender to find differences between the new list and the saved list. After reading the chain you referenced, I realized that the rsync command should be identical (no change in options) for this to work without side-effects, since I infer that the file list generated depends on the options. One possibility is to store the rsync command used to generate the list along with the saved list, to check that the list is still valid. I'm sure there are many rsync users out there who would benefit from this. Still rsync is greatly appreciated. Peter On 10/1/07, Peter Salameh [EMAIL PROTECTED] wrote: Has the rsync team considered an rsync option which would remember the last file list on both ends, and only send changes to the list? Perhaps a more natural approach is to use the delta-transfer algorithm to send the file list. Jamie Lokier suggested this approach here: http://lists.samba.org/archive/rsync/2007-August/018345.html Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: reducing file list bytes transferred
I agree that rsync being stateless is a good thing. I have found it very reliable and predictable for years. I certainly wouldn't change to unison (which apparently is currently not maintained) just to speed up the file list transfer. Delta-transfer of the file list seems to make good sense for rsync, especially for one-way backup involving lots of files. Is there any chance that this will happen, or any way to gauge interest in this? Peter Unison ( http://www.cis.upenn.edu/~bcpierce/unison/ ) is a stateful two-way synchronizer that does essentially this by default; you can use Unison even for your one-way copy to get the performance benefit. Rsync is meant to be stateless, so if it were enhanced to reduce the amount of file list transferred, I think delta-transferring the file list would be more in keeping with its mission than saving a last file list. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
reducing file list bytes transferred
Hello, This is my first posting to the rsync list. I mirror a database containing directories which contain a very large number of files (say 30,000), and sending the file list can often take longer than transferring the new files.(Rsync ends up sending nearly the same file list on every transfer, with only the addition of a few new files.) Has the rsync team considered an rsync option which would remember the last file list on both ends, and only send changes to the list? In my case this would reduce the number of files in the file list from 300,000 per day to under 1,000 per day. Might be a good addition to rsync-3.0. Thanks in advance. Peter Salameh [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[U] RE: RE: Rsync Errors
UNCLASSIFIED We are not using the -z or --compress option. I have turned up the logging when running a job but I am not sure what I should be seeing. I have the log from both the machine that is pulling the data and the machine that the data is being pulled from. On the machine pulling data... It looks like that we get an invalid file index and then the job dies out. On the machine data is being pulled... I am seeing renaming of files, changing of modtimes, recv_files and recv_generator going at the same time. Then our final error: 2007/09/05 10:28:10 [15233] rsync: connection unexpectedly closed (165735 bytes received so far) [generator] 2007/09/05 10:28:10 [15233] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [generator=2.6.9] 2007/09/05 10:28:10 [15233] _exit_cleanup(code=12, file=io.c, line=453): about to call exit(12) Any more ideas? What is going to be the best way to get us back on track? Should I restart the daemon, would that help anything; I wouldn't think so if it is a file problem with index? Theresa Peter Unix Team CSI - Data Services ISA-USAAC [EMAIL PROTECTED] (502) 626-0649 -Original Message- From: Wayne Davison [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 04, 2007 6:12 PM To: Peter, Theresa M Ms CONT USAAC Cc: rsync@lists.samba.org Subject: Re: [U] RE: Rsync Errors On Tue, Sep 04, 2007 at 05:56:11PM -0400, Peter, Theresa M Ms CONT USAAC wrote: Invalid checksum length 39976960 [sender] Invalid remainder length 39976960 [sender] Invalid file index: 2046101527 (count=1730) [sender] These are all symptoms of the same problem -- the binary data protocol is no longer in sync. Whether it is caused by dropped bytes in the connection or a programatical error on rsync's part is the big question, and can't be determined by us. Are you using the -z (--compress) option? If so, try turning it off. ..wayne.. UNCLASSIFIED smime.p7s Description: S/MIME cryptographic signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[U] RE: RE: RE: Rsync Errors
UNCLASSIFIED We were using compression on the network; through using a WAS device. We have turned that off and we are no longer seeing the problem. Why would compression cause these issues? If we turn the WAS (network compression) back on is this going to cause more issues. What do you mean by the binary data protocol is no longer in sync? Thanks Theresa Peter Unix Team CSI - Data Services ISA-USAAC [EMAIL PROTECTED] (502) 626-0649 -Original Message- From: Peter, Theresa M Ms CONT USAAC Sent: Wednesday, September 05, 2007 10:46 AM To: 'Wayne Davison'; 'rsync@lists.samba.org' Subject: [U] RE: RE: Rsync Errors UNCLASSIFIED We are not using the -z or --compress option. I have turned up the logging when running a job but I am not sure what I should be seeing. I have the log from both the machine that is pulling the data and the machine that the data is being pulled from. On the machine pulling data... It looks like that we get an invalid file index and then the job dies out. On the machine data is being pulled... I am seeing renaming of files, changing of modtimes, recv_files and recv_generator going at the same time. Then our final error: 2007/09/05 10:28:10 [15233] rsync: connection unexpectedly closed (165735 bytes received so far) [generator] 2007/09/05 10:28:10 [15233] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [generator=2.6.9] 2007/09/05 10:28:10 [15233] _exit_cleanup(code=12, file=io.c, line=453): about to call exit(12) Any more ideas? What is going to be the best way to get us back on track? Should I restart the daemon, would that help anything; I wouldn't think so if it is a file problem with index? Theresa Peter Unix Team CSI - Data Services ISA-USAAC [EMAIL PROTECTED] (502) 626-0649 -Original Message- From: Wayne Davison [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 04, 2007 6:12 PM To: Peter, Theresa M Ms CONT USAAC Cc: rsync@lists.samba.org Subject: Re: [U] RE: Rsync Errors On Tue, Sep 04, 2007 at 05:56:11PM -0400, Peter, Theresa M Ms CONT USAAC wrote: Invalid checksum length 39976960 [sender] Invalid remainder length 39976960 [sender] Invalid file index: 2046101527 (count=1730) [sender] These are all symptoms of the same problem -- the binary data protocol is no longer in sync. Whether it is caused by dropped bytes in the connection or a programatical error on rsync's part is the big question, and can't be determined by us. Are you using the -z (--compress) option? If so, try turning it off. ..wayne.. UNCLASSIFIED UNCLASSIFIED smime.p7s Description: S/MIME cryptographic signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[U] Rsync Errors
UNCLASSIFIED I am seeing the following errors occasionally when doing an rsync between servers (Solaris 9) at the local and remote site. Has anyone seen these errors before or know what they are? rsync error: error in rsync protocol data stream (code 12) at io.c(1122) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(954) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(960) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at sender.c(174) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(954) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(960) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at sender.c(174) [sender=2.6.9] Both servers are using 2.6.9 The only changes that I can think of that have been made is the amount of data is continuously growing. We have had the networking group take a look at their switches and routers bc we see Broken Pipe errors soon after we get one of the above errors, but they are not seeing anything on their end. Does anyone have any ideas? Thanks Theresa Peter Unix Team CSI - Data Services ISA-USAAC [EMAIL PROTECTED] (502) 626-0649 UNCLASSIFIED smime.p7s Description: S/MIME cryptographic signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[U] RE: Rsync Errors
UNCLASSIFIED I am still trying to figure out whether or not we have that savetransfer option already on the box. It may take me a bit to get a new package installed due to software cycle that we have to go through before putting it on the box. Looking into this more... The following errors were the first that we saw: rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(1122) [sender=2.6.9] rsync: read errors mapping /soinc/soinc_std_001 (in CYA): No data available (61) rsync error: some files could not be transferred (code 23) at main.c(683) [sender=2.6.9] Since then we have been seeing the following errors on every rsync: Invalid checksum length 39976960 [sender] Variable lengths listed rsync error: protocol incompatibility (code 2) at io.c(954) [sender=2.6.9] Invalid remainder length 39976960 [sender] rsync error: protocol incompatibility (code 2) at io.c(960) [sender=2.6.9] Invalid file index: 2046101527 (count=1730) [sender] rsync error: protocol incompatibility (code 2) at sender.c(174) [sender=2.6.9] I am hoping that with the additional log entries...someone might have an idea what is going on and the best way to get it back to normal. Theresa Peter Unix Team CSI - Data Services ISA-USAAC [EMAIL PROTECTED] (502) 626-0649 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Tuesday, September 04, 2007 2:31 PM To: Peter, Theresa M Ms CONT USAAC Cc: rsync@lists.samba.org Subject: Re: [U] Rsync Errors On 9/4/07, Peter, Theresa M Ms CONT USAAC [EMAIL PROTECTED] wrote: I am seeing the following errors occasionally when doing an rsync between servers (Solaris 9) at the local and remote site. Has anyone seen these errors before or know what they are? rsync error: error in rsync protocol data stream (code 12) at io.c(1122) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(954) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(960) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at sender.c(174) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(954) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at io.c(960) [sender=2.6.9] rsync error: protocol incompatibility (code 2) at sender.c(174) [sender=2.6.9] Both servers are using 2.6.9 We have had the networking group take a look at their switches and routers bc we see Broken Pipe errors soon after we get one of the above errors, but they are not seeing anything on their end. The Broken Pipe and the error in rsync protocol data stream are both aftermath of a previously occurring error. The other errors all mean that the sender received something invalid from the generator. That could indicate either a bug in rsync or corruption in the network connection. You could dump the data that each side sends and receives over the network using the savetransfer program that comes in support/ of the rsync source package ( http://rsync.samba.org/ftp/rsync/rsync-2.6.9.tar.gz ) to determine whether network corruption is occurring. Matt UNCLASSIFIED smime.p7s Description: S/MIME cryptographic signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync always gets index.html
Matt McCutchen wrote: Pass the -i, --itemize-changes option to see why rsync thought the files needed copying: `s' indicates a difference in size and `t' indicates a difference in mtime. If you get `t', make sure you are using the -t, --times option to set mtimes of destination files equal to those of the source files so that rsync can later trust that the data is the same. thnx a lot. This lead me to the actual problem: I had a second script running on the rsync-target-machine that should delete all index.html in a different folder which was running wild due a stupid error of the sysadmin (which happens to be me), so every copied file with index.html was deleted a few hours later :) I'd completely forgotten about this second old script, but with the -i option I discovered that rsync threats every index.html as new file, and so I started to think thnx peter -- mag. peter pilsl - goldfisch.at IT-Consulting Tel: +43-650-3574035 Tel: +43-1-8900602 Fax: +43-1-8900602-15 skype: peter.pilsl [EMAIL PROTECTED] www.goldfisch.at -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync always gets index.html
I use to sync several servers with rsync for years now. One strange thing I still havent figured out: while rsync does its job perfectly will, it cant refuse to transfer all files with index.html in it, even if the files havent change. example: the following files are the same for years, but every night rsync copies them again: opt_noraid/marcus/usr/local/apache2-140a/manual/vhosts/ opt_noraid/marcus/usr/local/apache2-140a/manual/vhosts/index.html opt_noraid/marcus/usr/local/apache2-140a/manual/vhosts/index.html.de opt_noraid/marcus/usr/local/apache2-140a/manual/vhosts/index.html.en opt_noraid/marcus/usr/local/apache2-140a/manual/vhosts/index.html.ja.jis opt_noraid/marcus/usr/local/apache2-140a/manual/vhosts/index.html.ko.euc-kr opt_noraid/marcus/usr/local/apache2-140a/manual/vhosts/index.html.ru.koi8-r it doesnt due much harm cause these small files dont use much bandiwidth, but its just confusing me and I dont have any clue why rsync copies these files again again. I read the docs and tried to search google, but its difficult to search for a term index.html ;) thnx peter -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync daemon only accessable as root
Hi, I'm trying to setup an rsync daemon such that I can synchronize my home pc (or other remote system) with my pc at work. I'm fairly new to linux and am just starting to learn its amazing possibilities. I followed an online toturial to create a rsyncd.conf file which reads: #motd file = /etc/rsyncd.motd log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock refuse options = delete [test] path = /home/user/test comment = rsync test use chroot = yes gid = nobody uid = nobody read only = no list = yes auth users = user secrets file = /etc/rsyncd.secrets # Limit access from LAN hosts allow=127.0.0.1, home-ip hosts deny=* Then I start the daemon as: sudo rsync -daemon (as user it doesn't seem to start). Now when I am at home I can only access the daemon using: rsync -av -e ssh -l root user@host-ip::test /dest i.e. it will only function as root, which is undesirable. I'd like to use something like: rsync -av -e ssh -l ssh-user user@host-ip::test /dest, but this produces the error rsync: connection unexpectedly closed (0 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(463) [receiver=2.6.8] What am I overlooking? I hope someone can help me, Regards, Peter -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: error with rsync and ssh
Le Jeudi 22 Mars 2007 20:55, Wayne Davison a écrit : On Thu, Mar 22, 2007 at 06:22:11PM -0300, Roberto Pereyra wrote: Seems like a ssh issue because if I use the standard rsync port (without shh) works well. Those are two different things. Without ssh, you're connecting to an existing rsync daemon. With ssh (via -e), you're using a remote shell to login to a remote host and start a NEW rsync process to handle a single request. I'd imagine that this new rsync process can't find its config file (it looks in the home directory of the user by default). I have been having the same problem. Some WinXP systems have cwRsync installed and everything worked well. Then, one laptop started exhibiting trouble. These are client laptops so I don't have much control on what gets done to them. This is the command I run inside a MS bat file: cmd /K rsync.exe -avr --rsh=ssh --progress --stats --delete /cygdrive/d/My Documents [EMAIL PROTECTED]::user_data I get this error locally: Quote: Timeout, server not responding And on the server: Quote: 2007/01/24 11:46:18 [59320] connect from FQDN (192.168.0.110) 2007/01/24 11:46:18 [59320] rsync: connection unexpectedly closed (0 bytes received so far) [receiver] 2007/01/24 11:46:18 [59320] rsync error: error in rsync protocol data stream (code 12) at io.c(462) [receiver=2.6.9] I have been troubleshooting with a test directory both locally and remotely (client and server). I have also upgraded rsync on the server as well as re-installed cwRsync on the laptop. I have attached a tcpdump dump file of the client-server exchange. .101 is the server and .110 is the client. To make the dump a little more understandable, I allowed about 10 seconds to elapse between launching the command and submitting the user password. Pedro rsync.dump Description: Binary data -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync or other tool
Le Lundi 29 Janvier 2007 00:08, Preeti Joshi a écrit : Hi, I am using rsync-2.5.5 on power pc system. It is an embedded system with very low memory few MBs. Most of the time only 4-5 MB memory is free (as shown using free command). I have an application that periodically (every 20 sec) fires rsync command from system command i.e. system (rsync..) to a remote m/c connected to it. The data I have to rsync has hardly 60 files with total size of 4MB. My problem is that even with this short file list and such small sized files, system leads to OUT OF MEMORY error within 2 hours killing my application and many other applications! Is there any solution? Should I look for some other tool which is equivalent for rsync but can be used for such small optimized transfers? Why not just copy the files over? Peter -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Timeout, server not responding
I have several WinXP laptops which have been backing up to a FreeBSD 6.0 server via rsync for a good while now. Now one laptop is giving me grief. I am running cwRsync and invoking a remote rsync daemon via a shell command: cmd /K rsync.exe -avr --rsh=ssh --progress --stats --delete /cygdrive/d/My Documents [EMAIL PROTECTED]::user_data I get this error locally: Quote: Timeout, server not responding And on the server: Quote: 2007/01/24 11:46:18 [59320] connect from FQDN (192.168.0.110) 2007/01/24 11:46:18 [59320] rsync: connection unexpectedly closed (0 bytes received so far) [receiver] 2007/01/24 11:46:18 [59320] rsync error: error in rsync protocol data stream (code 12) at io.c(462) [receiver=2.6.9] I have been troubleshooting with a test directory both locally and remotely (client and server). I have also upgraded rsync on the server as well as re-installed cwRsync on the laptop. Any comments welcome, Peter -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
strange rsync-error
When syncing huge data-folders (~50GB) with a remote server on a daily base I frequently (every 2nd to 3rd time) get the following error: Read from remote host goldfisch.at: Connection timed out rsync: writefd_unbuffered failed to write 4092 bytes: phase unknown [generator]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(1099) rsync error: received SIGUSR1 or SIGINT (code 20) at main.c(985) The connection itself is stable. There might be a small interruption for 1 or 2 seconds maybe from time to time but nothing unusual. The rsync-versions are: on local side: rsync version 2.6.6 protocol version 29 on remote side: rsync version 2.6.9 protocol version 29 Any idea? thnx peter -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Error while transfering large files
Hi,
I'm using rsync to backup my data from my Linux machine (SUSE10.1) and
Windows (XP)
I've mounted a windows share on my linux and I'm trying now to copy files to
the windows with rsync.
The windows share is a NTFS filesystem
It's all working except I have a error on large files.
This is the command that I'm trying:
rsync -a --no-o --delete /public/Games/Call_of_Duty2/
/backup/public/Games/Call_of_Duty2/
and this is the error I'm receiving on this command:
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: write failed on /backup/public/Games/Call_of_Duty2/dev-cod2.iso:
File too large (27)
rsync error: error in file IO (code 11) at receiver.c(258) [receiver=2.6.9]
rsync: connection unexpectedly closed (76 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453)
[generator=2.6.9]
rsync: connection unexpectedly closed (36 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453)
[sender=2.6.9]
Then I tried (as described in on the website) with the strace in front. The
command format was as followed:
strace -f rsync -a --no-o --delete /public/Games/Call_of_Duty2/
/backup/public/Games/Call_of_Duty2/ 2./tracedump.log
This is a part of the tracedump.log file. Where it went wrong.
[pid 826] select(1, [0], [], NULL, {60, 0} unfinished ...
[pid 824] ... write resumed ) = 32768
[pid 826] ... select resumed ) = 1 (in [0], left {60, 0})
[pid 824] select(5, NULL, [4], [4], {60, 0} unfinished ...
[pid 826] read(0, \326\3357\233`\v\t\26\3352G\33\210\314
\221\371\373\20..., 8184) = 8184
[pid 826] select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left {60, 0})
[pid 826] read(0,
\232\t\316S\177\300\373\362\350\26\371\311\255v\352\212..., 8184) = 8184
[pid 826] select(1, [0], [], NULL, {60, 0}) = 1 (in [0], left {60, 0})
[pid 826] read(0, unfinished ...
[pid 824] ... select resumed ) = 1 (out [4], left {60, 0})
[pid 826] ... read resumed
\344\31\f\7\360e\17\301\227\25\202/\233\324i\256\246\0..., 8184) = 8184
[pid 824] write(4, \0\200\0\0, 4 unfinished ...
[pid 826] select(1, [0], [], NULL, {60, 0} unfinished ...
[pid 824] ... write resumed ) = 4
[pid 826] ... select resumed ) = 1 (in [0], left {60, 0})
[pid 824] read(3, unfinished ...
[pid 826] read(0, unfinished ...
[pid 824] ... read resumed
\31\310A\232o\356\30r\223p\215B\27\313\337\357\371\364..., 262144) =
262144
[pid 826] ... read resumed
\3068\340\203\337\254\360\374%\217\321\320\352\343q\17..., 8184) = 8184
[pid 824] select(5, NULL, [4], [4], {60, 0} unfinished ...
[pid 826] select(1, [0], [], NULL, {60, 0} unfinished ...
[pid 824] ... select resumed ) = 1 (out [4], left {60, 0})
[pid 826] ... select resumed ) = 1 (in [0], left {60, 0})
[pid 824] write(4,
\31\310A\232o\356\30r\223p\215B\27\313\337\357\371\364..., 32768
unfinished ...
[pid 826] read(0, unfinished ...
[pid 824] ... write resumed ) = 32768
[pid 826] ... read resumed
\347\235\303/\210G\365\n\304\325\327\335A\374\272\31\320..., 8184) = 8184
[pid 824] select(5, NULL, [4], [4], {60, 0} unfinished ...
[pid 826] write(1,
x\17\240\2267\376m\257G\271\373\217\264\375\325\312\302..., 262144) =
262143
[pid 826] write(1, 5, 1)= -1 EFBIG (File too large)
[pid 826] --- SIGXFSZ (File size limit exceeded) @ 0 (0) ---
[pid 826] write(4, ^\0\0\10rsync: write failed on \/bac..., 98) = 98
[pid 826] rt_sigaction(SIGUSR1, {SIG_IGN}, unfinished ...
[pid 825] ... select resumed ) = 1 (in [3], left {2, 488000})
[pid 826] ... rt_sigaction resumed NULL, 8) = 0
[pid 825] read(3, unfinished ...
[pid 826] rt_sigaction(SIGUSR2, {SIG_IGN}, NULL, 8) = 0
[pid 826] unlink(.dev-cod2.iso.GrkNfL unfinished ...
[pid 825] ... read resumed ^\0\0\10, 4) = 4
[pid 825] select(4, [3], [], NULL, {60, 0}) = 1 (in [3], left {60, 0})
[pid 825] read(3, rsync: write failed on \/backup/..., 94) = 94
[pid 825] select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left {60, 0})
[pid 825] write(1, ^\0\0\10rsync: write failed on \/bac..., 98) = 98
[pid 825] time(NULL) = 1164306617
[pid 825] select(4, [3], [], NULL, {60, 0} unfinished ...
[pid 826] ... unlink resumed ) = 0
[pid 826] write(4, L\0\0\10rsync error: error in file I..., 80) = 80
[pid 826] exit_group(11) = ?
Process 826 detached
[pid 824] ... select resumed ) = 1 (out [4], left {59, 88})
[pid 824] write(4, \0\200\0\0, 4) = -1 EPIPE (Broken pipe)
[pid 824] --- SIGPIPE (Broken pipe) @ 0 (0) ---
[pid 824] write(2, rsync: writefd_unbuffered failed..., 76rsync:
writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)) = 76
[pid 824] write(2, \n, 1
) = 1
[pid 824] select(6, [5], [], NULL, {30, 0}) = 1 (in [5], left {30, 0})
[pid 825] ...
comparing 2 dirs and placing result in another dir
Is it possible to have rsync compare two directories and then place that difference in a separate directory? Peter __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: cross platform rsync avz lists all directory names in tree
Wayne Davison wrote: On Tue, May 23, 2006 at 10:18:23AM +0100, Peter Risdon wrote: another/path/ another/path/withoutanychangedfiles/ yet/anotherpath/withoutanychangedfiles/ ... If this is caused by a change in timestamps, you can avoid the updating of directory timestamps by using the -O option. If you still see updates to the directories, then using -i will at least tell you what attributes rsync thinks are different (and, as Matt indicated, you could filter the -i output to get rid of any updating-attribute lines). One other potential cause for a timestamp difference: if you're copying to a FAT filesystem, you should use the option --modify-window=1 because the timestamps can't store an odd value. ..wayne.. Thanks very much for the replies. In the end I used the -0 option. Peter. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
cross platform rsync avz lists all directory names in tree
Hi, I have tried googling and reading the docs and examples but haven't yet managed to solve this issue. rsyncing from Mac OS X to FreeBSD and from Linux to Windows XP, with the arguments avz I get the same thing in the reports (mailed from cron): path/ path/to/ path/to/directories/ path/to/directories/changedfile.txt another/path/ another/path/withoutanychangedfiles/ yet/anotherpath/ yet/anotherpath/withoutanychangedfiles/ ... and so on. With biggish directories this results in huge lists which have to be trawled through if manual checks are to be made that changed files were successfully copied/updated. I have a feeling that this stems from differences in permission/uid/gid values but haven't managed to find a combination of arguments that give a useful report. rsyncing between Linux and FreeBSD boxes is fine (of course). Can anyone advise how to limit the output to just files/directories that have been added, removed (if --delete is used) or changed when running across platforms like this? TIA. Please cc me, because I'm not a list subscriber. Regards, Peter Risdon. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync ok with open files?
Hi gang. I am synchronizing Windows machines (cwRsync/cygwin) with FreeBSD server. A batch file is used to initiate the process (shell invoked rsync daemon). My question is whether I need to tell my users to close any affected applications (ex: Outlook; I am synchronizing the outlook.pst file) during the procedure? Peter __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync ok with open files?
--- Tony Abernethy [EMAIL PROTECTED] wrote: Fundamental difference between unix and windows. Windows locks against simultaneous whatever. Unix assumes you know what you're doing. Unix allows deleting a file while someone is writing to it. Also, you will likely need to stop BOTH outlooks. There's usually one running that you do not see. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Tevfik Karag�lle Sent: Wednesday, May 10, 2006 2:34 PM To: 'Peter'; 'rsync' Subject: RE: rsync ok with open files? AFAIK, rsync cannot copy open files. Outlook must be stopped in your case. Rgrds Tev -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Sent: Wednesday, May 10, 2006 6:06 PM To: rsync Subject: rsync ok with open files? Hi gang. I am synchronizing Windows machines (cwRsync/cygwin) with FreeBSD server. A batch file is used to initiate the process (shell invoked rsync daemon). My question is whether I need to tell my users to close any affected applications (ex: Outlook; I am synchronizing the outlook.pst file) during the procedure? Peter Thanks Tony. I'll have my users shut off their apps when they perform the backup. And I'll keep my eye out for the second Outlook. Peter __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: file has vanished Chinese characters...
--- Tevfik Karag�lle [EMAIL PROTECTED] wrote: Hi, There is a UTF-8 patched cygwin available from http://www.okisoft.co.jp/esc/utf8-cygwin/download.html I haven't used it. Maybe it is worth to try :-) Rgrds Tev Background info: http://translate.google.com/translate?hl=enu=http://www.okisoft.co.jp/esc/c ygwin-20.html http://sourceware.org/ml/cygwin/2006-05/msg00068.html http://www.cygwin.com/ml/cygwin-patches/2002-q3/msg00037.html http://www.itefix.no/phpws/index.php?module=phpwsbbPHPWSBB_MAN_OP=viewPHPW S_MAN_ITEMS[]=396 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Sent: Friday, May 05, 2006 5:08 AM To: [EMAIL PROTECTED]; Wayne Davison Cc: rsync@lists.samba.org Subject: RE: file has vanished Chinese characters... --- [EMAIL PROTECTED] wrote: This is entirely guesswork, but maybe, just maybe From something unixy, do an smb mount of the Windows share (Cygwin does the incredible, but it's still not unix) There is probably some kind of option so that you see the filenames and do the rsync from the unixy side. I think I ran into it doing something with Knoppix and mounting an NTFS partition with chinese characters in it. I'm guessing that there is a similar option somewhere on the Samba side. something like -o unicode for smbmount might help. Looks like there's iocharset= and codepage= Good Luck. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Peter Sent: Thursday, May 04, 2006 9:18 PM To: Wayne Davison Cc: rsync@lists.samba.org Subject: Re: file has vanished Chinese characters... --- Wayne Davison [EMAIL PROTECTED] wrote: On Wed, May 03, 2006 at 11:27:50AM -0400, Peter wrote: During a synchronization I get many file has vanished messages; the filename contains question marks. This cygwin/Windows induced bug has come up before. One potentially useful older message: http://lists.samba.org/archive/rsync/2005-May/012638.html The root of the problem is that Windows is lying to rsync about what the filenames are, and no one has offered some advice on how to make the lying stop (universally). Thanks. Looks like I'm at the end of the road on this one. You would think that we could figure out a workaround. Well I'm using neither Samba nor Cygwin but cwRsync. The WinXP users are initiating backups/synchronizations to a server at their discretion via a simple Windows batch file/script. Although I am using Samba as a basic fileserver. I'm willing to try but I might have some difficulty with the instructions! __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: file has vanished Chinese characters...
--- Wayne Davison [EMAIL PROTECTED] wrote: On Wed, May 03, 2006 at 11:27:50AM -0400, Peter wrote: During a synchronization I get many file has vanished messages; the filename contains question marks. This cygwin/Windows induced bug has come up before. One potentially useful older message: http://lists.samba.org/archive/rsync/2005-May/012638.html The root of the problem is that Windows is lying to rsync about what the filenames are, and no one has offered some advice on how to make the lying stop (universally). Thanks. Looks like I'm at the end of the road on this one. You would think that we could figure out a workaround. Peter __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: file has vanished Chinese characters...
--- [EMAIL PROTECTED] wrote: This is entirely guesswork, but maybe, just maybe From something unixy, do an smb mount of the Windows share (Cygwin does the incredible, but it's still not unix) There is probably some kind of option so that you see the filenames and do the rsync from the unixy side. I think I ran into it doing something with Knoppix and mounting an NTFS partition with chinese characters in it. I'm guessing that there is a similar option somewhere on the Samba side. something like -o unicode for smbmount might help. Looks like there's iocharset= and codepage= Good Luck. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Peter Sent: Thursday, May 04, 2006 9:18 PM To: Wayne Davison Cc: rsync@lists.samba.org Subject: Re: file has vanished Chinese characters... --- Wayne Davison [EMAIL PROTECTED] wrote: On Wed, May 03, 2006 at 11:27:50AM -0400, Peter wrote: During a synchronization I get many file has vanished messages; the filename contains question marks. This cygwin/Windows induced bug has come up before. One potentially useful older message: http://lists.samba.org/archive/rsync/2005-May/012638.html The root of the problem is that Windows is lying to rsync about what the filenames are, and no one has offered some advice on how to make the lying stop (universally). Thanks. Looks like I'm at the end of the road on this one. You would think that we could figure out a workaround. Well I'm using neither Samba nor Cygwin but cwRsync. The WinXP users are initiating backups/synchronizations to a server at their discretion via a simple Windows batch file/script. Although I am using Samba as a basic fileserver. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
file has vanished Chinese characters...
Hi gang. I am running cwRsync on Windows XP systems and synching to a FreeBSD server. One user uses unicode because he needs Chinese characters. During a synchronization I get many file has vanished messages; the filename contains question marks. Is there any way to deal with this? Peter __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
file has vanished Chinese characters...
Hi gang. I am running cwRsync on Windows XP systems and synching to a FreeBSD server. One user uses unicode because he needs Chinese characters. During a synchronization I get many file has vanished messages; the filename contains question marks. Is there any way to deal with this? Peter __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: access denied from rsync server
--- Martin Jeppesen [EMAIL PROTECTED] wrote: Hi, Can someone help me out, if I want to connect to a rsync server, and download / from it? It is started in /etc/rc.local by su -c rsync --daemon and /etc/rsyncd.conf contains: log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock [tlf] path = / comment = export uid = root gid = root read only = yes hosts allow = 192.168.0.12 list = yes auth users = rsroot secrets file = /etc/rsyncd.secrets and /etc/rsyncd.secrets contains rroot:rroot On the computer where the data should be downloaded to do I use: rsync -avWHRx 192.168.0.6::tlf /media/backup/latest-tlf/ After entering the rroot password I get this error: @ERROR: auth failed on module tlf rsync error: error starting client-server protocol (code 5) at main.c(1296) [receiver] If I remove the auth users and secrets file it works. Can anyone help me out? =) Who is rsroot? __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to compile Rsync with acl support on Cygwin
I downloaded just the acls.diff and it compiles and builds fine. I was able to do the double --acls entry successfully. All special bits and acls rsync in both directions. ACL only changes get recognized and rsynced too. (BTW, the xattr patch may need to be updated as it patches with fuzz due to the changes created by the acls.diff updates) Nice Job on the fixes. In my mind, this is a real boon to the open source backup community. Now windows backups/restores can be more reliably and efficiently done using open source Linux/UNIX/Mac server solutions. Thanks again, Pete -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to compile Rsync with acl support on Cygwin
Problem: --- For sometime now, I have wanted to be able to rsync ACLs of files on windows machines to a Linux server. I never saw anyone post anything about how to make it work. I tried, but I was never able to compile on Cygwin when I patched the source with the acls.diff patch. ./configure would error out after displaying: Checking whether to support ACLS... checking for acl_get_file in -lacl... no checking for ACL support... no This issue was referenced in this post, but I never saw a follow up or resolution: http://www.cygwin.com/ml/cygwin/2004-01/msg01329.html Solution: --- Well I did some poking around the acls.diff patch and started examining the way cygwin implements ACLs to see what could be patched to make it work. I found that cygwin ACLs are implemented using the Solaris implementation: http://www.ee.adfa.edu.au/staff/hrp/webDesignHelp/cygwin-ug-net-nochunks.html#OV-HI-PERM Corinna Vinschen also mentioned that the cygwin ACLs were modeled using Solaris ACLs and to reference the sun documentation site for implementation http://sources.redhat.com/ml/cygwin/2001-06/msg01243.html Seeing that the acls.diff patch already had all the code for doing Solaris ACLs, it struck me that maybe it might work for Cygwin. I edited the acls.diff patch and it compiled (albeit with a few warnings). I have since tested it using the 2.6.6 and 2.6.7 code successfully. I was able to rsync acls intact to and from my Linux server. Here is the only edit needed for acls.diff to work on Cygwin --- Edit the following line + *solaris*) + AC_MSG_RESULT(Using solaris ACLs) + AC_DEFINE(HAVE_SOLARIS_ACLS, 1, [true if you have solaris ACLs]) to read : + *solaris*|*cygwin*) + AC_MSG_RESULT(Using solaris ACLs) + AC_DEFINE(HAVE_SOLARIS_ACLS, 1, [true if you have solaris ACLs]) Open Issue 1) The only issue Ive seen so far with rsync is that setuid, setgid and sticky bits dont get restored when restoring to windows with -acls . If you restore once with acls and once more without, it restores both acls and setuid/setgid/sticky bits. 2) I also had to use the old way (autoconf and autoheader) with 2.6.7 as the ./prepare-source did not regenerate the configure file Thought someone might want to test it out and merge the change into CVS. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to compile Rsync with acl support on Cygwin
Wayne wrote: The latest version of the acls.diff patch in CVS has the chmod() after the ACL-setting call, and I've just enhanced the code to work with a system where the special mode bits get cleared (though I didn't have an easy way to test this yet). I also fixed a few compiler warnings that were output by a Solaris build. This fixed the special mode bits. They are kept intact now. It also fixed another issue. Previously files with changes in ACLs alone would not be synced. Now the ACL only changes are being rsynced and uodated on the destination. Cool. I've checked in your suggested change to the CVS version of the ACL patch, so now Cygwin will choose to use Solaris ACLs. (You can also grab this via the latest nightly tar file, if you like.) 1) I grabbed the nightly build and compiled it in Cygwin. It works great for local Cygwin rsyncs (i.e. local folder to local folder). 2) I tried doing rsyncs to a linux server running 2.6.6 (my ultimate goal here) Linux rsync version: --- # rsync --version rsync version 2.6.6 protocol version 29 Copyright (C) 1996-2005 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, ACLs, xattrs, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. But Only dirs retained their ACLs and I got these errors on the files: - $ ./rsync.exe -avvz --acls ./test [EMAIL PROTECTED]:/test building file list ... done delta-transmission enabled test/ test/file1 test/file2 test/file3 test/file4 test/file5 set_acl: sys_acl_set_file(test/.file1.fd94pI, SMB_ACL_TYPE_ACCESS): Invalid argument set_acl: sys_acl_set_file(test/.file2.a7AIlJ, SMB_ACL_TYPE_ACCESS): Invalid argument set_acl: sys_acl_set_file(test/.file3.HG2nhK, SMB_ACL_TYPE_ACCESS): Invalid argument set_acl: sys_acl_set_file(test/.file4.w5D5cL, SMB_ACL_TYPE_ACCESS): Invalid argument set_acl: sys_acl_set_file(test/.file5.PxTn9L, SMB_ACL_TYPE_ACCESS): Invalid argument total: matches=0 hash_hits=0 false_alarms=0 data=0 sent 420 bytes received 136 bytes 74.13 bytes/sec total size is 0 speedup is 0.00 rsync error: some files could not be transferred (code 23) at main.c(892) [sender] Source dir: -- $ ls -l test total 0 -rwxrT+ 1 Owner mkgroup 0 Mar 17 00:41 file1 -rwxr-S---+ 1 Owner mkgroup 0 Mar 17 00:41 file2 -rwsr-+ 1 Owner mkgroup 0 Mar 17 00:41 file3 -rw-r--r--+ 1 Owner mkgroup 0 Mar 17 00:41 file4 -rw-r--r--+ 1 Owner mkgroup 0 Mar 17 01:05 file5 drwxr-xr-x+ 2 Owner mkgroup 0 Mar 17 01:19 test $ getfacl.exe test/* # file: test/file1 # owner: Owner # group: mkgroup user::rwx user:Guest:r-- group::r-- mask:rwx other:--- # file: test/file2 # owner: Owner # group: mkgroup user::rwx user:Guest:r-- group::r-- mask:rwx other:--- # file: test/file3 # owner: Owner # group: mkgroup user::rwx user:Guest:r-- group::r-- mask:rwx other:--- # file: test/file4 # owner: Owner # group: mkgroup user::rw- user:Guest:r-- group::r-- mask:rwx other:r-- # file: test/file5 # owner: Owner # group: mkgroup user::rw- user:Guest:r-- group::r-- mask:rwx other:r-- # file: test/test # owner: Owner # file: test/test # owner: Owner # group: mkgroup user::rwx group::r-x mask:rwx other:r-x default:user::rwx default:group::r-x default:other:r-x Dest Dir # ls -l test/ total 8 -rwxrT 1 1003 Domain Users0 Mar 17 00:41 file1 -rwxr-S--- 1 1003 Domain Users0 Mar 17 00:41 file2 -rwsr- 1 1003 Domain Users0 Mar 17 00:41 file3 -rw-r--r-- 1 1003 Domain Users0 Mar 17 00:41 file4 -rw-r--r-- 1 1003 Domain Users0 Mar 17 01:05 file5 drwxr-x---+ 2 1003 Domain Users 4096 Mar 17 01:19 test # getfacl test/* # file: test/file1 # owner: 1003 # group: Domain Users user::rwx group::r-- other::--- # file: test/file2 # owner: 1003 # group: Domain Users user::rwx group::r-- other::--- # file: test/file3 # owner: 1003 # group: Domain Users user::rwx group::r-- other::--- # file: test/file4 # owner: 1003 # group: Domain Users user::rw- group::r-- other::r-- # file: test/file5 # owner: 1003 # group: Domain Users user::rw- group::r-- other::r-- # file: test/test # owner: 1003 # group: Domain Users user::rwx group::r-x other::--- default:user::rwx default:group::r-x default:other::r-x Was wondering if you knew the source of the problems in this situation? Thanks so much for your help. Pete -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Help understanding rsync and cwrsync
--- Doug Lochart [EMAIL PROTECTED] wrote: Windows (cw)Rsync Client -- Linux Rsync server 1) Should we abandon cwrsync for cygwin + rsync? Have the disparities between the two versions been resolved? I am using cwrsync and it runs fine connecting to Slackware server. 2) What are the risks with using cwrsync (we want a limited footprint on the client)? What risks? The footprint is small. 3) Assuming we use straight cygwin + rsync how does the linux rsync server handle the permissions and ACL's of an NTFS files system? Are they restorable back to a windows machine? The linux server won't have any of the users plus the ACLS stuff is totally different that unix. Can't say. I don't use cygwin. We will be using rsync over an SSH tunnel to start but will eventually switch to rsync over SSL. Why? What is wrong with SSH? I do not believe rsync supports SSL natively yet. We want to be able to add module definitions to the rsyncd.conf file as we go along without having to restart the rsync daemon or inetd service. We want this so that we can add new client machines without interfering with other clients use of the server. Is what we are trying to do workable? If so how should we run our rsync server. The config file containing the module settings is read by rsync at every connection so you do not need to restart the daemon. Alternatively, you can run rsyncd on a single-transfer basis. The remote user invokes the daemon. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Help understanding rsync and cwrsync
--- Doug Lochart [EMAIL PROTECTED] wrote: Ok, how does cwrsync handle ACLs, users and permissions as on the windows box? How/where is that info kept in the filesystem on the linux box? Is it in meta-data? I figure basic file permissions are stored with the file but how is the user stored? I can't imagine we would have to create a user account for that end user ... or would we? I can get back to you later on this important question. I like to hear what others on this list are saying on this point. The config file containing the module settings is read by rsync at every connection so you do not need to restart the daemon. Alternatively, you can run rsyncd on a single-transfer basis. The remote user invokes the daemon. Could you explain how the last part works? (the remote user invokes the daemon)? Most examples I have seen mention running it in daemon mode or via inetd? On Windows box user invokes like this assuming src directory is on D: drive, rsync.exe -avr --rsh=ssh /cygdrive/d/src dir remote user@server::module --rsh is to specify encryption (optional). If you instead contact a standalone or inetd-awoken daemon then encryption is performed by default. The key for remote user invokes the daemon is the double colon followed by a module name. The user then supplies local password just as if he's logging in via SSH. In such case, single-transfer daemon is brought up and he checks user home directory for instructions (rsyncd.conf). To avoid a second (rsync) authentication do not use 'auth users' line in that file. This way each user has their own configuration. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync logging
--- Philippe BEAU [EMAIL PROTECTED] wrote: Hello all, I would like to know for each module i define in my rsyncd.conf (on my backup server side) if it is possible to have the transfer logging (host, ip address, ...) in my mysql server or in a separate log file for each module. I try to add log file = on each module, it was not working. I also try to modify logging format with log format in my rsyncd.conf and nothing work ! the logging is always the same as default. Philippe, You can specify a log file for each user (and each module if there is only one module per user) if rsync is launched as a single-transfer daemon. This implies that your rsync receiving machine is not running a standalone daemon. When the user contacts the server the user's home directory will be searched for a config file (~philippe/rsyncd.conf). Then you specify the log file in the global section. -- Peter __ Find your next car at http://autos.yahoo.ca -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
preventing rsync transfers
I sync some client stations to a central server at the discretion of the users. They run a single-transfer daemon by logging in via their ssh accounts. All is well. Now I also allow a certain user to rsync this accumulated client data (which end up as subdirectories of a single directory on the server) to an external hard drive. This user connects via ssh and runs a remote shell script on the server. (The script is needed to manage rotation of backups and to make sure the drive is mounted/unmounted properly.) All is well. Now I'm thinking: What if these operations overlap? What can I do to prevent this? Or is this even a problem to begin with? -- Peter __ Find your next car at http://autos.yahoo.ca -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
a few basic questions
Hi folks. Running rsync on Slackware. I'm going over the man pages yet some aspects remain murky. Hopefully someone can set me straight. 1. The --rsh option, The man page explains that you can invoke a single-transfer daemon if you want some features that a daemon provides: $ rsync -av --rsh=ssh host::module /dest The it says that you need to slap on an ssh user if the daemon/module user is different from the remote shell user: $ rsync -av -e ssh -l ssh-user [EMAIL PROTECTED]::module /dest Ok, but what happened to the --rsh stuff? I can use this second form with or without --rsh and I don't perceive any difference. 2. Encryption, Are transfers always encrypted or is encryption activated with a specific incantation? Thanks for any input. -- Peter __ Find your next car at http://autos.yahoo.ca -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: a few basic questions
--- Matt McCutchen [EMAIL PROTECTED] wrote: On Wed, 2006-02-15 at 19:12 -0500, Peter wrote: $ rsync -av -e ssh -l ssh-user [EMAIL PROTECTED]::module /dest Ok, but what happened to the --rsh stuff? -e is another name for --rsh. Are transfers always encrypted or is encryption activated with a specific incantation? Rsync performs no encryption of its own, but if you transfer over SSH, SSH encrypts the entire rsync data stream. And that is done with '-e ssh -l ssh-user' or '--rsh=ssh' or is it done unless I use --rsh=non ssh shell? __ Find your next car at http://autos.yahoo.ca -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync command: help with windows - unix
--- Tevfik Karag�lle [EMAIL PROTECTED] wrote: Try C:\rsync.exe /cygdrive/d/mike/* servername::antec_win2k_mike (double quotes) Best regards |: Tev :| -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Sent: 14. februar 2006 07:32 To: rsync Subject: rsync command: help with windows - unix Hi folks, I have a win2k acting as rsync client (I installed cwrsync) and FreeBSD 5.4 acting as rsync server and I'm having trouble with basic usage. -- My /etc/rsyncd.conf: log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock [antec_win2k_mike] path = /vol3/antec_win2k_mike uid = nobody gid = nobody read only = no list = yes auth users = mmat secrets file = /etc/rsync/rsyncd.secrets -- /etc/rsync/rsyncd.secrets: -- mmat:password1 -- I have a local user 'mike' on the server with password 'password2'. When I run this command I get my files dumped into the user's home directory, /home/mike/antec_win2k_mike (this antec directory is created): C:\rsync.exe /cygdrive/d/mike/* [EMAIL PROTECTED]:antec_win2k_mike I am prompted for the server user's password 'password2' but not the module password 'password1'. So it's like the configuration file is not being read. I am starting rsync on the server like this: # rsync --daemon --config=/etc/rsyncd.conf What should I do here? Anyone? Ah yes. Now I'm getting it. I use that form because I have a daemon running on the server side. Thanks. __ Find your next car at http://autos.yahoo.ca -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync command: help with windows - unix
Hi folks, I have a win2k acting as rsync client (I installed cwrsync) and FreeBSD 5.4 acting as rsync server and I'm having trouble with basic usage. -- My /etc/rsyncd.conf: log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock [antec_win2k_mike] path = /vol3/antec_win2k_mike uid = nobody gid = nobody read only = no list = yes auth users = mmat secrets file = /etc/rsync/rsyncd.secrets -- /etc/rsync/rsyncd.secrets: -- mmat:password1 -- I have a local user 'mike' on the server with password 'password2'. When I run this command I get my files dumped into the user's home directory, /home/mike/antec_win2k_mike (this antec directory is created): C:\rsync.exe /cygdrive/d/mike/* [EMAIL PROTECTED]:antec_win2k_mike I am prompted for the server user's password 'password2' but not the module password 'password1'. So it's like the configuration file is not being read. I am starting rsync on the server like this: # rsync --daemon --config=/etc/rsyncd.conf What should I do here? Anyone? -- Peter __ Find your next car at http://autos.yahoo.ca -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
(no subject)
Hi, I think, there is a typo in the rsyncd.conf man page. In the 'auth users' option section... The plain text usernames are passwords are stored Maybe:-) The plain text usernames and passwords are stored Have a nice day Peter -- Peter Strazovecwww.contal.sk/peters Contal OK, Ltd.www.contal.sk -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: -signs in rsync as well as in embedded ssh command
Manuel López-Ibáñez wrote: Is it very complicated to use several -o before each option? If you try: rsync -e ssh -o Port=22 -o ClearAllForwardings=yes -o ConnectTimeout=5 target myfile :/remote/path/ It works perfectly!! However, is there any possible scenario where the argument of -e should be interpreted by a shell? The ProxyCommand option in ssh usually contains space-characters, so it would not work in your example. Interpretation by a shell would solve this, but there are other possible sollutions as well. Anyway, does the shell would understand the nested quotes at all? If not, then even if the -e argument is interpreted by a shell, it won't make any difference in this case, will it? Lets take for example the next command on my interactive shell: rsync -e ssh -o \ProxyCommand corkscrew myhttpProxy\ testfile.txt targetcomputer.domain:receive Then my shell will call rsync with the following parameters: 1 -e 2 ssh -o ProxyCommand corkscrew myhttpProxy 3 testfile.txt 4 targetcomputer.domain:receive (Note that the shell uses and removes the -signs and un-escapes the embedded \-signs) (Note that -e and the value are passed as 2 parameters, this is ok) If rsync would give this command to another shell, then this shell would call ssh with parameters: 1 -o 2 ProxyCommand corkscrew myhttpProxy 3+ parameters from rsync to ssh (like hostname) (Note that the shell uses and removes the -signs.) (Note that -o and the value are passed as 2 parameters, this is ok) This should work. But rsync will split the ssh command on spaces and call ssh with parameters: 1 -o 2 ProxyCommand 3 corkscrew 4 myhttpProxy 5+ parameters from rsync to ssh (like hostname) (Note that -o option values is split over 4 parameters, ssh will not understand this.) (Note that rsync ignores and passes the -signs.) This could not work, because ssh cannot understand the -o option. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-signs in rsync as well as in embedded ssh command
Hello, Today I succesfully transfered some testfile on my local computer with the following command: rsync -e ssh testfile.txt localhost:receive I can also succesfully connect with SSH to a remote server through an http proxy with the following command: ssh -o ProxyCommand corkscrew myhttpProxy 8080 targetcomputer.domain 22 targetcomputer.domain But I'm so far unable to combine them, mostly because the -signs in the SSH command seem to conflict with the -signs in the rsync command. I've tried the following combinations with the following results: command: rsync -e ssh -o \ProxyCommand corkscrew myhttpProxy 8080 targetcomputer.domain 22\ testfile.txt targetcomputer.domain:receive results in: command-line: line 0: Bad configuration option: ProxyCommand rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(420) command: rsync -e ssh -o \\ProxyCommand corkscrew myhttpProxy 8080 targetcomputer.domain 22\\ testfile.txt targetcomputer.domain:receive results in: command-line: line 0: Bad configuration option: \\ProxyCommand rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(420) command: rsync -e ssh -o \\\ProxyCommand corkscrew myhttpProxy 8080 targetcomputer.domain 22\\\ testfile.txt targetcomputer.domain:receive results in: command-line: line 0: Bad configuration option: \\ProxyCommand rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(420) command: rsync -e ssh -o ProxyCommand corkscrew myhttpProxy 8080 targetcomputer.domain 22 testfile.txt targetcomputer.domain:receive results in: command-line line 0: Missing argument. rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(420) command: rsync -e ssh -o ProxyCommand\ corkscrew\ myhttpProxy\ 8080\ targetcomputer.domain\ 22 testfile.txt targetcomputer.domain:receive results in: command-line: line 0: Bad configuration option: ProxyCommand\\ rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(420) I've read the firewalls section in the manual and the thread from yesterday about multiple SSH-hops, but unfortunately none of the sollutions there need any -signs in the ssh command. Does anybody here has another suggestion? Regards, Peter van der Meer -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: -signs in rsync as well as in embedded ssh command
Quoting Paul Slootman [EMAIL PROTECTED]: On Fri 21 Oct 2005, Peter van der Meer wrote: Does anybody here has another suggestion? Yes: rsync -e ssh -o 'ProxyCommand corkscrew myhttpProxy 8080 targetcomputer.domain 22' testfile.txt targetcomputer.domain:receive This also results in the following errormessage on my system: command-line: line 0: Bad configuration option: 'ProxyCommand rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(420) So I'm affraid it suffers from the same problem as when using \ for the options for ssh. Regards, Peter -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Feature suggestion: --write-batch --dry-run
AFAICS, --write-batch logs the changes made while doing an rsync operation from place A to place B, and lets you then use the batch file to apply those changes to place C if it were hitherto the same as B. I reckon it'd be a good feature to have a mode where the changes can be logged *without* updating B, so that the update to B itself can be done from the batch file. The motivation for this is wanting to synchronise a large directory tree between Site A and Site B. They can only communicate directly through a slow network link, but I often *visit* Site B from Site A while carrying several Gbytes' worth of Ipod. So it would be cool if Site A could use the network connection to work out what needed doing, but not actually send any new files over -- instead just writing a batch file which I could then carry over to B and apply. (I can't just rsync Site A onto my Ipod, and then rsync it again at the other end, because the Ipod is much smaller than the total size of the directory tree, even though it's much larger than any one update would be. Nor do I really want to have *two* copies of the directory tree at Site A in order to use traditional --write-batch.) Even better, of course, would be to find out that this sort of operation was *already* possible using some rsync features I missed when looking into this... Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Why does rsync think my files are much bigger than they are ?
Thanks - I'd compiled it myself with gcc 2.96 on a redhat 7.2 system, and tried reverting to the RedHat RPM and it works fine.Guess it's a compiler issue! Cheers, P Wayne Davison wrote: On Tue, Sep 28, 2004 at 03:47:09PM +1000, Peter Skipworth wrote: send_files mapped /IFX/llog/logs.109051.gz of size 17592186044416 This line comes directly from this code: rprintf(FINFO, send_files mapped %s of size %.0f\n, safe_fname(fname), (double)st.st_size); The st variable is a structure that was populated by stat(), so the problem is either (1) stat() is returning bogus values, or (2) your compiler is not converting the value to a double properly. I think the former is more likely -- e.g. if the size of the variables actually returned by stat() are not what the header files used for the compilation claim they should be. Did you compile rsync yourself? If so, this is a basic compiler problem that should affect more programs than just rsync. If not, you should try compiling your own version and running that. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Why does rsync think my files are much bigger than they are ?
Has anyone ever seen this happen when using rsync ? I'm trying to send a series of =~ 300k files down the line, but for some reason, rsync thinks each file is 17592186044416 bytes long! The same behaviour occurs when using rsync to copy files locally. Partial output follows...any help appreciated! Cheers, P [EMAIL PROTECTED] llog]# ls -al /IFX/llog/logs.109051.gz -rw-rw1 informix informix 349590 Sep 27 02:07 /IFX/llog/logs.109051.gz [partial output of rsync -azvvv -e ssh /source/dir server:/dest/dir follows...] send_files mapped /IFX/llog/logs.109051.gz of size 17592186044416 calling match_sums /IFX/llog/logs.109051.gz llog/logs.109051.gz rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(225) _exit_cleanup(code=20, file=rsync.c, line=225): about to call exit(20) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsyncing very large files - extremely slow at end
Hello, I am trying to rsync a tree containing very large files. The first one encountered is a file that is 127 GB on the remote (authorative) end and around 80 GB on the receiving end. rsync version is 2.6.2, patched to fix the bug mentioned at: https://bugzilla.samba.org/show_bug.cgi?id=1529 But even so transfer is extremely slow towards the end. It will sync approximately (or exactly, not sure) as much data as is contain in the local file (i.e, 80 GB in this case) at normal multi-MB-per-second speeds. And then the speed suddently goes down to perhaps 3kb/sec. I tried increasing the block size to 25k and instead of 3kb it's now 100 kb/sec or so, but still extremely slow. I googled and found some posts indicating there are problems with extremely large block sizes, so I haven't tried going way above that - plus I am unsure of whether it affects the probability of false matches. Is it supposed to be possible to syncornize files this large at normal speeds? If not, what would help? Increasing the currently 2^16 hash table size while retaining a small block size perhaps? Something else? Any input would be appreciated. Thanks! -- / Peter Schuller, InfiDyne Technologies HB PGP userID: 0xE9758B7D or 'Peter Schuller [EMAIL PROTECTED]' Key retrieval: Send an E-Mail to [EMAIL PROTECTED] E-Mail: [EMAIL PROTECTED] Web: http://www.scode.org -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync'ing large files
On Fri, 23 Apr 2004, Paul Slootman wrote: I don't think that the overhead of one sqrt() per file will make much difference either way... It might over tens of thousands of files... :-) -Pete -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync'ing large files
On Fri, 23 Apr 2004, Wayne Davison wrote: Rsync doesn't call sqrt -- it just does a simple little integer estimation. It is probable that using a hardware sqrt call would be faster than rsync's estimator, but taking only 0.0041 seconds per file instead of 0.0232 seconds per file(*) is not going to make much difference, even with a really large number of really large files. Foo. Here I was looking for a cheap and fun performance boost. Or, and excuse to compile *something* useful that would be G5-specific. Oh well. :-) -Pete -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync'ing large files
On Thu, 22 Apr 2004, Wayne Davison wrote: * Per-file dynamic block size is now sqrt(file length). The per-file checksum size is determined according to an algorithm provided by Donovan Baarda which reduces the probability of rsync algorithm corrupting data and falling back using the whole md4 checksums. (J.W. Schultz, Donovan Baarda) Oooo... I need to go compile 2.6 on a G5 running OS X (unless somebody beat me to it). IIRC, the G5 has hardware sqrt. This could be quite useful. -Pete -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
symlink bug still not fixed
Hi. I found and reported a bug about a year ago regarding symbolic links but haven't seen any mention of it since and it is still present in 2.6.1-pre2. Just want to make sure it isn't forgotten. It can be reproduced by synchronizing two directories, one of which contains a normal file and the other has a symlink of the same name. With the archive, update and backup options set, rsync will always replace the file with the symlink even when the file is newer (and it will not save a backup either). This can lead to unexpected loss of data. In other words: mkdir foo1 foo2 ln -s apples foo1/oranges touch foo1/apples foo2/apples foo2/oranges rsync -avub foo1/ foo2/ and voila, the file foo2/oranges is overwritten by the symlink. Thanks, peter __ Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25¢ http://photos.yahoo.com/ph/print_splash -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: symlink bug still not fixed
--- Wayne Davison [EMAIL PROTECTED] wrote: ... The way rsync currently works, it doesn't consider a file and a symlink to be the same thing, so the -u option will not prevent a file from being replaced by a symlink. ... Hmm. But the symlink is older. I would expect the symlink to overwrite an older file, but not a newer one, which it does. Well, maybe it is philosophical, but I see this as a problem. Firstly, and most obviously, the unexpected loss of a symlink is not as tragic as the loss of a file with potentially lots of data in it. In fact, the symlink will replace a directory too, so you could lose an entire directory tree! At least a note to this effect should show up in a README or man page, no? Secondly, symlinks are often used to avoid duplicate identical files. It isn't unreasonable to then assume that if one of them needs a slight modification, one would replace the symlink with a regular file and make that modification. Now, if you use rsync to synchronize your desktop and laptop, say, then the modification is destroyed. I therefore believe that this behavior is counter-intuitive and strongly suggest that it be changed so that a symlink doesn't replace anything else with a newer date stamp when the -u option is used. After all, -u stands for update! Peter __ Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25¢ http://photos.yahoo.com/ph/print_splash -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: symlink bug still not fixed
--- Wayne Davison [EMAIL PROTECTED] wrote: I will consider such a change for the future, but I'll have to spend time contemplating the repercussions. Thanks. If you decide against changing the behavior, then please add a note in the man page (perhaps where the -u option is explained and in the section devoted to symlinks) just to clearly document what rsync does. And thanks for the tip about unison; I had not heart of it before. Peter __ Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25¢ http://photos.yahoo.com/ph/print_splash -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: earliest use of rsync?
My guess would be that it began to be used after it was written - long after the internet was invented. If you look at the original tech report, it was (I'll hazard a guess) after November 1998. Just FYI, not all the good software was there from day one. That's why people like Andrew are so handy. :-) Cheers, -Pete - Peter L. Wargo | [EMAIL PROTECTED] | http://www.basenji.com/pwargo There ought to be limits to freedom - George W. Bush On Mon, 29 Mar 2004, raghu wrote: Dear Folks, Is there a documented case of earliest use of rsync over internet? I presume rsync was used almost as soon as internet was invented as a file transfer and bandwidth efficient backup solution. If anyone has any information, please respond. -- Raghu Kulkarni -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Odd behavior with rsync/ssh/--delete
I've just about googled my brains out over this one, and banged heads with several other SA buddies. I have a nightly rsync of a DMZ system (Solaris 8 SPARC[1]) to an internal system (RedHat ES 3.0 [2]). The internal system runs a cron job and pulls changes off of the DMZ system via ssh. (To be honest, I've also seen this going between two Solaris systems.) However, my syncs are much bigger then they should be. I'm getting a bunch more than I expect - files that haven't changed in a long time are being deleted and re-sync'd. Here's an example of the command being used: rsync -alvx --delete --rsh=ssh [EMAIL PROTECTED]:/ /backups0/hosts//rsync/ (I use -x to avoid crossing filesystems, as I only back up certain ones. The refers to the hostname of the DMZ system.) What I see are files like this getting hit every night in the log: (real user name blanked.) [...] deleting home/xuserx/public_html/vat5.JPG [...] Then later the file is written. If you look at the file on the host: # ls -la vat5.JPG -rw-rw-r-- 1 xuserx xgroupx 39816 Jan 3 2000 vat5.JPG This is strange. And it's happening for thousands of files. The systems have the same date and time, I checked that out... I'm at wit's end on this one. If anybody has any ideas, I'd be happy to hear them. Also, I can provide more output if desired. Cheers, -Pete Wargo [1] rsync version 2.5.6 protocol version 26. Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6, 64-bit system inums, 64-bit internal inums [2] rsync version 2.5.6 protocol version 26. Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[PATCH] write only
We use rsync to provide an rsync share on a linux server with a javascript
program on Windows clients which ensures all client data is backed up each
morning.
After making no progress with my posting last month regarding making a module
upload only to prevent someone from manually downloading the rsync share
obtaining other users work (exclude = * wasn't possible because we need
--delete to work) I have had a go at making a patch to add this functionality
to rsync.
I've added a write only option to rsyncd.conf which is essentially the opposite
of read only. This means that although it's possible someone on our allowed IP
range could overwrite (or delete) files on the rsync share, they would not be
allowed to download any files (my primary concern).
Below is the patch for rsync-2.6.0 however there are some points I'd like to
make:
- I don't know c :) (I once did a quick program for an assignment 10 years
ago),
- I've never made a diff patch before so I might be doing it wrong,
- I don't really understand rsync's internals or all the ramifications of these
changes,
- The patch doesn't change man pages etc.
It's extremely unlikely I've done everything right - so if any one would like
to point areas I'm falling down on I'd appreciate it.
J
--- rsync-2.6.0/main.c.orig Thu Feb 12 11:47:38 2004
+++ rsync-2.6.0/main.c Thu Feb 12 11:48:16 2004
@@ -347,12 +347,19 @@ static void do_server_sender(int f_in, i
int i;
struct file_list *flist;
char *dir = argv[0];
+extern int module_id;
extern int relative_paths;
extern int recurse;
if (verbose 2)
rprintf(FINFO,server_sender starting pid=%d\n,(int)getpid());
+if (am_daemon lp_write_only(module_id) !am_sender) {
+rprintf(FERROR,ERROR: module is write only\n);
+exit_cleanup(RERR_SYNTAX);
+return;
+}
+
if (!relative_paths !push_dir(dir, 0)) {
rprintf(FERROR, push_dir %s failed: %s (3)\n,
full_fname(dir), strerror(errno));
--- rsync-2.6.0/loadparm.c.orig Thu Feb 12 11:47:17 2004
+++ rsync-2.6.0/loadparm.c Thu Feb 12 11:48:11 2004
@@ -119,6 +119,7 @@ typedef struct
char *comment;
char *lock_file;
BOOL read_only;
+BOOL write_only;
BOOL list;
BOOL use_chroot;
BOOL transfer_logging;
@@ -151,6 +152,7 @@ static service sDefault =
NULL,/* comment */
DEFAULT_LOCK_FILE,/* lock file */
True,/* read only */
+ False, /* write only */
True,/* list */
True,/* use chroot */
False, /* transfer logging */
@@ -276,6 +278,7 @@ static struct parm_struct parm_table[] =
{lock file,P_STRING, P_LOCAL, sDefault.lock_file, NULL, 0},
{path, P_PATH,P_LOCAL, sDefault.path,NULL, 0},
{read only,P_BOOL,P_LOCAL, sDefault.read_only, NULL, 0},
+ {write only, P_BOOL,P_LOCAL, sDefault.write_only, NULL, 0},
{list, P_BOOL,P_LOCAL, sDefault.list,NULL, 0},
{use chroot, P_BOOL,P_LOCAL, sDefault.use_chroot, NULL, 0},
{ignore nonreadable,P_BOOL, P_LOCAL, sDefault.ignore_nonreadable,
NULL, 0},
@@ -353,6 +356,7 @@ FN_LOCAL_STRING(lp_comment, comment)
FN_LOCAL_STRING(lp_path, path)
FN_LOCAL_STRING(lp_lock_file, lock_file)
FN_LOCAL_BOOL(lp_read_only, read_only)
+FN_LOCAL_BOOL(lp_write_only, write_only)
FN_LOCAL_BOOL(lp_list, list)
FN_LOCAL_BOOL(lp_use_chroot, use_chroot)
FN_LOCAL_BOOL(lp_transfer_logging, transfer_logging)
--- rsync-2.6.0/proto.h.origThu Feb 12 11:48:01 2004
+++ rsync-2.6.0/proto.h Thu Feb 12 11:48:23 2004
@@ -125,6 +125,7 @@ char *lp_comment(int );
char *lp_path(int );
char *lp_lock_file(int );
BOOL lp_read_only(int );
+BOOL lp_write_only(int );
BOOL lp_list(int );
BOOL lp_use_chroot(int );
BOOL lp_transfer_logging(int );
__
Do you Yahoo!?
Yahoo! Finance: Get your refund fast by filing online.
http://taxes.yahoo.com/filing.html
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
running rsync in $HOME/bin?
Hi, what exactly does rsync when logging in to a remote system? Is this a normal login? So it starts a shell and reads $HOME/profile and $HOME/.bashrc? I must use rsync in my own account and copied rsync to $HOME/bin. But it seems like rsync, started on another machine, does not find the rsync binary. What can i proceed, to run rsync out of $HOME/bin? Thank you for your attention! Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to trigger rsync after file changes?
Hi, what is the best way to trigger rsync after a file upload? verbose: I want users to upload files to Server A. If all upload is done, it should rsync itself to another server. The upload is done via ftp and users don´t want to use rsync themselves, so what I am missing now ist something that triggers rsync if something changed. An alternative would be to use cron, but not a good one for similar reasons. Maybe someone got this problem before? Thank you very much! Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to avoid rsyncing incomplete uploads? (best practice on fileservers?)
Hi, some big files are regularily updated on server A. Rsync should sync that with Server B. If rsync starts in the middle of an upload process, will it copy that incomplete file to Server B? First tests showed it will. How can this be avoided? Comments on rsync-usage with big files and very active users would be very appreciated! Thanks! Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to avoid rsyncing incomplete uploads? (best practice on fileservers?)
Hi, some big files are regularily updated on server A. Rsync should sync that with Server B (wow! :) If rsync starts in the middle of an upload process, will it copy that incomplete file to Server B? First tests showed it will. How can this be avoided? Comments on rsync-usage with big files and very active users would be very appreciated! Thanks! Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to mask @ symbol in username?
On Sun, 18 Jan 2004 13:17:01 -0800, you wrote: rsync -e 'ssh -l [EMAIL PROTECTED]' server.com:/home /backup thank you very much, that worked! Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to mask @ symbol in username?
Hi, I am trying to access an account with rsync that has a username like this: [EMAIL PROTECTED] Please note: this IS ONLY the username, so to login vie ssh I have to type [EMAIL PROTECTED] I think it´s Ensim Webppliance, that handles virtual accounts that way. So now, if I waqnt to access this account with rsync, I must type something like: rsync -e ssh [EMAIL PROTECTED]@server.com:/home /backbup that leads to an error, because obviously the second @ in the username/server combo is not understood by rsync. What can i do now? How can i mask the first @ ??? Thank you very much for your attention! Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
2.6.0 fails to configure if --disable-ipv6 is specified...
...because the code which sets $EGREP is not hit in this case -- Peter Fales Lucent Technologies, Room 1C-436 N9IYJ 2000 N Naperville Rd PO Box 3033 internet: [EMAIL PROTECTED] Naperville, IL 60566-7033 work: (630) 979-8031 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
In my absence...
Greetings, I've left my position as Senior Web Developer for O'Reilly Associates, effective October 15, 2002. Please direct any work-related email to Nancy Abila ([EMAIL PROTECTED]), if you need to contact me for other reasons, please email me at [EMAIL PROTECTED] Thanks, Peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
bug: rsync always writes temp-files to current directory sincecvs-20030326
Hi, The documentation states that rsync writes temp-files to the recieving directory, unless --temp-dir is specified. This is not true since cvs-20030326, when receiver.c was modified. ( http://cvs.samba.org/cgi-bin/cvsweb/rsync/receiver.c#rev1.44 ) You will get this error when you don't have write access to the current directory: mkstemp .somefile.UQVWQd failed: Permission denied rsync error: some files could not be transferred (code 23) at main.c(1045) IMHO the fix introduced, should either be removed or modified to work as documented. Please see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=192899 best regards peter -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
