Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination
On Thu 01 Sep 2005, Xuehai Zhang wrote: If (similar) tasks are run in parallel, then the data of the files being handled may still be in the buffer cache so that it doesn't need to get read in from disk again. This will save time... I agree with you that caching effect might be the cause. The buffer cache you mentioned refers to the cache on the data source, right? Yes. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination
On Wed 31 Aug 2005, xuehai zhang wrote: results. Why the time of transferring the file to 2N nodes is shorter than twice of the time of transferring the same file to N nodes? Does it make If the network is not the bottleneck, then cpu or the disks are. If (similar) tasks are run in parallel, then the data of the files being handled may still be in the buffer cache so that it doesn't need to get read in from disk again. This will save time... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync problem
On Mon 29 Aug 2005, Zhang, Shu wrote: Thanks for your reply. Here is the info. [snip] I don't see anything that would cause the output you gave... Please upgrade to the newest rsync (2.6.6) and try again. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: difference between src and destination sizes when rsync was run
On Wed 31 Aug 2005, Venkatesh.S wrote: i have two machines, one is a rsync server and another is rsync client when i run the rsync client for backup, the src dir is showing 4.8G and the destination dir to which i am backingup is showing 5.5G can anyone tell me why is this. Filesystem differences will do this, if your filesystem rounds up to e.g. 8k blocks for file allocation, then 1000 files of 1 byte will show 8MB; another filesystem that doesn't waste space like that (e.g. reiserfs with tail option) will show a few k. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync problem
On Mon 29 Aug 2005, Zhang, Shu wrote: The version of the rsync rsync version 2.5.2 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others That's pretty ancient, that was released 26 Jan 2002. Error message we are getting --- RSYNC FAILURE - RSYNC Error Code: 5888 RSYNC Failed: file1, file2, file3, file4, file5 Mon 8/29/2005 0:0:11 This doesn't look like a typical rsync error message, it would seem that perhaps some script or so is being used to call rsync, and it's processing the real rsync output into this form? Please give the actual command used to run rsync when it gives this message. Also give details about the runtime environment (operating system, version of such, etc.). And please consider upgrading to the latest version available (2.6.6), there have been many bugs fixed in the meantime. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: unable to open configuration file rsyncd.conf
On Fri 26 Aug 2005, Wayne Davison wrote: On Fri, Aug 26, 2005 at 10:55:11AM +0200, Paul Slootman wrote: This isn't stated that clearly in the manpage, this might be mentioned in the CONNECTING TO AN RSYNC SERVER OVER A REMOTE SHELL PROGRAM section... the distinction between that section and the RUNNING AN RSYNC SERVER OVER A REMOTE SHELL PROGRAM section is a bit vague as well. That was improved in the last release. See the on-line version if you don't have the 2.6.6 version handy: Hmm, I must have forgotten to upgrade that system at work where I happened to type man rsync :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: unable to open configuration file rsyncd.conf
On Thu 25 Aug 2005, [EMAIL PROTECTED] wrote: Today from the client I want to just copy over to the server what files have changed today. So my command from the client is the following: /usr/local/bin/rsync --partial --progress --stats -az -e ssh -p 22 /directory on the client [EMAIL PROTECTED]::modulename I do get a connection to Box B but after I enter the user's password thru ssh, I get the following the client: [...] this is what is produced on the server /var/adm/messages: rsyncd[19422]: [ID 702911 daemon.warning] rsync: rsync: unable to open configuration file rsyncd.conf: No such file or directory (2) rsyncd[19422]: [ID 702911 daemon.warning] rsync error: syntax or usage error (code 1) at clientserver.c(512) rsyncd[19436]: [ID 702911 daemon.warning] rsync: rsync: unable to open configuration file rsyncd.conf: No such file or directory (2) rsyncd[19436]: [ID 702911 daemon.warning] rsync error: syntax or usage error (code 1) at clientserver.c(512) On the server I am running rsync in daemon mode by using this command, /usr/local/bin/rsync --deamon --config /etc/rsyncd.conf, and the daemon is running, but the client is still get the unable to find /etc/rsyncd.conf. If you supply -e ssh ... as parameters to rsync in combination with the :: syntax, ssh is used to start a new rsync in daemon mode. If you're not doing that as root, the default location for the config file is rsyncd.conf in the current directory (typically $HOME). Hence it can't find rsyncd.conf in ~USER/. This isn't stated that clearly in the manpage, this might be mentioned in the CONNECTING TO AN RSYNC SERVER OVER A REMOTE SHELL PROGRAM section... the distinction between that section and the RUNNING AN RSYNC SERVER OVER A REMOTE SHELL PROGRAM section is a bit vague as well. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --files-from... Option gone?
On Mon 15 Aug 2005, Jamie Pratt wrote: Hi i had an app using --files-from, now it appears that option has been deprecated, or is not available in the latest version (rh es3 package) == Does anyone know the latest equivalent of how to get rsync to feed off a text file so i can get this working again? [EMAIL PROTECTED] cgi-bin]# rsync --version rsync version 2.5.7 protocol version 26 2.5.7 is pretty ancient... 2.6.6 is current, and has the --files-from option. I suggest you upgrade your rsync, and/or go bother rh about providing up to date stuff. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ACL and RSYNC and -a
On Thu 11 Aug 2005, Marc Perkel wrote: Forgive me if this has already been asked but - shouldn't the '-a' switch include -X and -A for ACLs and extended attributes? I thought that -a means everything - so when everything grows -a should grow with it. -a does not mean everything... e.g. it does not include -H as that incurs a lot more processing. -X and -A aren't (yet?) understood by standard rsync versions, so I don't think it's useful to include those in -a yet. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: segment fault with 2.6.6 or CVS
On Fri 29 Jul 2005, Helmut Jarausch wrote: I've rebuilt rsync (which was running just fine for quite some time) and it works if used via ssh or if the daemon is started via ssh. But it crashed in daemon mode (started standalone or by xinetd) I just tested it, and for me it works, started standalone as /usr/bin/rsync --no-detach --daemon --config /etc/rsyncd.conf --address=192.168.1.2 Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: hanging problem on cygwin
On Wed 27 Jul 2005, Hari Krishna Dara wrote: I am now more confused, could you please clarify something? There are three separate sections in the rsync man page called: CONNECTING TO AN RSYNC SERVER OVER A REMOTE SHELL PROGRAM RUNNING AN RSYNC SERVER RUNNING AN RSYNC SERVER OVER A REMOTE SHELL PROGRAM I am now clear that the first approach is what is causing the rsync to hang (because it is going through buggy cygwin pipes), and I agree that I should avoid it. My understanding was that the next two approaches are identical wrt to the protocol, as in both, the client will connect to the rsync server over tcp/ip instead of pipes (no remote shell involved). But you were saying that the 3rd one also involves using pipes, just like the 1st one, right? Could you please confirm again? Doesn't the below from man rsync page mean otherwise? Well, the description of the 3rd is ... OVER A REMOTE SHELL PROGRAM, so you're wrong in assuming (no remote shell involved). Using a remote shell will of course imply pipes... From the user's perspective, using rsync in this way is the same as using it to connect to an rsync server, except that you must explicitly set the remote shell program on the command line with --rsh=COMMAND. (Setting RSYNC_RSH in the environment will not turn on this functional- ity.) You're missing the From the user's perspective. the syntax and behaviour is the same as when connecting to an rsync server, but under the hood remote shells and pipes are involved now. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete problem
On Tue 26 Jul 2005, jdd sur free wrote: testing rsync, I use the following line: rsync --rsync-path=/usr/bin/rsync -e ssh --delete -valptz --safe-links * [EMAIL PROTECTED]:~/temp/test between a mandrake 10.1 and a suse 9.0 Why use * and not . ? a create a dummy file in start dir, run rsync, file is transfered. I delete the file in start dir, run rsync, the file is _not_ delete on the other side. The --delete option only deletes stuff that doesn't exist in directories being transferred; the manpage says: This tells rsync to delete extraneous files from the receiving side (ones that aren't on the sending side), but only for the directories that are being syn- chronized. You must have asked rsync to send the whole directory (e.g. dir or dir/) without using a wildcard for the directory's contents (e.g. dir/*) since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files' parent directory. That says it all! Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync to a Samba/CIFS filessytem hangs
On Fri 15 Jul 2005, Robert Gasch wrote:
The output from strace and lsof would be helpful.
However, my impression is that the CIFS filesystem is deadlocking
somewhere...
[EMAIL PROTECTED] root]# lsof -p 16327
COMMAND PID USER FD TYPE DEVICESIZENODE NAME
rsync 16327 root cwdDIR 0,14 0 64419
/mnt/backup/backup_www
rsync 16327 root rtdDIR3,14096 2 /
rsync 16327 root txtREG3,8 558667 3482979 /usr/local/bin/rsync
rsync 16327 root memREG3,1 35648 44022
/lib/libnss_files-2.3.3.so
rsync 16327 root memREG3,6 178476 411662
/usr/share/locale/ISO-8859-1/LC_CTYPE
rsync 16327 root memREG3,1 1165108 44042
/lib/tls/libc-2.3.3.so
rsync 16327 root memREG3,1 60804 44034
/lib/libresolv-2.3.3.so
rsync 16327 root memREG3,1 529609 43989 /lib/ld-2.3.3.so
rsync 16327 root0u unix 0xe16fb680 8778239 socket
rsync 16327 root2u CHR 136,2 4 /dev/pts/2
rsync 16327 root4u unix 0xe16fb380 8778248 socket
OK, this rsync process doesn't have any files open...
However, I expect that there should be a second rsync process as well?
[EMAIL PROTECTED] root]# strace -p 16327
Process 16327 attached - interrupt to quit
select(1, [0], [], NULL, {27, 137000}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
select(1, [0], [], NULL, {60, 0} unfinished ...
Process 16327 detached
This rsync process is waiting for data to come in on that socket, which
most probably should be supplied by the other rsync process. Hence,
could you repeat the exercise, but then for all rsync processes?
Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: @ERROR: access denied
On Thu 14 Jul 2005, [EMAIL PROTECTED] wrote: [repositories] comment = Subversion Repositories path = /usr/local/repositories read only = no list = yes hosts allow = 127.0.0.1 auth users = username secrets file = /etc/rsyncd.secrets However running rsync [EMAIL PROTECTED]::repositories I get: @ERROR: access denied to repositories from localhost (::1) You allow an IPv4 address '127.0.0.1', however the connection appears to come from an IPv6 address '::1'. Maybe use ::1 instead of 127.0.0.1 in the hosts allow line? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync to a Samba/CIFS filessytem hangs
On Thu 14 Jul 2005, Robert Gasch wrote: I'm using rsync to backup a Linux Mandrake 10.1 (kernel 2.6.10) ext3 filesystem (+- 5GB of content, lots of little files) to a CIFS filesystem mounted with samba 3.0.10. The exact invocation of rsync is: The CIFS filesystem is mounted on the linux system? Then samba doesn't really enter the picture. On what system is the CIFS filesystem located? Could you show the output of 'mount'? /usr/local/bin/rsync -v -a --copy-links --delete /var/www /mnt/backup/backup_www Using the system provided rsync 2.6.3 and a self-compiled 2.6.5 this process runs for a while and then simply hangs. What's even worse, Did you try using strace? Does lsof -p $pid show anything? when I try to kill the job, the process becomes owned by pid 1, can't be killed anymore and thus the memory it holds doesn't get released anymore (forcing me to reboot the machine about once a month or so). It becomes a zombie process. It should be reaped by init (pid 1), however for some reason init doesn't seem to be doing its work... A zombie process only holds an entry in the process table so that it can return its exit status to its parent (who hasn't waited for its child yet). It shouldn't take up any memory or other resources... The output from strace and lsof would be helpful. However, my impression is that the CIFS filesystem is deadlocking somewhere... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Can Rsync do this?
On Thu 14 Jul 2005, Stuart Halliday wrote: The program in question has a settings directory so I thought I could get Rsync to sync this directory between the two machines. But Rsync seems to always force the files on the Rsync server to go to the slave regardless that the slave has the most upto date file time-wise. That's the point of rsync: make the destination the same as the source. So I guess I need a way for Rsync to compare the two directories and if one file is more recent than the other then force the older file to be overwritten by the most recent regardless of which location it is in. You could either use the --update option, or (perhaps better suited to your situation) have a look at unison, which is meant to keep two systems in sync, regardless of where changes take place (so long there aren't any conflicts, of course). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Other possible solutions to: rsync memory usage, paid feature request
On Wed 06 Jul 2005, David Favro wrote: 1) Free: break your rsync's into several executions rather than one huge one. Do several sub-directory trees, each separately. If your data [...] 2) Cheap: buy more swap space. These days random-access magnetic [...] 4) Expensive: buy more solid-state memory. Possibly still cheaper than [...] None of these proposals would have helped when I wanten to move two year's worth of Debian archive images to another system using rsync. The Debian archive is currently around 88.000 files (at least what we mirror of it). Every day a snapshot is taken; common files are hardlinked across days. This means an incredible amount of directory entries and hunderds of thousands of distinct files. Doing 1) was not feasible, as that would result in very many hardlinks being lost and files effectively duplicated, leading to wasted space. Doing 2) was tried (actually: creating swap files on disk), but then we ran into the virtual address limitations of the 32-bit system: 3GB wasn't enough by far. Doing 3) would have the same problem as 2). Going to a 64-bit system might have helped, but I think that the memory usage would have exceeded what's reasonable in solid-state memory, and using swap would have slowed it all down horribly as the lists in memory are apparently transversed quite regularly. As it is, it took a couple of days before the virtual memory limit was reached... I ended up rsyncing the days separately, and using a perl program to build a tree of md5sums which were hardlinks to the corresponding files. With each new directory the md5sums could be compared and hardlinks recreated. However, I would *love* to see rsync be more memory-efficient... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error ?
On Wed 06 Jul 2005, [EMAIL PROTECTED] wrote: Thanks for the reply, this version of rsync is going to be used by a group of people, I need to convince them that this is not a real error, do you have any document which talks about this rsync behavior ? Let's put it another way: where in the rsync manpage synopsis do you see that you could run rsync without any options? Isn't it common sense that you need to supply arguments to rsync to let it work? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync .gz files?
On Wed 29 Jun 2005, Diane Rolland wrote: Interactively, I get the following error: :rsync error: some files could not be transferred (code 23) at main.c(1045) In the /etc/rsyncd.log, for each of the files, I get: send_files failed to open /backup/exp_dbase_2005-06-2 8.gz: Permission denied So it seems that the sending side isn't allowed to open the files in question. My command is: rsync --verbose --progress --stats --rsh=/usr/bin/ssh --recursive --delete prsvr01::prod-db/backup/* /prod/db/backup Tip: leave off the '*', you're already telling rsync to be recursive. My module on the source host is: [prod-db] path = /prod/db comment = /prod/db uid = nobody Is user 'nobody' allowed to read those files? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync question
On Thu 30 Jun 2005, Judith Flo wrote: Well, i've been just working with rsync also, and want to use the --files-from option and write the same command. And my question is: wouldn't it be the correct behavior to provide just one file with the --files-from without writting a source dir? I mean, the content of the files-from file shows the sources which i want to rsync so, why do i have to provide anything else? Read the manpage, the --files-from section, in particular this: The file names that are read from the FILE are all relative to the source dir -- any leading slashes are removed and no .. references are allowed to go higher than the source dir. If you don't supply a source dir, to what directory should the file names be relative to? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync process - writing to source file system
On Tue 28 Jun 2005, Diane Rolland wrote: My theory is that perhaps rsync is writing/changing file attributes on the source file system. Since the EMC process thinks changes are being made, it keeps trying to do the clone. As data changes, the cloning continues. Can anyone tell me if rsync makes any updates to the source file system? rsync itself will not update anything on the source. However, unixen will typically update the file's access time of any files / directories / etc. that rsync accesses, perhaps that's your problem. See if you can mount the file system noatime (that's for linux); check to see that no applications depend on access times though (e.g. mail programs compare modification time and access time to see whether there's new mail). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: adding a new log-format escape
On Mon 27 Jun 2005, YOSHIFUJI Hideaki / ?$B5HF#1QL@ wrote:
In article [EMAIL PROTECTED] (at Mon, 27 Jun 2005 00:07:19 -0600), Andrew
Shewmaker [EMAIL PROTECTED] says:
+ for (j = 0; j SUM_LENGTH; j++ ) {
+ snprintf(buf2 + j * 2, sizeof buf2,
fmt, file-u.sum[j]);
file-u.sum[j] 0xff
+ for (j = 0; j b; j++ ) {
+ snprintf(buf2 + j * 2, sizeof buf2,
fmt, file-u.sum[j]);
ditto.
Note also that to preserve the protection offered by snprintf, the
sizeof buf2 needs to be changed to sizeof buf2 - j * 2.
Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
OT: Re: Your email requires verification verify#UWUgyaRs5eQ0Ozo94QTL8FjSRuQdWxt
Could someone remove [EMAIL PROTECTED] (most probably) from the list, as people who subscribe to lists but require each and every poster to the list to jump through hoops to have their messages delivered to the person in question don't deserve a subscription... Paul On Mon 27 Jun 2005, [EMAIL PROTECTED] wrote: From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Your email requires verification verify#UWUgyaRs5eQ0Ozo94QTL8FjSRuQdWxts Date: Mon, 27 Jun 2005 08:14:13 -0400 The message you sent requires that you verify that you are a real live human being and not a spam source. To complete this verification, simply reply to this message and leave the subject line intact. The headers of the message sent from your address are show below: From [EMAIL PROTECTED] Mon Jun 27 08:14:13 2005 Received: from azera by host1.namelessdns.com with local-bsmtp (Exim 4.50) id 1DmsVA-7m-Nn for [EMAIL PROTECTED]; Mon, 27 Jun 2005 08:14:13 -0400 Received: from [66.70.73.150] (helo=lists.samba.org) by host1.namelessdns.com with esmtp (Exim 4.50) id 1DmsVA-7e-LI for [EMAIL PROTECTED]; Mon, 27 Jun 2005 08:14:12 -0400 Received: from dp.samba.org (localhost [127.0.0.1]) by lists.samba.org (Postfix) with ESMTP id 63C05163933 for [EMAIL PROTECTED]; Mon, 27 Jun 2005 12:14:16 + (GMT) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: please help -- Variable syntax error
On Fri 03 Jun 2005, Bhadri N Govindarajan wrote: I am using rsync to sync w machines. I had no problem untill. Now i am trying the same script to sync 2 differnt machines. The code is the same. I just changed the machine names. When i execute the script i get Variable syntax rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(150) I am able to scp or ssh to the from the machines. I have proper keys. Where does the Variable syntax come from., I have 5 folders in include file. So this error is repeated 5 times. Variable syntax is a csh error message telling you that the way you're trying to use shell variables is wrong somehow. I tried rsync -vv testfile mydestmachine:/export/opt/geneva/temp/testfile even for this command i get the same error. You typed this at the commandline, exactly as shown, and it gave you this error? That's strange, as no variables are involved here. 'rsync' doesn't happen to be an alias, does it? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync a remote site with ssh root login disable
On Mon 30 May 2005, spiv007 wrote: but if i want to go in /etc/ssh/sshd_config and disable root login how can I still rsync the remote location if sshd_config root is diables: Either consider running rsync as a daemon, or set PermitRootLogin to forced-commands-only. Implementing the last is left as an exercise to the reader :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how to reduce rsync system usage
On Mon 30 May 2005, dtra wrote: ok i did all as suggested, and tried rsync -a [EMAIL PROTECTED]::rsync_module/ /path/to/bak/ and rsync -a --password-file=/path/to/pwfile [EMAIL PROTECTED]::rsync_module/ /path/to/bak/ but it says @ERROR: access denied to rsync_module from unknown (ip.add.re.ss) for both So look at the hosts allow and/or hosts deny options in the rsyncd.conf manpage. (The manual is there for a reason :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and cron : connection unexpectedly closed
On Mon 30 May 2005, Maurice Lucas wrote: I use rsync with the following command #/usr/bin/rsync -ta rsync.server.domain::dir/* /destination Note: it's always better to let rsync do wildcard expansion in such cases, and -a implies -t already; i.e. use: /usr/bin/rsync -a rsync.server.domain::dir/ /destination from the command prompt without any problem but if I use the same rsync command from cron I get the following error daemon.warn: May 30 15:18:42 rsyncd[24703]: rsync: connection unexpectedly closed (0 bytes received so far) [receiver] daemon.warn: May 30 15:18:42 rsyncd[24703]: rsync error: error in rsync protocol data stream (code 12) at io.c(420) Could somebody tell me why. Try adding -v options (often -vv is enough) to show more details. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how to reduce rsync system usage
On Fri 27 May 2005, dtra wrote: when i try this, it says unable to find rsyncd.conf nice -n 19 rsync -a --rsh=ssh -l remoteuser -c blowfish [EMAIL PROTECTED]::rsync_module/files /path/to/bak/ Wayne's point of saying use a daemon (to paraphrase a bit :-) is to eliminate use of ssh. By explicitly passing an --rsh option you tell rsync to connect via ssh, and then start a one-off rsync daemon for this transfer. Try starting the rsync daemon separately first on the remote, i.e. something like: rsync --daemon --config /etc/rsyncd.conf Then do the transfer like this: rsync -a [EMAIL PROTECTED]::rsync_module/files /path/to/bak/ Add nice -n 19 where appropriate :-) (on one or both ends, depending on what you want). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: RSYNC doesn't like Unicode?
On Wed 25 May 2005, Stuart Halliday wrote: Unicode isn't the reserve of the Chinese language. So I'm more concerned that no one else in the *Western* world hasn't came across this bug! I mean it doesn't appear to work with European accent characters does it? Rsync couldn't give a toss what bytes are used in filenames. The names are transferred as-is, and I've had no problems transferring filenames with spaces, accents, strange non-printing characters in them with rsync. That was between linux systems. The common issue seems to be windows systems, as far as I can tell here. Perhaps transferring files (or rather, filenames) between windows systems with differing locales (or language settings) is the problem, and someone with intimate knowledge of how to manipulate filenames on windows needs to investigate this. The problem seems to be that there aren't too many people that fall into that category. Could there be hundreds of Rsync servers silently skipping those odd dozen or so files with the occasional foreign letter in them? ;-) No. Or maybe Rsync isn't used as much as we believe? I believe it's used more than you might think. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: timestamps
On Wed 25 May 2005, Juergen Busam wrote: I'm syncing a windows share from a NetApp filer to a local partition on my RHEL3 box. I observe the following behavior: - rsync syncs more than neccessary, files that haven't changed since years - files that have been synced, aged for an hour or two on the netapp host I can't quite parse that last sentence... Environment: cifs share mounted to /backup/sync (mount -t smbfs -o username=user,password=pwd //server/share /backup/sync) Why use CIFS? Netapp understands NFS like no one else... so use NFS! this user has only read rights on the windows share mirror the mounted cifs share to a local partition (rsync -a --delete-after /backup/sync/ /dest/dir) Using rsync over a network drive isn't that useful, rsync is meant to optimize network traffic between de sender and the receiver, possibly at the cost of more disk IO. In this case, disk IO on the sender *is* network traffic... Use -v (one or more times) to see exactly what's going on. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: RSYNC doesn't like Unicode?
On Wed 25 May 2005, Stuart Halliday wrote: I was using Rsync to copy favourites from one english UK XP sp2 machine to a Windows 2000 sp4 english UK machine. No different language settings involved. It just so happened that I had placed in my favourites some URLs with a few European characters in their name. I wouldn't be surprised if e.g. W2000 is using wide characters, while XP is using UTF8 (or some other combination of different encodings). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Symlink options (was Mysterious (bogus?) rsync(d) errors ...)
While on the subject of symlinks, it would be nice to have an option to convert absolute symlinks (pointing to an object inside the tree being transferred) to relative ones during the copy, as those probably won't be useful. Example: $ cd /tmp $ find /tmp/a -ls (edited a bit for width) drwxr-xr-xMay 20 10:37 /tmp/a drwxr-xr-xMay 20 10:37 /tmp/a/b -rw-r--r--May 20 10:37 /tmp/a/b/foo lrwxrwxrwxMay 20 10:37 /tmp/a/bar - /tmp/a/b/foo $ rsync -a a a-copy $ find /tmp/a-copy -ls drwxr-xr-xMay 20 10:40 /tmp/a-copy drwxr-xr-xMay 20 10:37 /tmp/a-copy/a drwxr-xr-xMay 20 10:37 /tmp/a-copy/a/b -rw-r--r--May 20 10:37 /tmp/a-copy/a/b/foo lrwxrwxrwxMay 20 10:40 /tmp/a-copy/a/bar - /tmp/a/b/foo I'd like an option to make that bar symlink be converted from /tmp/a/b/foo to b/foo ... --convert-symlinks ? This could e.g. be useful when using a backup of a full system which gets placed somewhere under a subdirectory (there are plenty of absolute symlinks in the average system). Also a Debian user asked for this, to handle e.g. software whose installation procedure creates absolute symlinks (although that case might be fixable with a perl script that modifies the symlinks in place); see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=148967 Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Close list to outsider's posts?
On Fri 20 May 2005, Shachar Shemesh wrote: I'm assuming that Wayne is the obvious destination for this request. Can we make the mailing list reject emails from non-subscribers? This would drastically reduce the amount of spam we receive. I don't think I get all that much spam via this list (if I do, almost all of it gets caught by spamassassin :-) Closing the list will make it harder for people to quickly get help. In my experience that has a negative impact on the reputation of the project the list is about. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug#305932: rsync on a directory transfers the files of this directory
On Thu 19 May 2005, Wayne Davison wrote: On Thu, May 19, 2005 at 11:04:23AM +0200, Paul Slootman wrote: --files-from transferring the contents of a dir (i.e. including the files in it) specified in the input, even though the files aren't listed in the input. That's as intended, since a trailing slash means the contents of the directory. This is not unique to --files-from, but also happens when -d is specified and a directory on the command-line has a trailing slash (obviously -r must not be specified, or even deeper files would also be copied, not just the directory's immediate contents). The behavior was first introduced in 2.6.4 when --dirs (-d) was added. The man page Ah... The example given states: If /tmp/foo contains the string bin (or even /bin), the /usr/bin directory will be created as /backup/bin on the remote host (but the contents of the /usr/bin dir would not be sent unless you specified -r or the names were explicitly listed in /tmp/foo). I'd suggest changing the last line to: -r or the names were explicitly listed in /tmp/foo; or the bin has a trailing slash added so that the implied --dirs option causes the immediate contents of that directory to also be transferred). These implied options, while being terribly useful, are a disaster when trying to understand what a particular option does. Hence I think it can't hurt to sometimes be a bit more explicit, especially in the examples. thanks, Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Close list to outsider's posts?
On Fri 20 May 2005, Steve Sether wrote: From the web archives I've seen there's really not much spam at all. Well, Jaugen Leushyn from Bulgaria(?) seemed to want to make a point just now ;-( I just deleted about 20 of his unreadable spam. However, I'm still for not closing the list. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug#305932: rsync on a directory transfers the files of this directory
Hi, I got the following report from a Debian user, about --files-from transferring the contents of a dir (i.e. including the files in it) specified in the input, even thugh the files aren't listed in the input. This happens only when the dir name ends with a slash. I asked him to cook up a script to reproduce this (as it wasn't quite clear to me at first what happened exactly). Any ideas? Please include [EMAIL PROTECTED] in the recipients of any replies, so that the discussion is recorded with the bug. Thanks, Paul Slootman On Thu 19 May 2005, Vincent Lefevre wrote: On 2005-05-18 11:40:40 +0200, Paul Slootman wrote: Could you reproduce this with a fixed list, i.e. give a script that creates the directories and files, generates the list (e.g. with find) and then calls rsync? That might help to make it clear what exactly is going wrong... The following script shows the bug. #!/bin/zsh set -e cd $HOME mkdir test cd test mkdir src d1 d2 mkdir src/dir mkdir src/dir/dir touch src/dir/file touch src/dir/dir/file cd src echo dir | rsync --files-from=- -zuv --progress -e ssh . localhost:test/d1/ echo dir/ | rsync --files-from=- -zuv --progress -e ssh . localhost:test/d2/ cd .. ls -Fld **/* I get: drwxr-xr-x 3 lefevre lefevre 4096 2005-05-19 09:18:29 d1/ drwxr-xr-x 2 lefevre lefevre 4096 2005-05-19 09:18:29 d1/dir/ drwxr-xr-x 3 lefevre lefevre 4096 2005-05-19 09:18:32 d2/ drwxr-xr-x 3 lefevre lefevre 4096 2005-05-19 09:18:32 d2/dir/ drwxr-xr-x 2 lefevre lefevre 4096 2005-05-19 09:18:32 d2/dir/dir/ -rw-r--r-- 1 lefevre lefevre0 2005-05-19 09:18:32 d2/dir/file drwxr-xr-x 3 lefevre lefevre 4096 2005-05-19 09:18:26 src/ drwxr-xr-x 3 lefevre lefevre 4096 2005-05-19 09:18:26 src/dir/ drwxr-xr-x 2 lefevre lefevre 4096 2005-05-19 09:18:26 src/dir/dir/ -rw-r--r-- 1 lefevre lefevre0 2005-05-19 09:18:26 src/dir/dir/file -rw-r--r-- 1 lefevre lefevre0 2005-05-19 09:18:26 src/dir/file So, the problem occurs only when the directory ends with a slash (this is the case for d2 in the above script). -- Vincent Lefèvre [EMAIL PROTECTED] - Web: http://www.vinc17.org/ 100% accessible validated (X)HTML - Blog: http://www.vinc17.org/blog/ Work: CR INRIA - computer arithmetic / SPACES project at LORIA -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: easiest way to do incrimentals?
On Wed 18 May 2005, Wayne Davison wrote: On Wed, May 18, 2005 at 04:57:27PM +0200, Paul Slootman wrote: What I've been wondering is whether --fuzzy will also work with --link-dest, i.e. hard-link to files whose names are slightly different but are the same for other purposes Fuzzy doesn't really do much with --link-dest at the moment. It can cause an alternate basis file to be found to help improve the transfer, but it only looks in the destination directory (which should be empty when using --link-dest in its normal form), and it doesn't cause any differently named files to be linked together. OK, thanks; I thought as much. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: easiest way to do incrimentals?
On Wed 18 May 2005, Wayne Davison wrote: It can be better to do a --compare-dest or --link-dest into a new (empty) hierarchy, as this gives you a hierarchy with either (for --compare-dest) just the new version of changed files in it or (for --link-dest) all the files with identical files hard-linked together with the older files. What I've been wondering is whether --fuzzy will also work with --link-dest, i.e. hard-link to files whose names are slightly different but are the same for other purposes (e.g. log files that are rotated every day to log.2.gz, log.3.gz, etc.)? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: -x option inoperative with bind mounts
On Wed 18 May 2005, Louis-David Mitterrand wrote: I have this mount defined in /etc/fstab: /backup/current/usr/local/share/premier /var/www/g5/trunk/Naxos none bind,ro When I backup /var with rsync using (among other) the -x option (one filesystem) then the whole contents of /backup/current/usr/local/share/premier are also backed up as they are mounted on /var/www/g5/trunk/Naxos. Is this expected behavior? I thought -x would exclude any mount, including bind mounts. No, -x means stay on this filesystem. You *are* on the same filesystem... This is determined by looking at the device number as returned by stat() (on linux you may have the stat command that displays that info). That corresponds to the block device on which the filesystem resides, and that doesn't change with a bind mount. You'd need to add an explicit --exclude for the bind mount point. (Does the concept of bind mount exist on systems other than linux?) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync doesn't exit
On Fri 13 May 2005, Ken Gillett wrote: Sorry to have to ask again, but I'm still completely stuck on this. I'd suggest at least first upgrading to the latest rsync version, there have been plenty of fixes since the versions you're using. Can anyone suggest why rsync doesn't exit when there is no output console, even though it's not trying to display anything? I can only think it must be a quirk of the rsync code, but is there a solution? Having strace output would help. At the least, what strace shows when you attach it to the hung rsync processes. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: not deleting from the root
On Fri 13 May 2005, Mike Zupan wrote: I have a bit of an issue with rsync. I am using to keep directories in sync via another server for backup. Here is the server config [w1] path = /w1 comment = w1 web dir [w2] path = /w2 comment = w2 web dir Now on the client i run this command rsync -avv --delete --force domain.com::w1/ /w1/ It will NOT delete anything that is no on the server anymore.. for From the rsyncd.conf manpage: read only The read only option determines whether clients will be able to upload files or not. If read only is true then any attempted uploads will fail. If read only is false then uploads will be possible if file permissions on the server allow them. The default is for all modules to be read only. Note the last sentence... rsync -avv --delete --force domain.com::w1/apache/ /w1/apache/ I'm stumped to why that works.. but the one above doesn't That's because you're not accessing it via the module to delete that way. Add a line read only = false to the w1 module definition and it should work. However, you may also need to take a look at the uid description in the same man page, in case it still doesn't work... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync: select: Bad file number
On Mon 09 May 2005, Holger Bartsch wrote: I want to try to sync files from one SUN box to another. Both running rsync version 2.5.5 protocol version 26 Note that that is a pretty ancient version. A normal ssh connection can be established without problems. Can anyone point me into the right direction, please! Is rsync in the path of the remote user? (Just a random idea.) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How about a --min-size option, next to --max-size
On Thu 28 Apr 2005, Wayne Davison wrote: On Mon, Apr 25, 2005 at 03:10:52PM +0200, Paul Slootman wrote: Would a patch for --min-size be acceptable? Firstly, thanks for the patch -- I've tweaked it a bit and checked it I just saw that my patch for the manpage copied the example from --max-size a bit too literally...: +may be a fractional value (e.g. bf(--max-size=1.5m)). That should probably be something like: +may be a fractional value (e.g. bf(--min-size=2.5k)). Additionally, the --max-size description could now simply refer to the --min-size text for explanation of the SIZE value, something like: dit(bf(--max-size=SIZE)) This tells rsync to avoid transferring any file that is larger than the specified SIZE. See bf(--min-size) above for the description of SIZE. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 2675] --backup --suffix=.foo --delete-after doesn't backup deleted files
On Mon 02 May 2005, [EMAIL PROTECTED] wrote: --- Additional Comments From [EMAIL PROTECTED] 2005-05-02 09:36 --- It helps to look at the NEWS file from CVS so that you won't duplicate bugs that have already been fixed (this is mentioned in the checklist on the bug-reporting page, complete with a link to the unpacked NEWS file). Hmm, I looked through the messages to the mailing list since 2.6.4 was released, and couldn't find anything relevant. I had figured that something like this would have been mentioned... BTW, there was a bug reported about the fact that rsync now shows non-7bit-ascii chars as '?' now... I tried checking the NEWS and OLDNEWS files to see when that had been implemented (I thought 2.6.4 but wasn't sure), but I can't find any mention of this change. Am I missing it? As for excluding the backup suffix when using --delete-after, that has always been required -- I've considered adding an automatic exclude for it, but have left it compatible with older versions for the time being. It was quite unexpected, I had expected that rsync made a list of the existing files on the destination before any transfer of files (and hence before the creation of the backup files). So rsync scans all the files twice apparently? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
non-7bit-ascii chars in filenames
On Mon 02 May 2005, Paul Slootman wrote: BTW, there was a bug reported about the fact that rsync now shows non-7bit-ascii chars as '?' now... I tried checking the NEWS and OLDNEWS BTW (again): could a log-format option be added that prints the filename unmodified? That could help in determining supposedly strange behaviour as well in some cases, in addition to helping those people who use the log output for further processing in their backup scripts. I'll look into writing a patch... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How about a --min-size option, next to --max-size
On Thu 28 Apr 2005, Wayne Davison wrote: As for accepting --min-size, I'm considering it. It would be nice to have a more general solution to the problem of limiting the transfer based on a file's attributes (rather than a profusion of rsync options), but that doesn't seem to be on the horizon at the moment. I'm thinking that the best way might be to integrate part of find's code, and to implement a --limit-find=list of find options way, e.g. rsync -av --limit-find=-size -1 -mtime +10 -user pete src dest That way we don't have to describe all the possible find options, we simply refer to the find(1) manpage :-) I don't know how easily find's source code can be extracted and used in this way... Especially important for keeping up with changes in find... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug#306368: filter rules are too modern for remote rsync (which is 2.5.6)
I received the following bug report for the Debian rsync package today. I wouldn't have expected 2.6.4 to refuse to talk to even a 2.6.2 in this way... Perhaps Wayne could comment? Paul Slootman On Tue 26 Apr 2005, Alexey Feldgendler wrote: Rsync fails to push filters to the the remote host if the remote rsync is older than the Debian's. I've tried to connect to a remote host with the following rsync: rsync version 2.5.6 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums (that was a Fedora Core 1 box) Here is the command I've tried to execute on my Debian system: rsync -Cavvv ~/ports-utils bfc1: (bfc1 is the remote Fedora Core 1 host with rsync 2.5.6) Here is the output from rsync on Debian: opening connection using /home/feldgendler/bin/ssh bfc1 rsync --server -vvvlogDtprC . [sender] add_rule(-C ) [sender] add_rule(- RCS) [cvsignore] [sender] add_rule(- SCCS) [cvsignore] [sender] add_rule(- CVS) [cvsignore] [sender] add_rule(- CVS.adm) [cvsignore] [sender] add_rule(- RCSLOG) [cvsignore] [sender] add_rule(- cvslog.*) [cvsignore] [sender] add_rule(- tags) [cvsignore] [sender] add_rule(- TAGS) [cvsignore] [sender] add_rule(- .make.state) [cvsignore] [sender] add_rule(- .nse_depinfo) [cvsignore] [sender] add_rule(- *~) [cvsignore] [sender] add_rule(- #*) [cvsignore] [sender] add_rule(- .#*) [cvsignore] [sender] add_rule(- ,*) [cvsignore] [sender] add_rule(- _$*) [cvsignore] [sender] add_rule(- *$) [cvsignore] [sender] add_rule(- *.old) [cvsignore] [sender] add_rule(- *.bak) [cvsignore] [sender] add_rule(- *.BAK) [cvsignore] [sender] add_rule(- *.orig) [cvsignore] [sender] add_rule(- *.rej) [cvsignore] [sender] add_rule(- .del-*) [cvsignore] [sender] add_rule(- *.a) [cvsignore] [sender] add_rule(- *.olb) [cvsignore] [sender] add_rule(- *.o) [cvsignore] [sender] add_rule(- *.obj) [cvsignore] [sender] add_rule(- *.so) [cvsignore] [sender] add_rule(- *.exe) [cvsignore] [sender] add_rule(- *.Z) [cvsignore] [sender] add_rule(- *.elc) [cvsignore] [sender] add_rule(- *.ln) [cvsignore] [sender] add_rule(- core) [cvsignore] [sender] add_rule(- .svn/) [cvsignore] [sender] parse_filter_file(/home/feldgendler/.cvsignore,8380,0) [sender] add_rule(- semantic.cache) [cvsignore] filter rules are too modern for remote rsync. rsync error: syntax or usage error (code 1) at exclude.c(1119) _exit_cleanup(code=1, file=exclude.c, line=1119): about to call exit(1) Even adding --protocol=26 to the above command did not help. I've also tried the same with another remote Fedora Core 1 host which had: rsync version 2.6.2 protocol version 28 Copyright (C) 1996-2004 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums and got a similar error message. Only installing rsync 2.6.4 (I actually copied the binary from Debian to Fedora Core 1) resolved the problem. So, rsync 2.6.4 only works with rsync 2.6.4, and this renders the package almost unusable because most remote osts run different (older) versions of rsync. Alexey. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug#306368: filter rules are too modern for remote rsync (which is 2.5.6)
On Tue 26 Apr 2005, Wayne Davison wrote: On Tue, Apr 26, 2005 at 10:25:14AM +0200, Paul Slootman wrote: I wouldn't have expected 2.6.4 to refuse to talk to even a 2.6.2 in this way... It shouldn't, and (interestingly) it wouldn't have if --delete had been specified (apparently I did all my backward-compatibility testing using --delete). Amazing how little things can be significant :-) Thanks for the quick fix! Paul -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How about a --min-size option, next to --max-size
There's a rather old bug report in Debian's bug tracking system (see http://bugs.debian.org/27126) about wanting to be able to specify the maximum file size, as well as the minimum file size. Here's the text: Sometimes, it's useful to specify a file size range one is interested in. For example, I'd like to keep an up-to-date mirror of Debian, but I currently have size problems, so I'd like to skip all files some limit. I've also had the opposite problem (with ordinary mirror) - a server where there were lots of small junk files I was not interested in. Both cases has its merits. Would a patch for --min-size be acceptable? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Need help with rsync
On Mon 25 Apr 2005, VC123 wrote: : rsync /export/home/amg/* [EMAIL PROTECTED]:/export/home/amg [EMAIL PROTECTED]'s password: rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(359) Try running with -vvv Is the rsync binary in a standard place in the PATH for [EMAIL PROTECTED] Does /export/home/amg/* expand to files, or also directories? If so, is it your intention to also transfer the contents of those directories? Then you will need some extra options. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncd server daemon not allowing connections
On Mon 25 Apr 2005, [EMAIL PROTECTED] wrote: I'm trying to run rsync in server mode and it appears to start normally, but it refuses all connections (refuses connection when I tried telnetting in on localhost 873!). How exactly do you start rsync then? I used the following command: rsync --daemon --server --config-file=/etc/rsyncd.conf . It responds normally: @RSYNC 28 Hopefully not like this... --config-file isn't a recognized option. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
address option in rsyncd.conf
I wanted to restrict rsync to listen only on one IP address on a multi-homed system. I put an address aa.bb.cc.dd option in the (single) module definition, as the manpage shows that address is a module option, not a global one. However, lsof showed that rsync had bound to * instead of the specified IP address. Moving the address line to the global part did the right thing. On the one hand I understand the current behaviour (restricting the use of a specific module to one of the addresses, thus letting one network access one module and another network access another module), however in the case of a single module (or perhaps when all modules have the same address line), it would make sense to effectively promote the module definition to a global one. In short, the behaviour was unexpected at first sight, but thinking about it, it is actually logical. Maybe the manpage's description of the address option should be expanded a bit, e.g. add this line: To make the rsync daemon listen to a single IP address, put the address option in the global section (even if there's only one option). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync over ftp
On Tue 19 Apr 2005, roger tremblay wrote: I would like to use rsync client over ftp. I know how to use it with ssh but it just can't work with ftp or ncftp. The purpose of usign it with ftp is simple: most webhosting soft. like cpanel or plesk don't monitor ssh bandwidth usage. You need to monitore it in order to suspend an over-quota account. I tried the following: rsync -avze '/usr/bin/ftp -nv' /tmp/testdir [EMAIL PROTECTED]:./testrsync/ I dont get any result. That will never work... ftp is a completely different protocol to rsh or ssh, think of it as batch-oriented (GET... wait PUT... wait) instead of interactive. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Spam to this list
On Tue 19 Apr 2005, Andrew Gideon wrote: Paul Slootman wrote: There's a difference between giving a 5xx response during SMTP, and first accepting a message and then later bouncing it to the (supposed) envelope sender. I believe spamcop is protesting the latter, not the first. I agree with them. 20% of the junk I get are bogus bounces. For good or ill, SMTP is a store-and-forward mechanism. The node in the process of delivering to the node which issues the rejection is no longer in a position to issue its own rejection. Instead, it must send a separate bounce message to the - claimed, unfortunately - sender. Yes, and if the first system in the chain would do it, spam wouldn't leave the originating system. (The first system is often an open relay, or my system in case of zombie PCs that are sending out the spam.) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Spam to this list
Continuing this off-topic issue: On Mon 18 Apr 2005, Shachar Shemesh wrote: John E. Malmberg wrote: The I.P. address is listed in bl.spamcop.net as hitting spamtraps. Just so you know, spamcop view bounces as spam. According to them, you should never send bounces. I believe the right approach is to convince admins to drop spamcop from their RBL list, rather than remove the very essential NACK SMTP has from all servers, as per spamcop's request. There's a difference between giving a 5xx response during SMTP, and first accepting a message and then later bouncing it to the (supposed) envelope sender. I believe spamcop is protesting the latter, not the first. I agree with them. 20% of the junk I get are bogus bounces. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync duplicates after file removal
On Thu 14 Apr 2005, Jon Essen-Moller wrote:
After I have removed some files from the backup computer with the command
find /home/user/.Maildir/.Spam/* -mtime +30 -type f -exec rm {} \;
does the command:
rsync -av -e ssh -l root host:/home/* /home
create many duplicates.
Does it create those duplicates in the /home/user/.Maildir/.Spam/
directory, or elsewhere?
BTW, I'd use:
rsync -av -e ssh -l root host:/home/ /home
Let rsync take care of the recursion, as you've given it the -a option.
What I expect is for it to just restore the backup to what it was
like before.
So would I, so I'd be interested in seeing what files are duplicated
where.
Paul Slootman
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Tue 12 Apr 2005, Jeff Schoby wrote: What the maximum filesize rsync can transfer? 2GB (4GB?) should always be possible, when built with the appropriate options for large file support 4GB files are no problem. I'm trying to rsync one of my servers to another but the rsync is croaking on a file that's barely 1GB. Well, it shouldn't... Please define croaking? Any diagnostic messages? Try running with -vvv Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Tue 12 Apr 2005, Christophe Kalt wrote: FWIW, I just upgraded to 2.6.4, and that has solved a problem i'd been having for a few weeks where 2.6.3 repeatedly failed to synchronize a 5GB file. Don't remember seeing anything in the NEWS or other making me think upgrading would help, but it did. Curious, I too have had reports of things that previously mysteriously failed, now started working with 2.6.4 :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
trailing slash on module name
A trailing slash on a module name has no effect, which is on the one hand logical as it's not a directory name; on the other hand it's not consistent either (as experienced by a user). I suggest at least adding some comment to the manpage where trailing slashes are discussed. Perhaps also mention it in the rsyncd.conf manpage, where perhaps it may be noticed sooner. If I find the time this week I may come up with an appropriate text :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
does --files-from transfer files in the order given?
There are one or two open bug reports in the Debian bug tracking system that boil down to being able to specify the order files are trasferred. One example where this is useful is to first transfer the packages, then the list of packages. That means that the available list of packages is always consistent, without any packages listed being temporarily unavailable. I'm sure Wayne knows this without having to look at the code :-) If it does, a note in the manpage would be useful. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: does --files-from transfer files in the order given?
On Sun 10 Apr 2005, Wayne Davison wrote: On Sun, Apr 10, 2005 at 03:38:42PM +0200, Paul Slootman wrote: There are one or two open bug reports in the Debian bug tracking system that boil down to being able to specify the order files are trasferred. There is no support in rsync for allowing the user to control the order the files are transferred in a set of files -- the transfer always happens in the internal sort order. Ah :-( OK... As I guess that this is not going change anytime soon, I'll report that back to the relevant reporters. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
PATCH: cosmetic fix to output with --existing -vv
Currently, rsync --delete -avvn test1/ test2/ --existing
will give output such as:
not creating new file newdir
not creating new file newdir/newfile
not creating new file newdir/subdir
It would make more sense if it said:
not creating new directory newdir
not creating new file newdir/newfile
not creating new directory newdir/subdir
This following simple patch fixes this.
Paul Slootman
--- generator.c.orig2005-04-10 21:15:07.0 +0200
+++ generator.c 2005-04-10 21:15:08.0 +0200
@@ -665,7 +665,8 @@
if (only_existing statret == -1 stat_errno == ENOENT) {
/* we only want to update existing files */
if (verbose 1) {
- rprintf(FINFO, not creating new file \%s\\n,
+ rprintf(FINFO, not creating new %s \%s\\n,
+S_ISDIR(file-mode) ? directory : file,
safe_fname(fname));
}
return;
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Full replication
On Fri 08 Apr 2005, Beach Computers wrote: Running Rsync on a Windows box. Trying to do replication, and it doesn't seem to be working as expected. I'm using -r. I put test.txt in the source dir. It copied over just fine. Deleted test.txt from the source. And it still exists in the target. Any ideas? Use --delete (it's all in the docs). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Synchronise trees
On Thu 07 Apr 2005, Eugene Kramer wrote: take a look at --delay-updates option at http://rsync.samba.org/ftp/rsync/rsync.html: Another possibility is --link-dest to build a new tree in parallel. That's what we do for a local Debian mirror. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rückruf: [rsync-announce] Rsync 2.6.4 released
On Thu 31 Mar 2005, Markus Kemkes wrote: Markus Kemkes möchte die Nachricht [rsync-announce] Rsync 2.6.4 released zurückrufen. Unfortunately it's not possible to cancel messages sent to a mailing list :-) Paul -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: pauses sync'ing between tmpfs and disk on Linux 2.4.x
On Wed 23 Mar 2005, Ray Van Dolson wrote: No swap in use... however, keep in mind that this is a tmpfs filesystem which I know differs from a ramfs in that it can be swapped. However, one would think this would show up in the 'free' output above which it does not... However, maybe my understanding of that is incorrect. I could switch to an actual ramfs filesystem to eliminate swapping as a possibility. However, that would require a reboot unfortunately so I'd have to try it later. :-) You could try removing the swap, if it's not needed anyway... swapoff -a Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync with 8bit filename characters or UTF8
On Tue 08 Mar 2005, Gregory Bleiker wrote: I'm having a problem that is driving me nuts. I am syncing directories that have characters which are not 7-bit Latin encoded, ie. äöü ' and the likes. I'm using a windows/cygwin client machine on one side and a FreeBSD server on the other side. I use an include/exclude list to specify which directories to sync. If I have a c:\backup\löl c:\backup\bla I put + /backup/löl + /backup/ - /backup/* - /* in my include/exclude file list I then call rsync something like rsync --include-from=files_c.inc /cygdrive/c/ [EMAIL PROTECTED]:data/current/c (some params omitted for readability's sake) Now c:\backup\löl will not be synced because the -/backup/* rule excludes it. It seems this is because rsync is seeing /backup/löl as /backup/lvl, which is the non-extended ascii version of this path (ö is 246 in ascii extended, 246-128=118, which is v). I have tried putting + /backup/lvl in the include file, without success. You mention UTF-8... That's different from iso8859-1, which is what you're using when you say that ö is 246 in ascii extended. If the filesystem stores names as UTF-8, then you need to use UTF-8 in anything that refers to filenames rsync doesn't care about special chars in filenames (or in the contents of files), it does a simple byte-for-byte match. In UTF-8 ö will be stored as two bytes... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync write behaviour / efficiency
On Mon 07 Mar 2005, [EMAIL PROTECTED] wrote: If, for example, you have a 500MB file (say an ISO) whose modification consists solely of a few bytes added on to the beginning of the file. Will it create an entirely new 500MB file? Or will it somehow know how to insert the bytes at the beginning of the remote file? Inserting bytes at the beginning of a file can't be done, at least not on unix-like systems. That can only be done by copying the file. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: wierd duration shown in progress with 0 byte files
On Sat 05 Mar 2005, Wayne Davison wrote: I don't quite see the point of setting diff to 1ms if it's zero... Because an elapsed time of 0 means that any data that arrived, arrived very quickly. Setting the rate to 0 when the elapsed time is 0 is the opposite of what we need -- an infinite data rate. I like the idiom of changing the elapsed time from 0 to 1 so that it divides safely into the quantity of sent data, and thus gives us a non-0 rate (if any data was actually sent). Yes, I get that, but the else branch of the if did a diff ? bla /diff : 0 i.e. only using diff iff it's not zero, while the then branch went about it another way (i.e. setting diff to 1 if it was zero). I don't like inconsistencies like that in programs, esp. so close together... Doing the division only if diff is 0 saves an assignment and a division in the case it was zero... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --one-file-system problem
On Tue 01 Mar 2005, Chuck Wolber wrote: rsync commandline: /usr/bin/rsync -e /usr/bin/ssh --archive --compress --sparse --verbose --stats --delete --numeric-ids --partial --relative --one-file-system target.host:/ /destination/path/ target rsync version: 2.6.3 destination rsync version: 2.6.2 The server we're trying to synchronize contains directories within / that are mounted to other locations within /. When the sync occurs, the Hmm, do you mean bind mounts? Those _are_ effectively the same filesystem in your case. mounted directories get copied, despite the fact that we use the --one-file-system argument. Is this a bug, or have I misunderstood the use of the --one-file-system argument? The check for one-file-system is generally done by comparing the device number to the one on the starting point, not by checking for mount points (I haven't checked how rsync does it exactly). Such a check won't discover bind mounts... I can wrap my mind around the fact that the mounted directory is actually a part of the filesystem that it is mounted to, and thus can't be divorced from the concept of one file system. If that's truly the case, is it worth my time to come up with a --really-one-file-system patch? I think there wouldn't be much demand for it, it's a pretty specific case. I'm guessing it would also impact performance, unless e.g. you use /proc/mounts (on Linux) to generate an internal --exclude list... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsyncing really large files
On Thu 24 Feb 2005, Wayne Davison wrote: It would certainly be possible to change the algorithm to not cache the data (and thus only allow the current block to be compared), but I don't think that idea has general enough interest for me to work on for inclusion in rsync. You might want to look into coding it up for yourself. I think that this would be useful enough in itself, e.g. when syncing database storage files. The chance that blocks move around (without changing) isn't that large. I've been considering something like that a while... Useful when syncing a 40GB database when there's mainly only insertions. I never had the time to persue it, though... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Incremental Backups
On Wed 16 Feb 2005, [EMAIL PROTECTED] wrote: I simply want to maintain a dated backup of a server so that I could always go back to a certain date. I would like to keep this structure for each day for the last seven days, then one weekly snapshot for each week in the month and then each month I would like to have as well. Take a look at http://www.dirvish.org/ which makes it easy to do what you want. It's sort of a wrapper around rsync. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: getting rsync to work +automating password line
On Thu 10 Feb 2005, Gil Naveh wrote: The --rsync-path=PATH did the work. :) Is there a doc that shows all the options that comes with rsync? - when I tried #rsync --help I did not get the --rsync-path option. Hmm, my copy does show that... (2.6.3) However, the man page is the usual location for full documentation with unix commands. Finally, the next step for me is to automate rsync through crontab. But when I type rsync -a e ServerB/... I get a prompt for #password: I tried rsync with the option --password-file=/name/of/file/with/password but it still asked me for password! The file contains a single line with the password? There is newline after the password? The file is readable for the process (but not for others!) ? You could also put the password in the environment variable RSYNC_PASSWORD. Check that the environment can't be read by all users on your system. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: getting rsync to work
On Thu 10 Feb 2005, Gil Naveh wrote: I am trying to do a very simple thing, just transfer a file from machine A to machine B using rsync with ssh. This is what I'm typing: # rsync -a -e ssh serverB:/tmp/rsync/test1 n serverA:/tmp/rsync/ # password: X This is what I get: # bash: rsync: command not found # rsync: connection unexpectedly closed (0 bytes received so far) [receiver] # rsync error: error in rsync protocol data stream (code 12) at io.c(359) Basically it seems to go wrong after logging into the remote machine when it says bash: rsync: command not found. rsync is definitely installed on both machines which are Solaris9 and is in the user's environment path. So I don't understand why it says rsync: command not found...? .profile etc. is typically not read when executing a command via ssh. That's why the --rsync-path=PATH option was invented. Add that (with the correct path to the rsync binary on the remote!) and things should start working. PS: what's the 'n' doing between /test1 and serverA: in your command? I hope it's a typo... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncd.conf is not created although successful installation of rsync on debian but
On Sun 06 Feb 2005, alok barnwal wrote: I have installed rysnc-2.6.3 on my debian linux ( 2.2.20-idepci #1 Sat Apr 20 12:45:19 EST 2002 i686). While configuration it was successful and i installed it without any error. Now, in default installation it should create rsyncd.conf as detault configuration file.. but it has not been created anywhere in system. No, it shouldn't create rsyncd.conf as detault configuration file. If you want an rsyncd.conf, there's an example in /usr/share/doc/rsync/examples/rsyncd.conf . [EMAIL PROTECTED]:/tmp/temp/rsync-2.6.3# which rsync /usr/local/bin/rsync Ah, you're building your own rsync. (Why?) Please let me know how to create configuration file and what could be reason that ? Read the doc for rsyncd.conf and create it yourself by hand, it's impossible to create an rsyncd.conf file that does what you want. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: RSync with /really/ long file lists
On Fri 03 Dec 2004, Brandon Knitter wrote: rsync 2.5.7 linux RHES v3 We need to sync over 4 Million files, and when we run rsync we run out of memory! :( Try the latest version, there have been improvements in memory efficiency since 2.5.7. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync - Copying Every File Every Time
On Mon 10 Jan 2005, Gabby James wrote: On my client I use the following command: /usr/local/bin/rsync --verbose --progress --stats --recursive --links --perms --compress 10.X.X.X::ftpscript/ /home/g/test I would have expected to see the --times option also... Or just the -a option. I pasted output of my rsync command below. The FAQ mentioned people thinking files were transferred every time when in reality only permissions or group were being updated. When I compare the files on my client server, the permissions, owner group are the same. What am I missing? Why is every file being transferred every time? Are they actually? Check the ctime of the files before and after the transfer (ls -lc). Also, are any of the files (source or dest) hardlinked? Also, the files are pretty tiny, I wouldn't be surprised that transferring the files is more efficient than checking for changes :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: date problem with ntfs or vfat mounted file sytem
On Tue 04 Jan 2005, jean-philippe proux wrote: under linux I have to mount windows filesystem (ntfs or vfat). but with rsync (or with a single touch command as well) I can not write a file with a correct date on /mnt/windows_vfat or /mnt/windows_ntfs with ntfs -- only write file system ! NTFS support in linux is still limited to read-only (does it really say only write?!). There is an experimental option for write support in the kernel, but AFAIK that's limited to overwriting files with the exact same length. Hence pretty useless for rsyncing to. There is a commercial NTFS filesystem module available that has full r/w support, but I doubt you're using that. with vfat -- wrong date ! so rsync does a full copy each time. Show some cut paste examples of the wrong date... VFAT only has a 2-second resolution for the time for some bizarre reason; try using rsync's --modify-window option. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: can not start rsync, address already in use
On Mon 20 Dec 2004, [EMAIL PROTECTED] wrote: I am running the following command to start the rsync server manually rsync --daemon Then I am getting the following error 2004/12/20 22:41:40 [3396] rsyncd version 2.5.7 starting, listening on port 873 2004/12/20 22:41:40 [3396] rsync: open inbound socket on port 873 failed: Addres s already in use 2004/12/20 22:41:40 [3396] rsync error: error in socket IO (code 10) at socket.c (394) Can anybody help me. It sounds pretty clear to me: something is already listening to port 873. Perhaps an rsync daemon is already running? Perhaps you first had inetd configured to listen to port 873? On some systems you can do fuser 873/tcp to show what's using that port. Aside: while you're at it, why not upgrade to the latest version of rsync. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and broken symlinks
On Thu 16 Dec 2004, Wayne Davison wrote: As for the fix, one could argue that ignore nonreadable has a bug in it when dealing with symlinks -- if readlink() worked, the symlink was readable, so there is no need to call access() on it. The appended patch changes this so that rsync doesn't ever use access() on a symlink. Comments or disagreement? Just to be certain - readlink() is only called if --copy-links is NOT specified? In that case, I agree. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync: a bit of confusion
On Mon 06 Dec 2004, Payal Rathod wrote: This is the first time I have setup rsync.conf like, [...] Then from 192.168.10.10, I tried, rsync -avz -e ssh [EMAIL PROTECTED]:/var/qmail/control/* . and it worked. So far so good. But then again it worked from 192.168.10.11 Rsyncd.conf is only used when running rsync as a daemon; when invoking it via ssh like you do, the rsyncd.conf isn't used at all. (It's possible to start rsync in daemon mode over an ssh connection, but that's a bit exotic IMHO.) Also note that you passed a wildcard, it's better to let rsync do the expansion, which it will do as you gave the -a option. You connect to the rsync daemon by using a command line like: rsync -avz 192.168.10.1::qmail-control . Of course, you will have to have started the daemon on 192.168.10.1 first. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync progress indicator?
On Tue 07 Dec 2004, Dag Rune Sneeggen wrote: I've been reading through the man pages for rsync, yet I can't seem to find a way to provide progress indication and/or current download speed for total and/or individual files... How about --progress? Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: man page: -u: mention when the dates are the same
On Tue 30 Nov 2004, Dan Jacobson wrote: Paste error: --size-only Normally rsync will skip any files that are Normally rsync will This was a patch error in de Debian 2.6.3-1 version that has already been fixed in the 2.6.3-2 version. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Uninitialized static structure in generator.c/write_sum_head
On Sat 27 Nov 2004, John E. Malmberg wrote:
In the module generator.c, there is a static struct sum_struct null_sum
that is not initialized by any way that I can determine in the routine
write_sum_head.
A static data structure is guaranteed to be initialised to zero...
Shouldn't it more correctly be:
const struct sum_struct null_num = {0, 0, 0, 0, 0, NULL};
Perhaps more readable, but in no way more correct.
Paul Slootman
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: patch for replacing non-printable chars in filenames
On Fri 26 Nov 2004, Stefan Nehlsen wrote: On Tue, Nov 23, 2004 at 05:29:57PM +0100, Paul Slootman wrote: +/* Replace non-printing chars in the string, most probably due to + * wierd filenames. Skip the first and last chars, they may be \n */ +int i; +for (i=1; ilen-1; i++) +if (!isprint(buf[i])) +buf[i] = '?'; Is looping over strings a good idea in times of UTF-8? It is if you don't know the strings are in UTF-8, and you want to prevent garbage chars reaching the tty (the whole point of this exercise :-) Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: patch for replacing non-printable chars in filenames
On Tue 23 Nov 2004, Wayne Davison wrote:
On Tue, Nov 23, 2004 at 05:29:57PM +0100, Paul Slootman wrote:
Here's a patch. Opinions?
I think that a better place to munge the name would be in the
safe_fname() routine in utils.c (which already munges newlines
characters into question marks). The reason I didn't change
any other characters was because I feared that it would mangle
foreign filenames that use high-bit characters. I'd want some
feedback from such users before accepting such a patch.
Not all filenames that are printed are passed through safe_fname()
AFAICS, e.g. a random piece of code from rsync.c:166 :
if (verbose 2) {
if (change_uid) {
rprintf(FINFO,
set uid of %s from %ld to %ld\n,
fname, (long)st-st_uid, (long)file-uid);
}
if (change_gid) {
rprintf(FINFO,
set gid of %s from %ld to %ld\n,
fname, (long)st-st_gid, (long)file-gid);
}
}
Note that isprint() will take into account the locale in effect, i.e.
when using the FR_fr locale things like é should be recognized as
printable. At least, under linux that would seem to be the case; from
the NOTE section of isprint's manpage:
The details of what characters belong into which class depend on
the current locale. [...]
setlocale(LC_CTYPE, NULL) probably needs to be called during program
startup, however...
The bug reporter (a frenchman I believe) was agreeable to all non-ASCII
chars being replaced however; that's preferable to having his tty messed
now and again.
Making it depend on whether stdout is a tty may also be useful.
Paul Slootman
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
patch for replacing non-printable chars in filenames
There's a bug reported in Debian about the tty being screwed up by wierd
filenames, see http://bugs.debian.org/bug=242300
On the one hand, find will also do this. On the other hand, ls will
replace such chars with a question mark. Upon inspection, it appears to
be fairly simple to also do this in rsync (in the rwrite() function).
Here's a patch. Opinions? Perhaps don't do it unconditionally, i.e.
offer some way to turn it off?
Paul Slootman
--- log.c.orig 2004-10-04 11:51:37.0 +0200
+++ log.c 2004-11-23 17:27:29.0 +0100
@@ -180,6 +180,15 @@
buf[len] = 0;
+if (code == FINFO) {
+/* Replace non-printing chars in the string, most probably due to
+ * wierd filenames. Skip the first and last chars, they may be \n
*/
+int i;
+for (i=1; ilen-1; i++)
+if (!isprint(buf[i]))
+buf[i] = '?';
+}
+
if (am_server msg_fd_out = 0) {
/* Pass the message to our sibling. */
send_msg((enum msgcode)code, buf, len);
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: specifying --bwlimit together with --daemon fails
On Wed 17 Nov 2004, Wayne Davison wrote: See the recent discussion about this: http://www.mail-archive.com/rsync@lists.samba.org/msg11850.html I need to subscribe a gmail account to the rsync list, I didn't find that message in my own archive :-( Thanks. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
specifying --bwlimit together with --daemon fails
I received a bug report that starting with 2.6.3, rsync fails when started from inetd with the --bwlimit option (actually, I narrowed it down to that :-). Earlier rsync versions did work. What I see from the strace output is that the rsync --daemon process sets fd 0 to ON_NONBLOCK, then does a select() on fd 0 for write (which shows it's ready for writing), and then does a write of zero bytes, which returns 0 (of course), and that is apparently interpreted as a failure, at which point it syslogs the following: Nov 17 19:07:38 preston rsyncd[610]: rsync: writefd_unbuffered failed to write 12 bytes: phase unknown [receiver]: Success (0) Nov 17 19:07:38 preston rsyncd[610]: rsync error: error in rsync protocol data stream (code 12) at io.c(909) I see it thinks it's trying to write 12 bytes, but strace tells otherwise... I guess that the bwlimit algorithm is working against us here. Unfortunately I can't look into the code right now; I'll look tomorrow (GMT) if no one else has beaten me to it... For reference, the bug report can be found at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug= 281519 (remove the space after bug= ; these pages are regularly harvested by spammers :-( ). Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: timestamp question
On Fri 12 Nov 2004, Mark Watts wrote: Server D is mirroring Server B. We'd like to change Server D to be mirroring Server C instead. Previously when I've swapped between servers like this, vast swathes of the archive are deleted and redownloaded (I use -av --delete-after) for no reason other than the timestamp being different. If rsync is used with -a, the timestamps _shouldn't_ be different. If I use -av -I, will it ignore the timestamps and only go on file size? Yes. The archive in question is a several hundred GB linux distro archive so I'd rather not have to redownload it :) Oops... I tried to move a local Debian mirror (with daily snapshots that has unchanged files hardlinked to the previous day) to a larger system. I ran into the problem that the address space was too small for rsync's memory requirements (the process grew to 4GB then crashed on out of memory). I ended up doing each day separately with a --link-dest pointing to the previous day... Only took 5 days :-/ Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Copying Oracle data files
On Wed 10 Nov 2004, Guilhem Bichot wrote: What MySQL or Oracle can bear is if your copy is instant. On Linux there is LVM for that: you create the LVM snapshot, rsync files, discard the snapshot. It does work. Here's an intro to LVM: It's also advisable to put oracle into backup mode before making the snapshot. That's what we did at a large oracle place where I used to work (using Solaris and a NetApp filer). Making the snapshot only takes a couple of seconds, so this has negligible impact on the performance. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --rsh not reading .ssh/config
On Thu 11 Nov 2004, Bill Moseley wrote: Again, the problem seems that on this machine .ssh/config is not being read, but only when ssh is run via rsync. My guess is this is just a problem with running the older rsync. Ah, you mean .ssh/config is being read, but not when ssh is run via rsync.. Your sentence is a bit confusing :-) Is that the case, and is there a workaround other than upgrading rsync? What works a lot better is understanding the rsync syntax. Here's some examples of how it's not reading my .ssh/config: ... $ rsync -a --rsh=ssh -F $HOME/.ssh/config -i $HOME/.ssh/mysqldump remotehost:: unknown host: remotehost When you use a double colon host::, then you're telling ssh to connect directly to the rsync daemon running on that host using port 873. In other words, ssh does not come into the equation, so it's pretty logical that the .ssh config isn't being read... Use a single colon, and it should be as you expect. This is covered in basically the first page of the manual. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: pb with fat FS on linux with rsync
On Thu 04 Nov 2004, jean-philippe Proux wrote: if I do : [EMAIL PROTECTED] tmp]$ rsync -aH /tmp/jppjppjpp ~jezequel/sauvNT chown /home/jezequel/sauvNT/jppjppjpp failed: Operation not permitted mkstemp /home/jezequel/sauvNT/jppjppjpp/.mlkjmlkj.dKl4JR failed: Operation not permitted chown /home/jezequel/sauvNT/jppjppjpp failed: Operation not permitted rsync error: some files could not be transferred (code 23) at main.c(632) I've got a error message ! /tmp is on a ext3 filesustem type and sauveNT is a mony point of fat partition Of course I can do : date~jezequel/sauvNT Yes, but a chmod will fail (as FAT doesn't understand that concept). The -a option implies preserve user, group, times, permissions, all concepts that are not (or almost not, in the case of permissions) supported by FAT. Symlinks also aren't possible. The -H option is also useless, as FAT has no concept of hard links. Perhaps (if it's just for a backup) it's better to use tar to write a tar file to the FAT partition. Otherwise don't use -aH, but only use those options that are possible with FAT. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync doesn't delete unreferenced directories.
On Mon 01 Nov 2004, Steve Sether wrote: I'm using rsync to backup multiple directories, i.e. rsync -azq --delete --relative /dog /cat /fish [EMAIL PROTECTED]::modulename Everything works fine, except if I remove a directory from the list of those being backed up, it isn't deleted on the other side. For example: [...] I realize this behaviour is probbably by design but is there a way to get rsync to delete directories no longer being backed up? Perhaps use --delete-excluded and --exclude the directory? Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete option
On Fri 29 Oct 2004, [EMAIL PROTECTED] wrote: I'm a bit paranoid, but had to ask-- How safe is it to use the --delete option? ie, when using it locally (or via a mounted network volume), will I *ever* have to worry about my actual source files being delete mistakenly (instead of dest. files)? Source files will never be deleted. Of course, if you screw up the parameters to rsync sufficiently, you might end up passing the source dir as the destination, but there's nothing any software can do about that sort of error... When in doubt, use the --dry-run option to show what rsync is going to do without actually doing anything. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 8 bit filenames?
On Wed 27 Oct 2004, Dan Stromberg wrote: Are there any characters that can occur in filenames that will choke rsync? We're transferring lots of data, and some of our users' filenames appear to have their high bit set. I don't expect it to cope with filenames having a \0 or / in them (sometimes created over appletalk shares - strange but true), but it'd be great to know if rsync can handle all other characters, or only normal characters. rsync doesn't do anything with the individual bytes in a filename; to rsync it's just a string of characters. That said, a null byte will of course serve to terminate the string there... and a slash will be interpreted as a directory separator (although I'm not familiar enough with the rsync internals to know whether rsync explicitly checks for '/'). I've transferred filenames with all sorts of wierd characters with rsync without any problem. One potential problem is if the character set on the two systems is different. Because rsync does nothing with the bytes, a system that uses e.g. UTF8 and a system that uses iso8859-1 will show the same string of bytes differently if there are any 8-bit chars used... Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsyncing files
On Thu 28 Oct 2004, Ryan Holowaychuk wrote: I want to to an Rsync with 4 servers. I want to sync the passwd and shadow. But I only want to do certain records to sync. basically all the user info. Is there away that this can be done? or do I have to do the whole file? Rsync handles data per file, it has no concept of records or such. I suggest you make some other way of generating the passwd and shadow files from different input files (system entries and user entries), so that you can transfer the files with just the user records. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question about 2.6.3pre2's --link-by-hash behaviour
One thing that the link-by-hash patch needs is an additional close();
without that, I quickly ran into too many open files.
--- hashlink.c.old 2004-09-24 10:59:12.0 +0200
+++ hashlink.c 2004-09-24 10:59:20.0 +0200
@@ -280,6 +280,7 @@
}
hashfile = compare_hashfiles(fd, hashfiles);
hashfiles = NULL;
+ close(fd);
if (hashfile) {
first = 0;
This is a patch to the patch (I hand-edited the patch)
--- link-by-hash.diff.orig 2004-09-24 10:58:38.0 +0200
+++ link-by-hash.diff 2004-09-24 11:03:41.0 +0200
@@ -24,7 +24,7 @@
popt/popthelp.o popt/poptparse.o
--- orig/hashlink.c2004-08-13 18:04:59
+++ hashlink.c 2004-08-13 18:04:59
-@@ -0,0 +1,342 @@
+@@ -0,0 +1,343 @@
+/*
+ Copyright (C) Cronosys, LLC 2004
+
@@ -307,6 +307,7 @@
+ }
+ hashfile = compare_hashfiles(fd, hashfiles);
+ hashfiles = NULL;
++ close(fd);
+
+ if (hashfile) {
+ first = 0;
Paul Slootman
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question about 2.6.3pre2's --link-by-hash behaviour
The --link-by-hash patch is a bit defective, I think. If I run the following command: rsync --link-by-hash=/tmp/hash 192.168.1.1::mirrors/ps1 /tmp I get the following output: (1) linkname = /tmp/hash/0fb9ca1a/3cc6ec7f5a2de3a0235b585f/0 link-by-hash (new): /tmp/ps1 - /tmp/hash/0fb9ca1a/3cc6ec7f5a2de3a0235b585f/0 If I then run it again, I get the following: (1) linkname = /tmp/hash/8e923961/e8d9e056eee8b5d083a579e4/0 link-by-hash (new): /tmp/ps1 - /tmp/hash/8e923961/e8d9e056eee8b5d083a579e4/0 Upon investigation, the patch calculates the checksum separately from the rsync file checksum, and I wonder why. The patch's calculated checksum is also always different to the rsync checksum. It forces the checksum-seed to 12345, although I can't see why. I get the feeling the patch may need to be rewritten from scratch... Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question about 2.6.3pre2's --link-by-hash behaviour
On Wed 22 Sep 2004, Erik Jan Tromp wrote: On Wed, 22 Sep 2004 13:21:31 +0200 Paul Slootman [EMAIL PROTECTED] wrote: I had hoped to use it both for my rotating backups for my (unofficial) slackware mirror. Hmmm... For a slackware mirror I expect that it would be fine. To my eyes, a mirror implies a duplicate fileset indistinguishable from the original. Having files show up with different modes dates - due, in this case, to the sort order of the file list the mode/date of the first file used for linking by hash - doesn't fit that scenario. Backups are even more sensitive as user.group information _must_ be retained. If a mirror of a distribution has different attributes for files that are the same in contents, I'd say that mirror is buggy That's why I qualified my response with For a slackware mirror. I agree that for general use, this is not useful. I tried downloading the patch from the web cvs, but I got this: Error: Unexpected output from cvs co: cvs [checkout aborted]: Absolute module reference invalid: `/rsync/patches/link-by-hash.diff' Check whether the directory /cvsroot/CVSROOT exists and the script has write-access to the CVSROOT/history file if it exists. The script needs to place lock files in the directory the file is in as well. An alternative would be to use: rsync://rsync.samba.org/ftp/unpacked/rsync/patches/link-by-hash.diff Unfortunately that seems to have tabs expanded, and at one point a line was wrapped. However I managed to apply the patch by hand now. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
