Re: meaning of IO Error: skipping the delete....
Dear Mr. Martin, Thanks for the replies. Our problems are sorted out upto an extend. We are facing one more problem i.e. while transferring the files, sometimes the transfer breaks in between and gives us the error message: readerror: connection reset by peer We are having a large file system (approx. 100 GB). Kindly let us know the reason behind this. Regards Nitin Agarwal Martin Pool wrote: On 25 Jan 2002, Nitin Agarwal [EMAIL PROTECTED] wrote: Dear Mr. Martin, Thanks for the reply. We tried as per your advice. Now, its not showing us any errors. But, we observed that while copying the files from the main server to the backup machine, rsync is changing the file names of those files having last extension as any single digit integer (e.g. .3, .2, .1). While copying it shows us copying the file with name 12345.1234554321.3 to the destination as 12345.1234554321.0003. i.e. it is adding 000 to the extension. This is very strange behaviour. I cannot at the moment think of any aspect of rsync that would cause it to manipulate the filenames in this way. What operating system and what version of rsync are you using? Can you please find a minimal set of instructions that recreate the problem: in other words, the commands to run to create files with the names that are causing problems, and then the rsync command that copies them. -- Martin
Re: Limit consumption of space on destination?
quotas arent that bad really - check this doc out and you'll have it going in no time.. http://linuxnewbie.org/nhf/intel/tools/edquota.html jamie *** REPLY SEPARATOR *** On 2/2/2002 at 12:33 AM Anonymous wrote: I've just started using rsync, and it's way cool. I checked the FAQ, the broken mailing list archive search function, and scanned the archives, but didn't see this mentioned. I presume it's a fairly common need: I'm mirroring files automatically and daily from one site to another, and anticipate the day when the size of the files may exceed the size of the destination disk. Rather than take up the whole disk (which is a root partition) and bring my destination system down, I'd prefer that the rsync process fail gracefully and not copy files when there's only a small amount of space left on the destination disk. I realize I could accomplish this by creating a partition on the destination disk, but I don't like partitions because changing them means taking down a production server and reformatting the drive. I took a look at quotas, which seem to fit the bill more closely, but either the documentation out there on them is weak or I'm just not smart enough to figure out whether they'll give me the assurance I'm looking for. How have other rsync users solved this problem?
Re: using rsync with Mac OS X
OK, I'm brand new to this group, brand new to rsync, brand new to unix in general. I'm trying to play catch up with this discussion so there are likely many misconceptions that I have about these issues. My goal is to create a tool that does backup and restore only transferring changes. It will connect to a server running Linux from Mac OS X and preserve all metadata without the user ever knowing there is an issue. I've found the rsync algorithm is a good start and it sounds like you all have the same idea. I don't think I like the idea of the MacBinary solution, in that I can see some configuration of the tool that the user will have to worry about. We obviously don't want the overhead of flattening files without forks or files that have FileInfo that can be determined from other metadata strategies. The user might have to maintain a list of files they use... How do I handle this file or that (á la mac cvs tools). I see another user experience issue with the MacBinary solution and the protocol change. What do the files look like when they get backed up? If I connect to the server via the finder am I going to see a bunch of files that are 'archived' or do I get the real deal. As a user I wouldn't use rsync if I couldn't just go and grab the files that got backed up. Not that running the file through stuffit is a big deal but it going to seems a bit clunky even if the solution is in fact much more extensible. What format is this new protocol going to produce? Will the only way to get to the files be to use the rsync client? Sorry, that's just not acceptable. The only solution left is to pre-process the file by splitting it before before creating the change lists so that comparisons can be made if the file is split on the server. There will have to be some intelligence about what method of splitting is used on the server but I'm positive that couldn't be too hard to determine. Directory metadata just has to be handled in another file as well, isn't that what .DSInfo files are? I'm starting to think that what I'm proposing is more of a combination of 2) and 3). Wouldn't it be great if we could support ACL's as well. Please tell me if I'm way off base here. One other question that I'm sure will show my ignorance of Darwin development. What is the issue with using the high level API's if the output is compatible with the other platforms running rsync. What is the advantage of trying for posix purity or code at the "Darwin level" if the code is only going to be used on Macs running the higher level stuff anyway? If you don't have a forked file system why would you care if you don't know how to handle forks? I'm planning on taking this project on full time and we would all benefit if we can all agree on a direction. Lets get this thing going, Terrence Geernaert Mark Valence wrote: 1) convert (on the fly) all files to MacBinary before comparing/sending them to the destination. MacBinary is a well documented way to package an HFS file into a single data file. The benefits with this method are compatibility with existing rsync versions that are not MacBinary aware, while the drawbacks are speed, maintainability, and that directory metadata is not addressed at all. 2) Treat the two forks and metadata as three separate files for the purposes of comparison/sending, and then reassemble them on the destination. Same drawbacks and benefits of the MacBinary route. This would also take more memory (potentially three times the number of files in the flist). 3) Change the protocol and implementation to handle arbitrary metadata and multiple forks. This could be made sort-of compatible with existing rsync's by using various tricks, but the most efficient way would be to alter the protocol. Benefits are that this would make the protocol extensible. Metadata can be "tagged" so that you could add any values needed, and ignore those tags that are not understood or supported. Any number of forks could be supported, which gives a step up in supporting NTFS where a file can have any number of "data streams". In fact, forks and metadata could all be done in the same way in the protocol.
Re: rsync dir in _both_ directions?
There's no such option. Rember that Rsync was devised as an efficient way to mirror ftp sites. What you're looking for is unison. Some of the guys on here use it. It's at http://www.cis.upenn.edu/~bcpierce/unison;, and looks promising for your application. Tim Conway [EMAIL PROTECTED] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), .\n ' There are some who call me Tim? Jack McKinney [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 02/04/2002 10:58 AM To: [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject:rsync dir in _both_ directions? Classification: I felt certain there was an option to do this, but I cannot find it. I want to rsync a directory on machine A over to machine B, and then rsync the directory on machine B back to machine A. The idea is this: I read my email using mutt, which is set to save my email in ~/Mail in maildir format. This means that each message is in its own file, instead of being appended to an existing file. Thus, if I backup my email by rsync'ing it to another machine, only the new messages are copied over. Mailboxes need not be. Before you say that rsync only sends changed blocks, and the changed blocks are at the end if the message is appended, remember that mail readers write status into the headers of the messages. If I change the status on the first message in the box (reply to it, delete it), then the ENTIRE mailbox is re-copied. Not so in maildir format. Well, I read my email on my laptop, and it is sometimes necessary to login to a central server to read my email (firewall issues). What I do is to peridocally sync with the central server (which we'll call server) by: rsync -e ssh -rptlv Mail server:. rsync -e ssh -rptlv server:Mail . I'd like to do this in one command. Is there a switch to rsync that says to make the local and remote directories identical, even if that means transfering data in both directions? -- There is no parameter that makes it impossibleJack McKinney for you to perform still more excellently. [EMAIL PROTECTED] -Mario Cuomo, on the lack of a clock in baseball http://www.lorentz.com 1024D/FBED2DAA 4096g/3F93879F 2002 Chicago Cubs magic number: 163 attj7zff.dat Description: Binary data
Re: SIGUSR1 or SIGINT error
I suspected that might be the case...now...how to determine the real problem? Does rsync log it somewhere? lsof shows that STDERR/STDOUT are going to /dev/null, so I hope it's not writing it there. Nothing informative in syslog, just the message about the SIG: Feb 5 09:49:41 hite rsyncd[9279]: [ID 702911 daemon.warning] rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(229) Any clues? Thanks, David. - On Tue, 5 Feb 2002, Dave Dykstra wrote: On Mon, Feb 04, 2002 at 12:24:02PM -0500, David Birnbaum wrote: Howdy, We occassionally get the following error when running our nightly backups: rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(229) This happens more on one or two machines than on any of the others. We've looked high and low to see if we're mistakenly sending these signals, but nothing is that we can find. Does anyone know what this might be from? Is it the server or the client that's complaining about the signal? The side of rsync that receives files forks into two processes, and one sends the other a SIGUSR1 when it experiences a problem. So this is probably a symptom of your real problem, not the real problem. - Dave Dykstra
Error from rsync-2.5.2
Hi, I have compiled the new version 2.5.2 rsync in our servers, then ran rsync last night, there were some files not copied to destination server, both source and destination servers are running 2.5.2, I got the following error messages, can you please let me know what would be caused the errors? I copied back 2.3.2 version on both servers, and rsync went well. Thanks for the help, here is the errors: ch_what Warning: unexpected read size of 0 in map_ptr Warning: unexpected read size of 0 in map_ptr collect_tests Warning: unexpected read size of 0 in map_ptr Warning: unexpected read size of 0 in map_ptr Thanks, Jennifer
2.5.2 will not compile
Trying to compile rsync 2.5.2 after ./configure --prefix=/usr I get the following make errors: gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c rsync.c -o rsync.o In file included from rsync.c:23: rsync.h:339: warning: no semicolon at end of struct or union rsync.h:339: parse error before `inode' rsync.h:341: parse error before `dev' rsync.h:341: warning: type defaults to `int' in declaration of `dev' rsync.h:341: warning: data definition has no type or storage class rsync.h:344: parse error before `rdev' rsync.h:344: warning: type defaults to `int' in declaration of `rdev' rsync.h:344: warning: data definition has no type or storage class rsync.h:347: `basename' redeclared as different kind of symbol /usr/include/string.h:317: previous declaration of `basename' rsync.h:350: `link' redeclared as different kind of symbol /usr/include/unistd.h:678: previous declaration of `link' rsync.h:352: parse error before `}' rsync.h: In function `flist_up': rsync.h:420: dereferencing pointer to incomplete type rsync.c: In function `set_perms': rsync.c:165: dereferencing pointer to incomplete type rsync.c:168: dereferencing pointer to incomplete type rsync.c:178: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:184: dereferencing pointer to incomplete type rsync.c:188: dereferencing pointer to incomplete type rsync.c:189: dereferencing pointer to incomplete type rsync.c:206: dereferencing pointer to incomplete type rsync.c:208: dereferencing pointer to incomplete type rsync.c:152: warning: `change_uid' might be used uninitialized in this function rsync.c:152: warning: `change_gid' might be used uninitialized in this function rsync.c: In function `finish_transfer': rsync.c:245: dereferencing pointer to incomplete type make: *** [rsync.o] Error 1 Some info about the build environment: uname -sr Linux 2.2.20 gcc --version egcs-2.91.66 ls -d /lib/libc* /lib/libc-2.1.2.so /lib/libc.so.6 /lib/libcrypt.so.1 make --version GNU Make version 3.77, by Richard Stallman and Roland McGrath. Output of configure appended below. I'd like to be able to upgrade to the new version due to the security concerns.Any help will be greatly appreciated. Regards, Richard (output of ./configure --prefix=/usr 21) configure: Configuring rsync 2.5.2 checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking target system type... i686-pc-linux-gnu checking for gcc... gcc checking for C compiler default output... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for executable suffix... checking for object suffix... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking how to run the C preprocessor... gcc -E checking for a BSD compatible install... /usr/bin/ginstall -c checking for gcc option to accept ANSI C... none needed checking whether to include debugging symbols... yes checking for remsh... 0 checking for broken largefile support... yes checking ipv6 stack type... linux-glibc checking for library containing getaddrinfo... none required checking whether byte ordering is bigendian... no checking for dirent.h that defines DIR... yes checking for opendir in -ldir... no checking whether time.h and sys/time.h may both be included... yes checking for sys/wait.h that is POSIX.1 compatible... yes checking for sys/fcntl.h... yes checking for sys/select.h... yes checking for fcntl.h... yes checking for sys/time.h... yes checking for sys/unistd.h... yes checking for unistd.h... yes checking for utime.h... yes checking for grp.h... yes checking for compat.h... no checking for sys/param.h... yes checking for ctype.h... yes checking for sys/wait.h... (cached) yes checking for sys/ioctl.h... yes checking for sys/filio.h... no checking for string.h... yes checking for stdlib.h... yes checking for sys/socket.h... yes checking for sys/mode.h... no checking for glob.h... yes checking for alloca.h... yes checking for mcheck.h... yes checking for sys/sysctl.h... yes checking for arpa/inet.h... yes checking for arpa/nameser.h... yes checking for netdb.h... yes checking for malloc.h... yes checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... (cached) yes checking for string.h... (cached) yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... (cached) yes checking for int... yes checking size of
rsync fails all of a sudden
I've been rsync'ing the linux tree from mvista.com for some time. Recently, when doing an update I began to receive read_timeout errors. Everything works fine when I do a download into an empty directory but an update fails. The error is the following: rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(152) rsync: connection unexpectedly closed (213927 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(140) This is the error I get from 2.5.2. I was using 2.4.6, under which I was getting the unexpected eof read_timeout error. Any ideas? Thanks. -- John Tyner [EMAIL PROTECTED]
rsync with openssh
We've used rsync with an old ssh for a couple of years with no problems. Now trying to switch to openssh 3.0.2p1 so we can use protocol 2. Rsync makes the connection via ssh and does it's work. The problem is after rsync runs and exits on the server side the sshd session keeps running and isn't closing the connection. So the rsync client hangs waiting for the connection to end. Is anyone else having this problem? Know of a solution? Jim Ogilvie
Re: rsync dir in _both_ directions?
On Mon, Feb 04, 2002 at 11:58:04AM -0600, Jack McKinney wrote: I felt certain there was an option to do this, but I cannot find it. I want to rsync a directory on machine A over to machine B, and then rsync the directory on machine B back to machine A. The idea is this: I read my email using mutt, which is set to save my email in ~/Mail in maildir format. This means that each message is in its own file, instead of being appended to an existing file. Thus, if I backup my email by rsync'ing it to another machine, only the new messages are copied over. Mailboxes need not be. Before you say that rsync only sends changed blocks, and the changed blocks are at the end if the message is appended, remember that mail readers write status into the headers of the messages. If I change the status on the first message in the box (reply to it, delete it), then the ENTIRE mailbox is re-copied. Not so in maildir format. Are you sure about that? Rsync is able to locate identical blocks that are in a different position. Well, I read my email on my laptop, and it is sometimes necessary to login to a central server to read my email (firewall issues). What I do is to peridocally sync with the central server (which we'll call server) by: rsync -e ssh -rptlv Mail server:. rsync -e ssh -rptlv server:Mail . I'd like to do this in one command. Is there a switch to rsync that says to make the local and remote directories identical, even if that means transfering data in both directions? No there isn't. What's wrong with doing it in two commands? You may be interested in http://mailsync.sourceforge.net which is explicitly designed to sync mailboxes. - Dave Dykstra
rsync usage... and I did RTFM
Hey folks, I've been through the rsync documentation, and to the best of my ability, I dont see how to do exactly what I wish to do. I'd go through the mailing list archives, but there doesnt seem to be a search function. Essentially, I need to rsync securely between some sites, but I want logs written. The problem is, it appears that using rsyncd is the only way to get logs, but if you use rsyncd, there's no way to use ssh for secure transfers. Is there any way to both get logs written and use ssh at the same time? Thanks --Ross Becker
Re: Patch: update zlib/* to 1.1.3
On 29 Jan 2002, Jos Backus [EMAIL PROTECTED] wrote: This patch (apologies for the size) updates zlib/* to the files that ship with zlib 1.1.3. rsync includes a slightly modified and incompatible version of zlib. (PhD-project over-optimization...) To update to a newer zlib would require merging those changes across. It might be nice to go back to using standard zlib sometime: - can upgrade to newer versions - can use shlibs on systems that ship zlib - avoids people breaking -z by forcing a link to another zlib :-) but this probably requires a protocol revision. -- Martin
Re: configure --with-rsh=CMD and default blocking-IO support
On 30 Jan 2002, Wayne Davison [EMAIL PROTECTED] wrote: A while back I argued for adding a --with-rsh=CMD option to configure and got some general agreement that it would be a good thing (especially for systems that don't have rsh at all). However, the changes were never integrated into rsync. OK, I agree --with-rsh should go in, but I think putting magic characters into it is needlessly confusing. I would feel much better about a separate configure option to set the default O_NONBLOCK mode. -- Martin
RE: Error from rsync-2.5.2
Not sure if that would be the same for you, but the version 2.5.2 use the obsolete malloc.h instead of the stdlib.h and I got plenty of warning. This is only new in the 2.5.2 as I try the 2.5.1 and it doesn't do this. I did a diff -r between the two version and I saw that malloc was introduce in here. Why, that I do not know. So, I would suggest to you to try the 2.5.1 instead and see if that work for you. Just a thought. Daniel -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Jennifer Lu Sent: Tuesday, February 05, 2002 3:31 PM To: [EMAIL PROTECTED] Subject: Error from rsync-2.5.2 Hi, I have compiled the new version 2.5.2 rsync in our servers, then ran rsync last night, there were some files not copied to destination server, both source and destination servers are running 2.5.2, I got the following error messages, can you please let me know what would be caused the errors? I copied back 2.3.2 version on both servers, and rsync went well. Thanks for the help, here is the errors: ch_what Warning: unexpected read size of 0 in map_ptr Warning: unexpected read size of 0 in map_ptr collect_tests Warning: unexpected read size of 0 in map_ptr Warning: unexpected read size of 0 in map_ptr Thanks, Jennifer
Re: rsync dir in _both_ directions?
Big Brother tells me that Dave Dykstra wrote: On Mon, Feb 04, 2002 at 11:58:04AM -0600, Jack McKinney wrote: If I change the status on the first message in the box (reply to it, delete it), then the ENTIRE mailbox is re-copied. Not so in maildir format. Are you sure about that? Rsync is able to locate identical blocks that are in a different position. Yes. blocks are 512 bytes long. If I add a 14 byte header on a message near the beginning, then there are no identical blocks. No there isn't. What's wrong with doing it in two commands? It can take 5-30 minutes to finish the first, and I want it to then start the second one without having to watch for the first one. I prefer not to use ssh's feature to remember keys or to use .rhosts/.shosts. You may be interested in http://mailsync.sourceforge.net which is explicitly designed to sync mailboxes. This is designed for long mbox-format boxes. Maildirs obviate the need for it, except when you modify a rather long single message, in which case mailsync is useful. OTOH, though it is designed to solve a different problem, it may indirectly solve the both directions problem, for this specific case. I have other data (some binary) that I want to sync in the same way, though. -- There are two kinds of people inJack McKinney this world: Those with loaded guns, [EMAIL PROTECTED] and those who dig. http://www.lorentz.com -The Good, The Bad, and The Ugly 1024D/FBED2DAA 4096g/3F93879F 2002 Chicago Cubs magic number: 163 msg02968/pgp0.pgp Description: PGP signature
Re: proper syntax?
On 23 Jan 2002, Jamie Pratt [EMAIL PROTECTED] wrote: ...It all appears to work, but I'm not satisfied - dates dont seem to change on files? What do you mean? Also, what version and platform is this? -- Martin
