Re: Instructions on how rsync uses ssh to run remotely?
On Tue, Jul 10, 2001 at 02:24:57AM -0700, Ben Escoto wrote: About two weeks ago I asked about a way to use rsync to backup (I wanted to make something in-between a mirror and an incremental backup). I wrote a preliminary version of a program that does this (see http://www.stanford.edu/~bescoto/rdiff-backup), and am now considering adding remote filesystem support through ssh the way rsync does. Conceptually this may not be too hard, but there are probably issues that will arise that I don't foresee now. So if anyone knows of a description of how rsync does the ssh/rsh client/server bit, or maybe something more generic on how to use ssh in the way rsync does, I would appreciate a reference. I could look at the rsync source, but it doesn't seem to be overly commented, and I'm not exactly fluent in C and thought that this post might save me time in any event. Thanks. All that rsync uses ssh for is to open a bi-directional data pipe and run a command (another rsync program) on the remote side. It then ignores ssh (or rsh, the default) and does its own protocol over the pipe between the two sides. - Dave Dykstra
Re: Instructions on how rsync uses ssh to run remotely?
DD == Dave Dykstra [EMAIL PROTECTED] wrote the following on Tue, 10 Jul 2001 10:34:53 -0500 DD All that rsync uses ssh for is to open a bi-directional data DD pipe and run a command (another rsync program) on the remote DD side. It then ignores ssh (or rsh, the default) and does its DD own protocol over the pipe between the two sides. Hmm, doesn't rsync run two copies of itself on the remote side? Maybe it uses three (?) unidirectional pipes instead of one bi-directional one? And either way, it would be nice to have a primer on designing a protocol like this, since I'm sure many people have already figured out the best way to handle latency, errors, etc. -- Ben Escoto
rsync 2.4.6 hangs.
I see there is quite a bit discussion on the list about rsync hanging on a variety of conditions, one of which I seem to be having. We use rsync to update remote servers over night. Each system syncs with a master server holding around 4.0 gig of data. Normally there is very little change on the master so rsync only updates a few files each night. My problem is that I am now upgrading the servers and need to sync all the data to the new systems and rsync is hanging like crazy. Sometimes it moves a few hundred meg and sometimes it only moves one or two files and then hangs. My master server is RedHat 6.2 running rsync 2.4.6 (from source - no RPM). The remote server is RedHat 7.0 with rsync 2.4.6 (from source - no RPM). Things were actually better when the master was running rsync 2.4.3 but it still hung. The remote server is a PIII 850 with 512 megs of RAM so I doubt this is a memory problem. Even at 100 bytes per file would not consume all the memory on the remote system. The master only has 112 Meg but I assume the file list is kept on the remote and not the master.. My command is rsync 192.168.0.12:/home/image /home/image -e ssh2 -avz --delete Is there something I'm doing wrong? What can I do to prevent rsync from hanging all the time?? I've tried removing the --delete and -z options but it still hangs.. Any help would be greatly appreciated. --- Steve Ladendorf Network Manager sladendorf at blakeschool.orgThe Blake School As far as the laws of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality. --Albert Einstein
Re: Instructions on how rsync uses ssh to run remotely?
On Tue, Jul 10, 2001 at 10:58:21AM -0700, Ben Escoto wrote: DD == Dave Dykstra [EMAIL PROTECTED] wrote the following on Tue, 10 Jul 2001 10:34:53 -0500 DD All that rsync uses ssh for is to open a bi-directional data DD pipe and run a command (another rsync program) on the remote DD side. It then ignores ssh (or rsh, the default) and does its DD own protocol over the pipe between the two sides. Hmm, doesn't rsync run two copies of itself on the remote side? Maybe it uses three (?) unidirectional pipes instead of one bi-directional one? And either way, it would be nice to have a primer on designing a protocol like this, since I'm sure many people have already figured out the best way to handle latency, errors, etc. rsync forks itself into two copies on the receiver side, one process for generating checksums and one for receiving updates. It's still a single bi-directional pipe, it's just that on the receiver side one process is writing and one is mainly reading. I've never read it myself, but I imagine that Andrew Tridgell's PhD thesis covers the protocol design. I don't think it's written down anywhere else. You can find it on his home page, http://samba.org/~tridge. - Dave Dykstra
Re: rsync 2.4.6 hangs.
There are 2 patches on this mailing list that fix the 2.4.6 hangs for me. Check the archives and you should be able to find them. I hope to see a 2.4.7 out soon that includes some or all of those hang fixes. eric Steve Ladendorf wrote: I see there is quite a bit discussion on the list about rsync hanging on a variety of conditions, one of which I seem to be having. We use rsync to update remote servers over night. Each system syncs with a master server holding around 4.0 gig of data. Normally there is very little change on the master so rsync only updates a few files each night. My problem is that I am now upgrading the servers and need to sync all the data to the new systems and rsync is hanging like crazy. Sometimes it moves a few hundred meg and sometimes it only moves one or two files and then hangs. My master server is RedHat 6.2 running rsync 2.4.6 (from source - no RPM). The remote server is RedHat 7.0 with rsync 2.4.6 (from source - no RPM). Things were actually better when the master was running rsync 2.4.3 but it still hung. The remote server is a PIII 850 with 512 megs of RAM so I doubt this is a memory problem. Even at 100 bytes per file would not consume all the memory on the remote system. The master only has 112 Meg but I assume the file list is kept on the remote and not the master.. My command is rsync 192.168.0.12:/home/image /home/image -e ssh2 -avz --delete Is there something I'm doing wrong? What can I do to prevent rsync from hanging all the time?? I've tried removing the --delete and -z options but it still hangs.. Any help would be greatly appreciated. --- Steve Ladendorf Network Manager sladendorf at blakeschool.orgThe Blake School As far as the laws of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality. --Albert Einstein -- __ Eric T. Whiting AMI Semiconductors
Re: rsync 2.4.6 hangs.
On Tue, 10 Jul 2001, Steve Ladendorf wrote: Is there something I'm doing wrong? What can I do to prevent rsync from hanging all the time?? If you avoid using -v, that can help rsync not to hang. However, a better fix is to apply this patch: http://www.clari.net/~wayne/rsync-nohang1.patch This fixes a very common hang problem, and makes -v safe to use again. Since you are also transferring large quantities of files, I would be very interested in knowing if rsync still hangs after applying that patch. If it does, try applying this patch: http://www.clari.net/~wayne/rsync-nohang2.patch The second patch fixes a much rarer bug, and it would be great to know if someone had run into it in simple, real-world conditions. ..wayne..
Re: rsync 2.4.6 hangs.
Hi Wayne, On Tue, Jul 10, 2001 at 01:59:22PM -0700, Wayne Davison wrote: http://www.clari.net/~wayne/rsync-nohang1.patch http://www.clari.net/~wayne/rsync-nohang2.patch Are these to be applied against the stock 2.4.6? In particular, the second patch does not apply cleanly against yesterday's CVS. I'd like to build an up-to-date version of rsync with these patches. Thanks, -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer;
Re: rsync 2.4.6 hangs.
A month of sundays ago Steve Ladendorf wrote: variety of conditions, one of which I seem to be having. We use rsync to update remote servers over night. Each system syncs with a master server holding around 4.0 gig of data. Normally there is very little change on the master so rsync only updates a few files each night. My problem is that I am now upgrading the servers and need to sync all the data to the new systems and rsync is hanging like crazy. Sometimes it moves a few I always run one rsync per subdirectory, and impose a timeout. If you do that in a loop it should be fine. Or you could always drop back to the older 2.3 series, which I think didn't have the problem. Or apply the nohang patches. What's holding up a release with these patches in? Peter
Re: New protocol to avoid EOF?
On Tue, 26 Jun 2001, Martin Pool wrote:
On 25 Jun 2001, Wayne Davison [EMAIL PROTECTED] wrote:
I was wondering if the protocol should be updated to avoid ever
assuming that an EOF on the socket was OK. The only case I know of
where this allowed is when we're listing modules from an rsync server.
If we modified the protocol to have the daemon rsync send an EOF token
(such as @RSYNCD: EOF) at the end of the list, this would allow the
rsync client to always report an unexpected EOF as an error.
Yes, that sounds good. I think I applied the patch that clears up the
is eof OK? flag, so please go ahead and send the new one.
I haven't seen this show up in the CVS version at pserver.samba.org.
Here's (finally) a combo patch that incorporates my previous two patches
(that removed the bogus EOF error) while also adding an EOF-marker
message if the other rsync is using the newest protocol (eliminating the
need for us to ever allow an actual EOF to be silently ignored).
..wayne..
---8--8--8--8---cut here---8--8--8--8---
Index: rsync/authenticate.c
--- rsync/authenticate.c22 Jun 2001 10:16:04 - 1.17
+++ rsync/authenticate.c10 Jul 2001 22:41:28 -
@@ -224,7 +224,7 @@
io_printf(fd,%s%s\n, leader, b64_challenge);
- if (!read_line(fd, line, sizeof(line)-1)) {
+ if (!read_line(fd, line, sizeof(line)-1, 0)) {
return NULL;
}
Index: rsync/clientserver.c
--- rsync/clientserver.c7 May 2001 06:59:37 - 1.67
+++ rsync/clientserver.c10 Jul 2001 22:41:29 -
@@ -44,9 +44,9 @@
extern int am_sender;
extern struct in_addr socket_address;
extern char *shell_cmd;
+ extern int list_only;
if (argc == 0 !am_sender) {
- extern int list_only;
list_only = 1;
}
@@ -93,7 +93,7 @@
io_printf(fd,@RSYNCD: %d\n, PROTOCOL_VERSION);
- if (!read_line(fd, line, sizeof(line)-1)) {
+ if (!read_line(fd, line, sizeof(line)-1, 0)) {
return -1;
}
@@ -107,9 +107,11 @@
if (p) *p = '/';
while (1) {
- if (!read_line(fd, line, sizeof(line)-1)) {
+ if (!read_line(fd, line, sizeof(line)-1, list_only)) {
return -1;
}
+ if (strcmp(line,@RSYNCD: EOF) == 0)
+ return 0;
if (strncmp(line,@RSYNCD: AUTHREQD ,18) == 0) {
auth_client(fd, user, line+18);
@@ -277,7 +279,7 @@
argv[argc++] = rsyncd;
while (1) {
- if (!read_line(fd, line, sizeof(line)-1)) {
+ if (!read_line(fd, line, sizeof(line)-1, 0)) {
return -1;
}
@@ -377,10 +379,13 @@
{
int n = lp_numservices();
int i;
-
+ extern int remote_version;
+
for (i=0;in;i++)
if (lp_list(i))
io_printf(fd, %-15s\t%s\n, lp_name(i), lp_comment(i));
+ if (remote_version = 25)
+ io_printf(fd,@RSYNCD: EOF\n);
}
/* this is called when a socket connection is established to a client
@@ -418,7 +423,7 @@
io_printf(fd,\n);
}
- if (!read_line(fd, line, sizeof(line)-1)) {
+ if (!read_line(fd, line, sizeof(line)-1, 0)) {
return -1;
}
@@ -429,7 +434,7 @@
while (i == -1) {
line[0] = 0;
- if (!read_line(fd, line, sizeof(line)-1)) {
+ if (!read_line(fd, line, sizeof(line)-1, 0)) {
return -1;
}
Index: rsync/io.c
--- rsync/io.c 7 May 2001 06:59:37 - 1.85
+++ rsync/io.c 10 Jul 2001 22:41:41 -
@@ -178,12 +178,13 @@
if (n == 0) {
- if (eof_error) {
- rprintf(FERROR,
-%s: connection to server unexpectedly closed
- (%.0f bytes read so far)\n,
-RSYNC_NAME, (double)stats.total_read);
- }
+ extern int remote_version;
+ if (!eof_error remote_version 25)
+ exit_cleanup(0);
+ rprintf(FERROR,
+ %s: connection to server unexpectedly closed
+(%.0f bytes read so far)\n,
+ RSYNC_NAME, (double)stats.total_read);
exit_cleanup(RERR_STREAMIO);
}
@@ -560,9 +561,9 @@
write_buf(f,(char *)c,1);
}
-int read_line(int f, char *buf, int maxlen)
+int read_line(int f, char *buf, int maxlen, int eof_ok)
{
- eof_error = 0;
+ eof_error = !eof_ok;
while (maxlen) {
buf[0] = 0;
Index: rsync/proto.h
--- rsync/proto.h 7 May 2001
Re: rsync 2.4.6 hangs.
On Tue, Jul 10, 2001 at 03:19:51PM -0700, Wayne Davison wrote: They will apply to stock 2.4.6 with offsets (but no fuzz), if you so desire. They were created from the CVS version, though. OK. The nohang1 patch is already incorporated into the CVS version, so it is the only one that should have given you trouble. I was able to apply nohang2 to an fresh CVS version without a whimper. What CVS server are you using? I'm using :pserver:[EMAIL PROTECTED]:/cvsroot. I'm using the same CVS server. I think my problem is that I applied your rsync-nohang.patch first, whereas I should only be applying rsync-nohang2.patch against the current CVS sources. Correct? Thanks, -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer;
Re: rsync 2.4.6 hangs.
On Tue, Jul 10, 2001 at 04:13:11PM -0700, Jos Backus wrote: I'm using the same CVS server. I think my problem is that I applied your rsync-nohang.patch first, whereas I should only be applying rsync-nohang2.patch against the current CVS sources. Correct? To answer my own question: yes. rsync-nohang2.patch applies against CVS without problems. Thanks Wayne. -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer;
