Re: question about --bwlimit=
On Fri 21 May 2004, Wallace Matthews wrote:
Since --bwlimit depends upon sleep(1 second), I repeated the experiment with a file
that was 383 Megabyte so that when I am running unthrottled it takes significantly
longer than a second (ie. ~50 seconds) to complete. I get the same bi-modal behavior
but with different values for 4000 and 4001 respectively. The fact that the break
point stays fixed isnt intuitive (to me at least).
There have been earlier discussions about the --bwlimit behaviour, and
that it's not that well suited for e.g. slower ADSL lines because it's
rather bursty, a limit of 20 means it'll write out at full throttle
until it reaches the 20k, then it sleeps.
There's an alternative --bwlimit patch that was posted back then, that
takes a subtlely different approach (by also limiting the size of the
writes). This prompted a discussion of whether this may have impact on
the tcp packets going out on the wire, perhaps leading to extra tcp
overhead which is contrary to rsync's goal of reducing network traffic
at all costs...
In the most recent Debian versions I've made the other bwlimit
implementation available (via --bwlimit-mod, for modified). Of course,
both ends need the Debian hacked version. My private tests have shown it
to work pretty well, other people have also been happy.
I haven't tried it with larger limits than about 100, though...
I doubt whether it will have any effect on your test case.
The patch was basically this:
--- rsync-2.6.2.orig/io.c
+++ rsync-2.6.2/io.c
@@ -814,6 +814,8 @@
if (FD_ISSET(fd, w_fds)) {
int ret;
size_t n = len-total;
+ if (bwlimit n (unsigned)(bwlimit*100))
+ n = bwlimit*100;
ret = write(fd,buf+total,n);
if (ret 0) {
Paul Slootman
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: File sync/delete question
On Fri 21 May 2004, Markus Gaugusch wrote: Imagine this happenings: 10:00: file a.txt gets uploaded to ftp server 10:01: file a.txt is rsynced to samba server 10:02: file a.txt gets deleted from samba server (after rsync run) 10:03: file would again be synced :( Now I think that I need a new option for rsync: Transfer all files, and remotely delete those that must have been transferred before and are not here anymore. It may be that unison will do what you want. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: two-way synchronization accross a firewall fails
On Thu 20 May 2004, Tim Harsch wrote: firewall allows ssh connections if inititiated from I to O, but not if the other way. both machines have an /etc/rsyncd.conf of: [rt] path = /tmp/rsync_test comment = Test area O runs rsync daemon, I initiates a rsync cammnad like rsync -rvvv --delete --rsh=ssh O::rt /tmp/rsync_test which works great, but when flipped and run on machine I, like so: rsync -rvvv --delete --rsh=ssh /tmp/rsync_test O::rt One thing that strikes me first: If you're going to use ssh, don't use an rsync daemon (and hence no :: paths, only : paths). Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs in cron (not SSH-problem)
On Fri 21 May 2004, Pascal Nobus wrote: When used this command in cron 00 01 * * * rsync -av --delete /mnt/web1 /mass/kuurne/day 00 02 * * * rsync -av --delete /mnt/web2 /mass/kuurne/day etc.. Rsync hangs, it doesn't finish! Does it eat cpu time, of does it just sit there? Use commands like lsof, strace / truss to see what it's doing. The stdout/stderr of the commands should also be redirected to a tmp file somewhere, have a look if anything useful is written there. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bidirectional speed question
On Mon 17 May 2004, Greger Cronquist wrote: I'm wondering if the following is rsync-related or an issue with my supposedly synchronous internet connection: I have a server running an rsync daemon. When I simultaneously pull and push files to this server using two separate processes on the client (different directories), I get about half the transfer speed compared to just running one process. This also occurs when the files are new (so the rsync algorithm should not be active which otherwise causes data to be sent in both directions). Is this to be expected from rsync? Or is it (a) a bug or (b) connection related? Depending on the speeds of connection, transfer, cpu power, etc., it could simply be that running 2 instances of rsync is severly impacting the effective cpu time that is usefully available. In short, without much more data it's impossible to say what might be the matter. Paul Slootman -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bidirectional speed question
Paul Slootman wrote: On Mon 17 May 2004, Greger Cronquist wrote: I'm wondering if the following is rsync-related or an issue with my supposedly synchronous internet connection: I have a server running an rsync daemon. When I simultaneously pull and push files to this server using two separate processes on the client (different directories), I get about half the transfer speed compared to just running one process. This also occurs when the files are new (so the rsync algorithm should not be active which otherwise causes data to be sent in both directions). Is this to be expected from rsync? Or is it (a) a bug or (b) connection related? Depending on the speeds of connection, transfer, cpu power, etc., it could simply be that running 2 instances of rsync is severly impacting the effective cpu time that is usefully available. In short, without much more data it's impossible to say what might be the matter. Paul Slootman Thanks for your reply! 1. All files are new = virtually no cpu usage at either end. 2. Line speed is around 100 kbps but halves when transferring files in both directions. 3. Win2k server at both ends. 4. rsync 2.6.1cvs (just before 2.6.2) at both ends. Exactly the same binaries. /Greger -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bidirectional speed question
I agree with Paul. It's almost certainly hour WAN link. My own at home often gives sustained downloads in excess of 2Mbps, This seems to be throttled by the cube of the difference between upload speed and 16kbps. When I get up to 10kbps up, it's still useable. At 12, it's like a 21,400 dialup, at 14, it's like amateur packet radio, and at 15-15.9kbps upload, all downloads cease... even name resolution fails. Sprint Broadband Direct is an extreme example, but most home broadband has similar linkage between the pipes. You are likely to actually improve your overall performance by throttling the rsync with the --bwlimit= option. Turn the upstream one's speed down to about 70% of its maximum alone, and it should leave enough for the downstream to do well. Tim Conway Unix System Administration Contractor - IBM Global Services desk:3032734776 [EMAIL PROTECTED] 2. Line speed is around 100 kbps but halves when transferring files in both directions. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
File sync/delete question
I think this would be an excellent addition to rsync and could find many applications for it myself. However like Markus I haven't the time or the programing knowledge to make it myself. just wanted to get my two cents worth in, Matt Markus Gaugusch [EMAIL PROTECTED] 05/21/04 04:10PM Hi, I need to solve a specific problem and I think that I need an extension to rsync for this to work :) I have an FTP server in the DMZ and a samba server in my intranet. The ftp server has an incoming directory, where people can put uploads and it is also running rsync in daemon mode. The samba server syncs the incoming directory of the ftp server every minute to make access to the uploaded files easier (I don't want the internal people to deal with ftp/sftp to get uploaded files. In fact, there is also an outgoing directory, that is also on the samba server and mirrored to the ftp server, so incoming should work the same way). My problem is, that any file that is deleted on the samba server is synced again from the ftp server a minute later. I use rsync -rvt $FTPSRV /ftp_mirror/incoming for syncing. The --delete option just doesn't fit here, because I want to delete on the SENDING side, not on mine. Also, still-running uploads would not be completely rsync'd if the file would be deleted while being uploaded. Imagine this happenings: 10:00: file a.txt gets uploaded to ftp server 10:01: file a.txt is rsynced to samba server 10:02: file a.txt gets deleted from samba server (after rsync run) 10:03: file would again be synced :( Now I think that I need a new option for rsync: Transfer all files, and remotely delete those that must have been transferred before and are not here anymore. E.g. rsync -rvt --delete-remote-before `cat last-sync-date` ... last-sync-date is a file that contains the unix timestamp (or any other format) of the last syncronization. It could probably also be created by rsync after successful completion, or created on return code 0 by my script. Again my scenario from above: 10:00: file a.txt gets uploaded to ftp server 10:01: file a.txt is rsynced to samba server 10:02: file a.txt gets deleted from samba server (after rsync run) 10:03: last rsync run was at 10:02, but file a.txt on server was modified at 10:00, so it must already have been transferred -- delete it on server Is this feasible? Is anyone willing to implement this, or shall I try on my own (haha, of course you would like me to do, but I'm asking because I don't have too much time ;-) kind regards Markus Gaugusch -- __/\ Markus Gaugusch \ /ASCII Ribbon Campaign markus(at)gaugusch.at X Against HTML Mail / \ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: two-way synchronization accross a firewall fails
On Thu, May 20, 2004 at 11:46:35AM -0700, Tim Harsch wrote: both machines have an /etc/rsyncd.conf of: [rt] path = /tmp/rsync_test comment = Test area Note that there is no read only = no, so this area cannot be written to. That may well be why the upload command you mentioned failed. It is hard to know for sure, though, because the command you use creates its own rsync daemon, and it doesn't look for the rsyncd.conf file in /etc, but in the home dir of the user you are using for the ssh connection. O runs rsync daemon, I initiates a rsync cammnad like rsync -rvvv --delete --rsh=ssh O::rt /tmp/rsync_test As Tim said, there's no need to run a separate rsync daemon if you're using an ssh connection, so skip that part. However, he fails to point out that it is possible to use :: with -rsh=ssh if you want some of the features of daemon mode (such as named directories or logging) but still want to use ssh (just realize that users can do other copies besides what is allowed in the config file unless you restrict the remote commands that ssh allows to be run). If you just want to transfer the files, simplify to something like this: rsync -rvvv --delete --rsh=ssh O:/tmp/rsync_test /tmp rsync -rvvv --delete --rsh=ssh /tmp/rsync_test O:/tmp Also, you'll probably want -t (unless you don't plan on updating the files once they're transferred). ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question about --bwlimit=
On Fri, May 21, 2004 at 02:48:12PM -0400, Wallace Matthews wrote: I can repeat this time after time. If --bwlimit is 4000 (ie. 4005, 4025, 4050,5000,7500,1,10) real is in the same range as 4001. If --bwlimit is 4000 or under (ie. 3725, 2000, 1000, 100) real is in the same range as 4000. That is because of this calculation: tv.tv_usec = bytes_written * 1000 / bwlimit; Rsync calls this function after a lot of 4-byte writes, and thus the sleep time for 4 * 1000 / 4001 (or anything higher than 4001) is 0. Thus, rsync neglects a bunch of sleep calls (but not all of them). I'm looking into some of the old bwlimit patches to see about improving this. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question about --bwlimit=
On Mon, May 24, 2004 at 01:54:42PM -0700, Wayne Davison wrote:
I'm looking into some of the old bwlimit patches to see about
improving this.
Here's a potential patch to make --bwlimit better. This started with
Roger's idea on accumulating delay until we have enough to make a sleep
call without significant rounding error, but I modified it to keep count
in bytes written so that we should avoid the problem discovered when
bwlimit is 4001KBPS or larger. The patch subtracts out elapsed time
since the last call to sleep_for_bwlimit() (but only in a limited way)
and also makes note of any rounding after the sleep() call when it
resets the counter. I also changed the use of 1000 for K to 1024 so
that it would more closely match the value reported by the progress
output. Finally, I applied a modified version of the patch that Paul
just reminded us that Debian is using, though I decided to limit the
write size to bwlimit * 512 rather than bwlimit * 100 (at least for
now, but feel free to argue that a different value is better).
Comments? Is this overkill? Does it have flaws? In my limited testing
this made the bwlimit more accurate.
..wayne..
--- io.c15 May 2004 19:31:10 - 1.121
+++ io.c24 May 2004 22:58:14 -
@@ -739,10 +739,22 @@ unsigned char read_byte(int f)
* use a bit less bandwidth than specified, because it doesn't make up
* for slow periods. But arguably this is a feature. In addition, we
* ought to take the time used to write the data into account.
+ *
+ * During some phases of big transfers (file FOO is uptodate) this is
+ * called with a small bytes_written every time. As the kernel has to
+ * round small waits up to guarantee that we actually wait at least the
+ * requested number of microseconds, this can become grossly inaccurate.
+ * We therefore keep track of the bytes we've written over time and only
+ * sleep when the accumulated delay is at least 1 tenth of a second.
**/
static void sleep_for_bwlimit(int bytes_written)
{
- struct timeval tv;
+ static struct timeval prior_tv;
+ static long total_written = 0;
+ struct timeval tv, start_tv;
+ long elapsed_usec, sleep_usec;
+
+#define ONE_uSEC 100L
if (!bwlimit)
return;
@@ -750,11 +762,31 @@ static void sleep_for_bwlimit(int bytes_
assert(bytes_written 0);
assert(bwlimit 0);
- tv.tv_usec = bytes_written * 1000 / bwlimit;
- tv.tv_sec = tv.tv_usec / 100;
- tv.tv_usec = tv.tv_usec % 100;
+ total_written += bytes_written;
+
+ gettimeofday(start_tv, NULL);
+ if (prior_tv.tv_sec) {
+ elapsed_usec = (start_tv.tv_sec - prior_tv.tv_sec) * ONE_uSEC
++ (start_tv.tv_usec - prior_tv.tv_usec);
+ total_written -= elapsed_usec * bwlimit / (ONE_uSEC/1024);
+ if (total_written 0)
+ total_written = 0;
+ }
+ sleep_usec = total_written * (ONE_uSEC/1024) / bwlimit;
+ if (sleep_usec ONE_uSEC / 10) {
+ prior_tv = start_tv;
+ return;
+ }
+
+ tv.tv_sec = sleep_usec / ONE_uSEC;
+ tv.tv_usec = sleep_usec % ONE_uSEC;
select(0, NULL, NULL, NULL, tv);
+
+ gettimeofday(prior_tv, NULL);
+ elapsed_usec = (prior_tv.tv_sec - start_tv.tv_sec) * ONE_uSEC
++ (prior_tv.tv_usec - start_tv.tv_usec);
+ total_written = (sleep_usec - elapsed_usec) * bwlimit / (ONE_uSEC/1024);
}
@@ -812,6 +844,8 @@ static void writefd_unbuffered(int fd,ch
if (FD_ISSET(fd, w_fds)) {
int ret;
size_t n = len-total;
+ if (bwlimit n (size_t)bwlimit * 512)
+ n = (size_t)bwlimit * 512;
ret = write(fd,buf+total,n);
if (ret 0) {
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Mon May 24 18:38:05 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv7253 Modified Files: options.c Log Message: The various include/exclude options are not used on the server side, so if someone is trying something funny, just quit. Revisions: options.c 1.151 = 1.152 http://www.samba.org/cgi-bin/cvsweb/rsync/options.c.diff?r1=1.151r2=1.152 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
