rsyncd.conf: "timeout=" crazyness
Hi,
from time to time, in times like today where the whole world is grabbing
SUSE-9.2 and/or debian-30r4, I really like to condemn those other anon
rsync server admins (you know, the successors of the traditional unix ftp
server admins).
They usually have within their /etc/rsyncd.conf a line like
timeout =
because they are thinking "less" there is "better for my server health",
fully neglecting (better: not aware of) the situation of their partner
servers during those phases of general download hystery.
At ftp.gwdg.de, I have a load ("acceptance!", "trust"!) like never never
ever before currently, caused by the nearly simultanous releases of
SUSE-Linux-9.2 for both plastforms i386 and x89_64, including a DVD ISO
for the first time (15 GB + 3.3 GB), and debian-30r4 (lots og GBs too),
the latter with a huge amount of pure "ethical" acceptance, regardless of
functionality. the former with additional aspects of affinity...
I guess there is no real machine at this world which is capable to hold
all this extremely hot stuff in buffercache at once, so all the servers
serving SUSE plus DEBIAN (at least those) are stressed with disk I/O like
never before, like me.
You can NOT understand what I say if you only have a 100 MBit/sec
connection to the internat, and I beg you to imagine what would happen to
you if you had "unlimited" bandwidth if you like to undrrstand what I am
trying to point out.
Unlimited bandwith is not THE freedom - it is simply a joke to make you
see your REAL bottlenecks...
This "acceptance" is OK, this is wonderful, this is showing that a server
is worth the name, but actually almost every server belongs to lots of
other servers to serve his role, and so let me show the dark side of
those very very "social" happenings;
ftp.gwdg.de is running into timeouts of this kind:
rsync: connection unexpectedly closed (1345637 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(342)
while contacting the masters of almost all important primary sources
which get mirrored using rsync at ftp.gwdg.de.
It happens after building the "remote data base" and the "local database",
just during the start of the real "fetch" actions.
Why?
Regardless how busy the server is, the client fulfills the "remote data
base" phase first, and then he is starting his "local data base" phase
second.
With a high level of "servility" (f.e. 5000 FTP sessions. 900 HTTP
sessions and 200 RSYNC sessions in parallel and 4 TB of output a day,
like seen on ftp.gwdg.de all these days since saturday), this "local"
rsync phase may need a long long (currently very long long here) time
because the local filesystem at ftp.gwdg.de is very busy and the Linux
buffer cache performance is relatively bad in kernel 2.4.
After fulfilling it, we reach the state that we would like to fetch the
first piece of new data from the server.
What happens?
See above.
rsync: connection unexpectedly closed (1345637 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(342)
Again, why?
Because server admins like to sleep with sweet dreams and are willing to
help themseves to achieve that.
So they put into their own /etc/rsyncd,conf a line like
timeout=
Natural, true "human" behaviour, fully understandable.
Yes, but not very much "socially aware".
I see two solutions:
1. the rsync maintainer delivers the product with a default of
timeout=0
for /etc/rsyancd.conf and tries to awake social responsiveness by
presenting some special words with "man rsyncd.conf" mentioning the
situation of very busy clients.,
2. all the maintainers of "socially relevant" Linux ressources end their
small dsreams and enter the bigger one:
timeout=
to help the society which is nothing more than a thousand time JUST YOU
INDIVIDUAL.
The third solution could be an option with rsync which forces to turn the
database phases vice versa (build "local" first. then contact the
server), but this would not be the best solution.
So plese rsync admins, put
timeout=
into your /etc/rsyncd.conf.
ftp:3 03:47:14 ~ # grep timeout /etc/rsyncd.conf
timeout = 15000
ftp:3 03:47:16 ~ #
Cheeers -e
--
Eberhard Moenkeberg ([EMAIL PROTECTED], [EMAIL PROTECTED])
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Timezone error with chrooted rsync version 2.6.3pre1
Hi, I am not sure it this still is a point, but I just discovered that a chrooted rsync version 2.6.3pre1 is forgetting the time zone underways: Jan 12 07:00:01 ftp4 rsyncd[11091]: rsync on dobes/ from [EMAIL PROTECTED] (134.76.28.251) Jan 12 06:00:39 ftp4 rsyncd[11091]: wrote 732542 bytes read 182 bytes total size 553909487945 Logging is done with syslog. Time zone is GMT+1, the system is set up to run the hardware clock as GMT. You see, the session is starting under the correct time, but after chrooting and doing the work, time zone info "+1" is lost. This does not happen on a different system (same setup) with rsync version 2.6.2: Jan 12 00:00:08 ftp rsyncd[1090]: rsync on pub/linux/suse/ftp.suse.com/suse/i386/update/9.1 from email.medizinische-genetik.de (62.245.197.183) Jan 12 00:00:10 ftp rsyncd[1090]: wrote 82006 bytes read 117 bytes total size 1327942093 Cheers -e -- Eberhard Moenkeberg ([EMAIL PROTECTED], [EMAIL PROTECTED]) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
connection unexpectedly closed...
When I run rsync -v --exclude-from= -a --delete --numeric-ids -e ssh -i /root/.ssh/id_rsa /bin/ [EMAIL PROTECTED]:/export/home/scbackup/socnt01//bin/ I immediately get: rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(359) rsync -v --exclude-from= -a --delete -e ssh -i /root/.ssh/id_rsa /boot/ [EMAIL PROTECTED]:/export/home/scbackup/socnt01//boot/ rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(359) However, if I run exactly the same command, but with a normal user as the remote user instead of root, the rsync works as expected. Both root and the normal user have passphraseless keys set. the reason why I want the rsync to run with root as the remote user is that I want the UID and GID of the copied files to be preserved on the remote machine (which has users and UIDs quite different from the local machine). What could I be doing wrong which stops it working with root as the remote user? thanks for your hints! Nigel -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Protocol incompatibility with rsync 2.6.4 (cvs) and disk full
Hi, I got this error after a transaction was interrupted by me and a new (slightly updated) transaction was started. ... fedora/1/en/i386/base/pkglist.dag.bz2 1246954 100% 67.79kB/s0:00:17 (27, 1.7% of 201557) Invalid file index 1777522462 (count=201557) rsync error: protocol incompatibility (code 2) at sender.c(152) On both ends exactly the same rsync was used. It's a packaged release from CVS yesterday: http://dag.wieers.com/packages/rsync/ A second run (with again a slightly updated transaction) gives the same error on another file: ... fedora/2/en/x86_64/base/pkglist.dag.bz2 1503129 100% 57.80kB/s0:00:25 (64, 10.5% of 201595) Invalid file index -1487994937 (count=201595) rsync error: protocol incompatibility (code 2) at sender.c(152) A third run gives: fedora/2/en/x86_64/base/pkglist.dag.bz2 1503129 100% 12.00kB/s0:01:59 (64, 10.5% of 201595) Invalid file index -131924025 (count=201595) rsync error: protocol incompatibility (code 2) at sender.c(152) And then I noticed my partition was full, made some room, synced again and it miraculously worked. I'm not sure why it gives this weird error when the *sender* has a full partition but apparantly it did. Kind regards, -- dag wieers, [EMAIL PROTECTED], http://dag.wieers.com/ -- [all I want is a warm bed and a kind word and unlimited power] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Speed up rsync ,cwRsync and replay changes against a file
I found a post to a users group you made regarding cwrsync and incremental backups. You were trying to see if there was a way to capture only the data that rsync had viewed as changed on the file and then save this to a different location on the linux machine and therefore give you an incremental backup. I was curious if you ever found a way of doing so as I would like to try and do the same thing but have yet to figure out a way? Thanks, Aaron Dailey Systems Engineer West Hills Gastroenterology Associates P.C. 9155 S.W. Barnes Rd., #636 Portland, OR 97225-6687 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: read-only access without rsyncd?
On 2005-01-12 09:20:29 -0800, Wayne Davison wrote: > On Wed, Jan 12, 2005 at 02:53:47PM +0100, Martin Schr?der wrote: > > Can I limit the rsync command on the server to read-only? > > The "resources" page of the rsync website mentions Joe Smith's perl > script that lets you do what you want: > > http://www.inwap.com/mybin/miscunix/?rrsync Thanks! Best regards Martin -- Martin Schröder, [EMAIL PROTECTED] ArtCom GmbH, Lise-Meitner-Str 5, 28359 Bremen, Germany Voice +49 421 20419-44 / Fax +49 421 20419-10 http://www.artcom-gmbh.de -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: read-only access without rsyncd?
On Wed, Jan 12, 2005 at 02:53:47PM +0100, Martin Schr?der wrote: > Can I limit the rsync command on the server to read-only? The "resources" page of the rsync website mentions Joe Smith's perl script that lets you do what you want: http://www.inwap.com/mybin/miscunix/?rrsync You'll need to put a command like this into your .ssh/authorized_keys file: command="rrsync -ro /" ssh-rsa B3NzaC1yc2E... The "rrsync" is the name of the script, -ro means read-only, "/" is a directory for restricted access (since you wanted no restrictions), and the rest is the ssh-key info. Caveats: the script as written does not let you specify names with spaces nor multiple items to fetch. Also, the pathnames it constructs when you restrict the path to "/" will begin with "//", so as long as that doesn't cause problems on your OS, it should work OK for you. I'm working on some changes that should resolve these minor issues. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
read-only access without rsyncd?
Hi, I've a setup where a special account (limited by the ssh-key) can use rsync via ssh to make a backup of / (limited by sudo). Obviously it needs read-access to everything. However, since he can read everything, he could also easily _write_ everything. Can I limit the rsync command on the server to read-only? Thanks in advance Martin -- Martin Schröder, [EMAIL PROTECTED] ArtCom GmbH, Lise-Meitner-Str 5, 28359 Bremen, Germany Voice +49 421 20419-44 / Fax +49 421 20419-10 http://www.artcom-gmbh.de -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
