apt-proxy v2 and rsync
Can anyone explain why rsync is no longer considered an appropriate
method for fetching Packages files? It's the only mechanism I'm aware of
that makes "apt-get update" over a 56Kb/s connection complete in a
reasonable length of time. Am I missing something?
Begin forwarded message:
Date: Thu, 7 Oct 2004 07:10:28 -0700
From: Ian Bruce <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>, <[EMAIL PROTECTED]>
Subject: apt-proxy v2 and rsync
I was distressed to read the following in the documentation for the new
apt-proxy:
- rsync is not officially supported.
It can work with rsync and some LD_PRELOAD hack, although it has
not been tested for a while, and the shared library is not
included in the .deb package.
If someone convinces me that it is still usefull in the current
situation I may give another chance to rsync backends.
- '+' prefix or equivalent functionality is not supported.
This was specially usefull for rsyncing uncompressed Packages
files, which are not anymore available.
'rsyncpackages' config option was supposed to fill the gap, but
in the current situation, someone will have to convince me that
it is still usefull.
I don't understand why the utility of rsync for fetching Packages files
is in question. For anyone with a dialup internet connection, waiting to
download ten megabytes of compressed Packages files every time they do
an "apt-get update" is unacceptable. The rsync option solved this
problem.
I spent some time tracking down a bug in the old version to make this
work properly:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=166305
I'll have to go on using it if the new version won't support rsync.
Now that gzip has the "--rsyncable" option, wouldn't it be feasible to
rsync against compressed Packages files rather than having to keep the
uncompressed ones around for this purpose?
Unless I'm missing some important development ("current situation"?),
the rsync method for Packages files is still as important as it ever
was. Please reconsider your decision to drop support for this feature.
Not everyone has a high-speed internet connection.
-- Ian Bruce
Re: apt-proxy v2 and rsync
On Tuesday 26 October 2004 09.20, Ian Bruce wrote: > Can anyone explain why rsync is no longer considered an appropriate > method for fetching Packages files? IIRC the problem is that rsync is quite CPU-heavy on the servers, so while the mirrors have the (network) resources to feed downloads to 100s of users, they don't have the (CPU) resources for a few dozen rsyncs. regards -- vbi -- Oops pgp3ZmFWdUFgo.pgp Description: PGP signature
Re: apt-proxy v2 and rsync
On Oct 26, Ian Bruce <[EMAIL PROTECTED]> wrote: > Can anyone explain why rsync is no longer considered an appropriate > method for fetching Packages files? It's the only mechanism I'm aware of Because it's hard on servers, for a start. -- ciao, | Marco | [8782 diFcw3LT7Erlw] signature.asc Description: Digital signature
Re: apt-proxy v2 and rsync
On Thu, Oct 28, 2004 at 01:54:54PM +0200, Adrian 'Dagurashibanipal' von Bidder wrote: > IIRC the problem is that rsync is quite CPU-heavy on the servers, so > while the mirrors have the (network) resources to feed downloads to > 100s of users, they don't have the (CPU) resources for a few dozen > rsyncs. And shouldn't this be left as a decision for the mirror administrators? It's not like setting up a mirror _automatically_ allows rsync access to it, isn't it? Marcelo
Re: apt-proxy v2 and rsync
On Tue, Oct 26, 2004 at 12:20:19AM -0700, Ian Bruce said > Now that gzip has the "--rsyncable" option, wouldn't it be feasible to > rsync against compressed Packages files rather than having to keep the > uncompressed ones around for this purpose? You have to explicitly enable this option, which is not currently done. -rob -- Words of the day: strategic jihad argus airframe Sears Tower BLU-114/B
Synching mirrors and clients (was: Re: apt-proxy v2 and rsync)
> > Can anyone explain why rsync is no longer considered an appropriate
> > method for fetching Packages files?
>
> IIRC the problem is that rsync is quite CPU-heavy on the servers, so while
> the mirrors have the (network) resources to feed downloads to 100s of
> users, they don't have the (CPU) resources for a few dozen rsyncs.
>
Why do you keep on saying this without providing _any_ figures!
Why always synching the full mirror when only about 1% of the files
changes daily? Just change to single file synching and most of your CPU
load is gone. Single file rsync doesn't need any CPU power to discover
the changed files. Single file rsync touches only the changed files,
only about 1%, so at least disk access is much less while probably also
lowers the CPU load.
If gzip --rsyncable would be used the CPU load would dramatically be
lowered, much lower than with _any_ other synching. As a side effect the
use bandwidth would be equally well be lowered. IMO rsync is very useful
if don't right.
Prove of concept
To finally produce some figures and prove this concept two servers are
needed, the first one an ordinary source mirror, the second a secondary
mirror with different mirror directories for each of the test cases. On
the first server the CPU load is measured, on the second the different
sync scripts are run:
- Ordniary full mirroring rsynch as today in use
- DpartialMirror sync script ("http://dpartialmirror.sourceforge.net/";)
- Deb-mirror sync script
- ???
- Sync with wget, etc.
IMO this will show which is the best solution for full mirrors.
Now limit the secondary mirror to support only one architecture and do
the test above again. This will show the best solution for the commonly
used mirrors.
In a third step limit the packages to what an ordinary user has, just
use popularity-contest or I could provide my dpkg --getselections. This
will show the best solution for servers from clients impact.
Now if you feel advantous, repack as many package on the source mirror
with gzip --rsyncable and notice the difference.
O. Wyss
--
See a huge pile of work at "http://wyodesktop.sourceforge.net/";
Re: Synching mirrors and clients (was: Re: apt-proxy v2 and rsync)
On Thu, Nov 04, 2004 at 05:46:55PM +0100, Otto Wyss wrote: > > Now if you feel advantous, repack as many package on the source mirror > with gzip --rsyncable and notice the difference. Exactly how is this going to help? I can only see this as being useful when the files change. Files should never change. You get a completly new file. Unless you can somehow tell to use the old file as base, this is not going to help. Kurt
Re: Synching mirrors and clients (was: Re: apt-proxy v2 and rsync)
On Thu, Nov 04, 2004 at 06:35:40PM +0100, Kurt Roeckx wrote: > On Thu, Nov 04, 2004 at 05:46:55PM +0100, Otto Wyss wrote: > > > > Now if you feel advantous, repack as many package on the source mirror > > with gzip --rsyncable and notice the difference. > > Exactly how is this going to help? I can only see this as being > useful when the files change. Files should never change. You > get a completly new file. Unless you can somehow tell to use the > old file as base, this is not going to help. He's talking about the Packages etc. files. Those do change. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune
Re: Synching mirrors and clients (was: Re: apt-proxy v2 and rsync)
On Thu, Nov 04, 2004 at 06:35:40PM +0100, Kurt Roeckx wrote: > > Now if you feel advantous, repack as many package on the source mirror > > with gzip --rsyncable and notice the difference. > > Exactly how is this going to help? I can only see this as being > useful when the files change. Files should never change. You > get a completly new file. Unless you can somehow tell to use the > old file as base, this is not going to help. there was a patch by paul russell floating around to build a heuristic into rsync to allow it to figure out from the name which file might be similar and use this as the file to sync against. this should work pretty fine for debian packages, especially with --rsyncable cu robert -- Robert Lemmen http://www.semistable.com signature.asc Description: Digital signature
Re: Synching mirrors and clients (was: Re: apt-proxy v2 and rsync)
On Thursday 04 November 2004 17.46, Otto Wyss wrote: > Why do you keep on saying this without providing _any_ figures! Who is "you" here? Please pay attention to attribution on mailing list postings - especially if you're starting a new thread with your mail. I posted this statement about cpu load of rsync, and did that exactly once, so I don't "keep saying it". Also, I put in an IIRC, so you have clear indication that I'm not too sure - somebody asked about the reason, I answered with that I heard was the reason when the last person asked. -- vbi -- Oops pgp5WZplFp7Wy.pgp Description: PGP signature
Re: Synching mirrors and clients (was: Re: apt-proxy v2 and rsync)
> > > IIRC the problem is that rsync is quite CPU-heavy on the servers, so while > > > the mirrors have the (network) resources to feed downloads to 100s of > > > users, they don't have the (CPU) resources for a few dozen rsyncs. > > > > Why do you keep on saying this without providing _any_ figures! > > Who is "you" here? Please pay attention to attribution on mailing list > postings - especially if you're starting a new thread with your mail. I > posted this statement about cpu load of rsync, and did that exactly once, > so I don't "keep saying it". Also, I put in an IIRC, so you have clear > indication that I'm not too sure - somebody asked about the reason, I > answered with that I heard was the reason when the last person asked. > I don't meant you personally but this statement about the CPU load, mostly accompanied with some vage numbers is repeated all the time and whenever I ask for exact figures only excuses or even not an aswer is provided. Your statement about "feed downloads to 100s ...(CPU) resources for a few dozen rsyncs" implied you have actually seen such CPU loads. Now you say you just repeated from hear say. How much value did your message have to the OP? Does it have any value? Well the main problem is not your message but that nobody is able or willing to set up a reasonable test case and provide accurate figures. Maybe this is a non issue because Debian has more than enough systems and don't has to care about. O. Wyss -- See a huge pile of work at "http://wyodesktop.sourceforge.net/";
