Re: Debian mirror scripts
On Mon, Feb 14, 2005 at 11:05:56AM +1100, Brian May wrote: zsync looks suspiciously like it might have similar patent issues which killed the rproxy project. Then again I am no expert; Please tell me I am wrong... i'm not an expert either, but the zsync maintainer and i talked to a lot of people about this issue. and while you can never be sure about patents i am very confident that zsync is unproblematic. cu robert -- Robert Lemmen http://www.semistable.com signature.asc Description: Digital signature
Re: Debian mirror scripts
Goswin == Goswin von Brederlow [EMAIL PROTECTED] writes: Goswin zsync uses the algorithm described in the rsync technical paper Goswin afaik. Does rsync have a patent issue? Do we realy care about some Goswin stupid countries patents? My understanding is that rsync doesn't have problems, but rproxy (which is also based on rsync) does have problems because it does the calculations at the client side instead of the server side. Rusty Russell had a webpage up describing the history of the problems encountered with rproxy, but all I can find right now is an empty page: URL:http://ozlabs.org/~rusty/index.cgi/rproxy.html (linked from: URL:http://ozlabs.org/~rusty/index.cgi/IP/2004-10-14.html). IIRC the patent owner wasn't interested in this application of the patent but was threatening to enforce it regardless. However, based on Robert's response it would appear that patent issues have been considered for zsync and considered OK. I would speculate this is because information is pre-calculated at the server and stored in *.zsync files. -- Brian May [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Goswin == Goswin von Brederlow [EMAIL PROTECTED] writes: Goswin zsync has the option of looking into gziped files and Goswin rsync them as if they would be ungziped (while still just Goswin downloading chunks of the gziped file). Its a bit more Goswin complex algorithm but works even better than rsyncable Goswin files and rsync. zsync looks suspiciously like it might have similar patent issues which killed the rproxy project. Then again I am no expert; Please tell me I am wrong... -- Brian May [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Brian May [EMAIL PROTECTED] writes: Goswin == Goswin von Brederlow [EMAIL PROTECTED] writes: Goswin zsync has the option of looking into gziped files and Goswin rsync them as if they would be ungziped (while still just Goswin downloading chunks of the gziped file). Its a bit more Goswin complex algorithm but works even better than rsyncable Goswin files and rsync. zsync looks suspiciously like it might have similar patent issues which killed the rproxy project. Then again I am no expert; Please tell me I am wrong... -- Brian May [EMAIL PROTECTED] zsync uses the algorithm described in the rsync technical paper afaik. Does rsync have a patent issue? Do we realy care about some stupid countries patents? Maybe non-us isn't ready to die yet, changing from crypto to patented software. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
On Sun, Jan 02, 2005 at 11:59:20PM +0100, Goswin von Brederlow wrote: Why there isn't there already a rsync method for apt is probably a mystery nobody ever will solve. It is not wanted due to rsync causing excessive server load. If Debian would provide zsync files a zsync module could be provided in a matter of days. zsync files for Packages (don't know which suite, though) are available on zsync's homepage and anyway it would be trivial to hack archive scripts to provide .zsync files and/or generate them somewhere like p.d.o filippo -- Filippo Giunchedi GNU/PG key: 6B79D401 Random signature follows: Computer Science is no more about computers than astronomy is about telescopes. -- Edsger Dijkstra pgpQuAxVd14yi.pgp Description: PGP signature
Re: Debian mirror scripts
Filippo Giunchedi [EMAIL PROTECTED] writes: On Sun, Jan 02, 2005 at 11:59:20PM +0100, Goswin von Brederlow wrote: Why there isn't there already a rsync method for apt is probably a mystery nobody ever will solve. It is not wanted due to rsync causing excessive server load. If Debian would provide zsync files a zsync module could be provided in a matter of days. zsync files for Packages (don't know which suite, though) are available on zsync's homepage and anyway it would be trivial to hack archive scripts to provide .zsync files and/or generate them somewhere like p.d.o Yes it would. Please convince ftp-master of that. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
On Sat, 12 Feb 2005, Goswin von Brederlow wrote: Filippo Giunchedi [EMAIL PROTECTED] writes: On Sun, Jan 02, 2005 at 11:59:20PM +0100, Goswin von Brederlow wrote: Why there isn't there already a rsync method for apt is probably a mystery nobody ever will solve. It is not wanted due to rsync causing excessive server load. If Debian would provide zsync files a zsync module could be provided in a matter of days. zsync files for Packages (don't know which suite, though) are available on zsync's homepage and anyway it would be trivial to hack archive scripts to provide .zsync files and/or generate them somewhere like p.d.o Yes it would. Please convince ftp-master of that. Better yet, submit a patch! -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Goswin von Brederlow wrote: Why there isn't there already a rsync method for apt is probably a mystery nobody ever will solve. It is not wanted due to rsync causing excessive server load. That is simply not true. This statement is repeated all the time but nobody ever was able to show hard figures. Where rsync produces much load is during the phase when it collects all the files for synchronisation and not during MD5 computation but this is is only due to not well designed scripts. DpartialMirror doesn't impose this phase since it only require single-file transfers and does the file collecting phase on the client. New versions. The size of the Packages files is comparatively tiny compared to all the debs. Even the 1% saving for rsyncing debs is hardly worth it due to the extra traffic for the checksums and the server load it causes. Sorry rsync reports the overall use, incl. checksums etc. Of course 1% saving doesn't make much sense so that's the main reason I don't develop DpartialMirror further. Anyway the next time a distribution concept is designed it will be based on a p2p solution. zsync has the option of looking into gziped files and rsync them as if they would be ungziped (while still just downloading chunks of the gziped file). Its a bit more complex algorithm but works even better than rsyncable files and rsync. As long as zsync allows multi-file transfers it won't be better that rsync. O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wxguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Otto Wyss [EMAIL PROTECTED] writes: Goswin von Brederlow wrote: Why there isn't there already a rsync method for apt is probably a mystery nobody ever will solve. It is not wanted due to rsync causing excessive server load. That is simply not true. This statement is repeated all the time but nobody ever was able to show hard figures. Rsync by default uses ~3% of the file size in ram to store block checksums. Consider a new kde-i18n release with its 200Mb file (for an extrem case). 33 donloads waste 200MB ram, 330 download 2GB ram. When do you thing the mirror will start swapping? The ~3% are filled with the clients checksums first and then rsync reads in the full file computing the adler32 checksum per byte and a md4sum for each potential match. Even though adler32 and md4 are very fast they are more load than just sending out the file. And all that for just ~1% saving (without rsyncable). Worse is downloading multiple files (as you mention below) using include and exclude patterns. Downloading 1000 files at once through this takes somewhere around an hour just to build a file listing what to get doing a complete find over the archive (wasting tons of I/O). But downloading files seperately isn't that much better since then every file opens a new connect and forks a new rsync on the server. Starting a new full Debian-amd64 mirror with a 300ms ping reply (that is what I get roughly) to the server would waste 75 hours on waiting for the initial three-way handshake for a connect. And another 50 hours for the round-robin sending the name of a file and getting the data. You can say all this is bad design in rsync and the solution is dead simple (now): zsync. Where rsync produces much load is during the phase when it collects all the files for synchronisation and not during MD5 computation but this is is only due to not well designed scripts. DpartialMirror doesn't impose this phase since it only require single-file transfers and does the file collecting phase on the client. Depends on use and available client data. New versions. The size of the Packages files is comparatively tiny compared to all the debs. Even the 1% saving for rsyncing debs is hardly worth it due to the extra traffic for the checksums and the server load it causes. Sorry rsync reports the overall use, incl. checksums etc. What I ment is the extra outgoing traffic on the client side. For example for dsl users sending the checksums might actualy slow things more down than the 1% saving speeds things up. Of course 1% saving doesn't make much sense so that's the main reason I don't develop DpartialMirror further. Anyway the next time a distribution concept is designed it will be based on a p2p solution. zsync has the option of looking into gziped files and rsync them as if they would be ungziped (while still just downloading chunks of the gziped file). Its a bit more complex algorithm but works even better than rsyncable files and rsync. As long as zsync allows multi-file transfers it won't be better that rsync. O. Wyss zsync works via http. YOU have to know the filename to request, so no large find like rsync does for multiple files, but through keep-alive you can still do multiple files, even in parallel if you like and code it, with a single connection. Also note that zsync uses precomputed checksum files on the server side and does the adler/md4 computations on the client. Thats why it can use simple http/1.1. It's a win-win situation. MfG Goswin PS: I think the sarge/sid rsync has improved the multiple file download but that doesn't help woody. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Goswin von Brederlow [EMAIL PROTECTED] wrote: Otto Wyss [EMAIL PROTECTED] writes: reply (that is what I get roughly) to the server would waste 75 hours on waiting for the initial three-way handshake for a connect. And another 50 hours for the round-robin sending the name of a file and getting the data. Did you messure this figures on a real Debian mirror or is it just what you think? can use simple http/1.1. It's a win-win situation. Perfect go ahead. Even if DpartialMirror isn't developed further I use it almost daily and are quite happy with it. And I guess this won't change in the future. O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wxguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
[EMAIL PROTECTED] (Otto Wyss) writes: Goswin von Brederlow [EMAIL PROTECTED] wrote: Otto Wyss [EMAIL PROTECTED] writes: reply (that is what I get roughly) to the server would waste 75 hours on waiting for the initial three-way handshake for a connect. And another 50 hours for the round-robin sending the name of a file and getting the data. Did you messure this figures on a real Debian mirror or is it just what you think? That is pure mathematics. An initial tcp three way handshake has to travel tree times the distance between me and the server. That is 150ms * 3 just to connect and then one roundtrip to get the data flowing. Even if the 3rd packet of the handshake is combined with data (not sure if that happens just now) you still have to wait for the reply. So 4 or 5 times 150ms times the number of files (~60). But it's not only math, I've tested this (not with that many files obviously) for debmirror using ftp (one data connect per file), http (one connect per file), http (keep-alive), rsync (1 file per connect) and rsync (100 files per connect). Http with keep-alive wins. can use simple http/1.1. It's a win-win situation. Perfect go ahead. Even if DpartialMirror isn't developed further I use it almost daily and are quite happy with it. And I guess this won't change in the future. If Debian starts shipping zsync checksum files in its mirror then adapting to it should be trivial. Just add the Depends and replace the rsync call with zsync. I just hope some ftp-master will. O. Wyss MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
[EMAIL PROTECTED] (Otto Wyss) writes: Goswin von Brederlow [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (Otto Wyss) writes: Goswin von Brederlow [EMAIL PROTECTED] wrote: Sure? Anyway DpartialMirror http://dpartialmirror.sourceforge.net/; can. I guess mirrorer doesn't care for bandwith saving as DpartialMirror, correct me if I'm wrong. Currently it will always redownload the Packages/Sources files as gzip on every update to fix a bug in the apt methods. But I already suggested only updating those that don't match the Release file. And, unless you have an rsync method for apt, it won't rsync files. Why there isn't there already a rsync method for apt is probably a mystery nobody ever will solve. It is not wanted due to rsync causing excessive server load. If Debian would provide zsync files a zsync module could be provided in a matter of days. While rsyncing the Packages files sounds like a good idea to save traffic it actualy is a bit insignificant compared to the daily traffic of new sources and debs. Do you mean there are up to 100 new packages each day? I get between 50 - 150 packages updated each day for just i386. Or do you mean there are 100 new versions? DpartialMirror handles new versions of packages (sources and deps) in a way it save about 1% even when the packages are normal gzip'ed. It would save around 10% - 50% with rsyncable. New versions. The size of the Packages files is comparatively tiny compared to all the debs. Even the 1% saving for rsyncing debs is hardly worth it due to the extra traffic for the checksums and the server load it causes. zsync has the option of looking into gziped files and rsync them as if they would be ungziped (while still just downloading chunks of the gziped file). Its a bit more complex algorithm but works even better than rsyncable files and rsync. Is there any plan to add this feature to mirrored? O. Wyss I doubt it. Everyone can add support for it to apt and reprepo (mirrorer) can utilize it. I think that is the best way. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
[EMAIL PROTECTED] (Otto Wyss) writes: Goswin von Brederlow [EMAIL PROTECTED] wrote: Debmirror is purely a mirror tool. It will download the Meta files just like any other file. You can easily switch between mirror of equal contents but not create Packages files reflecting what is locally available. Sure? Anyway DpartialMirror http://dpartialmirror.sourceforge.net/; can. O. Wyss A note of caution: | 2004-04-03 (wyo) Since Debian does not change its policy to add | adequate support for rsync'ing package mirrors, I don't actively | develop DpartialMirror further. Any user of dpartialmirror should check out mirrorer from alioth. I only glanced at the webpage and haven't used dpartialmirror but now that mirrorer has filtering support it looks like mirrorer could replace dpartialmirror completly and I'm also thinking about retireing debmirror in favour of a wrapper to mirrorer. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Am 30.01.2005 um 16:01 schrieb Thiemo Seufer: Cajus Pollmeier wrote: Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. You might want to compare your script with the one for partial d-i test mirrors, available from d-i SVN in trunk/scripts/testmirror. Here's the result of some testing: debpool:is fine for new pools, but recreating the whole mirror with .deb and .udeb packages didn't work and I'm not the perl guy who's capable of fixing it. dak:too complicated for fire and forget mirrorer: not tested because alioth is still down departialmirror: not a package, after resolving some dependencies, it actually did not do what I wanted. Finally I fixed the quick'n dirty bash script. It lacks features like gpg signing, SHA entries in Releases, etc., but for a local test mirror its failrly enough ;-) If someone's interested, I can clean it up an put it somewhere. Thanks for your hints! Cheers, Cajus -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
On Sun, Jan 30, 2005 at 05:27:11PM +0100, Cajus Pollmeier wrote: Am 30.01.2005 um 16:29 schrieb Marc Haber: On Sun, 30 Jan 2005 14:54:16 +0100, Cajus Pollmeier [EMAIL PROTECTED] wrote: I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Have a look at the debpool package. Ah. Will do. Seems to be in experimental and the description is matching ;-) hat mode=on type=maintainer type=author package=debpool/ Glad folks are finding it useful. There's still a bunch of work to be done on it before I consider it a (semi-) finished work, and thus it lives in experimental rather than unstable; when it goes to 1.0, I'll upload it to sid. However, that shouldn't be read to imply that folks can't or shouldn't use it as it is - just that I don't (yet) guarantee it does all the nice things I want it to do. One thing to be aware of with this usage is that Debpool is (currently) aimed at handling private-upload archives, which means it expects a changes file, which can sometimes be interesting to get from the main archive. The --rebuild-files option will rebuild the Packages, Sources, and Release files (and compressed/signed versions of the same, if you have those options turned on) from the information stored in the meta files in the pool. I have not yet implemented a rebuild all meta files option, though if folks want it, feel free to file a wishlist bug. This would let you drop stuff straight into pool/ and then trigger a full file rebuild. Alternatively, grab the relevant *.changes file (say, off of the changes list or out of the mail archive), and drop it into incoming along with the files from the main archive; that should work just fine (modulo being careful about signed changes files if you have 'require_sigs_meta' turned on). Speaking of which, it's probably time I uploaded the next version (though it's largely cosmetic fixes to some annoying bits)... -- Joel Aelwyn [EMAIL PROTECTED] ,''`. : :' : `. `' `- signature.asc Description: Digital signature
Re: Debian mirror scripts
Goswin von Brederlow [EMAIL PROTECTED] wrote: Sure? Anyway DpartialMirror http://dpartialmirror.sourceforge.net/; can. A note of caution: | 2004-04-03 (wyo) Since Debian does not change its policy to add | adequate support for rsync'ing package mirrors, I don't actively | develop DpartialMirror further. :-( Sad, isn't it? Any user of dpartialmirror should check out mirrorer from alioth. I only glanced at the webpage and haven't used dpartialmirror but now that mirrorer has filtering support it looks like mirrorer could replace dpartialmirror completly and I'm also thinking about retireing debmirror in favour of a wrapper to mirrorer. I guess mirrorer doesn't care for bandwith saving as DpartialMirror, correct me if I'm wrong. O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wxguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Cajus Pollmeier [EMAIL PROTECTED] writes: Am 30.01.2005 um 16:01 schrieb Thiemo Seufer: Cajus Pollmeier wrote: Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. You might want to compare your script with the one for partial d-i test mirrors, available from d-i SVN in trunk/scripts/testmirror. Here's the result of some testing: debpool: is fine for new pools, but recreating the whole mirror with .deb and .udeb packages didn't work and I'm not the perl guy who's capable of fixing it. dak: too complicated for fire and forget mirrorer: not tested because alioth is still down departialmirror: not a package, after resolving some dependencies, it actually did not do what I wanted. Finally I fixed the quick'n dirty bash script. It lacks features like gpg signing, SHA entries in Releases, etc., but for a local test mirror its failrly enough ;-) If someone's interested, I can clean it up an put it somewhere. Thanks for your hints! Cheers, Cajus From the amd64 archive on alioth. Enjoy. MfG Goswin -- # Rebuild Packages, Sources and Release files for SUITE in testing unstable; do case $SUITE in testing) CODENAME=sarge;; unstable) CODENAME=sid;; esac echo $STRUCT/dists/$CODENAME/Release Origin: Debian-amd64 echo $STRUCT/dists/$CODENAME/Release Label: Debian-amd64 echo $STRUCT/dists/$CODENAME/Release Suite: $SUITE echo $STRUCT/dists/$CODENAME/Release Codename: $CODENAME echo $STRUCT/dists/$CODENAME/Release Date: `date -u` echo $STRUCT/dists/$CODENAME/Release Architectures: $ARCHS echo $STRUCT/dists/$CODENAME/Release Components: main main/debian-installer echo $STRUCT/dists/$CODENAME/Release Description: Debian amd64 port echo $STRUCT/dists/$CODENAME/Release MD5Sum: for FILE in main/source/Release main/source/Sources.gz main/binary-amd64/Release main/binary-amd64/Packages main/binary-amd64/Packages.gz main/debian-installer/binary-amd64/Packages main/debian-installer/binary-amd64/Packages.gz contrib/source/Release contrib/source/Sources.gz contrib/binary-amd64/Release contrib/binary-amd64/Packages contrib/binary-amd64/Packages.gz non-free/source/Release non-free/source/Sources.gz non-free/binary-amd64/Release non-free/binary-amd64/Packages non-free/binary-amd64/Packages.gz; do echo `md5sum $STRUCT/dists/$CODENAME/$FILE | cut -b0-32` `wc -c $STRUCT/dists/$CODENAME/$FILE` $FILE $STRUCT/dists/$CODENAME/Release done echo Creator: $0 $STRUCT/dists/$CODENAME/Release # sign Release file gpg --default-key Debian-amd64 archive key (2004) debian-amd64@lists.debian.org --armor --detach-sign $STRUCT/dists/$CODENAME/Release mv $STRUCT/dists/$CODENAME/Release.asc $STRUCT/dists/$CODENAME/Release.gpg done -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
[EMAIL PROTECTED] (Otto Wyss) writes: Goswin von Brederlow [EMAIL PROTECTED] wrote: Sure? Anyway DpartialMirror http://dpartialmirror.sourceforge.net/; can. A note of caution: | 2004-04-03 (wyo) Since Debian does not change its policy to add | adequate support for rsync'ing package mirrors, I don't actively | develop DpartialMirror further. :-( Sad, isn't it? Any user of dpartialmirror should check out mirrorer from alioth. I only glanced at the webpage and haven't used dpartialmirror but now that mirrorer has filtering support it looks like mirrorer could replace dpartialmirror completly and I'm also thinking about retireing debmirror in favour of a wrapper to mirrorer. I guess mirrorer doesn't care for bandwith saving as DpartialMirror, correct me if I'm wrong. Currently it will always redownload the Packages/Sources files as gzip on every update to fix a bug in the apt methods. But I already suggested only updating those that don't match the Release file. And, unless you have an rsync method for apt, it won't rsync files. While rsyncing the Packages files sounds like a good idea to save traffic it actualy is a bit insignificant compared to the daily traffic of new sources and debs. The good news is that Andreas Barth is working on enabling Packages/Sources diff files for the Debian archive and that would reduce Packages/Sources updates to ~30K a day instead of the 3MB download. Once the apt method for this is written mirrorer can use it. But again, that little traffic is insignificant for someone with a complete mirror for one or more archs. For a very partial mirror, e.g. only D-I stuff or only installed deb, it might look different. O. Wyss MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Cajus Pollmeier wrote: Here's the result of some testing: debpool:is fine for new pools, but recreating the whole mirror with .deb and .udeb packages didn't work and I'm not the perl guy who's capable of fixing it. dak:too complicated for fire and forget mirrorer:not tested because alioth is still down You can find it here as a workround for now : http://dept-info.labri.fr/~danjean/deb.html#reprepro As lots of stuff in this page, it is a quick packaging for my personnal use, so do not expect it reaches Debian standard. reprepro is the old name of mirrorer. I made this package from CVS sources last week after reading this thread. Note that my web page is automatically generated from the Pakages/Sources files with WML processing. If someones are interested, just ask for it. departialmirror: not a package, after resolving some dependencies, it actually did not do what I wanted. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
* Goswin von Brederlow ([EMAIL PROTECTED]) [050131 19:35]: [EMAIL PROTECTED] (Otto Wyss) writes: I guess mirrorer doesn't care for bandwith saving as DpartialMirror, correct me if I'm wrong. Currently it will always redownload the Packages/Sources files as gzip on every update to fix a bug in the apt methods. But I already suggested only updating those that don't match the Release file. And, unless you have an rsync method for apt, it won't rsync files. While rsyncing the Packages files sounds like a good idea to save traffic it actualy is a bit insignificant compared to the daily traffic of new sources and debs. The good news is that Andreas Barth is working on enabling Packages/Sources diff files for the Debian archive and that would reduce Packages/Sources updates to ~30K a day instead of the 3MB download. Once the apt method for this is written mirrorer can use it. There is code available from Anthony on http://ftp-master.debian.org/~ajt/untiffani and http://ftp-master.debian.org/~ajt/apt-qupdate that works well (well, except if merkel was down for a few days like just happened). Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Goswin von Brederlow [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (Otto Wyss) writes: Goswin von Brederlow [EMAIL PROTECTED] wrote: Sure? Anyway DpartialMirror http://dpartialmirror.sourceforge.net/; can. I guess mirrorer doesn't care for bandwith saving as DpartialMirror, correct me if I'm wrong. Currently it will always redownload the Packages/Sources files as gzip on every update to fix a bug in the apt methods. But I already suggested only updating those that don't match the Release file. And, unless you have an rsync method for apt, it won't rsync files. Why there isn't there already a rsync method for apt is probably a mystery nobody ever will solve. While rsyncing the Packages files sounds like a good idea to save traffic it actualy is a bit insignificant compared to the daily traffic of new sources and debs. Do you mean there are up to 100 new packages each day? I get between 50 - 150 packages updated each day for just i386. Or do you mean there are 100 new versions? DpartialMirror handles new versions of packages (sources and deps) in a way it save about 1% even when the packages are normal gzip'ed. It would save around 10% - 50% with rsyncable. Is there any plan to add this feature to mirrored? O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wxguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Debian mirror scripts
Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? Thanks, Cajus -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Cajus Pollmeier a écrit : Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? Thanks, Cajus Hi Cajus, Do you use 'dpkg-scanpackages' ? It rebuild Packages file for directories that contains..pakages. cheers, Aurelien -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Cajus Pollmeier wrote: Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. You might want to compare your script with the one for partial d-i test mirrors, available from d-i SVN in trunk/scripts/testmirror. Thiemo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
On Sun, 30 Jan 2005 15:44:57 +0100, Aurelien Labrosse [EMAIL PROTECTED] wrote: Do you use 'dpkg-scanpackages' ? It rebuild Packages file for directories that contains..pakages. dpkg-scanpackages is deprecated. The low-level tool up to the task is apt-ftparchive. There is a number of higher-level tools packaged, for example debpool, but afaik the code managing the real Debian archive is not yet published. Greetings Marc -- -- !! No courtesy copies, please !! - Marc Haber |Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom | http://www.zugschlus.de/ Nordisch by Nature | Lt. Worf, TNG Rightful Heir | Fon: *49 621 72739834
Re: Debian mirror scripts
On Sun, 30 Jan 2005 14:54:16 +0100, Cajus Pollmeier [EMAIL PROTECTED] wrote: I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Have a look at the debpool package. Greetings Marc -- -- !! No courtesy copies, please !! - Marc Haber |Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom | http://www.zugschlus.de/ Nordisch by Nature | Lt. Worf, TNG Rightful Heir | Fon: *49 621 72739834
Re: Debian mirror scripts
Cajus Pollmeier [EMAIL PROTECTED] writes: Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? Thanks, Cajus Check out mirrorer on alioth when it is back up. Its great for combining upstream mirrors with local packages. Release for sid/experimental is pending. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Cajus Pollmeier a écrit : Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? Thanks, Cajus So, debpool has an options to force 'administrative' files rebuild at each run. I may be what you want. Aurelien -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Am 30.01.2005 um 15:44 schrieb Aurelien Labrosse: Cajus Pollmeier a écrit : Hi, I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? Thanks, Cajus Hi Cajus, Do you use 'dpkg-scanpackages' ? It rebuild Packages file for directories that contains..pakages. Hi Aurelien, sure, I'm using dpkg-scanpackages. But there are some problems I couldn'r resolve yet. Cheers, Cajus
Re: Debian mirror scripts
Am 30.01.2005 um 16:29 schrieb Marc Haber: On Sun, 30 Jan 2005 14:54:16 +0100, Cajus Pollmeier [EMAIL PROTECTED] wrote: I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Have a look at the debpool package. Ah. Will do. Seems to be in experimental and the description is matching ;-) Thanks, Cajus -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Hello Cajus, * Cajus Pollmeier [EMAIL PROTECTED] [2005-01-30 15:26]: I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? what about debmirror? regards nico -- Nico Golde - [EMAIL PROTECTED] | GPG: 1024D/73647CFF ,'`. [EMAIL PROTECTED] | http://www.ngolde.de ( grml.org VIM has two modes - the one in which it beeps`._,' and the one in which it doesn't -- encrypted mail preferred
Re: Debian mirror scripts
* Marc Haber [Sun, 30 Jan 2005 16:31:15 +0100]: example debpool, but afaik the code managing the real Debian archive is not yet published. uh? cvs.d.o/dak has been there for a long time (not today, though ;-). there are even debian packages in NEW and [1]. [1] http://ganneff.de/dak -- Adeodato Simó EM: asp16 [ykwim] alu.ua.es | PK: DA6AE621 The most exciting phrase to hear in science, the one that heralds new discoveries, is not Eureka! (I found it!) but That's funny... -- Isaac Asimov -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. Look at DpartialMirror http://dpartialmirror.sourceforge.net/;. I use it regularly to build a second mirror. O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wxguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Nico Golde [EMAIL PROTECTED] writes: Hello Cajus, * Cajus Pollmeier [EMAIL PROTECTED] [2005-01-30 15:26]: I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? what about debmirror? regards nico Debmirror is purely a mirror tool. It will download the Meta files just like any other file. You can easily switch between mirror of equal contents but not create Packages files reflecting what is locally available. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian mirror scripts
Hello Goswin, * Goswin von Brederlow [EMAIL PROTECTED] [2005-01-30 21:23]: Nico Golde [EMAIL PROTECTED] writes: * Cajus Pollmeier [EMAIL PROTECTED] [2005-01-30 15:26]: I'm looking for a script that regenerates Packages* and Release files for a complete mirror. Due to some installer development, I currently need to switch the mirrors during installation in order to get up to date packages. Tried to work around this with a simple script that merges my packages into the local mirror and regenerates everything as needed. But sadly this doesn't seem to be perfect :-( The installer just doesn't want to get some of these packages, even if the md5's are correct. Switching from http to ftp gets some more of them and stucks some packages later. Grrr. So let's try this way: Is there a ready to use script which I didn't find by googling around yet? what about debmirror? regards nico Debmirror is purely a mirror tool. It will download the Meta files just like any other file. You can easily switch between mirror of equal contents but not create Packages files reflecting what is locally available. ok thanks, i understand. regards nico -- Nico Golde - [EMAIL PROTECTED] | GPG: 1024D/73647CFF ,'`. [EMAIL PROTECTED] | http://www.ngolde.de ( grml.org VIM has two modes - the one in which it beeps`._,' and the one in which it doesn't -- encrypted mail preferred signature.asc Description: Digital signature
Re: Debian mirror scripts
Goswin von Brederlow [EMAIL PROTECTED] wrote: Debmirror is purely a mirror tool. It will download the Meta files just like any other file. You can easily switch between mirror of equal contents but not create Packages files reflecting what is locally available. Sure? Anyway DpartialMirror http://dpartialmirror.sourceforge.net/; can. O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wxguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
