Re: Package distribution, a concept for a modern package distribution
Anthony DeRobertis [EMAIL PROTECTED] writes: Goswin von Brederlow wrote: For some time now (as in around a year) you can update your Packages files by downloading only the differences (ed script format diff) to your local file. Using that daily updates go down from the full 3+MB (2+Mb with bz2 now) to ~10K per day. Really? I don't see it documented in, e.g. the sources.list manpage. Google finds only an open bug requesting the feature. Nor do I find any diffs at, e.g., http://http.us.debian.org/debian/dists/testing/main/binary-i386/ The method isn't integrated into the DAK nor into apt. You can download the scripts to update Packages per diff from http://ftp-master.debian.org/~ajt/apt-qupdate http://ftp-master.debian.org/~ajt/untiffani The ed scripts itself are provided by Andreas Barth currently on (defined in apt-qupdate) http://merkel.debian.org/~aba/debian MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Package distribution, a concept for a modern package distribution
Goswin von Brederlow wrote: For some time now (as in around a year) you can update your Packages files by downloading only the differences (ed script format diff) to your local file. Using that daily updates go down from the full 3+MB (2+Mb with bz2 now) to ~10K per day. Really? I don't see it documented in, e.g. the sources.list manpage. Google finds only an open bug requesting the feature. Nor do I find any diffs at, e.g., http://http.us.debian.org/debian/dists/testing/main/binary-i386/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Package distribution, a concept for a modern package distribution
Otto Wyss [EMAIL PROTECTED] writes: Since around last October, I've considered to make my concept for a modern package distribution public but I wanted to wait until Debian/sarge was released which is now the case. And since the Debconf5 in Helsinki is just around the corner it's about the right time. The concept is based on an LDAP server (or simiar) as a replacement for the Packages file and on a P2P network for package distribution (see http://wyodesktop.sourceforge.net/index.php?page=pkgdist.html). IMO it would make a lot sense if this concept is discussed at the Debconf5. For some time now (as in around a year) you can update your Packages files by downloading only the differences (ed script format diff) to your local file. Using that daily updates go down from the full 3+MB (2+Mb with bz2 now) to ~10K per day. It is long overdue that this is added to the DAK but with it I would hardly call the Packages/Sources file a bottleneck. That said and from my experience with P2P software I doubt you can design anything as efficient as the current mirror network. P2P is good for dynamic topography and distribution, adapting to changing bandwith, sources and downloaders on the fly. But that will never beat a static setup such as the mirror network in efficiency. [Efficient for the internet, not neccesarily every user. E.g. it might be faster (you wish :) for 1000 australians to leech files from a million US peers but an australian mirror will use 1/1000th of the sparse Australian-US bandwith.] I'm not actively work on this concept and its implementation since I've _no_ time, sorry. If someone else is interested just say so. O. Wyss MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Package distribution, a concept for a modern package distribution
On Fri, Jun 24, 2005 at 06:14:46PM +0200, Otto Wyss wrote: The concept is based on an LDAP server (or simiar) as a replacement for the Packages file and on a P2P network for package distribution (see http://wyodesktop.sourceforge.net/index.php?page=pkgdist.html). IMO it would make a lot sense if this concept is discussed at the Debconf5. i actually wrote something that exported a local mirror/server's Packages.gz file into an LDAP directory[1], as well as wrote the beginnings of an add-on method to apt to query this server. as far as speed/transfer efficiency goes, we're an order or two of magnitude at the very least. however, there hasn't seemed to be much interest from others in my continuing it, so i've been focusing on other things lately. I've seen some messages about an LDAP implementation around last october but I couldn't find them. I'm quite sure an LDAP solution is much better than the current solution. But before implementing it, it has to be evaluated against other way so it's truely the best. also, there's a limitation in apt that it expects the list of packages to be retrieved through the same method as the packages themselves, which would get a little hairy with LDAP (you don't want to be holding the packages themselves, in LDAP of course). there could be a quick-hack workaround for this by having ldap-ftp/ldap-http methods that wrap around the ftp/http for the actual fetching, but a real fix would be to patch apt to allow for this. such a patch would also make it easier to distribute the packages list via other methodst too. I wouldn't base any work on apt but start a complete new way of package distribution. The advantage is the stabe apt will keep on working while the new solution won't be hampered by any current limitation. IMO a P2P network would be a much better solution but yet again it has to be checked if it this is really true or if there are better alternatives. anyway if there are more people interested in working on this, i'd be willing to put my code in cvs/svn and start up an alioth project. The best way to start a new package distribution is to name a place where it can be discussed off Debian-devel. Since this concept probably has many more implications, like what about Debian installer etc, I think it would make a lot of sense to first collect any pros and cons and discuss them at Debconf5 and possibly on a list. Since I can't attend Debconf5 myself I'd appreciate if someone could keep track of the discussion there and forward it to the list. So the first action should be to set up or name this package distribution list and start collecting arguments there. Besides, until a better place is found I'll keep http://wyodesktop.sourceforge.net/index.php?page=pkgdist.html; updated, so you may send any input to the wyodesktop-users list or directly to me (list would be better because less spam prone). O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wyoguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Package distribution, a concept for a modern package distribution
On Sun, Jun 26, 2005 at 09:16:42AM +0200, Otto Wyss wrote: I've seen some messages about an LDAP implementation around last october but I couldn't find them. I'm quite sure an LDAP solution is much better than the current solution. But before implementing it, it has to be evaluated against other way so it's truely the best. i don't believe there is a best way, actually. in many circumstances, one alternative is better than another, but in others vice versa. I wouldn't base any work on apt but start a complete new way of package distribution. The advantage is the stabe apt will keep on working while the new solution won't be hampered by any current limitation. IMO a P2P network would be a much better solution but yet again it has to be checked if it this is really true or if there are better alternatives. i don't see why we'd have to reinvent the wheel, and wouldn't be interested in doing so, frankly. apt provides a fairly easy way to extend itself to use new protocols (ldap, bt, whatever). the distributing/fetching of packages and package info is only one small aspect of package management, and i'd to keep it that way and let apt/aptitude/whatever handle all the really hard stuff. sean -- signature.asc Description: Digital signature
Re: Package distribution, a concept for a modern package distribution
On Fri, Jun 24, 2005 at 06:14:46PM +0200, Otto Wyss wrote: The concept is based on an LDAP server (or simiar) as a replacement for the Packages file and on a P2P network for package distribution (see http://wyodesktop.sourceforge.net/index.php?page=pkgdist.html). IMO it would make a lot sense if this concept is discussed at the Debconf5. i actually wrote something that exported a local mirror/server's Packages.gz file into an LDAP directory[1], as well as wrote the beginnings of an add-on method to apt to query this server. as far as speed/transfer efficiency goes, we're an order or two of magnitude at the very least. however, there hasn't seemed to be much interest from others in my continuing it, so i've been focusing on other things lately. also, there's a limitation in apt that it expects the list of packages to be retrieved through the same method as the packages themselves, which would get a little hairy with LDAP (you don't want to be holding the packages themselves, in LDAP of course). there could be a quick-hack workaround for this by having ldap-ftp/ldap-http methods that wrap around the ftp/http for the actual fetching, but a real fix would be to patch apt to allow for this. such a patch would also make it easier to distribute the packages list via other methodst too. anyway if there are more people interested in working on this, i'd be willing to put my code in cvs/svn and start up an alioth project. sean [1] see Package Lists via LDAP http://www.debian.org/News/weekly/2004/48/ -- signature.asc Description: Digital signature
Package distribution, a concept for a modern package distribution
Since around last October, I've considered to make my concept for a modern package distribution public but I wanted to wait until Debian/sarge was released which is now the case. And since the Debconf5 in Helsinki is just around the corner it's about the right time. The concept is based on an LDAP server (or simiar) as a replacement for the Packages file and on a P2P network for package distribution (see http://wyodesktop.sourceforge.net/index.php?page=pkgdist.html). IMO it would make a lot sense if this concept is discussed at the Debconf5. I'm not actively work on this concept and its implementation since I've _no_ time, sorry. If someone else is interested just say so. O. Wyss -- Development of frame buffer drivers: http://linux-fbdev.sf.net Sample code snippets for wxWidgets: http://wxcode.sf.net How to build well-designed applications: http://wyoguide.sf.net Desktop with a consistent look and feel: http://wyodesktop.sf.net -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
