Re: Apt-Torrent project
Mike Furr a écrit : This would probably help as long as you didn't abuse super-seeding. One solution may be to only super seed those packages which are smaller than some threshold and are also in base or have a priority > standard(or something). Like most things, the distribution of popular packages appears to have a zipf distribution(at least, according to popcon), so you could also gain efficiency by exploiting this data. That's interesting. At the moment I'm "super-seeding" everything since my tracker/seeder is running on my own home ADSL connection which is too slow to run a server even for testing purposes. [...] Well, I wouldn't edit the Packages.gz file directly since it will no longer match the hash in the Release file, I would have this in a separate file, if at all. Mmmm yes, or use two different lines in the sources.list file. One to apt-torrent and one regular. [..] I look forward to trying apt-torrent and hope that it works out well. Since it appears that you are not a debian developer, are you looking for someone to package/sponsor this? That would be very nice. I've already packaged my program (only for unstable): http://sianka.free.fr/download.html Well, I'm a sponsored maintainer (packages gnunet and libextractor). My sponsor is Glenn McGrath ([EMAIL PROTECTED]). I'm also in the applying Q to be DD. I don't know if we can have multiple sponsors :D ? Arnaud
Re: Apt-Torrent project
Arnaud Kyheng wrote: I don't agree with the little package problem with Bittorrent. With Bittornado I'm using as a backend, the super-seeder option answer to this problem since if the package is already well available on the network, it'll not answer to the client but let it download from peers. [...] And also, if you think that the tracker overhead not worth a 5k package, you could as well split the downloading system in two. I mean put only >= 5k packages files on the apt-torrent server and let the others be fetched directly in http. This would probably help as long as you didn't abuse super-seeding. One solution may be to only super seed those packages which are smaller than some threshold and are also in base or have a priority > standard(or something). Like most things, the distribution of popular packages appears to have a zipf distribution(at least, according to popcon), so you could also gain efficiency by exploiting this data. This can be done easily since apt-torrent is fetching the Packages.gz as usual. I mean I could add a special header in the Packages.gz description to tell the proxy where to download the package direct-http, or apt-torrent-server for example. Well, I wouldn't edit the Packages.gz file directly since it will no longer match the hash in the Release file, I would have this in a separate file, if at all. My original idea was to save bandwidth of the Debian server, and improve the downloading speed of the packages for users that are even far of a mirror. I found that the Bittorrent was really mature and will fit well. In the future, I could as well use GNUnet as a backend :) Although I personally get fantastic download speeds from the push primary mirrors, I guess this is not the case everywhere. I agree that moving some load off of the mirror network would be beneficial. I look forward to trying apt-torrent and hope that it works out well. Since it appears that you are not a debian developer, are you looking for someone to package/sponsor this? -Mike
Re: Apt-Torrent project
Mike Furr a écrit : Arnaud Kyheng wrote: | Hello, | | I love the Debian project, and I have worked on a new development for | it: Apt-Torrent :) Thank you for your contribution. However, I looked at doing something similar to this a little while ago and found that bittorrent is not very well suited for doing package downloads. Of the ~15k binary packages in Debian about 87% are under 1 meg in size and 98% are under 10 megs. The bit-torrent protocol works best on files significantly larger than this. Also, the protocol is not as efficient as it could be for the server hosting the .torrent file, which means it scales quite poorly when there are lots of requests for small files, as would be the case for Debian packages. I don't agree with the little package problem with Bittorrent. With Bittornado I'm using as a backend, the super-seeder option answer to this problem since if the package is already well available on the network, it'll not answer to the client but let it download from peers. Furthermore the .torrent files are very small ~2k and could be easily cached by ISPs. And also, if you think that the tracker overhead not worth a 5k package, you could as well split the downloading system in two. I mean put only >= 5k packages files on the apt-torrent server and let the others be fetched directly in http. This can be done easily since apt-torrent is fetching the Packages.gz as usual. I mean I could add a special header in the Packages.gz description to tell the proxy where to download the package direct-http, or apt-torrent-server for example. However, I do feel that having a p2p backend to apt is a very interesting and feasible distribution method. There is a lot of structure in the way Debian lays out its archive, from the Package files to the .deb's themselves, which can be exploited to make this very efficient. There has also been a myriad of research papers exploring overlay networks and application level multicast which could be adopted to form a package network that would provide advantages over our current mirror infrastructure. (I've actually done a little work on exploring this, but haven't gotten very far due to time constraints.) My original idea was to save bandwidth of the Debian server, and improve the downloading speed of the packages for users that are even far of a mirror. I found that the Bittorrent was really mature and will fit well. In the future, I could as well use GNUnet as a backend :) With apt-torrent you may also reduce the number of proximity servers. We don't need extreme latency response for 2k .torrent files, and except if the mirror is an ISP, it'll not overload the ISPs peers, by Bittorent scheme. Furthermore it'll load more the ISPs cache with the .torrent files. With lesser proximity servers/seeders, it might also save rsync bandwidth needed to sync Debian repository servers. Arnaud
Re: Apt-Torrent project
Matt Zimmerman a écrit : (CCing the BTS, where this feature request is already tracked) On Sat, Oct 30, 2004 at 10:35:53AM +0200, Arnaud Kyheng wrote: I love the Debian project, and I have worked on a new development for it: Apt-Torrent :) Apt-Torrent is an apt proxy to the Bittorrent network. For security, the package listing, and the .torrent files are downloaded from a regular http server, as usual for a package, but then the whole package is fetched via bittorrent protocol and forwarded to apt :) Interesting; I hadn't considered implementing this as a proxy. It would probably be better to subclass an existing HTTP server implementation, rather than to implement a subset of HTTP in your package. At the moment apt-torrent is implemented like a real http server. This line is added in /etc/apt/sources.list by the package: deb http://127.0.0.1:6968/debian/ unstable main From the apt point of view, it's a real http package repository, which could also be a package source repository (deb-src) as well. Arnaud
Re: Apt-Torrent project
Matthew Palmer a écrit : [...] If we can get individually-signed .debs, you won't even need to worry so much about getting the torrent files off a trusted mirror... That's the idea, since the .torrent are fetched from a trusted server. The .torrents contains a SHA1 checksum which is checked against downloaded file. At any rate, it looks good, and I, for one, hope to see a lot more about it in the future. Cool :). Atm I'm trying to set up a better seeder with some adsl friends connections, to have a decent downloading rate. Arnaud
Re: Apt-Torrent project
(CCing the BTS, where this feature request is already tracked) On Sat, Oct 30, 2004 at 10:35:53AM +0200, Arnaud Kyheng wrote: > I love the Debian project, and I have worked on a new development for > it: Apt-Torrent :) > > Apt-Torrent is an apt proxy to the Bittorrent network. For security, the > package listing, and the .torrent files are downloaded from a regular > http server, as usual for a package, but then the whole package is > fetched via bittorrent protocol and forwarded to apt :) Interesting; I hadn't considered implementing this as a proxy. It would probably be better to subclass an existing HTTP server implementation, rather than to implement a subset of HTTP in your package. -- - mdz
Re: Apt-Torrent project
On Sun, Oct 31, 2004 at 12:01:30AM -0500, Adam Majer wrote: > BT doesn't make too much sense here. It is only useful for large files > that do not change often, like woody iso images. I think there are ways that the bittorrent model can be applied successfully. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=199316 has some of my thoughts on the subject. -- - mdz
Re: Apt-Torrent project
On Sat, Oct 30, 2004 at 07:05:12PM +1000, Matthew Palmer wrote: > If we can get individually-signed .debs, you won't even need to worry so > much about getting the torrent files off a trusted mirror... You don't need individually-signed .debs for that; the existing (experimental) archive signing mechanism works fine in this scenario. -- - mdz
Re: Apt-Torrent project
Henrique de Moraes Holschuh wrote: >On Sun, 31 Oct 2004, Steinar H. Gunderson wrote: > > > >The Azureus Java client does this, so yes, it is possible. How bad this >interacts with the scatter-gatter logic of BT, I don't know. But the >.torrent files would be huge, and they would need to be updated daily, >which would break all torrents. So, it is useless for unstable. > > Most ISPs probably have caches for at least HTTP traffic. I would expect that these would cache most of the debs, that is, if enough people are downloading these packages. BT doesn't make too much sense here. It is only useful for large files that do not change often, like woody iso images. - Adam -- Building your applications one byte at a time http://www.galacticasoftware.com
Re: Apt-Torrent project
On Sun, 31 Oct 2004, Steinar H. Gunderson wrote: > On Sat, Oct 30, 2004 at 12:42:32PM -0400, Mike Furr wrote: > > However, I do feel that having a p2p backend to apt is a very > > interesting and feasible distribution method. There is a lot of > > structure in the way Debian lays out its archive, from the Package files > > to the .deb's themselves, which can be exploited to make this very > > efficient. > > An interesting experiment would be simply cat-ing all the .debs together and > extend BT in some way to just fetch the parts (ie. the .debs) that it needs > (and likewise, only mirror hotspots). I don't know enough about the BT > protocol to know if this is feasible or not, but it does at least sound > interesting :-) The Azureus Java client does this, so yes, it is possible. How bad this interacts with the scatter-gatter logic of BT, I don't know. But the .torrent files would be huge, and they would need to be updated daily, which would break all torrents. So, it is useless for unstable. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Re: Apt-Torrent project
On Sat, Oct 30, 2004 at 12:42:32PM -0400, Mike Furr wrote: > However, I do feel that having a p2p backend to apt is a very > interesting and feasible distribution method. There is a lot of > structure in the way Debian lays out its archive, from the Package files > to the .deb's themselves, which can be exploited to make this very > efficient. An interesting experiment would be simply cat-ing all the .debs together and extend BT in some way to just fetch the parts (ie. the .debs) that it needs (and likewise, only mirror hotspots). I don't know enough about the BT protocol to know if this is feasible or not, but it does at least sound interesting :-) /* Steinar */ -- Homepage: http://www.sesse.net/
Re: Apt-Torrent project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Arnaud Kyheng wrote: | Hello, | | I love the Debian project, and I have worked on a new development for | it: Apt-Torrent :) Thank you for your contribution. However, I looked at doing something similar to this a little while ago and found that bittorrent is not very well suited for doing package downloads. Of the ~15k binary packages in Debian about 87% are under 1 meg in size and 98% are under 10 megs. The bit-torrent protocol works best on files significantly larger than this. Also, the protocol is not as efficient as it could be for the server hosting the .torrent file, which means it scales quite poorly when there are lots of requests for small files, as would be the case for Debian packages. However, I do feel that having a p2p backend to apt is a very interesting and feasible distribution method. There is a lot of structure in the way Debian lays out its archive, from the Package files to the .deb's themselves, which can be exploited to make this very efficient. There has also been a myriad of research papers exploring overlay networks and application level multicast which could be adopted to form a package network that would provide advantages over our current mirror infrastructure. (I've actually done a little work on exploring this, but haven't gotten very far due to time constraints.) Cheers, - -Mike -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) iD8DBQFBg8R17ZPKKRJLJvMRAkC3AKCwqrhxnIyBJZQ5vKcZtbk+cN9SbwCgiVah 5jsTvglijsm3lyWJTvBULvk= =TLcJ -END PGP SIGNATURE-
Re: Apt-Torrent project
Matthew Palmer <[EMAIL PROTECTED]> writes:
> On Sat, Oct 30, 2004 at 12:00:16PM +0200, Marc 'HE' Brockschmidt wrote:
>> Matthew Palmer <[EMAIL PROTECTED]> writes:
>> [...]
>> > If we can get individually-signed .debs, you won't even need to worry so
>> > much about getting the torrent files off a trusted mirror...
>> dpkg-sig exists. Use it :)
> Thanks for that, and I know all about it and sign all of my
> internally-generated .debs for work. However, I don't bother doing it for
> my Debian-uploaded ones because (a) anything built by an autobuilder won't
> have any sigs in it,
This will (hopefully) change when enough people use dpkg-sig.
> (b) most other developers aren't signing
Well, everybody can use this as a reason, but it's no problem to call
dpkg-sig instead of debsign. [1]
Marc
Footnotes:
[1] And i would be very happy if the dpkg-buildpackage and debuild
maintainers would add dpkg-sig support, at least as option.
--
$_=')(hBCdzVnS})3..0}_$;//::niam/s~=)]3[))_$(rellac(=_$({pam(esrever })e$.)4/3*
)e$(htgnel+23(rhc,"u"(kcapnu ,""nioj ;|_- |/+9-0z-aZ-A|rt~=e$;_$=e${pam tnirp{y
V2ajFGabus} yV2ajFGa&{gwmclBHIbus}gwmclBHI&{yVGa09mbbus}yVGa09mb&{hBCdzVnSbus';
s/\n//g;s/bus/\nbus/g;eval scalar reverse #
pgpSXg4Iopaqh.pgp
Description: PGP signature
Re: Apt-Torrent project
On Sat, Oct 30, 2004 at 12:00:16PM +0200, Marc 'HE' Brockschmidt wrote: > Matthew Palmer <[EMAIL PROTECTED]> writes: > [...] > > If we can get individually-signed .debs, you won't even need to worry so > > much about getting the torrent files off a trusted mirror... > > dpkg-sig exists. Use it :) Thanks for that, and I know all about it and sign all of my internally-generated .debs for work. However, I don't bother doing it for my Debian-uploaded ones because (a) anything built by an autobuilder won't have any sigs in it, (b) most other developers aren't signing, and (c) at this point in time, signed Release files and tracking down from there is the best way of verifying packages, and it's the only way that has much of a chance (from picking a package at random) of working. - Matt signature.asc Description: Digital signature
Re: Apt-Torrent project
Matthew Palmer <[EMAIL PROTECTED]> writes:
[...]
> If we can get individually-signed .debs, you won't even need to worry so
> much about getting the torrent files off a trusted mirror...
dpkg-sig exists. Use it :)
Marc
--
$_=')(hBCdzVnS})3..0}_$;//::niam/s~=)]3[))_$(rellac(=_$({pam(esrever })e$.)4/3*
)e$(htgnel+23(rhc,"u"(kcapnu ,""nioj ;|_- |/+9-0z-aZ-A|rt~=e$;_$=e${pam tnirp{y
V2ajFGabus} yV2ajFGa&{gwmclBHIbus}gwmclBHI&{yVGa09mbbus}yVGa09mb&{hBCdzVnSbus';
s/\n//g;s/bus/\nbus/g;eval scalar reverse #
pgpmdZ4AV0GNr.pgp
Description: PGP signature
Re: Apt-Torrent project
On Sat, Oct 30, 2004 at 10:35:53AM +0200, Arnaud Kyheng wrote: > I love the Debian project, and I have worked on a new development for > it: Apt-Torrent :) > > Apt-Torrent is an apt proxy to the Bittorrent network. For security, the > package listing, and the .torrent files are downloaded from a regular > http server, as usual for a package, but then the whole package is > fetched via bittorrent protocol and forwarded to apt :) [...] > I would like to know what do you think of it :) Well, I don't do P2P myself so much, but from what I know of it, this sounds like a pretty cool thing, if it gets off the ground and gets a lot of users. If we can get individually-signed .debs, you won't even need to worry so much about getting the torrent files off a trusted mirror... At any rate, it looks good, and I, for one, hope to see a lot more about it in the future. - Matt signature.asc Description: Digital signature
