Re: [Cooker] Mirror madness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul Dorman wanted us to know: Could we get some kind of p2p solution up and running? With so many people wanting to keep a local cooker tree, it would seem self-evident This would be could, someone could investigate on that topic? Warly, I volunteer! I'd like to co-ordinate at least. I'll to set up a CookerP2P project if that is something that people would like to see (and preferably endorsed by the Mandrake developers!). Shall I use Sourceforge, or something else? I think SF is the way to go yes. I invite people interested in a CookerP2P app to email me, so I can notify them when things are moving. Suggestions and offers of help are very welcome of course! Consider this my letter of intent to be a tester, I doubt I could do much more than that. As for suggestions, there is a bittorrent c library project on sf named libbt: http://sourceforge.net/projects/libbt/ It's still very early in the development, but it looks like it supports the basic functions. Now to devise a way of handling: 1) an entire tree of files/directories 2) when a file changes mid-stream Item #1 will have to be handled with mods to the structure of the bittorrent itself and some intelligence for filesystem traversal. Item #2 will have to be handled with the invention of some form of message passing from master to clients to indicate that the bittorrent should be resynced (but it should not be an outbound connection, rather, the master waits for clients to check in with their status and then tells them that the torrent needs to be resynced). I have visions of using rsync functions to resync the torrents, then bittorrent functions to sync individual files. Note that some form of caching or indexing really should be used as well because the amount of filesystem thrashing when a 100 clients all ask for segments of different files will be extreme and on ALL clients, not just the server. Alternatively, use a two tier system where the first tier is a single bittorrent to keep everybody's tree defining torrents in sync with the master. Then the second tier is running from that synced torrent. I also suggest that we not tie this to Mandrake or Cooker, but try to make it a general mirroring solution (at least for anybody not behind a NAT'ing firewall that they don't have the ability to forward the required ports to their machine). - -- Blue skies... Todd http://www.mrball.net Public key: http://www.mrball.net/todd.asc There are only 10 types of people in this world. Those who understand binary and those who don't. Linux kernel 2.4.21-0.25mdk 8 users, load average: 0.03, 0.07, 0.08 -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.2 (GNU/Linux) Comment: http://www.mrball.net/todd.asc iD8DBQE/R+GcIBT1264ScBURAiM4AJ0ToVMKms2tlzC/LbV8ARDhlofpYACglEwD YNW9LPESUDt3sVwbZg9dqUQ= =rk92 -END PGP SIGNATURE-
Re: [Cooker] Mirror madness
On Thu, 2003-08-21 at 10:29, Warly wrote: Paul Dorman [EMAIL PROTECTED] writes: Hi all, I understand that mirror issues have been discussed many times before, but I would really like to get to the bottom of the current situation (and have an up-to-date cooker mirror as well!!). Yes we switch the mirror reference machine, and get some issue in the move. Currently being fixed. Have any changes been made in the structure? (I just noticed that my rsync is suddenly pulling in src RPMs). Wouter -- Wouter Lagerweij [EMAIL PROTECTED]
RE: [Cooker] Mirror madness
I think that is a splendid idea. If you think about it, it would be a distributed backup type system... So maybe can be used as such. But for cooker that would be great. -Original Message- From: Paul Dorman [mailto:[EMAIL PROTECTED] Sent: Thursday, August 21, 2003 1:04 PM To: [EMAIL PROTECTED] Subject: Re: [Cooker] Mirror madness On Thu, 21 Aug 2003 20:29, Warly wrote: Paul Dorman [EMAIL PROTECTED] writes: Hi all, I understand that mirror issues have been discussed many times before, but I would really like to get to the bottom of the current situation (and have an up-to-date cooker mirror as well!!). Yes we switch the mirror reference machine, and get some issue in the move. Currently being fixed. [...] Could we get some kind of p2p solution up and running? With so many people wanting to keep a local cooker tree, it would seem self-evident that we should be taking advantage of this technology. A cooker torrent would be excellent! (does BitTorrent handle large numbers of files easily?). This would be could, someone could investigate on that topic? Warly, I volunteer! I'd like to co-ordinate at least. I'll to set up a CookerP2P project if that is something that people would like to see (and preferably endorsed by the Mandrake developers!). Shall I use Sourceforge, or something else? I invite people interested in a CookerP2P app to email me, so I can notify them when things are moving. Suggestions and offers of help are very welcome of course! Regards, Paul Dorman.
Re: [Cooker] Mirror madness
I checked the two main mirrors and all the files from 2003-08-13 20:45 on have been deleted.
Re: [Cooker] Mirror madness
Paul Dorman [EMAIL PROTECTED] writes: Hi all, I understand that mirror issues have been discussed many times before, but I would really like to get to the bottom of the current situation (and have an up-to-date cooker mirror as well!!). Yes we switch the mirror reference machine, and get some issue in the move. Currently being fixed. [...] Could we get some kind of p2p solution up and running? With so many people wanting to keep a local cooker tree, it would seem self-evident that we should be taking advantage of this technology. A cooker torrent would be excellent! (does BitTorrent handle large numbers of files easily?). This would be could, someone could investigate on that topic? -- Warly
Re: [Cooker] Mirror madness
Am Donnerstag, 21. August 2003, 10:29:55 Uhr MET, schrieb Warly: Could we get some kind of p2p solution up and running? With so many people wanting to keep a local cooker tree, it would seem self-evident that we should be taking advantage of this technology. A cooker torrent would be excellent! (does BitTorrent handle large numbers of files easily?). This would be could, someone could investigate on that topic? The problem isn't the number of files but the frequent file changes. If one file changes you'll have to regenerate the torrent which involves the calculation of the md5 sums of the complete data. This takes a lot of processing time. Maybe bittorrent could be improved by making it cache some md5 sums for the unchanged chunks of data. -- What difference does it make to the dead, the orphans and the homeless, whether the mad destruction is wrought under the name of totalitarianism or the holy name of liberty or democracy? Mahatma Gandhi (1869 - 1948), Non-Violence in Peace and War
Re: [Cooker] Mirror madness
On Thu, 21 Aug 2003 16:28, Götz Waschk wrote: The problem isn't the number of files but the frequent file changes. If one file changes you'll have to regenerate the torrent which involves the calculation of the md5 sums of the complete data. This takes a lot of processing time. Maybe bittorrent could be improved by making it cache some md5 sums for the unchanged chunks of data. Götz, if you could blend rsync and BitTorrent into one protocol, you'd be an international hero. (-: Cheers; Leon
Re: [Cooker] Mirror madness
On Thu, Aug 21, 2003 at 10:29:55AM +0200, Warly wrote: Paul Dorman [EMAIL PROTECTED] writes: Hi all, I understand that mirror issues have been discussed many times before, but I would really like to get to the bottom of the current situation (and have an up-to-date cooker mirror as well!!). Yes we switch the mirror reference machine, and get some issue in the move. Currently being fixed. What is the recommended maaster source of the mirrors for cooker? I use ibiblio for my mirror at klid.dk - is that the best currently? best regards keld
Re: [Cooker] Mirror madness
On Thursday 21 August 2003 04:29, Warly wrote: Could we get some kind of p2p solution up and running? With so many people wanting to keep a local cooker tree, it would seem self-evident that we should be taking advantage of this technology. A cooker torrent would be excellent! (does BitTorrent handle large numbers of files easily?). This would kick ass. :) Especially if it was setup so that it can download from multiple sources at a time to increase speed. -- Jason Straight [EMAIL PROTECTED] icq: 1796276
Re: [Cooker] Mirror madness
On Thu Aug 21 10:28 +0200, Götz Waschk wrote: The problem isn't the number of files but the frequent file changes. If one file changes you'll have to regenerate the torrent which involves the calculation of the md5 sums of the complete data. This takes a lot of processing time. Maybe bittorrent could be improved by making it cache some md5 sums for the unchanged chunks of data. P2P rsync? -- Levi Ramsey [EMAIL PROTECTED] [EMAIL PROTECTED] Take due notice and govern yourselves accordingly. Currently playing: Metallica - St. Anger - The Unnamed Feeling Linux 2.4.21-3mdk 12:22:00 up 16 days, 21:40, 9 users, load average: 0.06, 0.53, 0.61
Re: [Cooker] Mirror madness
On Thu, 21 Aug 2003 20:29, Warly wrote: Paul Dorman [EMAIL PROTECTED] writes: Hi all, I understand that mirror issues have been discussed many times before, but I would really like to get to the bottom of the current situation (and have an up-to-date cooker mirror as well!!). Yes we switch the mirror reference machine, and get some issue in the move. Currently being fixed. [...] Could we get some kind of p2p solution up and running? With so many people wanting to keep a local cooker tree, it would seem self-evident that we should be taking advantage of this technology. A cooker torrent would be excellent! (does BitTorrent handle large numbers of files easily?). This would be could, someone could investigate on that topic? Warly, I volunteer! I'd like to co-ordinate at least. I'll to set up a CookerP2P project if that is something that people would like to see (and preferably endorsed by the Mandrake developers!). Shall I use Sourceforge, or something else? I invite people interested in a CookerP2P app to email me, so I can notify them when things are moving. Suggestions and offers of help are very welcome of course! Regards, Paul Dorman.
Re: [Cooker] Mirror madness
On 08/21/03 16:04:01, Paul Dorman wrote: I invite people interested in a CookerP2P app to email me, so I can notify them when things are moving. Suggestions and offers of help are very welcome of course! I have a mirror I'd be willing to share if usage was reasonable. My PLF mirror went out of control pretty fast (2GB/day). Austin -- Austin Acton Hon.B.Sc. Synthetic Organic Chemist, Teaching Assistant Department of Chemistry, York University, Toronto MandrakeClub Volunteer (www.mandrakeclub.com) homepage: www.groundstate.ca
Re: [Cooker] Mirror madness
Warly wrote: Yes we switch the mirror reference machine, and get some issue in the move. Currently being fixed. Any news about this. Files are still missing from the main mirrors.
Re: [Cooker] mirror madness
On 2003-08-21(Thu) 19:55:51 -0400, Austin wrote: Okay, so I try to update my mirror to see if the missing files have showed up yet, but no, twenty more RPMS get deleted: Seems whenever there's updated packages, what happened is that old packages get deleted, while new packages are not uploaded... this is the case in every RPM repository I see, including klama... Abel To be done: Delete cooker/i586/Mandrake/RPMS/evolution-1.4.4-2mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/evolution-devel-1.4.4-2mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/evolution-pilot-1.4.4-2mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/gnome-panel-2.3.6.2-1mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/hostap-0.0.1-1mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/libgnomeprint-2.2.1.3-1mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/libgnomeprint2-2_0-2.2.1.3-1mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/libgnomeui2-2.3.6-2mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/libgnomeui2_0-2.3.6-2mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/libgnomeui2_0-devel-2.3.6-2mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/libpanel-applet-2_0-2.3.6.2-1mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/perl-Digest-MD2-2.01-3mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/perl-Digest-SHA1-2.02-3mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/perl-Filter-1.29-3mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/wireless-tools-25-5mdk.i586.rpm Delete cooker/i586/Mandrake/RPMS/autoconf-2.13-18mdk.noarch.rpm Delete cooker/i586/Mandrake/RPMS/perl-MailTools-1.58-3mdk.noarch.rpm Delete cooker/i586/Mandrake/RPMS/perl-URI-1.23-4mdk.noarch.rpm Delete cooker/i586/Mandrake/RPMS/perl-ldap-0.28-2mdk.noarch.rpm Isn't that great? Austin -- Austin Acton Hon.B.Sc. Synthetic Organic Chemist, Teaching Assistant Department of Chemistry, York University, Toronto MandrakeClub Volunteer (www.mandrakeclub.com) homepage: www.groundstate.ca -- Abel Cheung Linux counter #256983 | http://counter.li.org GPG Key: (0xC67186FF) | http://deaddog.org/gpg.asc Key fingerprint: 671C C7AE EFB5 110C D6D1 41EE 4152 E1F1 C671 86FF pgp0.pgp Description: PGP signature
