[Mageia-dev] Mirror list for apps (LONG mail)
Hi, I wrote an application to register mirrors which is currently here: http://distrib-coffee.ipsl.jussieu.fr/mageia-mirrors/ I hadn't the time to really check mirror brain, I just had a quick look, so don't ask. In all case the goal of my application is to allow everyone to register a new mirror, and provide a list usuable by urpmi. It is time to write the part of code responsible of the list build. So come some questions. First let me explain the current mandriva process: 1) At time we have an /etc/product.id given by mandriva-release-Free-2011.0-0.1mdv2011.0 On my system it contains: vendor=Mandriva,distribution=Mandriva Linux,type=Basic,version=2011.0,branch=Devel,release=1,arch=x86_64,product=Free 2) The base URL list is hardcoded in urpmi: https://api.mandriva.com/mirrors/$product_type.$product_version.$arch.list; On my system the correct url is: https://api.mandriva.com/mirrors/basic.cooker.x86_64.list 3) The list look like: continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=distrib,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=updates,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 So, question and my own POV about this: 1) /etc/product.id * I don't like this file, I think it is redundant with /etc/mandriva-release * Why basic and not Free since it is the product flag ? Dunno * Notice cooker does not appear in this file 2) The url * I suggest to use http://mirror.mageia.org/api/, other proposal welcome (need some review with sys admin) * Obviously the valid list version is 'cooker' not 2011 * it is harcoded in urpmi 3) The content * tags seems be optional * mirror point to 2011 not cooker (a symlink at time) * I can produce same output as I have same information * Do we need specific line for update since now distribution include everything * Do we keep this format ? Keeping this format is more simple, just changing the line specifiying basis url in URPMI. By changing the the format we can add features, make it clearer, etc... But changing format to what ? XML ? YAML ? CSV ? something worst ? :) Changes can done later. At time, even made some criticisms, I do plan to redo exactly the same things, first for compability issue (update from mdv to mga) and because it will need only few changes in urpmi, making the thing easier. Comments, Suggests ? Regards. -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgp68rs6zBbyx.pgp Description: PGP signature
Re: [Mageia-dev] Mirror list for apps (LONG mail)
* Fabrice Facorat (fabrice.faco...@gmail.com) wrote: 2010/11/3 Olivier Thauvin nanar...@nanardon.zarb.org: Keeping this format is more simple, just changing the line specifiying basis url in URPMI. And allow to keep compatibility with urpmi without maintaining too much patch By changing the the format we can add features, make it clearer, etc... But changing format to what ? XML ? YAML ? CSV ? something worst ? :) Changes can done later. if really you want to change this you should : - ditch the url in favor of a XML-RPC or REST calls We can provide both, just matter of some code. So product.id could be kept unchanged, but the way urpmi talk to the mirror provider could be improved. Last but not least, by using XML-RC or REST calls, we could filter the output of mirror list ( for example retrieve the list of mirror for cooker in EU zone ) Notice it can done by passing argument to URL: http:///...list?continent=EU I do agree XML-RPC/REST make it clearer and probably more extensible. -- Close the World, Open the Net http://www.linux-wizard.net -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpmVUUXl7Oua.pgp Description: PGP signature
Re: [Mageia-dev] Mirror list for apps (LONG mail)
2010/11/3 Michael Scherer m...@mandriva.org: Le mercredi 03 novembre 2010 à 20:05 +0100, Fabrice Facorat a écrit : 2010/11/3 Olivier Thauvin nanar...@nanardon.zarb.org: Keeping this format is more simple, just changing the line specifiying basis url in URPMI. And allow to keep compatibility with urpmi without maintaining too much patch By changing the the format we can add features, make it clearer, etc... But changing format to what ? XML ? YAML ? CSV ? something worst ? :) Changes can done later. if really you want to change this you should : - ditch the url in favor of a XML-RPC or REST calls Given the fact we have slightly less ressources than Mandriva and that people on Mandriva side ( or at least, that's what I remind that fcrozat told me ) didn't have the ressources to have a more complex system, we should avoid this. - by doing this you will be able to user whatever version or name for the distribution. Indeed by using an url, you have to avoid space, or too fancy names. On top of that you can change too much the format You can use space in a url. sure, but it's ugly and you need to escape the space ( url encoding ) And if we use a XML-RPC api, we will not be able to change it either, so that trading the impossibility of changing the format for the impossibility to change the API. a good API won't be an issue :p URL parsing is harder and more error-prone This could be done on the client side. On the client side, this would mean a little bit more bandwidth used, a little more cpu, and on our side, it would be less CPU. And I think cpu is a more rare ressources than bandwidth. you can use cache on the server side, i.e provide a static file which is a cache of the query. -- Close the World, Open the Net http://www.linux-wizard.net
[Mageia-dev] New mirror -- maybe
If I was to approach our local university about mirroring Mageia, would I have to supply them some specs? What would they need to know and who would they have to get into contact with? They are mirroring the Mandriva KDE updates. University of Waterloo. The university just announced last week that their fundraising for the year was worth 600 million $$$Cdn. Maybe they could afford a little server space. Marc
Re: [Mageia-dev] Mirror list for apps (LONG mail)
* Michael Scherer (m...@zarb.org) wrote: Le mercredi 03 novembre 2010 à 19:47 +0100, Olivier Thauvin a écrit : Hi, I wrote an application to register mirrors which is currently here: http://distrib-coffee.ipsl.jussieu.fr/mageia-mirrors/ I hadn't the time to really check mirror brain, I just had a quick look, so don't ask. Have you check mirror manager ( https://fedorahosted.org/mirrormanager/ ) :) Misc, have seen this rabbit, have you seen it, go, run after it ! 3) The list look like: continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=distrib,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=updates,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 I would add the AS number, as this would allow urpmi to take a mirror that is really near from a network point of view. My software fetch IP address, so it can find AS number as soon we a way to find them. * it is harcoded in urpmi Could I suggest to not hardcode it anymore, and maybe place it in a file outside of urpmi ( so a distribution could override it more easily if needed ). I agree. 3) The content * tags seems be optional * mirror point to 2011 not cooker (a symlink at time) Well, it depend if we want to let people go on cooker by default or not. This URL is used by urpmi on cooker system, so people using cooker are aware about how it works (at least I hope). * I can produce same output as I have same information * Do we need specific line for update since now distribution include everything * Do we keep this format ? The format seems fine for me. I think there is some problem, for instance, what happend if a country name contains a ',' ? But beside this little disadvantages I don't see myself the need to change it. -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpEhHrbHX8xb.pgp Description: PGP signature
Re: [Mageia-dev] Mirror list for apps (LONG mail)
On 11/03/2010 12:28 PM, Michael Scherer wrote: if really you want to change this you should : - ditch the url in favor of a XML-RPC or REST calls Given the fact we have slightly less ressources than Mandriva and that people on Mandriva side ( or at least, that's what I remind that fcrozat told me ) didn't have the ressources to have a more complex system, we should avoid this. Not only that, but simple solutions are often much less fragile and break less often. I'm in favor of keeping a simple solution at this point. (Sorry if I'm overstepping my boundaries.) - Michael
Re: [Mageia-dev] Mirror list for apps (LONG mail)
My main points in this - seen from the user's side: 1. If an automatic mirrorlist is used by urpmi there must be the ability to jump to the next mirror if the first one is not available or outdated. This was already requested some time ago in Mandriva and Mageia should not start a mirrorlist system without this feature. 2. Whatever system is used, the user must have the option to select a mirror if he wants to (like in SmartUrpmi). In short: there must be the easy option to override the automatic system, like it is now with Mandriva. Example: if the next mirror (selected by the mirrorlist system) is too slow for the user's needs. wobo
