Re: [Mageia-dev] Mirror list for apps (LONG mail)
Wolfgang Bornath a écrit : My main points in this - seen from the user's side: 1. If an automatic mirrorlist is used by urpmi there must be the ability to jump to the next mirror if the first one is not available or outdated. This was already requested some time ago in Mandriva and Mageia should not start a mirrorlist system without this feature. Agree totally. My closest mirror is often not up to date (by a day or 2), and is down for a few hours every night. (It is hosted at a small university.) So often for updates, I get an error message about being unable to download. I have to wait a day or 2, or update manually from another mirror. 2. Whatever system is used, the user must have the option to select a mirror if he wants to (like in SmartUrpmi). In short: there must be the easy option to override the automatic system, like it is now with Mandriva. Example: if the next mirror (selected by the mirrorlist system) is too slow for the user's needs. Exactly. The closest mirror is also relatively slow, and only allows a single connection. The 2nd and 3rd closest mirrors belong to a large network of mirrors, with very fast download, multiple connexions, and transparently selects the particular mirror site. But currently, with Mandriva, I can't select it as my default. With Sourceforge, for example, every time I download I can select an alternate default mirror if I want to. Which remains the default the next time I download. That would be ideal. wobo - André
Re: [Mageia-dev] Mirror list for apps (LONG mail)
* Funda Wang (fundaw...@gmail.com) wrote: > Just a question, why couldn't we just use the same architecture with > opensuse, which is based on metalink? As I said, I haven't look yet. -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgphDMlbtTdcP.pgp Description: PGP signature
Re: [Mageia-dev] Mirror list for apps (LONG mail)
On Thu, Nov 4, 2010 at 08:30, Funda Wang wrote: > Just a question, why couldn't we just use the same architecture with > opensuse, which is based on metalink? http://mirrorbrain.org/metalinks/ & http://www.metalinker.org/ (for reference about this). Romain
Re: [Mageia-dev] Mirror list for apps (LONG mail)
Just a question, why couldn't we just use the same architecture with opensuse, which is based on metalink?
Re: [Mageia-dev] Mirror list for apps (LONG mail)
My main points in this - seen from the user's side: 1. If an automatic mirrorlist is used by urpmi there must be the ability to jump to the next mirror if the first one is not available or outdated. This was already requested some time ago in Mandriva and Mageia should not start a mirrorlist system without this feature. 2. Whatever system is used, the user must have the option to select a mirror if he wants to (like in SmartUrpmi). In short: there must be the easy option to override the automatic system, like it is now with Mandriva. Example: if the next mirror (selected by the mirrorlist system) is too slow for the user's needs. wobo
Re: [Mageia-dev] Mirror list for apps (LONG mail)
On 11/03/2010 12:28 PM, Michael Scherer wrote: if really you want to change this you should : - ditch the url in favor of a XML-RPC or REST calls Given the fact we have slightly less ressources than Mandriva and that people on Mandriva side ( or at least, that's what I remind that fcrozat told me ) didn't have the ressources to have a more complex system, we should avoid this. Not only that, but simple solutions are often much less fragile and break less often. I'm in favor of keeping a simple solution at this point. (Sorry if I'm overstepping my boundaries.) - Michael
Re: [Mageia-dev] Mirror list for apps (LONG mail)
Le jeudi 04 novembre 2010 à 01:03 +0100, Olivier Thauvin a écrit : > * Michael Scherer (m...@zarb.org) wrote: > > Le mercredi 03 novembre 2010 à 19:47 +0100, Olivier Thauvin a écrit : > > > > > 3) The list look like: > > > continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=distrib,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 > > > continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=updates,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 > > > > I would add the AS number, as this would allow urpmi to take a mirror > > that is really near from a network point of view. > > My software fetch IP address, so it can find AS number as soon we a way > to find them. Pascal sent a email about this, using dns. > > > 3) The content > > > * tags seems be optional > > > * mirror point to 2011 not cooker (a symlink at time) > > > > Well, it depend if we want to let people go on cooker by default or not. > > This URL is used by urpmi on cooker system, so people using cooker are > aware about how it works (at least I hope). What about people using RC and Beta ? -- Michael Scherer
Re: [Mageia-dev] Mirror list for apps (LONG mail)
* Michael Scherer (m...@zarb.org) wrote: > Le mercredi 03 novembre 2010 à 19:47 +0100, Olivier Thauvin a écrit : > > Hi, > > > > I wrote an application to register mirrors which is currently here: > > http://distrib-coffee.ipsl.jussieu.fr/mageia-mirrors/ > > > > I hadn't the time to really check mirror brain, I just had a quick look, > > so don't ask. > > Have you check mirror manager > ( https://fedorahosted.org/mirrormanager/ ) :) Misc, have seen this rabbit, have you seen it, go, run after it ! > > 3) The list look like: > > continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=distrib,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 > > continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=updates,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 > > I would add the AS number, as this would allow urpmi to take a mirror > that is really near from a network point of view. My software fetch IP address, so it can find AS number as soon we a way to find them. > > * it is harcoded in urpmi > > Could I suggest to not hardcode it anymore, and maybe place it in a file > outside of urpmi ( so a distribution could override it more easily if > needed ). I agree. > > > 3) The content > > * tags seems be optional > > * mirror point to 2011 not cooker (a symlink at time) > > Well, it depend if we want to let people go on cooker by default or not. This URL is used by urpmi on cooker system, so people using cooker are aware about how it works (at least I hope). > > * I can produce same output as I have same information > > * Do we need specific line for update since now distribution include > > everything > > * Do we keep this format ? > > The format seems fine for me. I think there is some problem, for instance, what happend if a country name contains a ',' ? But beside this little disadvantages I don't see myself the need to change it. -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpEhHrbHX8xb.pgp Description: PGP signature
Re: [Mageia-dev] Mirror list for apps (LONG mail)
Le mercredi 03 novembre 2010 à 19:47 +0100, Olivier Thauvin a écrit : > Hi, > > I wrote an application to register mirrors which is currently here: > http://distrib-coffee.ipsl.jussieu.fr/mageia-mirrors/ > > I hadn't the time to really check mirror brain, I just had a quick look, > so don't ask. Have you check mirror manager ( https://fedorahosted.org/mirrormanager/ ) :) > In all case the goal of my application is to allow everyone to register > a new mirror, and provide a list usuable by urpmi. > > It is time to write the part of code responsible of the list build. > So come some questions. > > First let me explain the current mandriva process: > > 1) At time we have an /etc/product.id given by > mandriva-release-Free-2011.0-0.1mdv2011.0 > On my system it contains: > vendor=Mandriva,distribution=Mandriva > Linux,type=Basic,version=2011.0,branch=Devel,release=1,arch=x86_64,product=Free > > > 2) The base URL list is hardcoded in urpmi: > "https://api.mandriva.com/mirrors/$product_type.$product_version.$arch.list"; > On my system the correct url is: > https://api.mandriva.com/mirrors/basic.cooker.x86_64.list > > 3) The list look like: > continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=distrib,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 > continent=EU,zone=DE,country=Germany,latitude=51.97,longitude=7.63,version=2011.0,arch=x86_64,type=updates,url=rsync://ftp.join.uni-muenster.de/mandrakelinux/devel/2011.0/x86_64 I would add the AS number, as this would allow urpmi to take a mirror that is really near from a network point of view. > So, question and my own POV about this: > 1) /etc/product.id > * I don't like this file, I think it is redundant with > /etc/mandriva-release > * Why "basic" and not "Free" since it is the "product" flag ? Dunno > * Notice "cooker" does not appear in this file > > 2) The url > * I suggest to use http://mirror.mageia.org/api/, other proposal > welcome (need some review with sys admin) > * Obviously the valid list version is 'cooker' not 2011 > * it is harcoded in urpmi Could I suggest to not hardcode it anymore, and maybe place it in a file outside of urpmi ( so a distribution could override it more easily if needed ). > 3) The content > * tags seems be optional > * mirror point to 2011 not cooker (a symlink at time) Well, it depend if we want to let people go on cooker by default or not. > * I can produce same output as I have same information > * Do we need specific line for update since now distribution include > everything > * Do we keep this format ? The format seems fine for me. -- Michael Scherer
Re: [Mageia-dev] Mirror list for apps (LONG mail)
2010/11/3 Michael Scherer : > Le mercredi 03 novembre 2010 à 20:05 +0100, Fabrice Facorat a écrit : >> 2010/11/3 Olivier Thauvin : >> >> > Keeping this format is more simple, just changing the line specifiying >> > basis url in URPMI. >> >> And allow to keep compatibility with urpmi without maintaining too much patch >> >> > By changing the the format we can add features, make it clearer, etc... >> > But changing format to what ? XML ? YAML ? CSV ? something worst ? :) >> > Changes can done later. >> >> if really you want to change this you should : >> - ditch the url in favor of a XML-RPC or REST calls > > Given the fact we have slightly less ressources than Mandriva and that > people on Mandriva side ( or at least, that's what I remind that fcrozat > told me ) didn't have the ressources to have a more complex system, we > should avoid this. > >> - by doing this you will be able to user whatever version or name for >> the distribution. Indeed by using an url, you have to avoid space, or >> too fancy names. On top of that you can change too much the format > > You can use space in a url. sure, but it's ugly and you need to escape the space ( url encoding ) > And if we use a XML-RPC api, we will not be able to change it either, so > that trading the impossibility of changing the format for the > impossibility to change the API. a good API won't be an issue :p URL parsing is harder and more error-prone > This could be done on the client side. On the client side, this would > mean a little bit more bandwidth used, a little more cpu, and on our > side, it would be less CPU. And I think cpu is a more rare ressources > than bandwidth. you can use cache on the server side, i.e provide a static file which is a cache of the query. -- Close the World, Open the Net http://www.linux-wizard.net
Re: [Mageia-dev] Mirror list for apps (LONG mail)
Le mercredi 03 novembre 2010 à 20:05 +0100, Fabrice Facorat a écrit : > 2010/11/3 Olivier Thauvin : > > > Keeping this format is more simple, just changing the line specifiying > > basis url in URPMI. > > And allow to keep compatibility with urpmi without maintaining too much patch > > > By changing the the format we can add features, make it clearer, etc... > > But changing format to what ? XML ? YAML ? CSV ? something worst ? :) > > Changes can done later. > > if really you want to change this you should : > - ditch the url in favor of a XML-RPC or REST calls Given the fact we have slightly less ressources than Mandriva and that people on Mandriva side ( or at least, that's what I remind that fcrozat told me ) didn't have the ressources to have a more complex system, we should avoid this. > - by doing this you will be able to user whatever version or name for > the distribution. Indeed by using an url, you have to avoid space, or > too fancy names. On top of that you can change too much the format You can use space in a url. And if we use a XML-RPC api, we will not be able to change it either, so that trading the impossibility of changing the format for the impossibility to change the API. > So product.id could be kept unchanged, but the way urpmi talk to the > mirror provider could be improved. > Last but not least, by using XML-RC or REST calls, we could filter the > output of mirror list ( for example retrieve the list of mirror for > cooker in EU zone ) This could be done on the client side. On the client side, this would mean a little bit more bandwidth used, a little more cpu, and on our side, it would be less CPU. And I think cpu is a more rare ressources than bandwidth. In fact, doing this on the client would be better on the privacy side. -- Michael Scherer
Re: [Mageia-dev] Mirror list for apps (LONG mail)
* Fabrice Facorat (fabrice.faco...@gmail.com) wrote: > 2010/11/3 Olivier Thauvin : > > > Keeping this format is more simple, just changing the line specifiying > > basis url in URPMI. > > And allow to keep compatibility with urpmi without maintaining too much patch > > > By changing the the format we can add features, make it clearer, etc... > > But changing format to what ? XML ? YAML ? CSV ? something worst ? :) > > Changes can done later. > > if really you want to change this you should : > - ditch the url in favor of a XML-RPC or REST calls We can provide both, just matter of some code. > So product.id could be kept unchanged, but the way urpmi talk to the > mirror provider could be improved. > Last but not least, by using XML-RC or REST calls, we could filter the > output of mirror list ( for example retrieve the list of mirror for > cooker in EU zone ) Notice it can done by passing argument to URL: http:///...list?continent=EU I do agree XML-RPC/REST make it clearer and probably more extensible. > > -- > Close the World, Open the Net > http://www.linux-wizard.net -- Olivier Thauvin CNRS - LATMOS ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ pgpmVUUXl7Oua.pgp Description: PGP signature
Re: [Mageia-dev] Mirror list for apps (LONG mail)
Romain d'Alverny skrev 3.11.2010 21:04: On Wed, Nov 3, 2010 at 19:47, Olivier Thauvin wrote: [...] 3) The content [...] * Do we need specific line for update since now distribution include everything If we happen to have mirrors that only mirror an update tree? Then its not a mirror. A mirror need to keep a complete tree so one package in /updates can depend on a package in /release so we can stop with the "must copy files from /releases to /updates to satisfy deps" ... So _no_ separate update tree... -- Thomas
Re: [Mageia-dev] Mirror list for apps (LONG mail)
2010/11/3 Olivier Thauvin : > Keeping this format is more simple, just changing the line specifiying > basis url in URPMI. And allow to keep compatibility with urpmi without maintaining too much patch > By changing the the format we can add features, make it clearer, etc... > But changing format to what ? XML ? YAML ? CSV ? something worst ? :) > Changes can done later. if really you want to change this you should : - ditch the url in favor of a XML-RPC or REST calls - by doing this you will be able to user whatever version or name for the distribution. Indeed by using an url, you have to avoid space, or too fancy names. On top of that you can change too much the format So product.id could be kept unchanged, but the way urpmi talk to the mirror provider could be improved. Last but not least, by using XML-RC or REST calls, we could filter the output of mirror list ( for example retrieve the list of mirror for cooker in EU zone ) -- Close the World, Open the Net http://www.linux-wizard.net
Re: [Mageia-dev] Mirror list for apps (LONG mail)
On Wed, Nov 3, 2010 at 19:47, Olivier Thauvin wrote: [...] > So, question and my own POV about this: > 1) /etc/product.id > * I don't like this file, I think it is redundant with > /etc/mandriva-release It appeared for several reasons, in order to have a consistent scheme to identify several Mandriva products: an ISO release, a specific media, a set of media, a physical product, etc. It used to be spec'ed, but not properly reviewed, neither strongly/fully implemented. See http://wiki.mandriva.com/en/product_id (it never left the draft stage actually). It could have been only an int, or a hash string, putted against a database to get a human readable label, but that would be then, less human readable in the first place. > * Why "basic" and not "Free" since it is the "product" flag ? Dunno > * Notice "cooker" does not appear in this file It could. But here, provided it's a stable release, "cooker" is not supposed to appear. Given the new situation, I suggest to use a /etc/release file, somehow like the /etc/mandriva-release file, but being strongly structured (/etc/mandriva-release wasn't, in the long run), so we can have at least: - product name (Mageia) - version - arch - branch (or something stating the unstable/stable release state) - build number (in case of automated builds, someday). with mandatory ones being product name, version and arch. > [...] > 3) The content > [...] > * Do we need specific line for update since now distribution include > everything If we happen to have mirrors that only mirror an update tree? > * Do we keep this format ? > > Keeping this format is more simple, just changing the line specifiying > basis url in URPMI. > By changing the the format we can add features, make it clearer, etc... > But changing format to what ? XML ? YAML ? CSV ? something worst ? :) > Changes can done later. It's not an optimal format, but it looks simple and good enough. What would be needed to extend/change it? Romain
