Re: [Cooker] reliable mirrors . . .
Warly wrote: John Allen <[EMAIL PROTECTED]> writes: On Thursday 06 November 2003 20:35, Blindauer Emmanuel wrote: Le Jeudi 06 Novembre 2003 18:14, Robert Fox a écrit : Maybe someone should be a mirror monitor and be the main point of contact with the mirror owners when something goes wrong! http://extasia.u-strasbg.fr/~blindaue/mirrors.php You don't do uninett.no Also it would be better to test the mirrors against the hdlist/synthesis of the master site (wherever that is) updates every 2 hours the status of mirrors, looking into hdlist.cz and looking for the files inside to be present in the directory of the ftp. But I can't test more than that. I tried to have a more efficient rsync process to our mirror reference machine (I decrease contrib frequency mirroring to test if main only can be kept correctly up to date). Maybe we will have to switch to a every 2 hour frequency if one hour delay is too short for mirror to catch on. I'm sorry, but this doesn't sound right. I'm behind a crappy cable connection and I'm capable of pulling a good mirror, every hour, from kenobi. If kenobi was updated every 10mins, I would probably sync every 10 mins. I can't imagine why cooker.mandrakesoft.com shouldn't be able to do the same... IMHO cutting down on the frequency won't help, it'll just worsen the problem. What you need to do is make sure that you don't mirror the same thing twice --> i.e. don't start a second mirror process if the first one is still running. I'm running the following cronjobs for my mirrors: 0 0-22/2 * * * if [ `ps -u $UID | grep all.sh | wc -l` = 0 ]; then ~/bin/all.sh cooker upload >/dev/null 2>&1 ; fi 0 1-23/2 * * * if [ `ps -u $UID | grep all.sh | wc -l` = 0 ]; then ~/bin/all.sh cooker >/dev/null 2>&1 ; fi on the even hours packages are uploaded (ftpcooker / ftpcontrib is invoked) on the odd hours they aren't. If the "all.sh" script is found running under the current user, then a new one doesn't start. Increasing the frequency will make the mirror more up2date and decrease the load on it --> if you need to tranfer 100Mb, it's better to do it 12* 8.125Mb spread over the hour than 1* 100Mb at :00. Again, just make sure you don't two "all.sh" running. please take this into consideration. regards, Stefan smime.p7s Description: S/MIME Cryptographic Signature
Re: [Cooker] reliable mirrors . . .
Perhaps the mtime of the hdlist (or synthesis) files should be compared, together with the SHA1 hash of these files. Since these files are regenerated a few times throughout the day, perhaps the version of these files can be tracked and compared. regards, Stefan
Re: [Cooker] reliable mirrors . . .
On Friday 07 November 2003 12:28, BLINDAUER Emmanuel wrote: > Le Vendredi 07 Novembre 2003 10:05, Stefan van der Eijk a écrit : > > > Le Jeudi 06 Novembre 2003 22:00, Stefan van der Eijk a écrit : > > >> Really nice site, very usefull, but what is your reference point? > > > > > > The reference is the hdlist on the ftp. > > > > Which ftp? > > The ftp which is being monitored. > My goal wasn't to look for mirror being in sync with kenobi, but being > coherent (hdlist on the ftp against the files on the ftp. It would be great to have both, coherent & in sync. -- John Allen, Email: mailto:[EMAIL PROTECTED] MandrakeClub Silver Member.
Re: [Cooker] reliable mirrors . . .
John Allen <[EMAIL PROTECTED]> writes: > On Thursday 06 November 2003 20:35, Blindauer Emmanuel wrote: >> Le Jeudi 06 Novembre 2003 18:14, Robert Fox a écrit : >> > Maybe someone should be a mirror monitor and be the main point of >> > contact with the mirror owners when something goes wrong! >> >> http://extasia.u-strasbg.fr/~blindaue/mirrors.php >> > > You don't do uninett.no > > Also it would be better to test the mirrors against the hdlist/synthesis of > the master site (wherever that is) > >> updates every 2 hours the status of mirrors, looking into hdlist.cz and >> looking for the files inside to be present in the directory of the ftp. >> >> But I can't test more than that. I tried to have a more efficient rsync process to our mirror reference machine (I decrease contrib frequency mirroring to test if main only can be kept correctly up to date). Maybe we will have to switch to a every 2 hour frequency if one hour delay is too short for mirror to catch on. -- Warly
Re: [Cooker] reliable mirrors . . .
Le Vendredi 07 Novembre 2003 10:05, Stefan van der Eijk a écrit : > > Le Jeudi 06 Novembre 2003 22:00, Stefan van der Eijk a écrit : > >> Really nice site, very usefull, but what is your reference point? > > > > The reference is the hdlist on the ftp. > > Which ftp? The ftp which is being monitored. My goal wasn't to look for mirror being in sync with kenobi, but being coherent (hdlist on the ftp against the files on the ftp.
Re: [Cooker] reliable mirrors . . .
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 John Allen wrote: | On Thursday 06 November 2003 20:35, Blindauer Emmanuel wrote: | |>Le Jeudi 06 Novembre 2003 18:14, Robert Fox a écrit : |> |>>Maybe someone should be a mirror monitor and be the main point of |>>contact with the mirror owners when something goes wrong! |> |>http://extasia.u-strasbg.fr/~blindaue/mirrors.php |> | | | You don't do uninett.no | | Also it would be better to test the mirrors against the hdlist/synthesis of | the master site (wherever that is) | Shouldn't we monitor all official cooker mirrors listed on the Mandrake site ? E.g. ftp-stud.fht-esslingen.de is missing as well (and usually out of sync). Regards, Jan - -- Jan Ciger VRlab EPFL Switzerland GPG public key : http://www.keyserver.net/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQE/q4tEn11XseNj94gRAqAmAKDe4Lvy1/BirsFLXIZYCFLtgHOu9gCfeGvV I2H8rAtEZJ72VxR3wSc6/nE= =DX4x -END PGP SIGNATURE-
Re: [Cooker] reliable mirrors . . .
> Le Jeudi 06 Novembre 2003 22:00, Stefan van der Eijk a écrit : >> Really nice site, very usefull, but what is your reference point? > > The reference is the hdlist on the ftp. Which ftp? > On a ftp, urpmi will get it (or synthesis). So after getting the file, all > files who can be DL are referenced in this file. > Having more files in directory than listed in the hdlist isn't bad, but > missing files makes trouble. > > So the current script: > * Download the hdlist > * Make a list of all rpm listed in the hdlist. > * Look for these files in the directory > * Count the missing files (files listed in hdlist, but not present on ftp) > > At end, the script print the number of missing files. > > This is done only for RPMS from cooker, and for contrib. > > > When someone want to install a cooker or work on it, he need a coherent > FTP > who can provide rpms in sync with the hdlist. This was the bad problem who > was present some months ago with mirrors (hdlist not sync-ed). > > Having a up to date mirror sync-ed with kenobi is more difficult, and for > me, > cannot be done, as rpms are updated continuously. > > Emmanuel
Re: [Cooker] reliable mirrors . . .
On Thursday 06 November 2003 20:35, Blindauer Emmanuel wrote: > Le Jeudi 06 Novembre 2003 18:14, Robert Fox a écrit : > > Maybe someone should be a mirror monitor and be the main point of > > contact with the mirror owners when something goes wrong! > > http://extasia.u-strasbg.fr/~blindaue/mirrors.php > You don't do uninett.no Also it would be better to test the mirrors against the hdlist/synthesis of the master site (wherever that is) > updates every 2 hours the status of mirrors, looking into hdlist.cz and > looking for the files inside to be present in the directory of the ftp. > > But I can't test more than that. > > Emmanuel -- John Allen, Email: mailto:[EMAIL PROTECTED] MandrakeClub Silver Member.
Re: [Cooker] reliable mirrors . . .
Le Jeudi 06 Novembre 2003 22:00, Stefan van der Eijk a écrit : > Really nice site, very usefull, but what is your reference point? The reference is the hdlist on the ftp. On a ftp, urpmi will get it (or synthesis). So after getting the file, all files who can be DL are referenced in this file. Having more files in directory than listed in the hdlist isn't bad, but missing files makes trouble. So the current script: * Download the hdlist * Make a list of all rpm listed in the hdlist. * Look for these files in the directory * Count the missing files (files listed in hdlist, but not present on ftp) At end, the script print the number of missing files. This is done only for RPMS from cooker, and for contrib. When someone want to install a cooker or work on it, he need a coherent FTP who can provide rpms in sync with the hdlist. This was the bad problem who was present some months ago with mirrors (hdlist not sync-ed). Having a up to date mirror sync-ed with kenobi is more difficult, and for me, cannot be done, as rpms are updated continuously. Emmanuel
Re: [Cooker] reliable mirrors . . .
Olivier Blin wrote: On Thu, 6 Nov 2003 23:04:34 +0100 Guillaume Rousse <[EMAIL PROTECTED]> wrote: I'd really like to have someting as RSS to publish mirrors list. Mandrake would publish it this way, third-party projects as PLF or JPackage would also, and tools as urpmi.setup could easily retrieve and use them... That could also be used by the network install, to be more friendly, it would let the user pick an entry in the mirror list, what do you think of it ? Let's do it! smime.p7s Description: S/MIME Cryptographic Signature
Re: [Cooker] reliable mirrors . . .
On Thu, 6 Nov 2003 23:04:34 +0100 Guillaume Rousse <[EMAIL PROTECTED]> wrote: > I'd really like to have someting as RSS to publish mirrors list. > Mandrake would publish it this way, third-party projects as PLF or > JPackage would also, and tools as urpmi.setup could easily retrieve > and use them... That could also be used by the network install, to be more friendly, it would let the user pick an entry in the mirror list, what do you think of it ? -- Olivier Blin
Re: [Cooker] reliable mirrors . . .
Ainsi parlait Stefan van der Eijk : > True, cause you don't have access to what should be your reference point. I'd really like to have someting as RSS to publish mirrors list. Mandrake would publish it this way, third-party projects as PLF or JPackage would also, and tools as urpmi.setup could easily retrieve and use them... -- Guillaume Rousse There is no remedy for sex but more sex -- Murphy's Laws on Sex n°6
Re: [Cooker] reliable mirrors . . .
Emmanuel, Maybe someone should be a mirror monitor and be the main point of contact with the mirror owners when something goes wrong! http://extasia.u-strasbg.fr/~blindaue/mirrors.php Really nice site, very usefull, but what is your reference point? Your site says that sunet is up2date, yet after a fresh (& illegal) mirror fron kenobi (IMHO the one and only true source) the following src.rpm are removed: 1449 files to consider deleting lftp-2.6.8-1mdk.src.rpm deleting kdepim-3.1.93-6mdk.src.rpm deleting kdemultimedia-3.1.93-4mdk.src.rpm deleting kdelibs-3.1.93-11mdk.src.rpm deleting kdebase-3.1.93-13mdk.src.rpm deleting kdeadmin-3.1.93-5mdk.src.rpm deleting gnopernicus-0.7.1-1mdk.src.rpm deleting gda2.0-1.0.1-1mdk.src.rpm deleting gail-1.4.1-1mdk.src.rpm deleting fetchmail-6.2.5-1mdk.src.rpm deleting evolution-1.4.5-1mdk.src.rpm deleting drakxtools-9.3-3mdk.src.rpm deleting drakconf-9.3-1mdk.src.rpm deleting clips-6.21-3mdk.src.rpm deleting atk1.0-1.4.1-1mdk.src.rpm My conclusion is that sunet.se is *not* up2date, the reference point needs to be adjusted to the systems at the office in Paris (rue d'Aboukir). updates every 2 hours the status of mirrors, looking into hdlist.cz and looking for the files inside to be present in the directory of the ftp. But I can't test more than that. True, cause you don't have access to what should be your reference point. regards, Stefan smime.p7s Description: S/MIME Cryptographic Signature
Re: [Cooker] reliable mirrors . . .
Le Jeudi 06 Novembre 2003 18:14, Robert Fox a écrit : > Maybe someone should be a mirror monitor and be the main point of > contact with the mirror owners when something goes wrong! http://extasia.u-strasbg.fr/~blindaue/mirrors.php updates every 2 hours the status of mirrors, looking into hdlist.cz and looking for the files inside to be present in the directory of the ftp. But I can't test more than that. Emmanuel
[Cooker] reliable mirrors . . .
This topic has appeared several times in the past, but needs to be addressed yet again. Now that Cooker is officially opened again for the 10.0 series, could we please try to guarantee more stable mirrors (especially the two "main" mirrors.)!! Right now - sunsite.uio.no and ftp.sunet.se are horribly out of sync with each other and with the latest updates. This makes it very challenging to keep up to date with rsync and test frequently. We need to have at least ONE main reference server which is reliable!!! Maybe someone should be a mirror monitor and be the main point of contact with the mirror owners when something goes wrong! Thx, R.Fox Thx, R.Fox -- Robert Fox <[EMAIL PROTECTED]> Fox Consulting Services
