Re: Automatic package mirror discovery implementation for pkg_add(1) tool
hmm, on Thu, Feb 11, 2010 at 12:35:05AM +0300, Igor Zinovik said that > Maybe it is not polite to answer to this old thread, but I've integrated > (somehow) AutoMirrorDiscovery functionality into pkg_add tool, just to > prove myself that i can do that without crashing pkg_add functionality. It is > very clumsy right now, but it works for me (with some bugs of course). this is a great idea i think. maybe you would want to rename it MirrorAutoDiscovery though :] i would also fill the cache file with all the mirrors, (not just the n fastest) in order from fastest to slowest. if the list of the mirrors is taken from www.openbsd.org everytime the list is generated, the list could be considered authoritative. -f -- go ahead, jump. 100,000 lemmings can't be wrong.
Re: Automatic package mirror discovery implementation for pkg_add(1) tool
08.01.2010 15:30, Bob Beck writes:
> 2010/1/8 Bob Beck :
> And what I mean by that Is that I would be willing to put together a similar
> trick for this on ftp.openbsd.org as I did for the installer - if
> someone was willing
> to integrate it into the pkg_add tools.
Maybe it is not polite to answer to this old thread, but I've integrated
(somehow) AutoMirrorDiscovery functionality into pkg_add tool, just to
prove myself that i can do that without crashing pkg_add functionality. It is
very clumsy right now, but it works for me (with some bugs of course).
This is how it works (currently there is no knob to shut it up):
(/tmp)[213]% sudo pkg_add -d
Pinging anga.funkfeuer.at...OK
Pinging carroll.cac.psu.edu...OK
Pinging filedump.se.rit.edu...OK
Pinging ftp-stud.fht-esslingen.de...OK
Pinging ftp.arcane-networks.fr...OK
Pinging ftp.aso.ee...no response
Pinging ftp.belnet.be...OK
Pinging ftp.bytemine.net...OK
Pinging ftp.ca.openbsd.org...no response
Pinging ftp.cc.uoc.gr...no response
Pinging ftp.chg.ru...OK
Pinging ftp.crans.org...OK
Pinging ftp.cs.pu.edu.tw...no response
Pinging ftp.cse.buffalo.edu...no response
Pinging ftp.das.ufsc.br...OK
Pinging ftp.df.lth.se...OK
Pinging ftp.dkuug.dk...no response
Use of uninitialized value $ret in numeric eq (==) at
/usr/libdata/perl5/OpenBSD/AutoMirrorDiscovery.pm line 58.
Pinging ftp.duth.gr...no response
Pinging ftp.esat.net...OK
Pinging ftp.estpak.ee...OK
Pinging ftp.eu.openbsd.org...OK
Pinging ftp.fmed.uc.pt...no response
Pinging ftp.fr.openbsd.org...OK
Pinging ftp.freebsdchina.org...OK
Pinging ftp.freenet.de...OK
Pinging ftp.fsn.hu...OK
Pinging ftp.gamma.ru...OK
Pinging ftp.heanet.ie...OK
Pinging ftp.iinet.net.au...OK
Pinging ftp.inet.no...OK
Pinging ftp.irisa.fr...OK
Pinging ftp.is.co.za...OK
Pinging ftp.jaist.ac.jp...OK
Pinging ftp.jyu.fi...OK
Pinging ftp.kaist.ac.kr...no response
Pinging ftp.kddlabs.co.jp...no response
Pinging ftp.lambdaserver.com...OK
Pinging ftp.mirrorservice.org...OK
Pinging ftp.netbsd.se...OK
Pinging ftp.nluug.nl...OK
Pinging ftp.obsd.si...OK
Pinging ftp.openbsd.dk...OK
Pinging ftp.openbsd.or.id...OK
Pinging ftp.openbsd.org.ar...no response
Pinging ftp.openbsd.org...OK
Pinging ftp.piotrkosoft.net...no response
Pinging ftp.plig.net...OK
Pinging ftp.rediris.es...OK
Pinging ftp.spline.de...OK
Pinging ftp.task.gda.pl...OK
Pinging ftp.tcc.edu.tw...OK
Pinging ftp.tpnet.pl...OK
Pinging ftp.tw.openbsd.org...OK
Pinging ftp.udc.es...no response
Pinging ftp.ulak.net.tr...OK
Pinging ftp.uninett.no...OK
Pinging ftp.wu-wien.ac.at...OK
Pinging ftp2.fr.openbsd.org...OK
Pinging ftp3.usa.openbsd.org...OK
Pinging ftp5.usa.openbsd.org...OK
Pinging gulus.usherbrooke.ca...no response
Pinging mirror.aarnet.edu.au...OK
Pinging mirror.cdmon.com...OK
Pinging mirror.corbina.net...OK
Pinging mirror.hostfuss.com...OK
Pinging mirror.iawnet.sandia.gov...no response
Pinging mirror.internode.on.net...OK
Pinging mirror.pacific.net.au...OK
Pinging mirror.planetunix.net...OK
Pinging mirror.public-internet.co.uk...no response
Pinging mirror.rit.edu...OK
Pinging mirror.roothell.org...OK
Pinging mirror.switch.ch...OK
Pinging mirrors.24-7-solutions.net...OK
Pinging mirrors.localhost.net.ar...no response
Pinging mirrors.nic.funet.fi...OK
Pinging mirrors.ucr.ac.cr...no response
Pinging obsd.cec.mtu.edu...OK
Pinging openbsd.arcticnetwork.ca...OK
Pinging openbsd.bsdforen.de...OK
Pinging openbsd.ftp.fu-berlin.de...no response
Pinging openbsd.mirror.frontiernet.net...OK
Pinging openbsd.mirrors.pair.com...OK
Pinging openbsd.mirrors.tds.net...OK
Pinging openbsd.noc.jgm.gov.ar...no response
(/tmp)[214]% cat /var/db/ftpmirror.cache
ftp://mirrors.nic.funet.fi/pub/OpenBSD/4.6/packages/i386/
ftp://ftp.gamma.ru/pub/OpenBSD/4.6/packages/i386/
ftp://ftp.eu.openbsd.org/pub/OpenBSD/4.6/packages/i386/
(/tmp)[228]% echo $PKG_PATH
(/tmp)[229]% sudo pkg_add -i fdm
No packages available in the PKG_PATH
fdm-1.6:tdb-1.0.6p0: ok (1 to go)
fdm-1.6: ok
Here is the code:
/usr/src/usr.sbin/pkg_add/OpenBSD/AutoMirrorDiscovery.pm
# ex:ts=8 sw=4:
# $OpenBSD$
#
# Copyright (c) 2009 Igor Zinovik
#
# Permission to use, copy, modify, and distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
# functionality for automatic fastest ftp mirror discovery
use strict;
use warnings;
package OpenBSD::AutoMirrorDiscovery;
use OpenBSD::Paths;
use Net::Ping;
use Net::FTP;
sub discover_ftp_mirrors()
{
Re: Automatic package mirror discovery implementation for pkg_add(1) tool
2010/1/8 Bob Beck : >> Apart from this, this is a really tough problem, because of infrastructure >> issues. Basically, our mirrors are not that reliable, and the closest >> one often won't have the packages you need... which is a reason why it's >> mostly some user settings... > > True - although we could do a very similar stunt to what we do with ftplist > now in the installer - including remembering your most recent choice from your > IP address, so it just figured out where you liked to get your packages from. > And what I mean by that Is that I would be willing to put together a similar trick for this on ftp.openbsd.org as I did for the installer - if someone was willing to integrate it into the pkg_add tools.
Re: Automatic package mirror discovery implementation for pkg_add(1) tool
> Apart from this, this is a really tough problem, because of infrastructure > issues. Basically, our mirrors are not that reliable, and the closest > one often won't have the packages you need... which is a reason why it's > mostly some user settings... True - although we could do a very similar stunt to what we do with ftplist now in the installer - including remembering your most recent choice from your IP address, so it just figured out where you liked to get your packages from.
Re: Automatic package mirror discovery implementation for pkg_add(1) tool
On 2009/12/05 22:22, Marc Espie wrote: > Apart from this, this is a really tough problem, because of infrastructure > issues. Basically, our mirrors are not that reliable, and the closest > one often won't have the packages you need... which is a reason why it's > mostly some user settings... For snapshots I totally agree, but I think this could work reasonably well for releases.
Re: Automatic package mirror discovery implementation for pkg_add(1) tool
On Sat, Dec 05, 2009 at 10:47:25PM +0300, Igor Zinovik wrote:
> if (-e "/var/db/pkg/ftpmirror.cache") {
> open my $fh, '<', "/var/db/pkg/ftpmirror.cache" or
> die("Permission denied");
> @mirrors = <$fh>;
> close $fh;
> print $mirrors[0];
> exit;
> }
...
> open $fh, '>', "/var/db/pkg/ftpmirror.cache" or
> die("Cannot create mirror cache file");
>
you can't do this. /var/db/pkg only holds packages. This will confuse
pkg_add when you try to update things.
Apart from this, this is a really tough problem, because of infrastructure
issues. Basically, our mirrors are not that reliable, and the closest
one often won't have the packages you need... which is a reason why it's
mostly some user settings...
Automatic package mirror discovery implementation for pkg_add(1) tool
Hello, tech@ readers. At work i use rpm-based linux distribution (i'm writing this not to fire up new holywar thread). It package manager is called `yum', which is something like pkg_* tools but all in one executable file. This yum thing has nice plugin (which is called `fastest_mirror') that allows it automatically discover package repositories across the Net. When i'm installing software I have to type something like # yum install tmux and it will automatically find fastest mirror with tmux rpm package. Main advantage of this is that i do not have to care about manual repository selection. Since i'm ill now (damn flu) i decided to implement such functionality in Perl for pkg_add(1) tool. I was always annoyed by the fact the i have to manually find fastest mirror for my newly openbsd installation by hands. If it fails to work i have change it by hands, again :-E But why i have to do things which computer can do? We are living in 21th century. Where are domestic laser beams, light sables, anti gravity cars and automatic package mirror discovery in pkg_add tool? :) So absence of automatic mirror discovery was very disappointing me. What ordinary OpenBSD use have to do when something annoys him? Right, shut up and hack. So i implemented basic functionality in separate perl program, which i think can be easily transformed into perl module that can be integrated into pkg_add. Please do not judge my work very strictly since this is my second program in Perl. First one was 'Hello, World!' which i wrote 5 minutes before i started writing AutoMirrorDiscovery.pl. I mostly programmed in C and PHP in my life, so i think i implemented C program in Perl and it is not written in usual Perl style. I tried to copy some style guidelines from pkg_* tools source code. So what it does? Here is the description of AutoMirrorDiscovery.pl logic: 1. Try to find cache file that was created by previous execution if it exists use first available mirror, then print it to stdout and exit otherwise: 2. Load in all available mirrors listed in /etc/ftp.mirrors (this file contains just list of all FTP mirrors listed in http://www.openbsd.org/ftp.html, each mirror on separate line). 3. Start pinging each mirror. For this stuff i use Net::Ping module. Unfortunately to use icmp ping you need root privileges. I also tried to use UDP pings (which is default for Net::Ping), but they work very slowly. I mean UDP ping shows much higher response time than ICMP ping. 4. Then mirrors are sorted by response time. Fastest are getting at the beginning of hash list. 5. Then it tries to find 3 available mirrors that contain packages for our release version and application architecture. It just tries to connect to selected mirror and goes into package directory. Why this is needed (cwd to package directory)? We have to be sure that mirror which we going to use *really* contains packages. For example i found that ftp.chg.ru still does not provide packages for my 4.6 release. While we are traversing through mirror list we are writing cache to the disk. Ok not actually cache, but just a flat text file that contains 3 lines. Each line contains mirror for our location. First one is the fastest, second one is a bit slower and third is slowest from this trinity (slowest fast mirror :). Cache is needed so that second invocation will return fastest mirror in a jiffy, because for `yum' is also not ideal since it tries to find fastest mirror on every invocation, which i think is an overkill. Since PKG_PATH is allowed to contain several mirrors delimited by `:' we can just set it to contains all three mirrors. I do not know how yum determines fastest mirror, i did not looked it source code. So i just implemented simplest idea that comed to my mind. OS can be shipped with /etc/ftp.mirrors and each user will have only execute something like following or this even can be done during installation process, if perl is available during install process, which i think is not: # sudo pkg_add -v -d Automatic FTP mirror discovery started Checking ftp.openbsd.org for availability.OK Checking ftp.chg.ru for availabilityOK ... Determined three fastest mirrors for our location: ftp://mirrors.nic.funet.fi/pub/OpenBSD/4.6/packages/i386 ftp://ftp.gamma.ru/pub/OpenBSD/4.6/packages/i386 ftp://ftp.nluug.nl/pub/OpenBSD/4.6/packages/i386 Cache is written successfully. Just type pkg_add foobar and magic will happen. # pkg_add foobar ...normal installation procedure... I think that this feature will be very appreciated by mobile openbsd users when they travel a lot. You do not have to adjust your PKG_PATH anymore when you change your location. Just type pkg_add -d and the thing is done, but before it have to be fully implemented :) Maybe i'm just reinventing the wheel. Maybe all folks use just export PKG_PATH='all_ftp_mirrors_separated_by_colons' in your .profile and are happy. I think that espie@ can explain why such functionality wasn't implemente
