Re: Aw: Re: [gentoo-user] local mirrors (summary, leading to more questions)
On Monday, 29 April 2019 09:03:15 BST Mick wrote: > Just to mention the PORTDIR variable is no longer meant to be declared in / > etc/portage/make.conf, but in /etc/portage/repos.conf/gentoo.conf, using the > 'location = ' directive. See URL below: > > https://wiki.gentoo.org/wiki/PORTDIR That muddies the water somewhat. Now we have PORTDIR defined in one place, and PKGDIR and DISTDIR in another. -- Regards, Peter.
Re: Aw: Re: [gentoo-user] local mirrors (summary, leading to more questions)
On Sunday, 28 April 2019 21:37:19 BST n952...@web.de wrote: > - Peter Humphrey suggests: > - http ftp proxy > > In what way is that different from rsync which I thought I'd already > configured? I assume that means you're using rsync to synchronise the portage database. Then, whatever you use to fetch distfiles for installation, it uses ftp or http transport to fetch them. Squid sits on the network with its cache of previous fetches and supplies whatever hasn't changed, thus at least halving network traffic if you have two or more Gentoo machines to maintain. You can also point your browsers at squid (that's what it's meant for, in fact); their responsiveness also improves. Since I had fibre broadband I hadn't bothered with squid, but after installing it on a local machine yesterday I found it still made an observable difference. Squid doesn't need a powerful machine to serve a small LAN; I have it running happily on an old single-core, 32-bit Atom N270 box. Although squid apparently scales well up to much larger user numbers, it's remarkably easy to set up - I had it running in under half an hour. -- Regards, Peter.
Re: Aw: Re: [gentoo-user] local mirrors (summary, leading to more questions)
Mick wrote: > On Sunday, 28 April 2019 22:49:42 BST Dale wrote: >> n952...@web.de wrote: >>> Thanks to all who reponded: >>> - Rich Freeman suggests: >>> - having apache serve the local distfiles. >>> >>> How? Just make them available in the apache root and give portage >>> the URL somewhere?> >>> - use "infra scripts", but I think that's for running an up-to-date >>> general-purpose mirror >>> >>> He explains later that repos.conf defines the single place where the >>> repository is defined but that is distinct from where the distfiles >>> come from, which is defined by make.conf:GENTOO_MIRRORS >>> >>> - Dale suggests: >>> - a package, http-replicator, which might do Freeman's first >>> >>> suggestion. >>> >>> - Neil Bothwick explains the difference between serving the "portage >>> >>> mirror, the repository of ebuild and associated files" and the >>> distfiles, and suggests using NFS, presumably instead of, or in >>> addition to /usr/portage/distfiles. >>> >>> Okay, I looked for the first time at the files directly under >>> /usr/portage. >>> Is it a correct assumption that all packages available for >>> installation will be represented there? And that's it? That's >>> basically all there is to a gentoo installation? - >>> /usr/portage/distfiles >>> - the ebuilds directly under /usr/portage >>> - /usr/portage/eclass >>> - /usr/portage/metadata >>> - /usr/portage/profiles >>> - /etc/portage >>> >>> If so, then I'm delighted in the simplicity of it. Like reading the >>> one page definition of tcl. Unfortunately, it doesn't come through so >>> well in the documentation, with things like layouts seemingly basic >>> to an understanding. >>> >>> - Peter Humphrey suggests: >>> - http ftp proxy >>> >>> In what way is that different from rsync which I thought I'd >>> already configured? >> Just to add some more info. By changing the settings in make.conf, you >> can move these directories wherever you like. As a example, I moved >> mine here: >> >> >> root@fireball / # ls -al /var/cache/portage/ >> total 160 >> drwxr-xr-x 5 rootroot 4096 Dec 20 2012 . >> drwxr-xr-x 13 rootroot 4096 Apr 24 03:18 .. >> drwxrwxr-x 3 portage portage 143360 Apr 26 04:06 distfiles >> drwxr-xr-x 105 portage portage 4096 Apr 26 04:04 packages >> drwxr-xr-x 174 portage portage 4096 Apr 22 23:32 tree >> root@fireball / # >> >> >> I have these settings in make.conf to match where I put things: >> >> >> >> DISTDIR="/var/cache/portage/distfiles/" >> PKGDIR="/var/cache/portage/packages" >> PORTDIR="/var/cache/portage/tree" > Just to mention the PORTDIR variable is no longer meant to be declared in / > etc/portage/make.conf, but in /etc/portage/repos.conf/gentoo.conf, using the > 'location = ' directive. See URL below: > > https://wiki.gentoo.org/wiki/PORTDIR > > I wasn't aware of the move or don't remember it if I read it somewhere. Thanks for noticing that since that could cause confusion. I bet if I ever move that again, I'll forget that it is in a new place. :/ Dale :-) :-)
Re: Aw: Re: [gentoo-user] local mirrors (summary, leading to more questions)
On Sunday, 28 April 2019 22:49:42 BST Dale wrote: > n952...@web.de wrote: > > Thanks to all who reponded: > > - Rich Freeman suggests: > > - having apache serve the local distfiles. > > > > How? Just make them available in the apache root and give portage > > the URL somewhere?> > > - use "infra scripts", but I think that's for running an up-to-date > > general-purpose mirror > > > > He explains later that repos.conf defines the single place where the > > repository is defined but that is distinct from where the distfiles > > come from, which is defined by make.conf:GENTOO_MIRRORS > > > > - Dale suggests: > > - a package, http-replicator, which might do Freeman's first > > > > suggestion. > > > > - Neil Bothwick explains the difference between serving the "portage > > > > mirror, the repository of ebuild and associated files" and the > > distfiles, and suggests using NFS, presumably instead of, or in > > addition to /usr/portage/distfiles. > > > > Okay, I looked for the first time at the files directly under > > /usr/portage. > > Is it a correct assumption that all packages available for > > installation will be represented there? And that's it? That's > > basically all there is to a gentoo installation? - > > /usr/portage/distfiles > > - the ebuilds directly under /usr/portage > > - /usr/portage/eclass > > - /usr/portage/metadata > > - /usr/portage/profiles > > - /etc/portage > > > > If so, then I'm delighted in the simplicity of it. Like reading the > > one page definition of tcl. Unfortunately, it doesn't come through so > > well in the documentation, with things like layouts seemingly basic > > to an understanding. > > > > - Peter Humphrey suggests: > > - http ftp proxy > > > > In what way is that different from rsync which I thought I'd > > already configured? > Just to add some more info. By changing the settings in make.conf, you > can move these directories wherever you like. As a example, I moved > mine here: > > > root@fireball / # ls -al /var/cache/portage/ > total 160 > drwxr-xr-x 5 rootroot 4096 Dec 20 2012 . > drwxr-xr-x 13 rootroot 4096 Apr 24 03:18 .. > drwxrwxr-x 3 portage portage 143360 Apr 26 04:06 distfiles > drwxr-xr-x 105 portage portage 4096 Apr 26 04:04 packages > drwxr-xr-x 174 portage portage 4096 Apr 22 23:32 tree > root@fireball / # > > > I have these settings in make.conf to match where I put things: > > > > DISTDIR="/var/cache/portage/distfiles/" > PKGDIR="/var/cache/portage/packages" > PORTDIR="/var/cache/portage/tree" Just to mention the PORTDIR variable is no longer meant to be declared in / etc/portage/make.conf, but in /etc/portage/repos.conf/gentoo.conf, using the 'location = ' directive. See URL below: https://wiki.gentoo.org/wiki/PORTDIR -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: Aw: Re: [gentoo-user] local mirrors (summary, leading to more questions)
n952...@web.de wrote: > Thanks to all who reponded: > > - Rich Freeman suggests: > - having apache serve the local distfiles. > How? Just make them available in the apache root and give portage the > URL somewhere? > - use "infra scripts", but I think that's for running an up-to-date > general-purpose mirror > > He explains later that repos.conf defines the single place where the > repository is defined but that is distinct from where the distfiles > come from, which is defined by make.conf:GENTOO_MIRRORS > > - Dale suggests: > - a package, http-replicator, which might do Freeman's first > suggestion. > > - Neil Bothwick explains the difference between serving the "portage > mirror, the repository of ebuild and associated files" and the > distfiles, and suggests using NFS, presumably instead of, or in > addition to /usr/portage/distfiles. > > Okay, I looked for the first time at the files directly under > /usr/portage. > Is it a correct assumption that all packages available for installation > will be represented there? > And that's it? That's basically all there is to a gentoo installation? > - /usr/portage/distfiles > - the ebuilds directly under /usr/portage > - /usr/portage/eclass > - /usr/portage/metadata > - /usr/portage/profiles > - /etc/portage > > If so, then I'm delighted in the simplicity of it. Like reading the one > page definition of tcl. > Unfortunately, it doesn't come through so well in the documentation, > with things like layouts seemingly basic to an understanding. > > - Peter Humphrey suggests: > - http ftp proxy > > In what way is that different from rsync which I thought I'd already > configured? > > Just to add some more info. By changing the settings in make.conf, you can move these directories wherever you like. As a example, I moved mine here: root@fireball / # ls -al /var/cache/portage/ total 160 drwxr-xr-x 5 root root 4096 Dec 20 2012 . drwxr-xr-x 13 root root 4096 Apr 24 03:18 .. drwxrwxr-x 3 portage portage 143360 Apr 26 04:06 distfiles drwxr-xr-x 105 portage portage 4096 Apr 26 04:04 packages drwxr-xr-x 174 portage portage 4096 Apr 22 23:32 tree root@fireball / # I have these settings in make.conf to match where I put things: DISTDIR="/var/cache/portage/distfiles/" PKGDIR="/var/cache/portage/packages" PORTDIR="/var/cache/portage/tree" If you need to move things to a place that is more convenient for you, it is doable. The only thing I don't think you can move is /etc/portage itself. I think that has to be there. Other than that, move things where you need them to be. There are others who have placed theirs in other places and as long as you tell portage where it is at, you can adjust them as needed. Hope that extra info helps. Dale :-) :-)
Aw: Re: [gentoo-user] local mirrors (summary, leading to more questions)
Thanks to all who reponded: - Rich Freeman suggests: - having apache serve the local distfiles. How? Just make them available in the apache root and give portage the URL somewhere? - use "infra scripts", but I think that's for running an up-to-date general-purpose mirror He explains later that repos.conf defines the single place where the repository is defined but that is distinct from where the distfiles come from, which is defined by make.conf:GENTOO_MIRRORS - Dale suggests: - a package, http-replicator, which might do Freeman's first suggestion. - Neil Bothwick explains the difference between serving the "portage mirror, the repository of ebuild and associated files" and the distfiles, and suggests using NFS, presumably instead of, or in addition to /usr/portage/distfiles. Okay, I looked for the first time at the files directly under /usr/portage. Is it a correct assumption that all packages available for installation will be represented there? And that's it? That's basically all there is to a gentoo installation? - /usr/portage/distfiles - the ebuilds directly under /usr/portage - /usr/portage/eclass - /usr/portage/metadata - /usr/portage/profiles - /etc/portage If so, then I'm delighted in the simplicity of it. Like reading the one page definition of tcl. Unfortunately, it doesn't come through so well in the documentation, with things like layouts seemingly basic to an understanding. - Peter Humphrey suggests: - http ftp proxy In what way is that different from rsync which I thought I'd already configured? > Gesendet: Freitag, 26. April 2019 um 07:42 Uhr > Von: n952...@web.de > An: gentoo-user@lists.gentoo.org > Betreff: Aw: Re: [gentoo-user] local mirrors > > What I'm hoping for, though, is a configuration flag that causes any of: > - inhibits a theoretical traversal of repos.conf databases (e.g. also > /usr/share/porrage/...) > - inhibits trying to upgrade beyond what's already installed in the mirror > server > > > Gesendet: Freitag, 26. April 2019 um 07:32 Uhr > > Von: n952...@web.de > > An: gentoo-user@lists.gentoo.org > > Betreff: Aw: Re: [gentoo-user] local mirrors > > > > Thank you for the explanation. I wonder what the local mirror page means > > when it talks about saving bandwidth. What *does* it serve if not the > > distfiles? And when /etc/portage/repos.conf points to my local server, why > > would portage disregard that? > > > > The rsync server on the mirror host points to the gentoo portage > > installation on that local mirror host. How can any metadata there know > > about anything that's not already resolved there? > > > > At the very least, I suspect that that local mirror page is wrong and > > rather refers to something that *could* be implemented (without extra > > packages being installed, just by configuration), but isn't yet. > > > > > Gesendet: Freitag, 26. April 2019 um 00:35 Uhr > > > Von: "Rich Freeman" > > > An: gentoo-user@lists.gentoo.org > > > Betreff: Re: [gentoo-user] local mirrors > > > > > > On Thu, Apr 25, 2019 at 5:44 PM wrote: > > > > > > > > So, I set up the rsync deamon on my "mirror server" host and the > > > > /etc/portage/repos.conf/gentoo.conf file on the client machine, run the > > > > recommended test and everything is just as described. I sync my > > > > client, and am very happy. > > > > > > > > By coincidence, I happen to look in /var/log/emerge-fetch.log on the > > > > client and discover that everything got fetched remotely, from, like > > > > gentoo.org and rubygems.org, etc. > > > > > > > > What a disappointment! Is there something I still have to do? > > > > > > > > > > The repository and distfiles both need to be separately mirrored. > > > Maintaining a distfiles mirror is a bit more complex as all your hosts > > > probably don't have the same packages installed, and fetching > > > distfiles for all the packages would use a ton of space (and > > > bandwidth). For Gentoo official mirrors this is exactly what is done, > > > but of course they get used by many users. > > > > > > What I have done at times is run apache on one host and serve out its > > > local distfiles cache, and then list this as the first mirror in the > > > list for my other hosts. So, they try to fetch from that host before > > > going out to the internet. However, that doesn't do anything to > > > ensure that the needed files are on that host. It just helps with > > > @system packages and other packages the hosts have in common. That is > > > an approach that doesn't really cost you anything and probably > > > provides 75% of the benefit. It is also easy to do if you have a > > > bunch of identical hosts and then yields 100% of the benefit (if > > > they're truly identical I'd go a step further and set up a binpkg > > > mirror as there is no point in building the same thing many times with > > > the same flags).
