Re: Aw: Re: Re: [gentoo-user] local mirrors (summary, leading to more questions)
On Monday, 29 April 2019 22:50:18 BST n952...@web.de wrote: > But that seems strange - why would I need both GENTOO_MIRRORS and > http_proxy? > > GENTOO_MIRRORS="http://myserver; > http_proxy="http://myserver:3142 > > Does the http_proxy imply that I'd need a proxy app, like squid. Between my > client and server, there won't be any appreciable traffic. I think there are two different concepts here, which have been conflated. You do not need a proxy server, unless you intend to set up a caching server. There are two configurations you could set up in your home network, depending what functionality you intend to achieve: 1. Local portage and local distfiles servers. You can set up an rsync server for portage's files and a web or ftp server for distfiles. This works well with identical systems. When your local server uses the same packages as your clients the corresponding distfiles will only be downloaded once from the Gentoo source mirrors to the local server and all clients will fetch them via web or ftp within your LAN, as and when they need them. However, when some of your clients have different packages installed, or in any case different than the local server, then each client will need to download such source files from the Internet. Say you have a server with no Gnome packages, but 10 out of 30 client PCs run a Gnome desktop. Then each and every of these 10 clients will try to fetch the Gnome distfiles from the local server, get a 404 and then try the next server you have specified for GENTOO_MIRRORS in their /etc/portage/make.conf. This way you end up downloading large source files multiple times, which defeats the purpose of running a local mirror. So, this set up works best for few LAN clients which have mostly the same packages installed as the local Gentoo server and quickly becomes suboptimal as the number of different packages x clients increases. A variation on the above is using rsync to duplicate the distfiles directory between server and client, or NFS/CIFS to share the same folder across PCs. 2. Local portage and local caching distfiles servers. In this configuration the portage files are served over rsync as above. The distfiles however are served via a caching proxy server. Using the previous example, when some clients want to install different packages they will always go via the local http_proxy server. If the server has cached said packages it will serve these to the requesting clients immediately. Otherwise, the server will fetch the missing packages from the Gentoo mirrors once only, cache them locally and then serve these to whichever client requested them. There are many different web servers you could use to serve distfiles. I have used boa for many years because the server machine did not have a web/ftp server already installed and the boa web server is exceedingly fast with a very small footprint. However, if you already have apache running, then it would make sense you use that. There are also many proxy caching servers you could choose. Squid has already been mentioned, net-misc/apt-cacher-ng is also mentioned in the wiki pages. HTH. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: Aw: Re: Re: [gentoo-user] local mirrors (summary, leading to more questions)
On Monday, 29 April 2019 22:50:18 BST n952...@web.de wrote: > But that seems strange - why would I need both GENTOO_MIRRORS and > http_proxy? > > GENTOO_MIRRORS="http://myserver; > http_proxy="http://myserver:3142 No. Here are my entries in make.conf: GENTOO_MIRRORS=" https://mirror.bytemark.co.uk/gentoo/ http://mirror.bytemark.co.uk/gentoo/ https://mirrors.evowise.com/gentoo/ http://mirrors.evowise.com/gentoo/ https://www.mirrorservice.org/sites/distfiles.gentoo.org/ http://www.mirrorservice.org/sites/distfiles.gentoo.org/ " http_proxy="http://192.168.1.2:3128; > Does the http_proxy imply that I'd need a proxy app, like squid. Yes; it's meant for people who're sitting behind a firewall and can't see the Internet. The proxy runs on the firewall, and you tell your apps on client machines to use the proxy when they need something via http. > Between my client and server, there won't be any appreciable traffic. The converse. Your client sends a request to the proxy, giving the destination address; the proxy sends the request on, but itself supplies any bits it already has. Thus, if portage is asking for a large distfile, such as kernel sources, it will come directly from the proxy if it's already been fetched for another machine. Don't ask me how the proxy synchronises itself with the current state of the target, as in a complex web page with both static and dynamic objects, because I don't know, not having investigated it - I just use it happily. Sorry :) -- Regards, Peter.
Aw: Re: Re: [gentoo-user] local mirrors (summary, leading to more questions)
But that seems strange - why would I need both GENTOO_MIRRORS and http_proxy? GENTOO_MIRRORS="http://myserver; http_proxy="http://myserver:3142 Does the http_proxy imply that I'd need a proxy app, like squid. Between my client and server, there won't be any appreciable traffic. > Gesendet: Montag, 29. April 2019 um 23:39 Uhr > Von: n952...@web.de > An: gentoo-user@lists.gentoo.org > Betreff: Aw: Re: Re: [gentoo-user] local mirrors (summary, leading to more > questions) > > > Gesendet: Montag, 29. April 2019 um 10:47 Uhr > > Von: "Peter Humphrey" > > An: gentoo-user@lists.gentoo.org > > Betreff: Re: Aw: Re: [gentoo-user] local mirrors (summary, leading to more > > questions) > > > > On Sunday, 28 April 2019 21:37:19 BST n952...@web.de wrote: > > > > > - Peter Humphrey suggests: > > > - http ftp proxy > > > > > > In what way is that different from rsync which I thought I'd > > > already > > > configured? > > > > I assume that means you're using rsync to synchronise the portage database. > > > Yes, as instructed here: > > https://wiki.gentoo.org/wiki/Local_Mirror#Setting_up_the_mirror > > > > Then, whatever you use to fetch distfiles for installation, it uses ftp or > > http transport to fetch them. ... > > > This page discusses local distfile servers: > >https://wiki.gentoo.org/wiki/Local_distfiles_cache > > using either of two approaches: > - apt-cacher-ng > - nginx > > Do they relate to the ftp and http you mention? > > I already have apache and don't want to install any more packages - it would > seem that if nginx can serve the files, apache should be able to, too. > I'm trying to configure it. > > This page: > https://wiki.gentoo.org/wiki/Local_distfiles_cache#Setting_up_clients > > says I should put this line into /etc/portage/make.conf: > > http_proxy="http://distfilescache:3142 > > Is that the URL it'll send? So, all I'd have to do is have a virtual host > responding to that port with a copy of /etc/portage/distfiles at its root? > > TIA, > > Tom > > > > > > > > > > -- > > Regards, > > Peter. > >
Aw: Re: Re: [gentoo-user] local mirrors (summary, leading to more questions)
> Gesendet: Montag, 29. April 2019 um 10:47 Uhr > Von: "Peter Humphrey" > An: gentoo-user@lists.gentoo.org > Betreff: Re: Aw: Re: [gentoo-user] local mirrors (summary, leading to more > questions) > > On Sunday, 28 April 2019 21:37:19 BST n952...@web.de wrote: > > > - Peter Humphrey suggests: > > - http ftp proxy > > > > In what way is that different from rsync which I thought I'd already > > configured? > > I assume that means you're using rsync to synchronise the portage database. Yes, as instructed here: https://wiki.gentoo.org/wiki/Local_Mirror#Setting_up_the_mirror > Then, whatever you use to fetch distfiles for installation, it uses ftp or > http transport to fetch them. ... This page discusses local distfile servers: https://wiki.gentoo.org/wiki/Local_distfiles_cache using either of two approaches: - apt-cacher-ng - nginx Do they relate to the ftp and http you mention? I already have apache and don't want to install any more packages - it would seem that if nginx can serve the files, apache should be able to, too. I'm trying to configure it. This page: https://wiki.gentoo.org/wiki/Local_distfiles_cache#Setting_up_clients says I should put this line into /etc/portage/make.conf: http_proxy="http://distfilescache:3142 Is that the URL it'll send? So, all I'd have to do is have a virtual host responding to that port with a copy of /etc/portage/distfiles at its root? TIA, Tom > > -- > Regards, > Peter.
