Re: [gentoo-user] Is distfile partial mirror with failover possible?
On 5/9/23 22:58, Walter Dnes wrote: On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote: It looks like remote-mounting /var/cache/distfiles might be the quick-n-dirty solution like Alan suggested. And I never have a need to have emerge run simultaneously on more than one machine. It seems the simplest suitable solution for your needs. And while we're at it, howsabout "emerge --sync" on one host, and then remote-mounting /var/db/repos/gentoo ? Similarly - I was mounting a single location on moosefs (network filesystem) for ~15 hosts. One host updates using --sync. One buildhost for each architecture building packages. Distfiles are handled the same way. Then "emerge -k" on the remainder checking for packages that needed an actual build. If none, parallel merge (via ansible) or else single builds. Resulting packages are also stored remotely per arch (various amd64, arm32, arm64 etc.) Working well for a few years, but I am in the process of converting to nfs with a similar structure as I have decommissioned the moosefs system and many machines that are no longer needed. A single machine will handle built packages, distfiles and repos which are mounted on the other systems via nfs. I will look into apt-cacher-ng as I was a fan of http-replicator until it was deprecated. BillK
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Tuesday, 5 September 2023 16:33:43 BST Walter Dnes wrote: > On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote > > > On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote: > > > And while we're at it, howsabout "emerge --sync" on one host, and then > > > > > > remote-mounting /var/db/repos/gentoo ? > > > > Take a look here: > > > > https://wiki.gentoo.org/wiki/Local_Mirror > > That's what I do right now. I figure that remote-mounting might make > things easier. There's more work required to set it up than just rsyncd and NFS requires you run the local mirror all the time if the clients expect it to be up & running when they boot. signature.asc Description: This is a digitally signed message part.
RE: [gentoo-user] Is distfile partial mirror with failover possible?
> > > From: Alan McKinnon alan.mckin...@gmail.com<mailto:alan.mckin...@gmail.com> > Sent: Monday, September 4, 2023 12:38 PM > To: gentoo-user@lists.gentoo.org<mailto:gentoo-user@lists.gentoo.org> > Subject: Re: [gentoo-user] Is distfile partial mirror with failover possible? > > BTW Welcome back Alan, but leave your dirty top-posting in Archland :P > > Eh, I use gmail in the browser . the blerry thing is built to top post, > like Outlook > > > Alan > I'm in the same boat, but Notepad++ on Windows and Vim on Linux both support column-mode editing, so a couple quick cut-n-pastes and I can preserve the illusion that I'm using a sensible client. LMP
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote > On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote: > > > > And while we're at it, howsabout "emerge --sync" on one host, and then > > remote-mounting /var/db/repos/gentoo ? > > Take a look here: > > https://wiki.gentoo.org/wiki/Local_Mirror That's what I do right now. I figure that remote-mounting might make things easier. -- I've seen things, you people wouldn't believe; Gopher, Netscape with frames, the first Browser Wars. Searching for pages with AltaVista, pop-up windows self-replicating, trying to uninstall RealPlayer. All those moments, will be lost in time like tears in rain... time to die.
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Tue, 05 Sep 2023 16:07:16 +0100, Michael wrote: > > And while we're at it, howsabout "emerge --sync" on one host, and > > then remote-mounting /var/db/repos/gentoo ? > > Take a look here: > > https://wiki.gentoo.org/wiki/Local_Mirror I used to do that, the server syncing with Gentoo then everything else syncing with that. However, when I switched to git syncing, it was so fast that there was no point in running a local mirror - everything syncs directly from Github now. -- Neil Bothwick BBS: (n.) a system for connecting computers and exchanging gossip, facts, and uninformed speculation under false names. pgpswfLEI66eX.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote: > On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote > > > On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote: > > > It looks like remote-mounting /var/cache/distfiles might be the > > > > > > quick-n-dirty solution like Alan suggested. And I never have a need to > > > have emerge run simultaneously on more than one machine. > > > > It seems the simplest suitable solution for your needs. > > And while we're at it, howsabout "emerge --sync" on one host, and then > remote-mounting /var/db/repos/gentoo ? Take a look here: https://wiki.gentoo.org/wiki/Local_Mirror signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Tuesday, 5 September 2023 10:06:42 BST hitachi303 wrote: > Am 04.09.23 um 15:04 schrieb Michael: > > PS. Is http_replicator still available/maintained? I see the wiki > > mentions > > apt-cacher-ng for local distfiles cache. > > IIRC it died when python or perl 2.x was no longer available. It's worth mentioning another option, not involving a full caching proxy as such. It may suit Walter's workflow so I mention it here for completeness. A lightweight browser can serve files directly from the local mirror's own /var/ cache/distfiles. Unlike a caching proxy, files not already downloaded/needed by the local mirror won't be available locally, so the clients will download these from external mirrors. The local server will need to be updated first, to fetch all the latest source file versions. If there is a large difference in package requirements between different clients and the server, then only a partial bandwidth efficiency will be achieved this way. Also if all the clients require the same packages not available on the server, then these will be downloaded by each client separately from external mirrors, further eroding any savings in overall bandwidth. Setting up a small footprint browser like gatling, or just a couple of lines of python http.server, will allow any light on resources gentoo machine to become a useful local mirror with minimal configuration. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote > On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote: > > > It looks like remote-mounting /var/cache/distfiles might be the > > quick-n-dirty solution like Alan suggested. And I never have a need to > > have emerge run simultaneously on more than one machine. > > It seems the simplest suitable solution for your needs. And while we're at it, howsabout "emerge --sync" on one host, and then remote-mounting /var/db/repos/gentoo ? -- I've seen things, you people wouldn't believe; Gopher, Netscape with frames, the first Browser Wars. Searching for pages with AltaVista, pop-up windows self-replicating, trying to uninstall RealPlayer. All those moments, will be lost in time like tears in rain... time to die.
Re: [gentoo-user] Is distfile partial mirror with failover possible?
Am 04.09.23 um 15:04 schrieb Michael: PS. Is http_replicator still available/maintained? I see the wiki mentions apt-cacher-ng for local distfiles cache. IIRC it died when python or perl 2.x was no longer available.
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote: > On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote > > I use apt-cacher-ng for this and it does what you are looking for. > > https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says... > > > Open issues > > > > * Apt-cacher-ng installs a cron job to delete unreferenced files > > from the cache. Since the Gentoo cache contains no index files, > > this probably deletes either everything or nothing from it. > > Have you run into ptoblems with this? Not that I've noticed. > It looks like remote-mounting /var/cache/distfiles might be the > quick-n-dirty solution like Alan suggested. And I never have a need to > have emerge run simultaneously on more than one machine. It seems the simplest suitable solution for your needs. -- Neil Bothwick A pessimist complains about the noise when opportunity knocks. pgpofPXiYb_oS.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > > The reference to a host makes me think Walter wants one machne to hold > the distfiles for all on the network. Correct. > I use apt-cacher-ng for this and it does what you are looking for. https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says... > Open issues > > * Apt-cacher-ng installs a cron job to delete unreferenced files > from the cache. Since the Gentoo cache contains no index files, > this probably deletes either everything or nothing from it. Have you run into ptoblems with this? It looks like remote-mounting /var/cache/distfiles might be the quick-n-dirty solution like Alan suggested. And I never have a need to have emerge run simultaneously on more than one machine. -- I've seen things, you people wouldn't believe; Gopher, Netscape with frames, the first Browser Wars. Searching for pages with AltaVista, pop-up windows self-replicating, trying to uninstall RealPlayer. All those moments, will be lost in time like tears in rain... time to die.
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, 4 Sept 2023 at 22:02, Mark Knecht wrote: > I'm a GMail user also. Sadly you'll want to not only bottom post, but also > select all text you're responding to, remove formatting (Ctrl-V) and then > type your response or you'll be down voted for responding in HTML. > > I hate it also, but this list is easily one of my favorites and I'm no longer > a Gentoo user. I also use both GMail and Outlook, and you can set up GMail to be plain text by default. It's not too difficult to click the three dots and type your reply where it belongs. Not so easy in Outlook which is actively hating its users. Regards, Arve
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, Sep 4, 2023 at 12:38 PM Alan McKinnon wrote: > > > > On Mon, Sep 4, 2023 at 8:26 PM Neil Bothwick wrote: >> >> On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote: >> >> > Quick n dirty solution: >> > >> > put all distfiles on a central server >> > FS mount that remote dir to /var/cache/distfiles on all hosts >> >> That's what I used to do, but had problems when downloading the same file >> from to clients at once. >> >> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P >> > > Eh, I use gmail in the browser . the blerry thing is built to top post, like Outlook > > > Alan > I'm a GMail user also. Sadly you'll want to not only bottom post, but also select all text you're responding to, remove formatting (Ctrl-V) and then type your response or you'll be down voted for responding in HTML. I hate it also, but this list is easily one of my favorites and I'm no longer a Gentoo user. Best wishes, Mark
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, Sep 4, 2023 at 8:26 PM Neil Bothwick wrote: > On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote: > > > Quick n dirty solution: > > > > put all distfiles on a central server > > FS mount that remote dir to /var/cache/distfiles on all hosts > > That's what I used to do, but had problems when downloading the same file > from to clients at once. > > BTW Welcome back Alan, but leave your dirty top-posting in Archland :P > > Eh, I use gmail in the browser . the blerry thing is built to top post, like Outlook Alan
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote: > Quick n dirty solution: > > put all distfiles on a central server > FS mount that remote dir to /var/cache/distfiles on all hosts That's what I used to do, but had problems when downloading the same file from to clients at once. BTW Welcome back Alan, but leave your dirty top-posting in Archland :P > > Alan > > On Mon, Sep 4, 2023 at 7:21 PM Neil Bothwick wrote: > > > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > > > > > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote: > > > > I may be misunderstanding, but it seems to me that local > > > > mirrors are all-or-nothing. In the interests of saving > > > > bandwidth, I'd like to have a client first check the host's > > > > /var/cache/distfiles directory for a source tarball file. If not > > > > found, then fail over to another mirror as per GENTOO_MIRRORS in > > > > /etc/portage/make.conf. This would require emerge doing the > > > > lookup and potential failover for each file. Is this possible? > > > > > > Unless I misunderstand what you're asking, isn't this what takes > > > place anyway? > > > > The reference to a host makes me think Walter wants one machne to hold > > the distfiles for all on the network. > > > > > PS. Is http_replicator still available/maintained? I see the wiki > > > mentions apt-cacher-ng for local distfiles cache. > > > > I use apt-cacher-ng for this and it does what you are looking for. > > > > > > -- > > Neil Bothwick > > > > The facts, although interesting, are usually irrelevant. > > > > -- Neil Bothwick Not one shred of evidence supports the notion that life is serious. pgpEycCmOOrWx.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is distfile partial mirror with failover possible?
Quick n dirty solution: put all distfiles on a central server FS mount that remote dir to /var/cache/distfiles on all hosts Alan On Mon, Sep 4, 2023 at 7:21 PM Neil Bothwick wrote: > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > > > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote: > > > I may be misunderstanding, but it seems to me that local mirrors are > > > all-or-nothing. In the interests of saving bandwidth, I'd like to > > > have a client first check the host's /var/cache/distfiles directory > > > for a source tarball file. If not found, then fail over to another > > > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would > > > require emerge doing the lookup and potential failover for each file. > > > Is this possible? > > > > Unless I misunderstand what you're asking, isn't this what takes place > > anyway? > > The reference to a host makes me think Walter wants one machne to hold > the distfiles for all on the network. > > > PS. Is http_replicator still available/maintained? I see the wiki > > mentions apt-cacher-ng for local distfiles cache. > > I use apt-cacher-ng for this and it does what you are looking for. > > > -- > Neil Bothwick > > The facts, although interesting, are usually irrelevant. > -- Alan McKinnon alan dot mckinnon at gmail dot com
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote: > > I may be misunderstanding, but it seems to me that local mirrors are > > all-or-nothing. In the interests of saving bandwidth, I'd like to > > have a client first check the host's /var/cache/distfiles directory > > for a source tarball file. If not found, then fail over to another > > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would > > require emerge doing the lookup and potential failover for each file. > > Is this possible? > > Unless I misunderstand what you're asking, isn't this what takes place > anyway? The reference to a host makes me think Walter wants one machne to hold the distfiles for all on the network. > PS. Is http_replicator still available/maintained? I see the wiki > mentions apt-cacher-ng for local distfiles cache. I use apt-cacher-ng for this and it does what you are looking for. -- Neil Bothwick The facts, although interesting, are usually irrelevant. pgpmQhUbItn3g.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> I may be misunderstanding, but it seems to me that local mirrors are
> all-or-nothing. In the interests of saving bandwidth, I'd like to have
> a client first check the host's /var/cache/distfiles directory for a
> source tarball file. If not found, then fail over to another mirror as
> per GENTOO_MIRRORS in /etc/portage/make.conf. This would require emerge
> doing the lookup and potential failover for each file. Is this possible?
Unless I misunderstand what you're asking, isn't this what takes place anyway?
Emerge will look at /var/cache/distfiles and if the source tarball is missing,
will try to fetch it from the list of GENTOO_MIRRORS you have configured. For
example, if you have:
GENTOO_MIRRORS="http://192.168.2.35/my_gentoo_distfiles/ https://
www.mirrorservice.org/sites/distfiles.gentoo.org/"
any host configured as above will first look at its own /var/cache/distfiles,
then at the first configured mirror which happens to be in the same LAN, then
whatever other external mirrors you have added. You could of course export
your local mirror's LAN ${DISTDIR} via NFS for your hosts to use.
PS. Is http_replicator still available/maintained? I see the wiki mentions
apt-cacher-ng for local distfiles cache.
signature.asc
Description: This is a digitally signed message part.
[gentoo-user] Is distfile partial mirror with failover possible?
I may be misunderstanding, but it seems to me that local mirrors are all-or-nothing. In the interests of saving bandwidth, I'd like to have a client first check the host's /var/cache/distfiles directory for a source tarball file. If not found, then fail over to another mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would require emerge doing the lookup and potential failover for each file. Is this possible? -- I've seen things, you people wouldn't believe; Gopher, Netscape with frames, the first Browser Wars. Searching for pages with AltaVista, pop-up windows self-replicating, trying to uninstall RealPlayer. All those moments, will be lost in time like tears in rain... time to die.
