Re: [gentoo-user] Is distfile partial mirror with failover possible?

[William Kenworthy]

On 5/9/23 22:58, Walter Dnes wrote:

On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote

On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:


   It looks like remote-mounting /var/cache/distfiles might be the
quick-n-dirty solution like Alan suggested.  And I never have a need to
have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.

   And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

Similarly - I was mounting a single location on moosefs (network 
filesystem) for ~15 hosts.  One host updates using --sync.  One 
buildhost for each architecture building packages.  Distfiles are 
handled the same way. Then "emerge -k" on the remainder checking for 
packages that needed an actual build.  If none, parallel merge (via 
ansible) or else single builds.  Resulting packages are also stored 
remotely per arch (various amd64, arm32, arm64 etc.)


Working well for a few years, but I am in the process of converting to 
nfs with a similar structure as I have decommissioned the moosefs system 
and many machines that are no longer needed.  A single machine will 
handle built packages, distfiles and repos which are mounted on the 
other systems via nfs. I will look into apt-cacher-ng as I was a fan of 
http-replicator until it was deprecated.


BillK

Re: [gentoo-user] tar exclude files question

[Walter Dnes]

On Tue, Sep 05, 2023 at 08:36:00PM +0200, Alan McKinnon wrote
> 
> Try it and see. My concern is that the man page implies that with
> --exclude-caches-under the subdirectories are excluded recursively,
> but the directory with the file called CACHEDIR.TAG is not.  I'm sure
> that's wrong but the man page says what it says.

  Starting off with...

[x8940][waltdnes][~] ll .cache
total 68
drwx--  12 waltdnes users  4096 Sep  5 17:45  .
drwxr-xr-x 141 waltdnes users 20480 Sep  5 18:38  ..
-rw-r--r--   1 waltdnes users44 Sep  5 17:42  CACHEDIR.TAG
drwx--   2 waltdnes users  4096 Aug 29 15:34  babl
drwxr-xr-x   2 waltdnes users  4096 Sep  5 08:52  fontconfig
drwxr-xr-x   3 waltdnes users  4096 Jun 11  2021  geeqie
drwx--   3 waltdnes users  4096 May 29  2021  gegl-0.4
drwxr-xr-x   3 waltdnes users  4096 May 29  2021  gimp
drwx--   3 waltdnes users  4096 May 27  2021  google-chrome
drwx--   2 waltdnes users  4096 Sep  5 10:23  mc
drwxr-xr-x 258 waltdnes users  4096 Mar 24  2022  mesa_shader_cache
drwx--   3 waltdnes users  4096 Sep  5 17:12 'moonchild productions'
drwx--   3 waltdnes users  4096 Nov 16  2021  thumbnails

tar --exclude-caches -cvzf x1.tgz .cache

...archives the following...

drwx-- waltdnes/users0 2023-09-05 17:45 .cache/
-rw-r--r-- waltdnes/users   44 2023-09-05 17:42 .cache/CACHEDIR.TAG

tar --exclude-caches-all -cvzf x2.tgz .cache

...archives absolutely nothing; zip... zilch... nada

tar --exclude-caches-under -cvzf x3.tgz .cache

...archives...

drwx-- waltdnes/users0 2023-09-05 17:45 .cache/

  I'll gladly take any of them.  One more thing; in this mode, you
*MUST* use a leading minus for "-cvzf".  The lazy "cvzf" will *NOT*
work, and throws a misleading error message.  See also my post to
Michael about the contents of the CACHEDIR.TAG file.

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] tar exclude files question

[Walter Dnes]

On Tue, Sep 05, 2023 at 07:38:54PM +0100, Michael wrote
> 
> Have a look at this page which explains what you need to do:
> 
> https://bford.info/cachedir/

  Thank you! Thank you! Thank you!  That page explains that any random
CACHEDIR.TAG file won't suffice, and why my attempts were all failing.

This file must be an ordinary file, not for example a symbolic link.
Additionally, the first 43 octets of this file *MUST* *MUST* *MUST*
consist of the following ASCII header string:

Signature: 8a477f597d28d172789f06886806bc55

  Otherwise it wil *NOT* work.  This *NOT* mentioned in "man tar".
https://www.gnu.org/software/tar/manual/tar.html#index-cachedir mentions
http://www.brynosaurus.com/cachedir/spec.html in passing, which
redirects to https://bford.info/cachedir/

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] tar exclude files question

[Dale]

Michael wrote:
> On Tuesday, 5 September 2023 16:28:31 BST Walter Dnes wrote:
>>   Every so often I tar up my home directory on my main machine, and push
>> it over to my "hot backup" machine, and then do a tap-dance with the
>> .ssh directory.  I notice oodles of cache files being tarred.  Do I
>> understand the man page correctly about the CACHEDIR.TAG "magic file"?
>> Assume I have a .cache directory like so...
>>
>> [x8940][waltdnes][~] ll .cache
>> total 64
>> drwx--  12 waltdnes users  4096 Sep  5 10:50  .
>> drwxr-xr-x 141 waltdnes users 20480 Sep  5 10:58  ..
>> -rw-r--r--   1 waltdnes users 0 Sep  5 10:50  CACHEDIR.TAG
>> drwx--   2 waltdnes users  4096 Aug 29 15:34  babl
>> drwxr-xr-x   2 waltdnes users  4096 Sep  5 08:52  fontconfig
>> drwxr-xr-x   3 waltdnes users  4096 Jun 11  2021  geeqie
>> drwx--   3 waltdnes users  4096 May 29  2021  gegl-0.4
>> drwxr-xr-x   3 waltdnes users  4096 May 29  2021  gimp
>> drwx--   3 waltdnes users  4096 May 27  2021  google-chrome
>> drwx--   2 waltdnes users  4096 Sep  5 10:23  mc
>> drwxr-xr-x 258 waltdnes users  4096 Mar 24  2022  mesa_shader_cache
>> drwx--   3 waltdnes users  4096 May 25  2021 'moonchild productions'
>> drwx--   3 waltdnes users  4096 Nov 16  2021  thumbnails
>>
>>   Would a script in /home like...
>>
>> #!/bin/bash
>> tar --exclude-caches-under cvzf wdexport.tgz waltdnes
>>
>> ...skip files in that directory?  I don't mind a few empty directories.
> Have a look at this page which explains what you need to do:
>
> https://bford.info/cachedir/
>
> Alternatively, use something like:
>
> --exclude=".cache/*"
>
> for top level .cache directories.


If it helps any, that looks like the way rsync does that as well.  It
could be they use the same coding even tho they do different things.  It
would be funny if the same person helps code both.  lol

Just thought it worth a mention. 

Dale

:-)  :-)

Re: [gentoo-user] tar exclude files question

[Michael]

On Tuesday, 5 September 2023 16:28:31 BST Walter Dnes wrote:
>   Every so often I tar up my home directory on my main machine, and push
> it over to my "hot backup" machine, and then do a tap-dance with the
> .ssh directory.  I notice oodles of cache files being tarred.  Do I
> understand the man page correctly about the CACHEDIR.TAG "magic file"?
> Assume I have a .cache directory like so...
> 
> [x8940][waltdnes][~] ll .cache
> total 64
> drwx--  12 waltdnes users  4096 Sep  5 10:50  .
> drwxr-xr-x 141 waltdnes users 20480 Sep  5 10:58  ..
> -rw-r--r--   1 waltdnes users 0 Sep  5 10:50  CACHEDIR.TAG
> drwx--   2 waltdnes users  4096 Aug 29 15:34  babl
> drwxr-xr-x   2 waltdnes users  4096 Sep  5 08:52  fontconfig
> drwxr-xr-x   3 waltdnes users  4096 Jun 11  2021  geeqie
> drwx--   3 waltdnes users  4096 May 29  2021  gegl-0.4
> drwxr-xr-x   3 waltdnes users  4096 May 29  2021  gimp
> drwx--   3 waltdnes users  4096 May 27  2021  google-chrome
> drwx--   2 waltdnes users  4096 Sep  5 10:23  mc
> drwxr-xr-x 258 waltdnes users  4096 Mar 24  2022  mesa_shader_cache
> drwx--   3 waltdnes users  4096 May 25  2021 'moonchild productions'
> drwx--   3 waltdnes users  4096 Nov 16  2021  thumbnails
> 
>   Would a script in /home like...
> 
> #!/bin/bash
> tar --exclude-caches-under cvzf wdexport.tgz waltdnes
> 
> ...skip files in that directory?  I don't mind a few empty directories.

Have a look at this page which explains what you need to do:

https://bford.info/cachedir/

Alternatively, use something like:

--exclude=".cache/*"

for top level .cache directories.

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] tar exclude files question

[Alan McKinnon]

On Tue, Sep 5, 2023 at 5:28 PM Walter Dnes  wrote:

>   Every so often I tar up my home directory on my main machine, and push
> it over to my "hot backup" machine, and then do a tap-dance with the
> .ssh directory.  I notice oodles of cache files being tarred.  Do I
> understand the man page correctly about the CACHEDIR.TAG "magic file"?
> Assume I have a .cache directory like so...
>
> [x8940][waltdnes][~] ll .cache
> total 64
> drwx--  12 waltdnes users  4096 Sep  5 10:50  .
> drwxr-xr-x 141 waltdnes users 20480 Sep  5 10:58  ..
> -rw-r--r--   1 waltdnes users 0 Sep  5 10:50  CACHEDIR.TAG
> drwx--   2 waltdnes users  4096 Aug 29 15:34  babl
> drwxr-xr-x   2 waltdnes users  4096 Sep  5 08:52  fontconfig
> drwxr-xr-x   3 waltdnes users  4096 Jun 11  2021  geeqie
> drwx--   3 waltdnes users  4096 May 29  2021  gegl-0.4
> drwxr-xr-x   3 waltdnes users  4096 May 29  2021  gimp
> drwx--   3 waltdnes users  4096 May 27  2021  google-chrome
> drwx--   2 waltdnes users  4096 Sep  5 10:23  mc
> drwxr-xr-x 258 waltdnes users  4096 Mar 24  2022  mesa_shader_cache
> drwx--   3 waltdnes users  4096 May 25  2021 'moonchild productions'
> drwx--   3 waltdnes users  4096 Nov 16  2021  thumbnails
>
>   Would a script in /home like...
>
> #!/bin/bash
> tar --exclude-caches-under cvzf wdexport.tgz waltdnes
>
> ...skip files in that directory?  I don't mind a few empty directories.
>

Try it and see. My concern is that the man page implies that with
--exclude-caches-under the subdirectories are excluded recursively, but the
directory with the file called CACHEDIR.TAG is not.
I'm sure that's wrong but the man page says what it says.

-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Michael]

On Tuesday, 5 September 2023 16:33:43 BST Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote
> 
> > On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> > >   And while we're at it, howsabout "emerge --sync" on one host, and then
> > > 
> > > remote-mounting /var/db/repos/gentoo ?
> > 
> > Take a look here:
> > 
> > https://wiki.gentoo.org/wiki/Local_Mirror
> 
>   That's what I do right now.  I figure that remote-mounting might make
> things easier.

There's more work required to set it up than just rsyncd and NFS requires you 
run the local mirror all the time if the clients expect it to be up & running 
when they boot.

signature.asc
Description: This is a digitally signed message part.

RE: [gentoo-user] Is distfile partial mirror with failover possible?

[Laurence Perkins]

>
>
> From: Alan McKinnon alan.mckin...@gmail.com
> Sent: Monday, September 4, 2023 12:38 PM
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Is distfile partial mirror with failover possible?
>
> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P
>
> Eh, I use gmail in the browser . the blerry thing is built to top post, 
> like Outlook
>
>
> Alan
>

I'm in the same boat, but Notepad++ on Windows and Vim on Linux both support 
column-mode editing, so a couple quick cut-n-pastes and I can preserve the 
illusion that I'm using a sensible client.

LMP

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Walter Dnes]

On Tue, Sep 05, 2023 at 04:07:16PM +0100, Michael wrote
> On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> > 
> >   And while we're at it, howsabout "emerge --sync" on one host, and then
> > remote-mounting /var/db/repos/gentoo ?
> 
> Take a look here:
> 
> https://wiki.gentoo.org/wiki/Local_Mirror

  That's what I do right now.  I figure that remote-mounting might make
things easier.

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

[gentoo-user] tar exclude files question

[Walter Dnes]

  Every so often I tar up my home directory on my main machine, and push
it over to my "hot backup" machine, and then do a tap-dance with the
.ssh directory.  I notice oodles of cache files being tarred.  Do I
understand the man page correctly about the CACHEDIR.TAG "magic file"?
Assume I have a .cache directory like so...

[x8940][waltdnes][~] ll .cache
total 64
drwx--  12 waltdnes users  4096 Sep  5 10:50  .
drwxr-xr-x 141 waltdnes users 20480 Sep  5 10:58  ..
-rw-r--r--   1 waltdnes users 0 Sep  5 10:50  CACHEDIR.TAG
drwx--   2 waltdnes users  4096 Aug 29 15:34  babl
drwxr-xr-x   2 waltdnes users  4096 Sep  5 08:52  fontconfig
drwxr-xr-x   3 waltdnes users  4096 Jun 11  2021  geeqie
drwx--   3 waltdnes users  4096 May 29  2021  gegl-0.4
drwxr-xr-x   3 waltdnes users  4096 May 29  2021  gimp
drwx--   3 waltdnes users  4096 May 27  2021  google-chrome
drwx--   2 waltdnes users  4096 Sep  5 10:23  mc
drwxr-xr-x 258 waltdnes users  4096 Mar 24  2022  mesa_shader_cache
drwx--   3 waltdnes users  4096 May 25  2021 'moonchild productions'
drwx--   3 waltdnes users  4096 Nov 16  2021  thumbnails

  Would a script in /home like...

#!/bin/bash
tar --exclude-caches-under cvzf wdexport.tgz waltdnes

...skip files in that directory?  I don't mind a few empty directories.

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Neil Bothwick]

On Tue, 05 Sep 2023 16:07:16 +0100, Michael wrote:

> >   And while we're at it, howsabout "emerge --sync" on one host, and
> > then remote-mounting /var/db/repos/gentoo ?  
> 
> Take a look here:
> 
> https://wiki.gentoo.org/wiki/Local_Mirror

I used to do that, the server syncing with Gentoo then everything else
syncing with that. However, when I switched to git syncing, it was so
fast that there was no point in running a local mirror - everything syncs
directly from Github now.


-- 
Neil Bothwick

BBS: (n.) a system for connecting computers and exchanging gossip,
 facts, and uninformed speculation under false names.


pgpswfLEI66eX.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Michael]

On Tuesday, 5 September 2023 15:58:05 BST Walter Dnes wrote:
> On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
> 
> > On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
> > >   It looks like remote-mounting /var/cache/distfiles might be the
> > > 
> > > quick-n-dirty solution like Alan suggested.  And I never have a need to
> > > have emerge run simultaneously on more than one machine.
> > 
> > It seems the simplest suitable solution for your needs.
> 
>   And while we're at it, howsabout "emerge --sync" on one host, and then
> remote-mounting /var/db/repos/gentoo ?

Take a look here:

https://wiki.gentoo.org/wiki/Local_Mirror


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Michael]

On Tuesday, 5 September 2023 10:06:42 BST hitachi303 wrote:
> Am 04.09.23 um 15:04 schrieb Michael:
> > PS. Is http_replicator still available/maintained?  I see the wiki
> > mentions
> > apt-cacher-ng for local distfiles cache.
> 
> IIRC it died when python or perl 2.x was no longer available.

It's worth mentioning another option, not involving a full caching proxy as 
such.  It may suit Walter's workflow so I mention it here for completeness.  A 
lightweight browser can serve files directly from the local mirror's own /var/
cache/distfiles.

Unlike a caching proxy, files not already downloaded/needed by the local 
mirror won't be available locally, so the clients will download these from 
external mirrors.  The local server will need to be updated first, to fetch 
all the latest source file versions.  If there is a large difference in 
package requirements between different clients and the server, then only a 
partial bandwidth efficiency will be achieved this way.  Also if all the 
clients require the same packages not available on the server, then these will 
be downloaded by each client separately from external mirrors, further eroding 
any savings in overall bandwidth.

Setting up a small footprint browser like gatling, or just a couple of lines 
of python http.server, will allow any light on resources gentoo machine to 
become a useful local mirror with minimal configuration.

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Walter Dnes]

On Tue, Sep 05, 2023 at 08:04:19AM +0100, Neil Bothwick wrote
> On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:
> 
> >   It looks like remote-mounting /var/cache/distfiles might be the
> > quick-n-dirty solution like Alan suggested.  And I never have a need to
> > have emerge run simultaneously on more than one machine.
> 
> It seems the simplest suitable solution for your needs.

  And while we're at it, howsabout "emerge --sync" on one host, and then
remote-mounting /var/db/repos/gentoo ?

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[hitachi303]

Am 04.09.23 um 15:04 schrieb Michael:

PS. Is http_replicator still available/maintained?  I see the wiki mentions
apt-cacher-ng for local distfiles cache.


IIRC it died when python or perl 2.x was no longer available.

Re: [gentoo-user] Is distfile partial mirror with failover possible?

[Neil Bothwick]

On Mon, 4 Sep 2023 22:54:38 -0400, Walter Dnes wrote:

> On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote

> > I use apt-cacher-ng for this and it does what you are looking for.  
> 
>   https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says...
> 
> > Open issues
> >
> > *  Apt-cacher-ng installs a cron job to delete unreferenced files
> > from the cache. Since the Gentoo cache contains no index files,
> > this probably deletes either everything or nothing from it.  
> 
>   Have you run into ptoblems with this?

Not that I've noticed.

>   It looks like remote-mounting /var/cache/distfiles might be the
> quick-n-dirty solution like Alan suggested.  And I never have a need to
> have emerge run simultaneously on more than one machine.

It seems the simplest suitable solution for your needs.


-- 
Neil Bothwick

A pessimist complains about the noise when opportunity knocks.


pgpofPXiYb_oS.pgp
Description: OpenPGP digital signature