Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, Sep 04, 2023 at 06:21:27PM +0100, Neil Bothwick wrote > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > > The reference to a host makes me think Walter wants one machne to hold > the distfiles for all on the network. Correct. > I use apt-cacher-ng for this and it does what you are looking for. https://wiki.gentoo.org/wiki/Local_distfiles_cache#Open_issues says... > Open issues > > * Apt-cacher-ng installs a cron job to delete unreferenced files > from the cache. Since the Gentoo cache contains no index files, > this probably deletes either everything or nothing from it. Have you run into ptoblems with this? It looks like remote-mounting /var/cache/distfiles might be the quick-n-dirty solution like Alan suggested. And I never have a need to have emerge run simultaneously on more than one machine. -- I've seen things, you people wouldn't believe; Gopher, Netscape with frames, the first Browser Wars. Searching for pages with AltaVista, pop-up windows self-replicating, trying to uninstall RealPlayer. All those moments, will be lost in time like tears in rain... time to die.
Re: [gentoo-user] dev-db/mysql fails to build, more than one version.
Michael Orlitzky wrote: > On Sun, 2023-09-03 at 09:35 -0500, Dale wrote: >> Anyone else having this? Is this mysql or is something else causing >> this and mysql is just a symptom? Given two versions are failing to >> build, is kinda interesting. > https://bugs.gentoo.org/912797 > > > > A comment on the bug report helped me work around the issue. I kinda figured something else was causing it besides mysql. Just one version, ok, bad ebuild or something. Two versions, something bigger. Thanks. Dale :-) :-)
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, 4 Sept 2023 at 22:02, Mark Knecht wrote: > I'm a GMail user also. Sadly you'll want to not only bottom post, but also > select all text you're responding to, remove formatting (Ctrl-V) and then > type your response or you'll be down voted for responding in HTML. > > I hate it also, but this list is easily one of my favorites and I'm no longer > a Gentoo user. I also use both GMail and Outlook, and you can set up GMail to be plain text by default. It's not too difficult to click the three dots and type your reply where it belongs. Not so easy in Outlook which is actively hating its users. Regards, Arve
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, Sep 4, 2023 at 12:38 PM Alan McKinnon wrote: > > > > On Mon, Sep 4, 2023 at 8:26 PM Neil Bothwick wrote: >> >> On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote: >> >> > Quick n dirty solution: >> > >> > put all distfiles on a central server >> > FS mount that remote dir to /var/cache/distfiles on all hosts >> >> That's what I used to do, but had problems when downloading the same file >> from to clients at once. >> >> BTW Welcome back Alan, but leave your dirty top-posting in Archland :P >> > > Eh, I use gmail in the browser . the blerry thing is built to top post, like Outlook > > > Alan > I'm a GMail user also. Sadly you'll want to not only bottom post, but also select all text you're responding to, remove formatting (Ctrl-V) and then type your response or you'll be down voted for responding in HTML. I hate it also, but this list is easily one of my favorites and I'm no longer a Gentoo user. Best wishes, Mark
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, Sep 4, 2023 at 8:26 PM Neil Bothwick wrote: > On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote: > > > Quick n dirty solution: > > > > put all distfiles on a central server > > FS mount that remote dir to /var/cache/distfiles on all hosts > > That's what I used to do, but had problems when downloading the same file > from to clients at once. > > BTW Welcome back Alan, but leave your dirty top-posting in Archland :P > > Eh, I use gmail in the browser . the blerry thing is built to top post, like Outlook Alan
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, 4 Sep 2023 19:49:56 +0200, Alan McKinnon wrote: > Quick n dirty solution: > > put all distfiles on a central server > FS mount that remote dir to /var/cache/distfiles on all hosts That's what I used to do, but had problems when downloading the same file from to clients at once. BTW Welcome back Alan, but leave your dirty top-posting in Archland :P > > Alan > > On Mon, Sep 4, 2023 at 7:21 PM Neil Bothwick wrote: > > > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > > > > > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote: > > > > I may be misunderstanding, but it seems to me that local > > > > mirrors are all-or-nothing. In the interests of saving > > > > bandwidth, I'd like to have a client first check the host's > > > > /var/cache/distfiles directory for a source tarball file. If not > > > > found, then fail over to another mirror as per GENTOO_MIRRORS in > > > > /etc/portage/make.conf. This would require emerge doing the > > > > lookup and potential failover for each file. Is this possible? > > > > > > Unless I misunderstand what you're asking, isn't this what takes > > > place anyway? > > > > The reference to a host makes me think Walter wants one machne to hold > > the distfiles for all on the network. > > > > > PS. Is http_replicator still available/maintained? I see the wiki > > > mentions apt-cacher-ng for local distfiles cache. > > > > I use apt-cacher-ng for this and it does what you are looking for. > > > > > > -- > > Neil Bothwick > > > > The facts, although interesting, are usually irrelevant. > > > > -- Neil Bothwick Not one shred of evidence supports the notion that life is serious. pgpEycCmOOrWx.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is distfile partial mirror with failover possible?
Quick n dirty solution: put all distfiles on a central server FS mount that remote dir to /var/cache/distfiles on all hosts Alan On Mon, Sep 4, 2023 at 7:21 PM Neil Bothwick wrote: > On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > > > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote: > > > I may be misunderstanding, but it seems to me that local mirrors are > > > all-or-nothing. In the interests of saving bandwidth, I'd like to > > > have a client first check the host's /var/cache/distfiles directory > > > for a source tarball file. If not found, then fail over to another > > > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would > > > require emerge doing the lookup and potential failover for each file. > > > Is this possible? > > > > Unless I misunderstand what you're asking, isn't this what takes place > > anyway? > > The reference to a host makes me think Walter wants one machne to hold > the distfiles for all on the network. > > > PS. Is http_replicator still available/maintained? I see the wiki > > mentions apt-cacher-ng for local distfiles cache. > > I use apt-cacher-ng for this and it does what you are looking for. > > > -- > Neil Bothwick > > The facts, although interesting, are usually irrelevant. > -- Alan McKinnon alan dot mckinnon at gmail dot com
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Mon, 04 Sep 2023 14:04:53 +0100, Michael wrote: > On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote: > > I may be misunderstanding, but it seems to me that local mirrors are > > all-or-nothing. In the interests of saving bandwidth, I'd like to > > have a client first check the host's /var/cache/distfiles directory > > for a source tarball file. If not found, then fail over to another > > mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would > > require emerge doing the lookup and potential failover for each file. > > Is this possible? > > Unless I misunderstand what you're asking, isn't this what takes place > anyway? The reference to a host makes me think Walter wants one machne to hold the distfiles for all on the network. > PS. Is http_replicator still available/maintained? I see the wiki > mentions apt-cacher-ng for local distfiles cache. I use apt-cacher-ng for this and it does what you are looking for. -- Neil Bothwick The facts, although interesting, are usually irrelevant. pgpmQhUbItn3g.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is distfile partial mirror with failover possible?
On Monday, 4 September 2023 11:12:51 BST Walter Dnes wrote:
> I may be misunderstanding, but it seems to me that local mirrors are
> all-or-nothing. In the interests of saving bandwidth, I'd like to have
> a client first check the host's /var/cache/distfiles directory for a
> source tarball file. If not found, then fail over to another mirror as
> per GENTOO_MIRRORS in /etc/portage/make.conf. This would require emerge
> doing the lookup and potential failover for each file. Is this possible?
Unless I misunderstand what you're asking, isn't this what takes place anyway?
Emerge will look at /var/cache/distfiles and if the source tarball is missing,
will try to fetch it from the list of GENTOO_MIRRORS you have configured. For
example, if you have:
GENTOO_MIRRORS="http://192.168.2.35/my_gentoo_distfiles/ https://
www.mirrorservice.org/sites/distfiles.gentoo.org/"
any host configured as above will first look at its own /var/cache/distfiles,
then at the first configured mirror which happens to be in the same LAN, then
whatever other external mirrors you have added. You could of course export
your local mirror's LAN ${DISTDIR} via NFS for your hosts to use.
PS. Is http_replicator still available/maintained? I see the wiki mentions
apt-cacher-ng for local distfiles cache.
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-user] Re: attic
On Mon, Sep 4, 2023 at 2:36 PM Rich Freeman wrote: > On Mon, Sep 4, 2023 at 4:38 AM William Kenworthy > wrote: > > > > On 4/9/23 16:04, Nuno Silva wrote: > > > > > > (But note that Rich was suggesting using the *search* feature of the > > > gitweb interface, which, in this case, also finds the same topmost > > > commit if I search for "reedsolomon".) > > > > > tkx, missed that! > > Note that in terms of indexing git and CVS have their pros and cons, > because they use different data structures. I've heard the saying > that Git is a data structure masquerading as an SCM, and certainly the > inconsistencies in the command line operations bear that out. > I'd always heard that Git is a file system and all useful side effects are pure luck -- Alan McKinnon alan dot mckinnon at gmail dot com
Re: [gentoo-user] Re: attic
On Mon, Sep 4, 2023 at 4:38 AM William Kenworthy wrote: > > On 4/9/23 16:04, Nuno Silva wrote: > > > > (But note that Rich was suggesting using the *search* feature of the > > gitweb interface, which, in this case, also finds the same topmost > > commit if I search for "reedsolomon".) > > > tkx, missed that! Note that in terms of indexing git and CVS have their pros and cons, because they use different data structures. I've heard the saying that Git is a data structure masquerading as an SCM, and certainly the inconsistencies in the command line operations bear that out. Git tends to be much more useful in general, but for things like finding deleted files CVS was definitely more time-efficient. The reason for this is that everything in git is reachable via commits, and these are reachable from a head via a linked list. The most recent commit gives access to the current version of the repository, and a pointer to the immediately previous commit(s). To find a deleted file, git must go to the most recent commit in whatever branch you are searching, then descend its tree to look for the file. If it is not found, it then goes to the previous commit and descends that tree. There are 745k commits in the active Gentoo repository. I think there are something like 2M of them in the historical one. Each commit is a random seek, and then each step down the directory tree to find a file is another random seek. In CVS everything is organized first by file, and then each file has its own commit history. So finding a file, deleted or otherwise, just requires a seek for each level in the directory tree. Then you can directly read its history. So finding an old deleted file in the gentoo git repo can require millions of reads, while doing so in CVS only required about 3. It is no surprise that the web interfaces were designed to make that operation much easier - if you do sufficiently complex searches in the git web interface it will time you out to avoid bogging down the server, which is why some searches may require you to clone the repo and do it locally. Now, if you want to find out what changed in a particular commit the situation is reversed. If you identify a commit in git and want to see what changed, it can directly read the commit from disk using its hash. It then looks at the parent commit, then descends both trees doing a diff at each level. Since everything is content-hashed only directory trees that contain differences need to be read. If a commit had changes to 50 files, it might only take 10 reads to figure out which files changed, and then another 100 to compare the contents of each file and generate diffs. If you wanted to do that in CVS you'd have to read every single file in the repository and read the sequential history of each file to find any commits that have the same time/author. CVS commits also aren't atomic so ordering across files might not be the same. Git is a thing of beauty when you think about what it was designed to do and how well-suited to this design its architecture is. The same can be said of several data-driven FOSS applications. The right algorithm can make a huge difference... -- Rich
[gentoo-user] Is distfile partial mirror with failover possible?
I may be misunderstanding, but it seems to me that local mirrors are all-or-nothing. In the interests of saving bandwidth, I'd like to have a client first check the host's /var/cache/distfiles directory for a source tarball file. If not found, then fail over to another mirror as per GENTOO_MIRRORS in /etc/portage/make.conf. This would require emerge doing the lookup and potential failover for each file. Is this possible? -- I've seen things, you people wouldn't believe; Gopher, Netscape with frames, the first Browser Wars. Searching for pages with AltaVista, pop-up windows self-replicating, trying to uninstall RealPlayer. All those moments, will be lost in time like tears in rain... time to die.
Re: [gentoo-user] Re: attic
On 4/9/23 16:04, Nuno Silva wrote: On 2023-09-04, William Kenworthy wrote: On 3/9/23 18:29, Rich Freeman wrote: On Sun, Sep 3, 2023 at 4:44 AM Michael wrote: On Sunday, 3 September 2023 07:49:36 BST William Kenworthy wrote: Hi , I used to be able to get old ebuilds from "the attic" but I cant find it on google - is it still around? Perhaps have a look here at the archives? https://gitweb.gentoo.org/ The archives will only contain data migrated from CVS - so only things from more than a few years ago. You want to look into the main repo for anything recently deleted. [...] This can be done via the website, though the search capability is a little limited. I ended up having to search from a local clone because your package name contains an error and the web search found nothing. To find your file, go to: https://gitweb.gentoo.org/repo/gentoo.git/ Go to the search box in the top right and search for: dev-python/reedsolomon (note that the package category is different from what was in your email) Find the commit one commit before the one that removed your package. (ie one that contains your package in its most recent version) If you find the one that deleted your file, then just look at the parent in the commit header and click on that to go back one version where it is still present. Click the tree hash to browse the historical version of the repository that existed before your file was deleted. For example, you can find v1.6.1 of that package at: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/reedsolomon/reedsolomon-1.6.1.ebuild?id=149a131188ebce76a87fd8363fb212f5f1620a02 [...] The web git interface is capable of displaying past commits. It just can't search for wildcards/etc. Thanks Rich, unfortunately the web interface isn't helpful - I cant just navigate the tree to find commits - "https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/reedsolomon/"; gives path not found - it looks like you have to know the commit first by downloading the git tree to search it - not friendly at all! With /log/ instead of /tree/ in the URL it at least shows the list of commits. From a quick check, this seems to include the commit removing the directory when it's removed instead of renamed, so hopefully it helps too with retrieving older ebuilds? (But note that Rich was suggesting using the *search* feature of the gitweb interface, which, in this case, also finds the same topmost commit if I search for "reedsolomon".) tkx, missed that! BillK
[gentoo-user] Re: attic
On 2023-09-04, William Kenworthy wrote: > On 3/9/23 18:29, Rich Freeman wrote: >> On Sun, Sep 3, 2023 at 4:44 AM Michael wrote: >>> On Sunday, 3 September 2023 07:49:36 BST William Kenworthy wrote: Hi , I used to be able to get old ebuilds from "the attic" but I cant find it on google - is it still around? >>> Perhaps have a look here at the archives? >>> >>> https://gitweb.gentoo.org/ >> The archives will only contain data migrated from CVS - so only things >> from more than a few years ago. >> >> You want to look into the main repo for anything recently deleted. [...] >> This can be done via the website, though the search capability is a >> little limited. I ended up having to search from a local clone >> because your package name contains an error and the web search found >> nothing. >> >> To find your file, go to: >> https://gitweb.gentoo.org/repo/gentoo.git/ >> Go to the search box in the top right and search for: >> dev-python/reedsolomon (note that the package category is >> different from what was in your email) >> Find the commit one commit before the one that removed your package. >> (ie one that contains your package in its most recent version) If you >> find the one that deleted your file, then just look at the parent in >> the commit header and click on that to go back one version where it is >> still present. >> Click the tree hash to browse the historical version of the repository >> that existed before your file was deleted. >> For example, you can find v1.6.1 of that package at: >> https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/reedsolomon/reedsolomon-1.6.1.ebuild?id=149a131188ebce76a87fd8363fb212f5f1620a02 [...] >> The web git interface is capable of displaying past commits. It just >> can't search for wildcards/etc. >> > Thanks Rich, > > unfortunately the web interface isn't helpful - I cant just navigate > the tree to find commits - > "https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/reedsolomon/"; > gives path not found - it looks like you have to know the commit first > by downloading the git tree to search it - not friendly at all! With /log/ instead of /tree/ in the URL it at least shows the list of commits. From a quick check, this seems to include the commit removing the directory when it's removed instead of renamed, so hopefully it helps too with retrieving older ebuilds? (But note that Rich was suggesting using the *search* feature of the gitweb interface, which, in this case, also finds the same topmost commit if I search for "reedsolomon".) -- Nuno Silva
