On 6/20/24 8:46 AM, Peter Humphrey wrote: > Hello list, > > While building a new KDE system (see my post a few minutes ago), I'm finding > the system stalling because it can't handle all its install jobs. I have this > set: > > $ grep '\-j' /etc/portage/make.conf > EMERGE_DEFAULT_OPTS="--jobs --load-average=30 [...]" > MAKEOPTS="-j16 -l16" > > The CPU has 24 threads and 64GB RAM, and lots of swap space, and those values > have worked well for some time. Now, though, I'm going to have to limit the > --jobs or the --load-average. > > On interrupting one such hang, I found that 32 install jobs had been waiting > to run; is this limit hard coded? I also saw "too many jobs" or something, > and > "could not read job counter". > > Is it now bug-report time?
It's not clear to me what "stalling" means here. Did portage stop doing any work as verified by ps or htop? Did it just spend a long time showing no progress? I do know what the 32 install jobs "waiting to run" is though. Or at least I'm pretty sure I know what it means. Recent portage has this change: https://gitweb.gentoo.org/proj/portage.git/commit/?id=825db01b91a37dcd9890ee5bf9f462ea524ac5cc "Add merge-wait FEATURES setting enabled by default" From the changelog: portage-3.0.62 (2024-02-22) -------------- * FEATURES: Add FEATURES="merge-wait", enabled by default, to control whether we do parallel merges of images to the live filesystem (bug #663324). If enabled, we serialize these merges. For now, this makes FEATURES="parallel-install" a no-op, but in future, it will be improved to allow parallel merges, just not while any packages are compiling. The purpose of this change is to solve cases where in one thread, portage is modifying /usr by deleting and installing files in a package such as "dev-python/setuptools-scm", while in another thread it is running a program that attempts to use those files and crashes because they are half-installed. When enabled (by default), portage will still compile lots of packages in parallel. But only when no packages are currently compiling, will they actually be merged to ROOT. Until then, they are in pending-merge state, and take up space in /var/tmp/portage. You can disable this FEATURES if you want. In https://bugs.gentoo.org/934382 portage is adding additional options: --jobs-merge-wait-threshold=X will cause portage to stop starting new jobs when X number of packages are in pending-merge state, and portage will copy the installed package files from the image to the root filesystem. Otherwise, portage will get there eventually but it might take a bit longer. -- Eli Schwartz
OpenPGP_0x84818A6819AF4A9B.asc
Description: OpenPGP public key
OpenPGP_signature.asc
Description: OpenPGP digital signature
