This PR is to address two different but related issues:
1) On 32bit systems, available virtual memory is in practise very limited and
easy to exhaust on systems with many CPUs. (RhBug:1729382)
2) There are systems with lot of CPUs (virtual or otherwise) but relatively
limited memory, where just looking at available CPUs causes severe trashing.
(RhBug:1118734)
This effectively caps the number of threads on 32bit to just four, but tunable
by changing the thread size estimate. Number of processes is capped to
gigabytes of memory or number of CPU's, whichever is smaller, and tunable by
changing the process size estimate.
The series adds a new macro primitive for getting system memory information and
builds bunch of heuristics on top of those. There's a lot of subtleties
involved and no doubt some of them I've gotten wrong here, so this PR is not
intended for immediate merging, but more as a basis for discussion and other
feedback.
What bothers me personally here is that it adds quite a bit of brittle
heuristics that we never needed before, heuristics that will inevitably go
wrong. The 32bit thread issue could be handled by just slapping a hard limit,
with just a couple of lines of code. OTOH, the many cpus but little memory
-case is legit and solving does require heuristics no matter what. And since
heuristics will go wrong sooner or later, there needs to be tunables, which is
why so much of this is done in macro level despite being somewhat painful.
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/821
-- Commit Summary --
* Move smp-related macros from platform to main macros
* Add some macro primitives for retrieving system memory information
* Limit build parallelism by available memory too, add tunables (#804)
-- File Changes --
M build/parseSpec.c (2)
M doc/manual/macros (7)
M macros.in (27)
M platform.in (11)
M rpmio/macro.c (67)
-- Patch Links --
https://github.com/rpm-software-management/rpm/pull/821.patch
https://github.com/rpm-software-management/rpm/pull/821.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/821
_______________________________________________
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
