Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
This is broken and not going to go anywhere soon. I'll resubmit once fixed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/821#issuecomment-581905200___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
Closed #821. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/821#event-3006041141___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
> Evidently better way is to emit an error. No kidding? :joy: -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/821#issuecomment-533457065___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
> Yeah, known. Missing/wrong arguments to built-in macros are wildly > inconsistent in how they behave, some emit errors, some just fail silently > etc. Evidently better way is to emit an error. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/821#issuecomment-526562551___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
> * The expansion of macro %{getmem:virt}" finishes with an error. It is
> because getmem returns 0. This is caused by the fact that function
> getmem_virt returns SIZE_MAX. After dividing by 1024^2 it is still to high to
> fit into return_type of getmem (unsigned int).
Right. I probably only tested this in 32bit context where its more relevant.
There might be more similar issues lurking (it's only an RFC for a reason)
>
> * If there is a wrong parameter after '%{getmem:' , then no warning or
> error is emitted
Yeah, known. Missing/wrong arguments to built-in macros are wildly inconsistent
in how they behave, some emit errors, some just fail silently etc.
> * Why you use a new type of built-in macro synatx %{getmem:} instead of
> something close to the existing built-in macros like %getmem_avail?
It's not a new syntax, it's a built-in with an argument like several others. I
don't see a point of adding multiple macros that all return different aspects
of the same thing, that's what a parameter is for. Purely a matter of style of
course.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/821#issuecomment-526558715___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
I tested this PR on my laptop.
- The expansion of macro %{getmem:virt}" finishes with an error. It is because
getmem returns 0. This is caused by the fact that function getmem_virt returns
SIZE_MAX. After dividing by 1024^2 it is still to high to fit into return_type
of getmem (unsigned int).
- If there is a wrong parameter after '%{getmem:' , then no warning or error is
emitted
- Why you use a new type of built-in macro synatx %{getmem:} instead of
something close to the existing built-in macros like %getmem_avail?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/821#issuecomment-526549042___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
for the record, there are already packages like ceph or firefox that do their own heuristics, and I'm not sure it's always correct, so having a system way is definitely step in the right direction -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/821#issuecomment-525673262___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)
This PR is to address two different but related issues: 1) On 32bit systems, available virtual memory is in practise very limited and easy to exhaust on systems with many CPUs. (RhBug:1729382) 2) There are systems with lot of CPUs (virtual or otherwise) but relatively limited memory, where just looking at available CPUs causes severe trashing. (RhBug:1118734) This effectively caps the number of threads on 32bit to just four, but tunable by changing the thread size estimate. Number of processes is capped to gigabytes of memory or number of CPU's, whichever is smaller, and tunable by changing the process size estimate. The series adds a new macro primitive for getting system memory information and builds bunch of heuristics on top of those. There's a lot of subtleties involved and no doubt some of them I've gotten wrong here, so this PR is not intended for immediate merging, but more as a basis for discussion and other feedback. What bothers me personally here is that it adds quite a bit of brittle heuristics that we never needed before, heuristics that will inevitably go wrong. The 32bit thread issue could be handled by just slapping a hard limit, with just a couple of lines of code. OTOH, the many cpus but little memory -case is legit and solving does require heuristics no matter what. And since heuristics will go wrong sooner or later, there needs to be tunables, which is why so much of this is done in macro level despite being somewhat painful. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/821 -- Commit Summary -- * Move smp-related macros from platform to main macros * Add some macro primitives for retrieving system memory information * Limit build parallelism by available memory too, add tunables (#804) -- File Changes -- M build/parseSpec.c (2) M doc/manual/macros (7) M macros.in (27) M platform.in (11) M rpmio/macro.c (67) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/821.patch https://github.com/rpm-software-management/rpm/pull/821.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/821 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
