Hi Aurelien, On Sun, Oct 11, 2026 at 05:08:50PM +0200, Aurelien Jarno wrote: > I think most of the confusion comes from here. I see two main use cases:
Thank you for spelling out the use cases. > 1) Avoid an OOM on a typical developer laptop with a lot of of cores, > but not that much memory, especially given there are many other > processes running. In that case we the number of jobs should really > determined from the maximum memory used by a single process during the > build. Given that I build very many packages locally (and some OOM), this definitely is a use case I had in mind. Note that Santiago Vila has been initially using quite small VMs for his archive rebuilds and encountered quite some failures that way. I'd like us to not only consider Debian's buildds as relevant machines for building but making builds just work locally such that users are enabled to easily modify packages. > 2) Optimize the build time by limiting the use of swap for active > processes. In that case the amount of memory is just a typical maximum > usage. We accept that some gcc/g++ process will just use swap if it > still improves the overall build time. This is what we used on the > buildd so far, they have enough swap that an OOM is almost impossible > (at least for the official buildds). This is not a use case I had in mind, but it sounds sensible to me. Note that when you get to a point where compilers are being swapped, you typically already swapped out all of tmpfs. I guess my own machines simple have insufficient swap. On a system with 32GB swap a fair number of packages stopped building about three years ago. 64GB swap likely is insufficient today. Does that match your numbers? > It is not fully clear what use case guess-concurrency is designed for. I > would say for the scenario 2, given it doesn't take the swap into > account (except for backing tmpfs with the change in git) and it returns > 1 instead of failing if requiring more memory than available. I fear it is more complex than this. We need to consider two variants of the use cases here. One is when the build tree is backed by tmpfs and the other is when it is not. Note that guess-concurrency does not presently distinguish them. When using tmpfs, a lot of RAM can be occupied with page cache. Several packages use more than 64GB of build trees. This needs to fit into swap. So in the tmpfs case, the assumption kinda is that swap is able to hold all of tmpfs usage and it is not accounted otherwise. When not using tmpfs, swap could be accounted indeed. Though as you point out in 2) doing so slows down the build at some point. > I then looked at a practical case, the vtk9 package. It makes almost > 11000 calls to gcc/g++ with an average of 215 MiB, and a median of > 171 MiB. Only 22 calls use more than 2GiB and only 3 calls use more > than 4 GiB, with a maximum of 6.1 GiB. In the scenario 1) we probably > want to compute the number of jobs using 6.1 GiB, while in the scenario > 2) we probably want to compute the number of jobs using 2 GiB or less as > the gain of doing most of the build with more jobs it not lost by > *possibly* running a few big jobs in parallel. Tricky. I don't think avoiding swapping at all cost is the way to go. A pessimistic parameter would be --require-mem 6g,5g,,4g,2g. On a machine with 16GB, this would likely produce a concurrency of 2. That quite definitely is not an ideal outcome. I have to admit that I'm not quite sure where to go from here. Quite evidently, the model that I used is too simplistic and the way it is being used in Debian causes harm to riscv64. I'm sorry about that. Last but not least swap is not all the same. Some build machines swap to rotating storage (maybe none of Debian's) while most swap to NVMe. I shall upload the pending change soon to provide some relief, but I doubt it to be sufficient. This bug will be marked as closed. Don't hesitate to reopen or open a new one if you think we need to change more. But we also need to change e.g. src:openbabel as --require-mem 16G is not sensible. Thanks for bearing with me Helmut
