Hi Helmut, On 2026-10-11 19:17, Helmut Grohne wrote: > Hi Aurelien, > > On Sun, Oct 11, 2026 at 05:08:50PM +0200, Aurelien Jarno wrote: > > I think most of the confusion comes from here. I see two main use cases: > > Thank you for spelling out the use cases. > > > 1) Avoid an OOM on a typical developer laptop with a lot of of cores, > > but not that much memory, especially given there are many other > > processes running. In that case we the number of jobs should really > > determined from the maximum memory used by a single process during the > > build. > > Given that I build very many packages locally (and some OOM), this > definitely is a use case I had in mind. > > Note that Santiago Vila has been initially using quite small VMs for his > archive rebuilds and encountered quite some failures that way. I'd like > us to not only consider Debian's buildds as relevant machines for > building but making builds just work locally such that users are enabled > to easily modify packages.
I understand this use case, as long as it doesn't mean having to significantly increase the resources we need at the build daemon level. Hosting build daemons is difficult those days. And buying additional hardware even more. > > 2) Optimize the build time by limiting the use of swap for active > > processes. In that case the amount of memory is just a typical maximum > > usage. We accept that some gcc/g++ process will just use swap if it > > still improves the overall build time. This is what we used on the > > buildd so far, they have enough swap that an OOM is almost impossible > > (at least for the official buildds). > > This is not a use case I had in mind, but it sounds sensible to me. Note > that when you get to a point where compilers are being swapped, you > typically already swapped out all of tmpfs. I guess my own machines > simple have insufficient swap. On a system with 32GB swap a fair number > of packages stopped building about three years ago. 64GB swap likely is > insufficient today. Does that match your numbers? It seems you are mixing swap and size of the tmpfs. In normal use case, you need to have at minimum as much swap as the size of your tmpfs used for building. Then you need extra swap for handling the cases were big processes run concurrently, even if the goal is to limit that case as much as possible. The minimum disk size needed to build all packages (being ext4, tmpfs or other fs), is around 150G on arm64 (for the linux package), and around 100GB on other architectures for linux, rustc and/or gcc-16. Therefore the tmpfs is sized to 150GB on all architectures except s390x where it is 100GB (as we currently do not have enough more diskk space). Then we configured the swap as tmpfs size + 20GB, so 170GB on all architectures, except s390x where it is 120GB. > > It is not fully clear what use case guess-concurrency is designed for. I > > would say for the scenario 2, given it doesn't take the swap into > > account (except for backing tmpfs with the change in git) and it returns > > 1 instead of failing if requiring more memory than available. > > I fear it is more complex than this. We need to consider two variants of > the use cases here. One is when the build tree is backed by tmpfs and > the other is when it is not. Note that guess-concurrency does not > presently distinguish them. When using tmpfs, a lot of RAM can be > occupied with page cache. Several packages use more than 64GB of build > trees. This needs to fit into swap. So in the tmpfs case, the assumption > kinda is that swap is able to hold all of tmpfs usage and it is not > accounted otherwise. When not using tmpfs, swap could be accounted > indeed. Though as you point out in 2) doing so slows down the build at > some point. We just use tmpfs because it's fast, doesn't need special file system option or wrapper like eatmydata to optimize the speed, doesn't break and always come clean after a reboot. In addition it allows a significant part of the packages to be built without disk I/O, an for other packages to have the I/O performed slowly in the background. Finally even if it lacks some features, it's an easy way to standardize the filesystem and ensure reproducibility. But to me the question of swap vs memory vs number of jobs, has to be decorrelated from tmpfs, the same issue exists with an ext4 build directory. At the end you should just ensure that the swap covers at least the size of the tmpfs. Seen otherwise, just subtract the size of the tmpfs from the swap to see how much memory can be overcommitted for the running processes. > > I then looked at a practical case, the vtk9 package. It makes almost > > 11000 calls to gcc/g++ with an average of 215 MiB, and a median of > > 171 MiB. Only 22 calls use more than 2GiB and only 3 calls use more > > than 4 GiB, with a maximum of 6.1 GiB. In the scenario 1) we probably > > want to compute the number of jobs using 6.1 GiB, while in the scenario > > 2) we probably want to compute the number of jobs using 2 GiB or less as > > the gain of doing most of the build with more jobs it not lost by > > *possibly* running a few big jobs in parallel. > > Tricky. I don't think avoiding swapping at all cost is the way to go. A > pessimistic parameter would be --require-mem 6g,5g,,4g,2g. On a machine > with 16GB, this would likely produce a concurrency of 2. That quite > definitely is not an ideal outcome. You are trying to define the required memory from 0.03% of the GCC calls (admittedly a bit more if we look at the CPU time). And I only looked at all the calls, not the makefiles. Maybe there is no chance at all that they run concurrently. > I have to admit that I'm not quite sure where to go from here. Quite > evidently, the model that I used is too simplistic and the way it is > being used in Debian causes harm to riscv64. I'm sorry about that. riscv64 is currently the most affected, and to be fair, we need to upgrade the hardware asap, but it's not always easy. That said if more packages starting using guess-concurrency in a bad way, other architectures will get affected relatively quickly, as it tends to be used in big packages. > Last but not least swap is not all the same. Some build machines swap to > rotating storage (maybe none of Debian's) while most swap to NVMe. We only have 2 build daemons with rotating storage. All the other are backed by NVMe, except s390x which is backed by amazingly fast storage. But at the end, even in the vtk9 case, the goal is to not get the running process to get swapped. It's just there for the unlikely case where the 3 bigs jobs end up running in parallel. > I shall upload the pending change soon to provide some relief, but I > doubt it to be sufficient. This bug will be marked as closed. Don't > hesitate to reopen or open a new one if you think we need to change > more. But we also need to change e.g. src:openbabel as --require-mem 16G > is not sensible. Thanks, that's already a significant improvement. Regards Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B [email protected] http://aurel32.net
