On Fri, Oct 09, 2026 at 06:33:44PM +0200, Helmut Grohne wrote: > Hi Adrian,
Hi Helmut, > On Fri, Oct 09, 2026 at 05:31:45PM +0300, Adrian Bunk wrote: > > Let me try to explain the severity of the problem caused by > > guess-concurrency. > > guess-concurrency is not breaking anything. It merely collects a pattern > that was vendored into several source packages into a binary package for > easier reuse. The guesswork (that people seem to be getting wrong) > remains with them. is guess-concurrency supposed to only replace various existing open coded implementations/limits, or is it intended that people add new guess-concurrency usage just because it exists? The former is an improvement, but the latter is what is actually happening. And this is the problem. > > IMHO the most important fix would be to make it clear in the > > documentation that guess-concurrency should only be used when > > ***really*** necessary. > > Expanding the documentation likely is a good thing. Unfortunately, we > first have to agree what guidance we want to give. What it means to be > really necessary is not that obvious. From my pov, it is necessary when > the build OOMs the machine. The tricky part is figuring out what the > lowest machine specs are that we want to support. Primarily we want to support our buildds. > Generally though, I agree that using guess-concurrency is more of an > exception as most packages should work with the default concurrency. There is a large overlap between the packages with huge build times, and the packages where the build uses a lot of memory. > > Sometimes guess-concurrency is used with an insanely high value > > (#1149937), people guessing numbers is just harmful. > > This is sad indeed. Currently in the archive: 3x --require-mem 4G 1x --require-mem 5G 1x --require-mem 8G (#1149937) 1x --require-mem 16G The only non-round number (5G) has a comment indicating that the maintainer tried to determine actual memory usage. > > From a decade of experience keeping packages alive on 32-bit I can tell > > that it is usually exactly one ld call or the one compilation of the > > swig-generated Python bindings for a C++ library that is the memory hog. > > I know about this behavior and took care of making guess-concurrency > somewhat support this common case. If you have a single large (e.g. 3GB) > link job and expect your other compiler invocations to not typically "not typically" or "not ever"? A semantic problem with guess-concurrency is that it is not clear whether it is supposed to estimate typical maximum usage, or whether it is supposed to set a hard maximum. The comment for the "--require-mem 5G" in nextpnr says: # Building the BBA files requires about 5GB in the worst case On a riscv64 buildds with 4 cores and 16 GB RAM I would guess that fastest might be to use all 4 cores when the worst case for a process is 5 GB, since it would be unusual that all processes reach the worst case maximum and reach it at the same time. nextpnr building on 32-bit architectures without any memory-saving tricks also makes me wonder how much margin above any actually observed number is in the "worst case" 5G value. The "--require-mem 5G" in nextpnr was added for OOM on the alpha qemu builders, this was not a problem on any release architecture. > exceed 1GB, you could pass --require-mem 3G,1G. If this is the final link then that is wrong, and --require-mem 3G,0,,1G would be correct. I'd call it a bug in guess_concurrency that it doesn't support the zero value needed here. > > Using guess-concurrency does not make much sense in this typical case. > > That's your view. My view is that suing guess-concurrency makes more > sense than having to manually retry your build several times due to it > being OOM killed. I've seen the vast majority of FTBFS on our buildds in the past 10 years, and builds failing due to being OOM killed on a release architecture happens basically never. Using guess_concurrency only when an actual problem has been observed would be good, but that's not what is happening. > > Packages where the maximum memory usage scales with the number of cores > > do exist, but they are rare. > > In my experience it is most packages that scale with the number of > cores. I think what you meant to imply here is "scales linearly". I > agree that a linear approximation often is a bad match, but an affine > approximation is accurate more often. Effectively, guess-concurrency > supports piecewise linear approximation - a generalization of affine > functions. The problem is not what guess-concurrency supports, the problem is how it is being used. > > On 32-bit architectures, requiring more memory per process than the > > address space available for a process is obviously wrong. > > This is a different problem not addressed by guess-concurrency in any > way. The package with "--require-mem 16G" builds successfully on sh4, where a process has 2 GB address space. > > In [1] the removal of guess-concurrency in my 9.6.2+ds-2+test1 package > > uses 6 instead of 2 cores on the buildd, resulting in a 2.76x build time > > difference. A 2.76x speedup for a 3x increase in the number of cores is > > as close to perfect scaling as might happen practice. > > That was using --require-mem=4G. And then to me there is a difference > between it building slowly and it requiring human intervention to build > at all. But building slowly is much worse than building faster, and that's what actually happened here. > > In practice we have to pick one of the following: > > 1. treat build time increases due to guess-concurrency as RC bugs, or > > 2. drop riscv64 as release architecture > > I don't follow how you reach this conclusion. riscv64 is already considered a problem by the release team due to buildd speed, there is no margin for even slower builds. > While debhelper implements > concurrent building by default, doing so still is not mandatory. Nearly all large packages do, and many of them were already using --parallel before compat 10 became available 10 years ago. > For > instance, sqlite3 still is annoyingly sequential and I haven't seen a > bug about this. Much less an rc one. Nor is sqlite3 being used as a > reason to demote riscv64. I did a lot of FTBFS reporting and fixing when packages switched to compat 10, and I do remember around ~1% of all packages having breakage due to that. For tiny old software where it was not trivial to fix a handwritten Makefile I was just adding --no-parallel to many packages. Some odd package like sqlite3 taking 3 hours instead of 1 hour on riscv64 is not a real problem. Many packages in two large transitions (fmtlib, boost1.92) are currently waiting for vtk9 to finish building before they can start building, and adding guess-concurrency usage did increase the build time of vtk9 from 4 days to 6 days. If that alone is not already the final straw for the release team to remove riscv64 from the forky release architectures, then an even larger buildd backlog due to increased build times for many packages where people add new guess-concurrency usage would do it. > What you probably see as a problem here is that pessimistic use of > guess-concurrency tends to clog the build queue. This has been a problem > before when guess-concurrency was open coded into debian/rules in > various ways. In my view, the fix is not to stop using it but to pass > more sensible memory guesses. No, the only reasonable fix is to stop guessing memory usage unless this is needed to actually fix an observed problem. > > And I mean 2. seriously, [2] increases the build time of a package > > currently involved in a transition from 4 to 6 days on the slower > > buildds due to only using 2 instead of 4 cores. Even on the new faster > > buildds[3] it would likely also push the build time above 2 days. > > The alternative that you do not highlight here is failing the build > after two days to have some developer give it back hoping that it does > not fail again. Not only does this waste buildd time, but also developer > time. I am not highlighting this because it does not happen. The first developer who sees a FTBFS on our buildds and looks at why it fails is very often me, and alpha is the only architecture where we sometimes have this problem. > > And riscv64 is just the tip of the iceberg: > > We spent a decade getting parallel builds working in nearly all packages, > > we don't want to lose a large part of that when people start modifying > > packages on a large scale with super-high guessed limits to > > guess-concurrency - requiring in the need for more buildds on multiple > > architectures. > > The vast majority of packages should not be using guess-concurrency. > Unfortunately, the ones where it does make sense are the ones that > typically take several days so any misuse delays builds a lot. Figuring > out what limits are sensible is a task that arises independently of > whether you implement it using guess-concurrency or not. That's only true when there was an actual problem observed that needed fixing. > > And you also want to be conservative regarding how to calculate the > > available memory. Not using all cores on a 4-core riscv64 buildd with > > 16 GB RAM because your proposed formula would use only 3 cores when > > an uploader used 4 GB as guessed value[4] - that's just bad. > > That's your view. My view is that less builds fail by OOMing the buildd > and thus *save* a lot of time. My observation is that builds failing by OOMing the buildd is not a problem we actually have on the buildds of release architectures. In the cases where this ever happened the maintainer already added a --max-parallel or -j years ago. Replacing a hardcoded or open coded value for that with guess-concurrency makes sense, but this is not how people are using guess-concurrency so far. > > Can we agree that guess-concurrency increasing build times is a bad > > thing that should usually result in an RC bug, or do you insist on > > killing riscv64? > > No, we actually disagree about this. > > Can we agree that the way guess-concurrency is presently being used in a > few packages is less than ideal? The 6 packages that are currently using guess-concurrency are not a promising start. > Note that the forky/sid manual page is expanded compared to trixie. It > now includes a ready example for the linker behavior. I guess that > guess that some users read the trixie one. Among the first 6 packages that use guess-concurrency is one that added "guess_concurrency --require-mem 16G" but builds on an architecture with 2 GB address space - when talking about how/when to use guess_concurrency it is important to keep in mind that the technical skills of many DDs are rather poor. Humans guessing numbers and then hardcoding them in debian/rules where they might stay unchanged for decades - that only makes sense when it is fixing an actual problem today. If replacing the hardcoded 1 in --max-parallel=1 in mlpack with something like "guess_concurrency --require-mem 4G" would reduce the build time on riscv64 from half a week to 1 day, then that would be a usage where guess-concurrency would bring real benefits. > Helmut cu Adrian
