> ---------------------------------------- > From: Robert Clausecker <f...@freebsd.org> > Date: Oct 24, 2023, 12:12:13 PM > To: <ports@freebsd.org> > Subject: We need to do something about build times > > > The build times have gone up to the point where they are unsustainable. > Frequent updates to key ports (like llvm*, rust, gcc*) make it so that > basically every time I prepare a new batch of commits, I have to rebuild > a variety of toolchain ports across 8 jails (amd64/i386/arm64/armv7 each > for FreeBSD 12.4 and 13.2). This takes multiple days. And I'm working > with hardware that's quite recent (for x86, an 8 thread Skylake box, for > arm, an 8 thread Windows 2023 dev kit). > > By the time the builds are done, some random update has usually caused > the ports to be out of date again, so if I were to rebase, I would have > to do all of this again. And again. And again. > > Particularly bad offenders are gcc and rust. Ccache is ineffective for > these as gcc has LTO turned on, which seems to more than triple the > regular build time to more than 24 hours even on a fast Skylake box. > This is single threaded as I build multiple ports at once; if I were to > build multi-threaded, the same amount of total CPU hours would have been > spent, so that would not fix my problem. Ccache is also ineffective for > rust of course. > > There's another issue in that ccache doesn't scale to large cache sizes > (my experiments show that anything larger than 20 GB seems to cause > problems as ccache repeatedly tries to scan the whole thing for evictions), > and the sizes that work are just not enough to be effective. What would > help is being able to have one cache for each combination of ports tree > and jail, but Poudriere has no support for that. > > Another bad offender is texlive. For some reason, texlive-texmf needs to > be rebuilt frequently, despite mostly comprising data that is just > unpacked and repacked. This takes forever and pegs the disk at 100% for > more than an hour as the texlive source tarball is repeatedly extracted > and then compressed into packages. I don't get why the texlive stuff is > not split in such a way that the stuff that is just repacked lives in its > own port with no dependencies so it only needs to be rebuilt on rare > texlive updates. > > And it seems I'm slowly killing my build SSD like that. After just about > 9 months, it is already at 100 TB of writes just from port builds. > Building with workdirs in memory is no longer an option as that frequently > kills my build server by filling all its RAM with build files until no > processes can be started anymore. Poudriere does not have an effective > mechanism to prevent this (tmpfs limits don't work as the ports in > question require very large workdirs, tend to take very long to build and > tend to be built all at the same time for multiple jails). > > Using prebuilt packages is not an option as they lag behind by several > days/weeks and lead to an inconsistent testing environment. It is also > not a good solution to chose non-default build options for these ports > as it is not clear if that would affect the validity of the testbuilds. > > How can we fix this problem and make ports development sustainable again? > > Some ideas: > > - disable LTO and other options by default that increase build times by > such a ridiculous degree. This would really make a huge impact with > very little work. I don't think LTO on toolchain ports improves build > times enough in comparison to the extra time it takes to build these. > > - for gcc, switch to single or no bootstrap by default. We have known > good toolchains we use to build gcc. There's really no reason to > build it multiple times just out of paranoia. The maintainer is > supposed to check that gcc is built correctly without bootstrapping > so consumers don't need to build it multiple times. > > - untangle some of the dependencies so that less ports may trigger > rebuilds of critical ports. For example, llvm docs could be moved to > separate ports so that updates in the documentation toolchain do not > trigger an LLVM rebuild. > > - reduce USES to chose lighter dependencies by default. E.g. USES=llvm > could depend on the light flavour by default. I'm sure only very few > ports need all of LLVM and the light flavour is faster to build. > > - rework Poudriere's rebuild detection to not rebuild every port for > every random bullshit thing. For example, I don't see why ports need > to be rebuilt for transitive changes in build dependencies. E.g. if > port A has build depends on port B which build depends on port C, and > C is updated, then A has to be rebuilt despite its direct dependencies > being unchanged. This does not appear to be reasonable. > > - unbundle libraries more thoroughly. We currently have dozens of > copies of LLVM, skia, webkit, and others in tree as ports just bundle > them instead of even making an attempt at unbundling. This means that > every time they need to be patched, it's a whackamole at finding all > copies. Plus build times suffer a lot. I know it's hard, but perhaps > something can be done. For example, I have given up on trying to make > electron work on armv7 as with every major version update, my patches > are randomly being dropped and I have to do it all again. Like all > chromium ports, electron takes over two days to build on my arm box > and my time is insufficient for that. > > - stop bulk bumping RUN_DEPENDS consumers when dependencies are updated, > or at least think carefully before doing so. RUN_DEPENDS are only > installed after the build and should not affect the build. For > example, sysutils/cdrtools uses the command line opus encoder and thus > depends on audio/opus. There is absolutely no reason to bump it when > audio/opus is updated. It just causes everybody to needlessly rebuild > and reinstall ports. Sure there's the odd case where that needs to be > done, but it seems like some maintainers just always do that, even > when it's not needed. > > - maybe add a system where ports can declare the oldest version of > themselves they are compatible to, in the sense that consumers only > need to be rebuilt if they were built against a version older than > that. For example, if a shared library is updated with a bug fix > that does not change the ABI, there's no need to rebuild all consumers. > > With great frustration, > Robert Clausecker > > -- > () ascii ribbon campaign - for an 8-bit clean world > /\ - against html email - against proprietary attachments
Well done diagnostics reporting, and the suggestions look reasonable to me as well. I suspect we're all getting frustrated with ever-increasing build-times, and now's a good time to address their underlying causes. -- Sent with https://mailfence.com Secure and private email
