Hi, On 2023-08-05 13:25:39 -0700, Andres Freund wrote: > We have some issues with CI on macos and windows being too expensive (more on > that soon in a separate email). For macos most of the obviously wasted time is > spent installing packages with homebrew. Even with the package downloads being > cached, it takes about 1m20s to install them. We can't just cache the whole > homebrew installation, because it contains a lot of pre-installed packages. > > After a bunch of experimenting, I found a way to do this a lot faster: The > attached patch installs macports and installs our dependencies from > that. Because there aren't pre-existing macports packages, we can just cache > the whole thing. Doing so naively wouldn't yield that much of a speedup, > because it takes a while to unpack a tarball (or whatnot) with as many files > as our dependencies have - that's a lot of filesystem metadata > operations. Instead the script creates a HFS+ filesystem in a file and caches > that - that's mounted within a few seconds. To further keep the size in check, > that file is compressed with zstd in the cache. > > As macports has a package for IPC::Run and IO::Pty, we can use those instead > of the separate cache we had for the perl installation. > > After the patch, the cached case takes ~5s to "install" and the cache is half > the size than the one for homebrew. > > The comparison isn't entirely fair, because I used the occasion to not install > 'make' (since meson is used for building) and llvm (we didn't enable it for > the build anyway). That gain is a bit smaller without that, but still > significant. > > > An alternative implementation would be to create the "base" .dmg file outside > of CI and download it onto the CI instances. But I didn't want to figure out > the hosting situation for such files, so I thought this was the best > near-medium term path.
Given how significant an improvement this is in test time, and the limited blast radius, I am planning to push this soon, unless somebody opposes that soon. Greetings, Andres Freund