On Fri, Jan 05, 2024 at 04:41:14PM +0000, Christopher Baines wrote: > > Ludovic Courtès <l...@gnu.org> writes: > > > Hi, > > > > Christopher Baines <m...@cbaines.net> skribis: > > > >> When asked by the data service, it seems to take Guix around 3 minutes > >> to compute cross derivations for all packages (to a single > >> target). Here's a simple script that replicates this: > > ... > > > One idiom that defeats caching is: > > > > (define (make-me-a-package x y z) > > (package > > …)) > > > > Such a procedure returns a fresh package every time it’s called, > > preventing caching from happening (because cache entries are compared > > with ‘eq?’). That typically leads to lower hit rates. > > > > Anyway, lots of words to say that I don’t see anything immediately > > obvious with cross-compilation, yet I wouldn’t be surprised if some of > > these cache-defeating idioms were used because we’ve payed less > > attention to this. > > I've got a feeling that performance has got worse since I looked at this > originally, I've finally got around to having a further look. > > I spent some time looking at various metrics, but it was most useful to > just write the cache keys of various types to files and have a read. > > The cross-base module was causing many issues, as all but one of the > procedures there produced new package records each time. There is also > make-rust-sysroot which showed up. > > I've sent some patches as #68266 to add memoization to avoid this, and > that seems to speed things up. > > Looking at other things in the cache, I think there are some issues with > file-append and local-file. The use of file-append in svn-fetch and > local-file in the lower procedure in the python build system both bloat > the cache for example, although I'm less sure about how to address these > cases. > > One thing I am sure about though, is that these problems will come > back. Maybe we could add some reporting in to Guix to look through the > cache at the keys, lower them all and check for equivalence. That way it > should be possible to automate saying that having [1] in the cache > several thousand times is unhelpful. The data service could then run > this reporting and store it. > > 1: #<file-append #<package subversion@1.14.2 > gnu/packages/version-control.scm:2267 7f294d908840> "/bin/svn">
I grabbed the patch for make-rust-sysroot to try it out: Native builds: time GUIX_PROFILING="object-cache" ./pre-inst-env guix build --no-grafts $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head -n 100) -d Object Cache: fresh caches: 21 lookups: 133146 hits: 130101 (97.7%) cache size: 3044 entries real 0m7.539s user 0m10.239s sys 0m0.327s Before: time GUIX_PROFILING="object-cache" ./pre-inst-env guix build --no-grafts $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head -n 100) --target=aarch64-linux-gnu -d Object Cache: fresh caches: 20 lookups: 221189 hits: 211390 (95.6%) cache size: 9798 entries real 0m18.215s user 0m14.492s sys 0m0.469s After: time GUIX_PROFILING="object-cache" ./pre-inst-env guix build --no-grafts $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head -n 100) --target=aarch64-linux-gnu -d Object Cache: fresh caches: 20 lookups: 138654 hits: 135291 (97.6%) cache size: 3362 entries real 0m7.753s user 0m10.248s sys 0m0.328s That's a massive drop in the size of the cache and a big decrease in the amount of time it took to calculate those 100 items. -- Efraim Flashner <efr...@flashner.co.il> רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted
signature.asc
Description: PGP signature
