Guix pull speed
Hi everyone, After looking a bit more into guix pull speed, or to be more precise the "Computing Guix derivation..." step, which is not substitutable. I've come to the conclusion that the thing that takes the majority of the time is loading the files that define the packages that the new Guix needs to build itself. These files are not compiled yet, and worse, loading just (gnu packages guile) ends up loading 361 other package files. You can generate a package graph in GraphML with `guix graph -t module -b graphml guile`, and use e.g. networkx to analyze it. You can compare with a compiled check-out of guix by just running the following in a `guix repl`: --8<---cut here---start->8--- (use-modules (guix self) (guix monad-repl)) ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1) --8<---cut here---end--->8--- which takes at most 5 seconds on my laptop. One idea I had was to move all the packages that are looked up in (guix self) to their own little bubble, to avoid having to load extra stuff. However, this is not currently possible because some of them do have non-trivial dependency graphs. I've identified these problematic inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it pulls in all other dependencies itself) po4a graphviz What could be done about this? Another solution would be to somehow build Guix without any of the dependencies and then add them in later, similar to what is done with build-aux/build-self.scm to be able to load (guix self) in the first place. That seems quite complex though. Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: Guix pull speed
Hi Josselin, On Tue, 29 Aug 2023 at 13:58, Josselin Poiret wrote: > After looking a bit more into guix pull speed, or to be more precise the > "Computing Guix derivation..." step, which is not substitutable. I've > come to the conclusion that the thing that takes the majority of the > time is loading the files that define the packages that the new Guix > needs to build itself. Well, on my machine, the bigger bottleneck seems the procedure name ’proxy’ which copies stuff around, IIUC. See [1]. > These files are not compiled yet, and worse, > loading just (gnu packages guile) ends up loading 361 other package > files. You can generate a package graph in GraphML with `guix graph -t > module -b graphml guile`, and use e.g. networkx to analyze it. Hum, is this ’graphml’ something you have not submitted? Or am I missing a point? Last time I played with “guix graph”, I wrote a small script for bridging with networkx. See [2]. > You can compare with a compiled check-out of guix by just running the > following in a `guix repl`: > --8<---cut here---start->8--- > (use-modules (guix self) (guix monad-repl)) > ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1) > --8<---cut here---end--->8--- > which takes at most 5 seconds on my laptop. Yeah, that’s fast. :-) For comparing, what would be the corresponding derivations that “guix pull” is building? > One idea I had was to move all the packages that are looked up in (guix > self) to their own little bubble, to avoid having to load extra stuff. > However, this is not currently possible because some of them do have > non-trivial dependency graphs. I've identified these problematic > inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it > pulls in all other dependencies itself) po4a graphviz > > What could be done about this? Another solution would be to somehow > build Guix without any of the dependencies and then add them in later, > similar to what is done with build-aux/build-self.scm to be able to load > (guix self) in the first place. That seems quite complex though. Thanks for looking at this. Well, as reported in [1], I am missing what is the aim of the procedure ’proxy’ and if it could be avoided. Cheers, simon 1: Slow guix pull Simon Tournier Wed, 23 Aug 2023 15:48:06 +0200 id:86o7ix3m5l@gmail.com https://lists.gnu.org/archive/html/guix-devel/2023-08 https://yhetil.org/guix/86o7ix3m5l@gmail.com 2: Some stats about the graph of dependencies zimoun Fri, 09 Dec 2022 18:29:43 +0100 id:874ju4qyd4@gmail.com https://lists.gnu.org/archive/html/guix-devel/2022-12 https://yhetil.org/guix/874ju4qyd4@gmail.com
Re: Guix pull speed
Hi Simon, Simon Tournier writes: > Hi Josselin, > > On Tue, 29 Aug 2023 at 13:58, Josselin Poiret wrote: > >> After looking a bit more into guix pull speed, or to be more precise the >> "Computing Guix derivation..." step, which is not substitutable. I've >> come to the conclusion that the thing that takes the majority of the >> time is loading the files that define the packages that the new Guix >> needs to build itself. > > Well, on my machine, the bigger bottleneck seems the procedure name > ’proxy’ which copies stuff around, IIUC. See [1]. Proxy is on the helper script side, it's just waiting for the actual build script to do its thing. >> These files are not compiled yet, and worse, >> loading just (gnu packages guile) ends up loading 361 other package >> files. You can generate a package graph in GraphML with `guix graph -t >> module -b graphml guile`, and use e.g. networkx to analyze it. > > Hum, is this ’graphml’ something you have not submitted? Or am I > missing a point? Last time I played with “guix graph”, I wrote a small > script for bridging with networkx. See [2]. I've committed it pretty recently, and should work OOTB now. >> You can compare with a compiled check-out of guix by just running the >> following in a `guix repl`: >> --8<---cut here---start->8--- >> (use-modules (guix self) (guix monad-repl)) >> ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1) >> --8<---cut here---end--->8--- >> which takes at most 5 seconds on my laptop. > > Yeah, that’s fast. :-) > > For comparing, what would be the corresponding derivations that “guix > pull” is building? It's building the same! Just that building the derivation takes way longer because it has first to load a bunch of uncompiled guile files. Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: Guix pull speed
Hi Josselin, On Wed, 06 Sep 2023 at 11:45, Josselin Poiret wrote: >> Well, on my machine, the bigger bottleneck seems the procedure name >> ’proxy’ which copies stuff around, IIUC. See [1]. > > Proxy is on the helper script side, it's just waiting for the actual > build script to do its thing. Do it copy on the fly? I mean, is it first written somewhere then moved? >> Hum, is this ’graphml’ something you have not submitted? Or am I >> missing a point? Last time I played with “guix graph”, I wrote a small >> script for bridging with networkx. See [2]. > > I've committed it pretty recently, and should work OOTB now. Cool! >>> You can compare with a compiled check-out of guix by just running the >>> following in a `guix repl`: >>> --8<---cut here---start->8--- >>> (use-modules (guix self) (guix monad-repl)) >>> ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1) >>> --8<---cut here---end--->8--- >>> which takes at most 5 seconds on my laptop. >> >> Yeah, that’s fast. :-) >> >> For comparing, what would be the corresponding derivations that “guix >> pull” is building? > > It's building the same! Just that building the derivation takes way > longer because it has first to load a bunch of uncompiled guile files. Well, I am not sure to follow. Is your point not about why it takes longer? If it is the same, these 5 seconds is not why “guix pull” is slow, no? Cheers, simon
Re: Guix pull speed
-- Simon Tournier writes: > Do it copy on the fly? I mean, is it first written somewhere then moved? It just copies the output of the building process, which shouldn't be too big. It's waiting for build output though, hence why it's taking so much time. >> It's building the same! Just that building the derivation takes way >> longer because it has first to load a bunch of uncompiled guile files. > > Well, I am not sure to follow. Is your point not about why it > takes longer? > > If it is the same, these 5 seconds is not why “guix pull” is slow, no? It's the same process, but in one case it's done with an already compiled tree, so loading the files is instantaneous, but on `guix pull` the tree is not compiled yet (that's the whole point of `guix pull`!) and so loading these 300+ guile files is extremely slow. Best, -- Josselin Poiret signature.asc Description: PGP signature
Re: Guix pull speed
Hi Josselin, On Thu, 7 Sept 2023 at 10:37, Josselin Poiret wrote: [...] Thanks for explaining. Cheers, simon
Re: Guix pull speed
Hello! Josselin Poiret skribis: > After looking a bit more into guix pull speed, or to be more precise the > "Computing Guix derivation..." step, which is not substitutable. I've > come to the conclusion that the thing that takes the majority of the > time is loading the files that define the packages that the new Guix > needs to build itself. These files are not compiled yet, and worse, > loading just (gnu packages guile) ends up loading 361 other package > files. You can generate a package graph in GraphML with `guix graph -t > module -b graphml guile`, and use e.g. networkx to analyze it. Nice (we should add this trick under “Invoking guix graph” for instance; we already mention xdot there.) > You can compare with a compiled check-out of guix by just running the > following in a `guix repl`: > > (use-modules (guix self) (guix monad-repl)) > ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1) > > which takes at most 5 seconds on my laptop. > > One idea I had was to move all the packages that are looked up in (guix > self) to their own little bubble, to avoid having to load extra stuff. > However, this is not currently possible because some of them do have > non-trivial dependency graphs. I've identified these problematic > inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it > pulls in all other dependencies itself) po4a graphviz Yeah, I don’t think the idea of an independent subset of the module graph is achievable, because sooner or later glibc starts depending on Python, which starts depending on Rust, which pulls in Qt. Compiling with ‘-Wunused-modules’ might allow us to clean up the module graph a bit though. > What could be done about this? Another solution would be to somehow > build Guix without any of the dependencies and then add them in later, > similar to what is done with build-aux/build-self.scm to be able to load > (guix self) in the first place. That seems quite complex though. Most of the time is spent evaluating (gnu packages …) modules; evaluation is pretty slow, and from what I remember of previous profiling sessions, this is largely due to psyntax. We should concentrate on that. Another approach: the resulting .drv is a deterministic result. Wouldn’t it be nice if we could make the whole “Computing Guix derivation” process a derivation itself? That way it could be substituted. Nix introduced (or considered introducing) “recursive Nix”, whereby a derivation build process can talk to the daemon “from the inside”, creating additional derivations on the way. This could be one possibility, but it’s a lot of work with unclear outcomes. In Guix, we can use the (guix …) modules inside the build process, and we have almost everything we need to compute a set of derivations “in the abstract” (we’d need a backend that, instead of making RPCs, would store .drv and “sources” in memory or in a file). So we could create a derivation that returns let’s say a nar containing the closure of the .drv (the “Guix derivation”), which we would then import. There’s “just” one gotcha with this plan: grafts. To compute grafts, we potentially need to build things, and for that we’d need to actually talk to the daemon. Maybe there are other possibilities, such as adding a “builtin:build-self” derivation builder in the daemon. Food for thought! Ludo’.
Re: Guix pull speed
Hi, On Thu, 14 Sep 2023 at 11:58, Ludovic Courtès wrote: >> What could be done about this? Another solution would be to somehow >> build Guix without any of the dependencies and then add them in later, >> similar to what is done with build-aux/build-self.scm to be able to load >> (guix self) in the first place. That seems quite complex though. > > Most of the time is spent evaluating (gnu packages …) modules; > evaluation is pretty slow, and from what I remember of previous > profiling sessions, this is largely due to psyntax. We should > concentrate on that. On my machine the step “Computing Guix derivation” is single-threaded. Is it not possible to exploit multi-thread here? Cheers, simon
