Hi,
On lun., 13 mai 2024 at 17:04, Edouard Klein <e...@rdklein.fr> wrote:
> - Why is this step not substitutable ? The inputs are known, a hash can
> be derived, a substitute server could be queried for an output of that
> hash ? What am I missing ? Does the guix derivation not end up in the
> store ? What makes it so special that it can't be served by a substitute
> server ?
Assume we are running two different Guix revisions, say A and B. And at
the end of our respective “guix pull”, we expect to have the same
revision, say C. We expect to then run the same Guix.
Other said, how can we “compile” the code of C using one machinery from
A and another potentially different from B and expect to have the same
result?
Somehow, we need an intermediary step: something minimal that is
independent of A and B but produces the same C. And it’s the aim of
“Computing derivation” with the script build-aux/build-self.scm.
The inputs are known, indeed. However, the ones from A and from B are
not necessary the sames. For instance, Guile of A might be different of
Guile of B. Somehow, that “Computing derivation” is what allows to time
travel.
Well, that’s my understanding and I could have missing something.
For instance, I am running:
$ guix describe
Generation 4 mars 25 2024 18:22:25 (current)
guix 929ddec
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 929ddec8f4a181be653152c7436581c2adc54eee
Assume it is revision A. From there, let run ’guix pull’.
--8<---------------cut here---------------start------------->8---
$ time guix pull
Updating channel 'guix' from Git repository at
'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 6ba29e0 (51 new commits)...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git 6ba29e0
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
module-import 2KiB 305KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import-compiled 1.2MiB 306KiB/s 00:04 ▕██████████████████▏ 100.0%
compute-guix-derivation 1015B 1.8MiB/s 00:00 ▕██████████████████▏ 100.0%
Computing Guix derivation for 'x86_64-linux'... /
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/115prkj2rp58gl2lr0qy6higpliw8xpy-profile.drv
/gnu/store/56dsm4f5snfmnfgkki4x41fj0xcl394m-guix-daemon.drv
/gnu/store/77r2c33r1cq03qrrrfsf3g6mvmjp0w92-guix-command.drv
/gnu/store/0y0fq1lmz3dwqi0fpbq4g6swkc9yljmp-guix-cli-modules.drv
/gnu/store/bbczy39rid3q6j191mj397ak589s1101-guix-system-modules.drv
/gnu/store/gx3ngbxmb6ayzjpxv9afsigvajc5d7h4-guix-extra-modules.drv
/gnu/store/i0b6inna8a8bxg75k3dxr4xl6jrn53cm-guix-cli-core-modules.drv
/gnu/store/iww6bi0ciaqkjjbqp26iclq2ijd1bgck-guix-system-tests-modules.drv
/gnu/store/x0ai22yrq99kp3il0x3nf1d3yj5fydqy-guix-packages-base-modules.drv
70,7 MB will be downloaded
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
[...]
guix-package-cache 872KiB 303KiB/s 00:03 ▕██████████████████▏ 100.0%
guix-cli 315KiB 241KiB/s 00:01 ▕██████████████████▏ 100.0%
guix-cli-core 922KiB 194KiB/s 00:05 ▕██████████████████▏ 100.0%
guix-cli-modules 260B 253KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-cli-core-modules 261B 450KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-extra 2.8MiB 363KiB/s 00:08 ▕██████████████████▏ 100.0%
guix-manual 5.2MiB 273KiB/s 00:19 ▕██████████████████▏ 100.0%
guix-extra-modules 261B 187KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-packages-base 17.4MiB 271KiB/s 01:06 ▕██████████████████▏ 100.0%
guix-system 7.9MiB 264KiB/s 00:31 ▕██████████████████▏ 100.0%
guix-packages-base-modules 265B 279KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-system-modules 262B 282KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-system-tests 921KiB 332KiB/s 00:03 ▕██████████████████▏ 100.0%
guix-system-tests-modules 267B 279KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-6ba29e021-modules 31.2MiB 299KiB/s 01:47 ▕██████████████████▏ 100.0%
guix-module-union 2KiB 385KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-command 655B 1.0MiB/s 00:00 ▕██████████████████▏ 100.0%
guix-daemon 402B 711KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-6ba29e021 802B 499KiB/s 00:00 ▕██████████████████▏ 100.0%
[...]
real 5m8,633s
user 1m53,567s
sys 0m1,530s
--8<---------------cut here---------------end--------------->8---
Ok, all is substitutable (wow! 70MiB that’s not nothing… another story)
except the part “Computing derivation”.
Now, from another revision B (here I choose C to make things more
confusing ;-)), let run:
--8<---------------cut here---------------start------------->8---
simon@lili$ time guix time-machine --commit=6ba29e0 -- describe
Updating channel 'guix' from Git repository at
'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
[...]
config.scm 507B 377KiB/s 00:00 ▕██████████████████▏ 100.0%
git.scm 105B 127KiB/s 00:00 ▕██████████████████▏ 100.0%
hash.scm 130B 267KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import 2KiB 557KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import 2KiB 577KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import-compiled 1.2MiB 302KiB/s 00:04 ▕██████████████████▏ 100.0%
module-import-compiled 1.2MiB 244KiB/s 00:05 ▕██████████████████▏ 100.0%
building
/gnu/store/icka5f4977darnyw75is2v716k0agp48-compute-guix-derivation.drv...
Computing Guix derivation for 'x86_64-linux'... |
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
The following derivations will be built:
/gnu/store/iq1xhzdjc2lhvfmg5ap65a1wx2mwbynw-profile.drv
/gnu/store/jkslyhj4m35bqrg0am8y5qlh9gsdamyv-guix-6ba29e021.drv
/gnu/store/8nc0vjaz96n3rixxdi57w0zk7ja5nh7h-guix-daemon.drv
/gnu/store/fryzibmg9vrc542qra59qmzf8d6n86sw-guix-command.drv
/gnu/store/qqnxqvqsbjm1mglisnmm9sb0n6043c9i-guix-module-union.drv
/gnu/store/q9p35jxy0jr4zaldm6nmkj91zkhsv0an-guix-6ba29e021-modules.drv
/gnu/store/5fz6m02fgjnyqf359wv47hwwfhfwk64b-guix-packages-modules.drv
/gnu/store/jm5il7lsxigw0d63cv1vwx1kxgm06580-guix-home-modules.drv
/gnu/store/jz26dqj8kai936h2fj0szdnkmnlhmfl3-guix-config-modules.drv
/gnu/store/lf4k886h1cqnj39774hmnpfsxxi40cnb-guix-config.drv
/gnu/store/lh77nxg1hl3snzfd8f5fngzfj6zbr8jn-guix-config-source.drv
/gnu/store/593nr6hirjgkvmpfyvfw8laqha2ycgfw-config.scm.drv
/gnu/store/jjqy49vzcfcvdkg59cpssvg643bsilkl-inferior-script.scm.drv
/gnu/store/pbfh548cj96kd7gxdyrs1q4ljdgdb1gn-profile.drv
2,4 MB will be downloaded
module-import-compiled 107KiB 145KiB/s 00:01 ▕██████████████████▏ 100.0%
guix-home 751KiB 180KiB/s 00:04 ▕██████████████████▏ 100.0%
guix-packages 1.5MiB 210KiB/s 00:07 ▕██████████████████▏ 100.0%
building /gnu/store/593nr6hirjgkvmpfyvfw8laqha2ycgfw-config.scm.drv...
building /gnu/store/jm5il7lsxigw0d63cv1vwx1kxgm06580-guix-home-modules.drv...
building /gnu/store/lh77nxg1hl3snzfd8f5fngzfj6zbr8jn-guix-config-source.drv...
building
/gnu/store/5fz6m02fgjnyqf359wv47hwwfhfwk64b-guix-packages-modules.drv...
building /gnu/store/lf4k886h1cqnj39774hmnpfsxxi40cnb-guix-config.drv...
building /gnu/store/jz26dqj8kai936h2fj0szdnkmnlhmfl3-guix-config-modules.drv...
building
/gnu/store/q9p35jxy0jr4zaldm6nmkj91zkhsv0an-guix-6ba29e021-modules.drv...
building /gnu/store/qqnxqvqsbjm1mglisnmm9sb0n6043c9i-guix-module-union.drv...
building /gnu/store/fryzibmg9vrc542qra59qmzf8d6n86sw-guix-command.drv...
building /gnu/store/8nc0vjaz96n3rixxdi57w0zk7ja5nh7h-guix-daemon.drv...
building /gnu/store/jkslyhj4m35bqrg0am8y5qlh9gsdamyv-guix-6ba29e021.drv...
[...]
guix 6ba29e0
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: 6ba29e02108ed144da1234b4b5512ee03865dcf6
real 1m34,408s
user 1m50,315s
sys 0m1,421s
--8<---------------cut here---------------end--------------->8---
Similar story. All is substitutatble except “Computing derivation”.
Now, if we scrutinize, let compare the script we run when type “guix”:
(A) /gnu/store/77r2c33r1cq03qrrrfsf3g6mvmjp0w92-guix-command.drv
and
(B) /gnu/store/fryzibmg9vrc542qra59qmzf8d6n86sw-guix-command.drv
The outputs are different (I did not expect that ;-))
(A) /gnu/store/mwv6xzx8ggyzb67jwj1pmxyr8zndq12x-guix-command
and
(B) /gnu/store/4jjd4zfyvp2g8gnp3wqzaxnn9xs9sdan-guix-command
Well, I was expecting the same guix-command. Why not the same? Because
one thing is different (aside the order of listed inputs):
(A) /gnu/store/39l6dl5q5287wnn063nx4qgd3gx1l6zy-guix-module-union.drv
and
(B) /gnu/store/qqnxqvqsbjm1mglisnmm9sb0n6043c9i-guix-module-union.drv
Let dive in more. The two outputs of these derivations look similar;
except one file: config.scm. (Aside .go binary compiled files which are
known to not be reproducible.) The difference between revision A and B
reads:
--8<---------------cut here---------------start------------->8---
(define %channel-metadata
(quote
(repository
(version 0)
(url "https://git.savannah.gnu.org/git/guix.git";)
(A) (branch "master")
(B) (branch #f)
(commit "6ba29e02108ed144da1234b4b5512ee03865dcf6")
(name guix)
(introduction
(channel-introduction
(version 0)
(commit "9edb3f66fd807b096b48283debdcddccfea34bad")
(signer "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))))
--8<---------------cut here---------------end--------------->8---
and that’s because one is from regular “guix pull”, and the other is
from “guix time-machine”. Well, maybe that could be considered as a
bug, that’s another story. ;-)
All the point is to build the same result starting from two different
machinery (different Guile, modules, etc) and ending to the same output.
Considering the combinatorial, it would not be possible to substitutes;
it would mean compute all the couples A-C and B-C.
That’s an hard topic and speed it up would be very nice. Bah it would
require to redesign stuff and perhaps find a process to build the same
output using two different toolchains, somehow.
Again, maybe I am missing something.
> - Is there a way (even a very dirty one, like hand copying stuff accross
> /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4
> minutes per user price ? As I said, this is downtime on my server, as
> I need to stop all other services to let guix pull finish.
Nothing I am aware.
For this situation with different users, are they all running the same
Guix revision and targeting the same final Guix revision?
If yes, indeed ’guix pull’ should be costly only once and then instantly
fast for all the others; if that’s not already the case.
If no, as explained, it seems hard to avoid the cost of “Computing
derivation“ for each user using a different Guix revision.
Well, maybe I am missing something.
Cheers,
simon
