Hi, Really interesting use case!
On Fri, 13 Jun 2025 at 18:38, Timothee Mathieu <[email protected]> wrote: > I finally concluded that there was a problem with AVX512 somewhere in > the dependencies graph but I gave up identifying where, as this seems > very complicated. For instance, I had the question: How many packages that python-mujoco depends on are tunable? The answer is none. :-) But Guix provides nice tools for manipulating all the graph. For example, let start an interactive session. --8<---------------cut here---------------start------------->8--- $ cat channels.scm (list (channel (name 'guix-science) (url "https://codeberg.org/guix-science/guix-science.git";) (branch "master") (commit "8bec6b3870dd4589dc4dc7c735f7cc611ac5c947") (introduction (make-channel-introduction "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc" (openpgp-fingerprint "CA4F 8CF4 37D7 478F DA05 5FD4 4213 7701 1A37 8446")))) (channel (name 'guix-science-nonfree) (url "https://codeberg.org/guix-science/guix-science-nonfree/";) (introduction (make-channel-introduction "58661b110325fd5d9b40e6f0177cc486a615817e" (openpgp-fingerprint "CA4F 8CF4 37D7 478F DA05 5FD4 4213 7701 1A37 8446")))) (channel (name 'guix) (url "https://codeberg.org/guix/guix";) (branch "master") (commit "6f14cf2abc6a3c3bdbf879baa0031fc236cfbe8c") (introduction (make-channel-introduction "9edb3f66fd807b096b48283debdcddccfea34bad" (openpgp-fingerprint "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))) $ guix time-machine -C channels.scm -- repl -L guix-rl --8<---------------cut here---------------end--------------->8--- >From there, it’s possible to traverse all the graph of dependencies and collect some packages depending on a predicate (PRED) procedure. --8<---------------cut here---------------start------------->8--- scheme@(guix-user)> (use-modules (guix) (gnu) (guix sets) (ice-9 match) (srfi srfi-1) ) (define (package-inputs* p) (filter-map (match-lambda ((_ (? package? x)) x) (_ #f)) (package-inputs p))) (define (traverse p pred) (let loop ((lst (list p)) (result '()) (visited (setq))) (match lst (() result) ((head . tail) (if (set-contains? visited head) (loop tail result visited) (let ((deps (append tail (package-inputs* head))) (res (if (pred head) (cons head result) result))) (loop deps res (set-insert head visited)))))))) --8<---------------cut here---------------end--------------->8--- Then let import the package python-mujoco and check which of its dependencies (and recursively, dependencies of dependencies, etc.) have the property tunable?. --8<---------------cut here---------------start------------->8--- scheme@(guix-user)> ,use(guix-rl packages mujoco) scheme@(guix-user)> (traverse python-mujoco (lambda (p) (assq-ref (package-properties p) 'tunable?))) $1 = () --8<---------------cut here---------------end--------------->8--- And none. Well, I think it’s expected. :-) Another example, which packages in the graph of dependencies of python-mujoco are “replaced”, i.e., grafts. --8<---------------cut here---------------start------------->8--- scheme@(guix-user)> ,pp (map package-full-name (traverse python-mujoco package-replacement)) $2 = ("[email protected]" "[email protected]" "[email protected]" "[email protected]") scheme@(guix-user)> --8<---------------cut here---------------end--------------->8--- In the same idea, it would possible to separate packages depending on their build system, etc. Or using ’package-mapping‘, rewrite various fields of some package definition. > I guess all of this should be a cautionary tale that sometimes it may > be needed to look carefully at the cpu flags in order to get > reproducibility with guix shell. Ideally I think when we want to have > something reproducible, we may want to also communicate the cpu flags > in addition to the manifest and channels file (or at least test that > changing the flags does not change the results). Maybe I miss something, if I understanding correctly [1] The solution is to select the most appropriate implementation of “hot” code at run time. Video players like MPlayer and number-crunching software like the GNU multiprecision library have used this “trick” since their inception: using the cpuid instruction, they can determine at run time which ISA extensions are available and branch to routines optimized for the available extensions. Many other applications include similar ad-hoc mechanism. Therefore, the CPU flag might be the same but not the run-time depending on the machine where it runs. The same binaries in the same pack produced by Guix might use AVX when available at run-time or do not otherwise. Is it not what you checked when setting on and off using QEMU? That said, yes CPU flags might change the result. For a “trivial” example, the C code, #include <stdio.h> #include <math.h> int main(){ printf("%E\n", j0f(0x1.33d152p+1f)); } compiled with or without the flags ’-lm -fno-builtin’ returns, 5.643440E-08 # no options 5.963430E-08 # -lm -fno-builtin because of turning on or off constant folding compiler pass. 1: https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/ > It would be great if someday a > feature to mask some CPU flag made its way to guix shell in order to > improve reproducibility but I guess my case of having a big difference > due to AVX512 is a limit case that does not happen often (?). This would mean that “guix shell” would run on the top of some emulator, no? Yeah I agree, it would nice to be able to launch on the fly an environment with some specific hardware properties. :-) Cheers, simon
