Hello! I’m happy to announce the new ‘guix challenge’ command! (Documentation below.)
The goal of the command is, ideally, to report malicious or corrupt substitute servers. In practice though, many package build processes are still non-deterministic, so chances are that, when a discrepancy reported by ‘guix challenge’, it is due to a non-deterministic build process–something we must fix. Here’s what it reports on my machine (compared to hydra.gnunet.org, which is one of the build machines behind hydra.gnu.org): --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix challenge --substitute-urls=http://hydra.gnunet.org finding garbage collector roots... determining live/dead paths... updating list of substitutes from 'http://hydra.gnunet.org'... 100.0% /gnu/store/1c96zwwg6lnh6v9n3m0q30pg0x2m0v5c-openssl-1.0.2d contents differ: local hash: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q http://hydra.gnunet.org/nar/1c96zwwg6lnh6v9n3m0q30pg0x2m0v5c-openssl-1.0.2d: 1zy4fmaaqcnjrzzajkdn3f5gmjk754b43qkq47llbyak9z0qjyim /gnu/store/3b7sjkz1ps719fkl53mcxzmjysd95i5c-emacs-deferred-0.3.2 contents differ: local hash: 0hh4d5plz4ph93yqcqvyy8dfbdidwdmy93kikjgnkglxr0lw8qss http://hydra.gnunet.org/nar/3b7sjkz1ps719fkl53mcxzmjysd95i5c-emacs-deferred-0.3.2: 0d0k5sl6zwbwb4yj0ns118bq6nxqaa8d4kr189w8lb7sma4ji5y1 /gnu/store/5zavfkmrax6z93q85q5nifbwkfz4704m-git-2.5.0 contents differ: local hash: 00p3bmryhjxrhpn2gxs2fy0a15lnip05l97205pgbk5ra395hyha http://hydra.gnunet.org/nar/5zavfkmrax6z93q85q5nifbwkfz4704m-git-2.5.0: 069nb85bv4d4a6slrwjdy8v1cn4cwspm3kdbmyb81d6zckj3nq9f /gnu/store/76yq0qvcbjjljk8my6x06ayssph573xx-pius-2.1.1 contents differ: local hash: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax http://hydra.gnunet.org/nar/76yq0qvcbjjljk8my6x06ayssph573xx-pius-2.1.1: 1cy25x1a4fzq5rk0pmvc8xhwyffnqz95h2bpvqsz2mpvlbccy0gs /gnu/store/avrrq8sl1ihq0vrhjrilszgmg7ifhqdw-guile-ssh-0.8.0 contents differ: local hash: 1qxjnirrd24djxwrh1wmsdg3qhhigymaqg673nri0d5dn87dihmc http://hydra.gnunet.org/nar/avrrq8sl1ihq0vrhjrilszgmg7ifhqdw-guile-ssh-0.8.0: 1js93xgwdrbq85mzl655dc7f6hb742bmvv46jij7b73npfzzjml9 /gnu/store/bc081jj8rx640ism3gm6wlpb7jamg080-dmd-0.2.01 contents differ: local hash: 1ngmjzscwva6w0wy3ahmq4r6svdpa8pq7f5kz273qld3k2xrg54v http://hydra.gnunet.org/nar/bc081jj8rx640ism3gm6wlpb7jamg080-dmd-0.2.01: 0hfcg2qvdfrzyfgwn1p1j7k0vv93xk5z4y9kr22gyakf9f9jjfix /gnu/store/cmb889dhvnx9kk6gdvqjm14w4j6pqalq-guix-0.8.3.abbe2c6 contents differ: local hash: 0p3mr40xk9q8bw26h8gkz2nx6n0csrmdh05zb1x85i0zs9pfd2f8 http://hydra.gnunet.org/nar/cmb889dhvnx9kk6gdvqjm14w4j6pqalq-guix-0.8.3.abbe2c6: 1aca8zkvcff38y3s9khf565mlcz8xgl8p6c93jbrin87k8z2islq /gnu/store/fkl97li2g7gqyw5bq09q2r0hkxla54lq-ath9k-htc-firmware-1.4.0 contents differ: local hash: 0r4lysb0skx7dpyh1qsk2mamkwwd4a7yldr1yy2dpr7bc0nk4z8n http://hydra.gnunet.org/nar/fkl97li2g7gqyw5bq09q2r0hkxla54lq-ath9k-htc-firmware-1.4.0: 1aia3539xvzaff4s11ga35f7iahch6rbryqzh5adlmx3gffpy2y7 /gnu/store/j0qc3ghc7ajja6k9c35y8ssxjzxrsy95-emacs-debbugs-0.7 contents differ: local hash: 0wfczpznl9pdalf4sp23dfp63pvrk9jsw7znvw8axj24wv9rbk4c http://hydra.gnunet.org/nar/j0qc3ghc7ajja6k9c35y8ssxjzxrsy95-emacs-debbugs-0.7: 19bagzab1d6xabhy3av863ab06zkcxya4lc7q62za74p7qpj112g /gnu/store/n5q0gbn01w5m2ic9rc2xq4kj2faglg6s-emacs-typo-1.1 contents differ: local hash: 1yly4y92vphdsnqnmdqwsz019fy11jkbz31sl6ysly9j3fmnhq2m http://hydra.gnunet.org/nar/n5q0gbn01w5m2ic9rc2xq4kj2faglg6s-emacs-typo-1.1: 1mj1h56pwf5cqbn397crlgsp7hdwjvrbqp4czha429y7pharjapl /gnu/store/q6xnjg9fd7lfhh7rq2l98grlyq7nbcf0-emacs-butler-0.2.4 contents differ: local hash: 14lb6cdfngjjlxrnipq961hbgnhwp47ap904a9mm0dj4q7pj23n2 http://hydra.gnunet.org/nar/q6xnjg9fd7lfhh7rq2l98grlyq7nbcf0-emacs-butler-0.2.4: 0vk41961frfvm5nrkmljhli3ibsz2lv0s09s6m29aiymixx6sw9g --8<---------------cut here---------------end--------------->8--- If we diff as explained in the doc below, we see that the Git discrepancies are due to timestamp in Perl’s POD files as well as sorted-by-inode-number ‘tclIndex’ files. For the Emacs modes, the problem is the autogenerated autoloads files, which include some sort of a timestamp as well. You’re welcome to help fix these issues! Comments welcome! Ludo’. 6.12 Invoking ‘guix challenge’ ============================== Do the binaries provided by this server really correspond to the source code it claims to build? Is this package’s build process deterministic? These are the questions the ‘guix challenge’ command attempts to answer. The former is obviously an important question: Before using a substitute server (*note Substitutes::), you’d rather _verify_ that it provides the right binaries, and thus _challenge_ it. The latter is what enables the former: If package builds are deterministic, then independent builds of the package should yield the exact same result, bit for bit; if a server provides a binary different from the one obtained locally, it may be either corrupt or malicious. We know that the hash that shows up in ‘/gnu/store’ file names is the hash of all the inputs of the process that built that file or directory—compilers, libraries, build scripts, etc. (*note Introduction::). Assuming deterministic build processes, one store file name should map to exactly one build output. ‘guix challenge’ checks whether there is, indeed, a single mapping by comparing the build outputs of several independent builds of any given store item. The command’s output looks like this: $ guix challenge --substitute-urls="http://hydra.gnu.org http://guix.example.org" updating list of substitutes from 'http://hydra.gnu.org'... 100.0% updating list of substitutes from 'http://guix.example.org'... 100.0% /gnu/store/…-openssl-1.0.2d contents differ: local hash: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q http://hydra.gnu.org/nar/…-openssl-1.0.2d: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q http://guix.example.org/nar/…-openssl-1.0.2d: 1zy4fmaaqcnjrzzajkdn3f5gmjk754b43qkq47llbyak9z0qjyim /gnu/store/…-git-2.5.0 contents differ: local hash: 00p3bmryhjxrhpn2gxs2fy0a15lnip05l97205pgbk5ra395hyha http://hydra.gnu.org/nar/…-git-2.5.0: 069nb85bv4d4a6slrwjdy8v1cn4cwspm3kdbmyb81d6zckj3nq9f http://guix.example.org/nar/…-git-2.5.0: 0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73 /gnu/store/…-pius-2.1.1 contents differ: local hash: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax http://hydra.gnu.org/nar/…-pius-2.1.1: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax http://guix.example.org/nar/…-pius-2.1.1: 1cy25x1a4fzq5rk0pmvc8xhwyffnqz95h2bpvqsz2mpvlbccy0gs In this example, ‘guix challenge’ first scans the store to determine the set of locally-built derivations—as opposed to store items that were downloaded from a substitute server—and then queries all the substitute servers. It then reports those store items for which the servers obtained a result different from the local build. As an example, ‘guix.example.org’ always gets a different answer. Conversely, ‘hydra.gnu.org’ agrees with local builds, except in the case of Git. This might indicate that the build process of Git is non-deterministic, meaning that its output varies as a function of various things that Guix does not fully control, in spite of building packages in isolated environments (*note Features::). Most common sources of non-determinism include the addition of timestamps in build results, the inclusion of random numbers, and directory listings sorted by inode number. See <http://reproducible.debian.net/howto/>, for more information. To find out what’s wrong with this Git binary, we can do something along these lines (*note Invoking guix archive::): $ wget -q -O - http://hydra.gnu.org/nar/…-git-2.5.0 \ | guix archive -x /tmp/git $ diff -ur /gnu/store/…-git.2.5.0 /tmp/git This command shows the difference between the files resulting from the local build, and the files resulting from the build on ‘hydra.gnu.org’ (*note Comparing and Merging Files: (diffutils)Overview.). The ‘diff’ command works great for text files. When binary files differ, a better option is Diffoscope (http://diffoscope.org/), a tool that helps visualize differences for all kinds of files. Once you’ve done that work, you can tell whether the differences are due to a non-deterministic build process or to a malicious server. We try hard to remove sources of non-determinism in packages to make it easier to verify substitutes, but of course, this is a process, one that involves not just Guix but a large part of the free software community. In the meantime, ‘guix challenge’ is one tool to help address the problem. If you are writing packages for Guix, you are encouraged to check whether ‘hydra.gnu.org’ and other substitute servers obtain the same build result as you did with: $ guix challenge PACKAGE ... where PACKAGE is a package specification such as ‘guile-2.0’ or ‘glibc:debug’. The general syntax is: guix challenge OPTIONS [PACKAGES…] The one option that matters is: ‘--substitute-urls=URLS’ Consider URLS the whitespace-separated list of substitute source URLs to compare to.