-------------------------------------------------------------------- o ⬋ ⬊ May 2024 in Reproducible Builds o o ⬊ ⬋ https://reproducible-builds.org/reports/2024-05/ o --------------------------------------------------------------------
Welcome to the May 2024 report from the Reproducible Builds [0] project! In these reports, we try to outline what we have been up to over the past month and highlight news items in software supply-chain security more broadly. As ever, if you are interested in contributing to the project, please visit our "Contribute" [1] page on our website. [0] https://reproducible-builds.org [1] https://reproducible-builds.org/contribute/ § TABLE OF CONTENTS * "A peek into build provenance for Homebrew" * Distribution news * Mailing list news * Miscellaneous news * Two new academic papers * "diffoscope" * Website updates * Upstream patches * Reproducibility testing framework § "A peek into build provenance for Homebrew" ------------------------------------------- Joe Sweeney and William Woodruff on the Trail of Bits [3] blog wrote an extensive post about build provenance [4] for Homebrew [5], the third- party package manager for MacOS. Their post details how each "bottle" (i.e. each release): > […] built by Homebrew will come with a cryptographically verifiable > statement binding the bottle’s content to the specific workflow and > other build-time metadata that produced it. […] In effect, this > injects greater transparency into the Homebrew build process, and > diminishes the threat posed by a compromised or malicious insider by > making it impossible to trick ordinary users into installing > non-CI-built bottles. The post also briefly touches on future work, including work on source provenance: > Homebrew’s formulae already hash-pin their source artifacts, but we > can go a step further and additionally assert that source artifacts are > produced by the repository (or other signing identity) that’s latent in > their URL or otherwise embedded into the formula specification. [3] https://www.trailofbits.com/ [4] https://blog.trailofbits.com/2024/05/14/a-peek-into-build-provenance-for-homebrew/ [5] https://brew.sh/ § Distribution news ----------------- In Debian this month, Johannes Schauer Marin Rodrigues (aka "josch") noticed that the Debian binary package bash version 5.2.15-2+b3 was "uploaded to the archive twice. Once to bookworm and once to sid but with differing content." [6] This is problem for reproducible builds in Debian due its assumption that the package name, version and architecture triplet is unique. However, josch highlighted that > This example with bash is especially problematic since bash is > Essential:yes, so there will now be a large portion of .buildinfo files > where it is not possible to figure out with which of the two differing > bash packages the sources were compiled. In response to this, Holger Levsen performed an analysis of all .buildinfo files and found that this needs almost 1,500 binNMUs [7] to fix the fallout from this bug. [6] https://bugs.debian.org/1072205 [7] https://wiki.debian.org/NonMaintainerUpload Elsewhere in Debian, Vagrant Cascadian posted about a Non-Maintainer Upload (NMU) sprint [8] to take place during early June, and it was announced that there is now a #debian-snapshot IRC channel on OFTC to discuss the creation of a new source code archiving service to, perhaps, replace snapshot.debian.org [9]. Lastly, 11 reviews of Debian packages were added, 15 were updated and 48 were removed this month adding to our extensive knowledge about identified issues [10]. A number of issue types have been updated by Chris Lamb as well. [11][12] [8] https://lists.reproducible-builds.org/pipermail/rb-general/2024-May/003404.html [9] https://snapshot.debian.org/ [10] https://tests.reproducible-builds.org/debian/index_issues.html [11] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/5fda7f6e [12] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/cf46a837 Elsewhere in the world of distributions, deep within a larger announcement from Colin Percival about the release of version 14.1-BETA2 [13], it was mentioned that the FreeBSD [14] kernels are now built reproducibly. [13] https://lists.freebsd.org/archives/freebsd-stable/2024-May/002133.html [14] https://www.freebsd.org/ In Fedora, however, the change proposal mentioned in our report for April 2024 [15] was approved, so, per the ReproduciblePackageBuilds [16] wiki page, the "add-determinism" [17] tool is now running in new builds for Fedora 41 ('rawhide'). The "add-determinism" tool is a Rust program which, as its name suggests, adds determinism to files that are given as input by "attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances". This is essentially the Fedora version of Debian's strip-nondeterminism. However, strip-nondeterminism is written in Perl, and Fedora did not want to pull Perl in the buildroot for every package. The add-determinism tool eliminates many causes of non- determinism and work is ongoing to continue the scope of packages it can operate on. [15] https://reproducible-builds.org/reports/2024-04/ [16] https://fedoraproject.org/wiki/Changes/ReproduciblePackageBuilds [17] https://github.com/keszybz/add-determinism § Mailing list news ----------------- On our mailing list [18] this month, regular contributor "kpcyrd" wrote to the list with an update on their source code indexing project, whatsrc.org [19]. The whatsrc.org [20] project, which was launched last month in response to the XZ Utils backdoor [21], now contains and indexes almost 250,000 unique source code archives. In their post, kpcyrd gives an example of its intended purpose, noting that it shown that whilst "there seems to be consensus about [the] source code for zsh 5.9" in various Linux distributions, it "does not align with the contents of the zsh Git repository". [18] https://lists.reproducible-builds.org/listinfo/rb-general/ [19] https://lists.reproducible-builds.org/pipermail/rb-general/2024-May/003407.html [20] https://whatsrc.org/ [21] https://en.wikipedia.org/wiki/XZ_Utils_backdoor Holger Levsen also posted to the list with a 'pre-announcement' of sorts for the 2024 Reproducible Builds summit [22]. In particular: > [Whilst] the dates and location are not fixed yet, however if you don' > help us with finding a suitable location *soon*, it is very likely that > we'll meet again in Hamburg in the 2nd half of September 2024 […]. [22] https://lists.reproducible-builds.org/pipermail/rb-general/2024-May/003411.html Lastly, Frederic-Emmanuel Picca wrote to the list asking for help understanding the "non-reproducible status of the Debian silx package" [23] and received replies from both Vagrant Cascadian [24] and Chris Lamb [25]. [23] https://lists.reproducible-builds.org/pipermail/rb-general/2024-May/003393.html [24] https://lists.reproducible-builds.org/pipermail/rb-general/2024-May/003394.html [25] https://lists.reproducible-builds.org/pipermail/rb-general/2024-May/003396.html § Miscellaneous news ------------------ strip-nondeterminism [26] is our tool to remove specific non- deterministic results from a completed build. This month strip- nondeterminism version 1.14.0-1 was uploaded to Debian unstable [27] by Chris Lamb chiefly to incorporate a change from Alex Muntada to avoid a dependency on Sub::Override to perform monkey-patching and break circular dependencies related to debhelper [28]. [26] https://tracker.debian.org/pkg/strip-nondeterminism [27] https://tracker.debian.org/news/1532251/accepted-strip-nondeterminism-1140-1-source-into-unstable/ [28] https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/4d235a6 Elsewhere in our tooling, Jelle van der Waa modified reprotest [29] because the pipes [30] module will be removed in Python version 3.13 [31]. [29] https://tracker.debian.org/pkg/strip-nondeterminism [30] https://docs.python.org/3/library/pipes.html [31] https://salsa.debian.org/reproducible-builds/reprotest/commit/37a13d9 It was also noticed that a new blog post by Daniel Stenberg [32] detailing "How to verify a Curl release [33]" mentions the SOURCE_DATE_EPOCH environment variable [34]. This is because: > The [curl] release tools document also contains another key > component: the exact time stamp at which the release was done – > using integer second resolution. In order to generate a correct > tarball clone, you need to also generate the new version using the > old version’s timestamp. Because the modification date of all files > in the produced tarball will be set to this timestamp. [32] https://daniel.haxx.se/ [33] https://daniel.haxx.se/blog/2024/05/23/how-to-verify-a-curl-release/ [34] https://reproducible-builds.org/docs/source-date-epoch/ Furthermore, Fay Stegerman filed a bug [35] against the Signal messenger app for Android [36] to report that their 'reproducible' builds cannot, in fact, be reproduced. However, Fay is quick to note that she has: > … found zero evidence of any kind of compromise. Some differences > are yet unexplained but everything I found seems to be benign. I am > disappointed that Reproducible Builds have been broken for months > but I have zero reason to doubt Signal's security in any way. [35] https://github.com/signalapp/Signal-Android/issues/13565 [36] https://github.com/signalapp/Signal-Android Lastly, it was observed that there was a concise and diagrammatic overview of "supply chain threats [37]" on the SLSA [38] website. [37] https://slsa.dev/spec/v1.0/threats-overview [38] https://slsa.dev/ § Two new academic papers ----------------------- Two new scholarly papers were published this month. Firstly, Mathieu Acher, Benoît Combemale, Georges Aaron Randrianaina and Jean-Marc Jézéquel of University of Rennes [39] on "Embracing Deep Variability For Reproducibility & Replicability" [40]. The authors describe their approach as follows: > In this short [vision] paper we delve into the application of > software engineering techniques, specifically variability > management, to systematically identify and explicit points of > variability that may give rise to reproducibility issues (e.g., > language, libraries, compiler, virtual machine, OS, environment > variables, etc.). The primary objectives are: i) gaining insights > into the variability layers and their possible interactions, ii) > capturing and documenting configurations for the sake of > reproducibility, and iii) exploring diverse configurations to > replicate, and hence validate and ensure the robustness of results. > By adopting these methodologies, we aim to address the complexities > associated with reproducibility and replicability in modern software > systems and environments, facilitating a more comprehensive and > nuanced perspective on these critical aspects. A PDF of this article is available [41]. [39] https://www.univ-rennes.fr/en/welcome-university-rennes [40] https://hal.science/hal-04582287 [41] https://hal.science/hal-04582287/document Secondly, Ludovic Courtès, Timothy Sample, Simon Tournier and Stefano Zacchiroli have collaborated to publish a paper on "Source Code Archiving to the Rescue of Reproducible Deployment" [42]. Their paper was motivated because: > The ability to verify research results and to experiment with > methodologies are core tenets of science. As research results are > increasingly the outcome of computational processes, software plays > a central role. GNU Guix [43] is a software deployment tool that > supports reproducible software deployment, making it a foundation > for computational research workflows. To achieve reproducibility, we > must first ensure the source code of software packages Guix deploys > remains available. A PDF of this article [44] is also available. [42] https://hal.science/hal-04586520 [43] https://guix.gnu.org/ [44] https://hal.science/hal-04582287/document § diffoscope ---------- diffoscope [46] is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions 266, 267, 268 and 269 to Debian, making the following changes: * New features: * Use xz --list to supplement output when comparing .xz archives; essential when metadata differs. (#1069329 [47]) * Include xz --verbose --verbose (ie. double) output. (#1069329 [48]) * Strip the first line from the xz --list output. [49] * Only include xz --list --verbose output if the xz has no other differences. [50] * Actually append the xz --list after the container differences, as it simplifies a lot. [51] * Testing improvements: * Allow Debian testing to fail right now. [52] * Drop apktool from Build-Depends; we can still test APK functionality via autopkgtests. (#1071410 [53]) * Add a versioned dependency for at least version 5.4.5 for the xz tests as they fail under (at least) version 5.2.8. (#374 [54]) * Fix tests for 7zip 24.05. [55][56] * Fix all tests after additon of xz --list. [57][58] * Misc: * Update copyright years. [59] [46] https://diffoscope.org [47] https://bugs.debian.org/1069329 [48] https://bugs.debian.org/1069329 [49] https://salsa.debian.org/reproducible-builds/diffoscope/commit/ac8a5070 [50] https://salsa.debian.org/reproducible-builds/diffoscope/commit/52919364 [51] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2acff705 [52] https://salsa.debian.org/reproducible-builds/diffoscope/commit/0a5a5cc3 [53] https://bugs.debian.org/1071410 [54] https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/374 [55] https://salsa.debian.org/reproducible-builds/diffoscope/commit/31a6a56a [56] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9b421991 [57] https://salsa.debian.org/reproducible-builds/diffoscope/commit/a6651ded [58] https://salsa.debian.org/reproducible-builds/diffoscope/commit/8443cb8c [59] https://salsa.debian.org/reproducible-builds/diffoscope/commit/1e782e18 In addition, James Addison fixed an issue where the HTML output showed only the first difference in a file, while the text output shows all differences [60][61][62], Sergei Trofimovich amended the 7zip version test for older 7z versions that include the string "[64]" [63][64] and Vagrant Cascadian relaxed the versioned dependency to allow version 5.4.1 for the xz tests [65] and proposed updates to guix for versions 267, 268 [66] and pushed version 269 to Guix [67]. [60] https://salsa.debian.org/reproducible-builds/diffoscope/commit/4a685bbb [61] https://salsa.debian.org/reproducible-builds/diffoscope/commit/e976c352 [62] https://salsa.debian.org/reproducible-builds/diffoscope/commit/067a8d1c [63] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2a361d7d [64] https://salsa.debian.org/reproducible-builds/diffoscope/commit/614c1b2c [65] https://salsa.debian.org/reproducible-builds/diffoscope/commit/fd7eed75 [66] https://issues.guix.gnu.org/71024 [67] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c7888f5361fbdbe5182e7dbe90ccc12e2d95d3c3 Furthermore, Eli Schwartz updated the diffoscope.org website [68] in order to explain how to install diffoscope on Gentoo [69]. [68] https://diffoscope.org/ [69] https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/a58b28f § Website updates --------------- There were a number of improvements made to our website this month, including Chris Lamb making the "print" CSS stylesheet nicer [70]. Fay Stegerman made a number of updates to the page about the SOURCE_DATE_EPOCH environment variable [71] [72][73][74] and Holger Levsen added some of their presentations to the "Resources" page [75]. [70] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/77b997d1 [71] https://reproducible-builds.org/docs/source-date-epoch/ [72] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/db010718 [73] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/c41667e6 [74] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/aa5e9da9 [75] https://reproducible-builds.org/docs/resources/ Furthermore, IOhannes zmölnig stipulated support for SOURCE_DATE_EPOCH in clang [76] version 16.0.0+ [77], Jan Zerebecki expanded the "Formal definition [78]" page and fixed a number of typos on the "Buy-in [79]" page [80] and Simon Josefsson fixed the link to Trisquel GNU/Linux [81] on the "Projects [82]" page [83]. [76] https://clang.llvm.org/ [77] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/7489a013 [78] https://reproducible-builds.org/docs/formal-definition/ [79] https://reproducible-builds.org/docs/buy-in/ [80] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a5ab5d35 [81] https://trisquel.info/ [82] https://reproducible-builds.org/who/projects/ [83] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/438f0ff9 § Upstream patches ---------------- This month, we wrote a number of patches to fix specific reproducibility issues, including: * Bernhard M. Wiedemann: * nauty [84] (CPU-detection issue) * emacs [85] (ASLR) [84] https://bugzilla.opensuse.org/show_bug.cgi?id=1225415 [85] https://mail.gnu.org/archive/html/emacs-devel/2024-05/msg01026.html * Chris Lamb: * #1070754 [86] filed against gensio [87]. * #1071064 [88] filed against tkgate [89]. * #1072094 [90] filed against ruby-pgplot [91]. [86] https://bugs.debian.org/1070754 [87] https://tracker.debian.org/pkg/gensio [88] https://bugs.debian.org/1071064 [89] https://tracker.debian.org/pkg/tkgate [90] https://bugs.debian.org/1072094 [91] https://tracker.debian.org/pkg/ruby-pgplot § Reproducibility testing framework --------------------------------- The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org [92] in order to check packages and other artifacts for reproducibility. In May, a number of changes were made by Holger Levsen: * Debian-related changes: * Enable the rebuilder-snapshot API on osuosl4. [94] * Schedule the i386 architecture a bit more often. [95] * Adapt cleanup_nodes.sh to the new way of running our build services. [96] * Add 8 more workers for the i386 architecture. [97] * Update configuration now that the infom07 and infom08 nodes have been reinstalled as "real" i386 systems. [98] * Make diffoscope [99] timeouts more visible on the #debian- reproducible-changes IRC channel. [100] * Mark the cbxi4a-armhf node as down. [101][102] * Only install the hdmi2usb-mode-switch package only on Debian bookworm and earlier [103] and only install the haskell- platform package on Debian bullseye [104]. [92] https://tests.reproducible-builds.org [94] https://salsa.debian.org/qa/jenkins.debian.net/commit/8cf39a1d7 [95] https://salsa.debian.org/qa/jenkins.debian.net/commit/3af751f20 [96] https://salsa.debian.org/qa/jenkins.debian.net/commit/127890236 [97] https://salsa.debian.org/qa/jenkins.debian.net/commit/33d7eca13 [98] https://salsa.debian.org/qa/jenkins.debian.net/commit/c9ebd1d46 [99] https://diffoscope.org [100] https://salsa.debian.org/qa/jenkins.debian.net/commit/9f3bcea14 [101] https://salsa.debian.org/qa/jenkins.debian.net/commit/2fa8b2402 [102] https://salsa.debian.org/qa/jenkins.debian.net/commit/a94ec1db5 [103] https://salsa.debian.org/qa/jenkins.debian.net/commit/14dbf963a [104] https://salsa.debian.org/qa/jenkins.debian.net/commit/1574a0fdf * Misc: * Install the ntpdate utility as we need it later. [105] * Document the progress on the i386 architecture nodes at Infomaniak [106]. [107] * Drop an outdated and unnoticed notice. [108] * Add live_setup_schroot to the list of so-called "zombie" jobs. [109] [105] https://salsa.debian.org/qa/jenkins.debian.net/commit/c1c3e6862 [106] https://www.infomaniak.com/en [107] https://salsa.debian.org/qa/jenkins.debian.net/commit/d34fd5c04 [108] https://salsa.debian.org/qa/jenkins.debian.net/commit/1388c366c [109] https://salsa.debian.org/qa/jenkins.debian.net/commit/6c048aa94 In addition, Mattia Rizzolo reinstalled the infom07 and infom08 nodes [110] and Vagrant Cascadian marked the cbxi4a node as online [111]. [110] https://salsa.debian.org/qa/jenkins.debian.net/commit/f79acda69 [111] https://salsa.debian.org/qa/jenkins.debian.net/commit/29da0a918 § If you are interested in contributing to the Reproducible Builds project, please visit our "Contribute" [112] page on our website. However, you can get in touch with us via: * IRC: #reproducible-builds on irc.oftc.net. * Twitter: @ReproBuilds [113] * Mastodon: @reproducible_bui...@fosstodon.org [114] * Mailing list: rb-general@lists.reproducible-builds.org [115] [112] https://reproducible-builds.org/contribute/ [113] https://twitter.com/ReproBuilds [114] https://fosstodon.org/@reproducible_builds [115] https://lists.reproducible-builds.org/listinfo/rb-general -- o ⬋ ⬊ o o reproducible-builds.org 💠 ⬊ ⬋ o