--------------------------------------------------------------------
o
⬋ ⬊ April 2024 in Reproducible Builds
o o
⬊ ⬋ https://reproducible-builds.org/reports/2024-04/
o
--------------------------------------------------------------------
Welcome to the April 2024 report from the Reproducible Builds [0]
project! In our reports, we attempt to outline what we have been up to
over the past month, as well as mentioning some of the important things
happening more generally in software supply-chain security.
As ever, if you are interested in contributing to the project, please
visit our Contribute [1] page on our website.
[0] https://reproducible-builds.org
[1] https://reproducible-builds.org/contribute/
§
Table of contents:
* New backseat-signed tool to validate distributions’ source inputs
* ‘NixOS is not reproducible’
* Certificate vulnerabilities in F-Droid’s fdroidserver
* Website updates
* ‘Reproducible Builds and Insights from an Independent Verifier for Arch
Linux’
* libntlm now releasing ‘minimal source-only tarballs’
* Distribution work
* Mailing list news
* diffoscope
* Upstream patches
* reprotest
* Reproducibility testing framework
* And finally...
§
New backseat-signed tool to validate distributions' source inputs
-----------------------------------------------------------------
kpcyrd announced a new tool called "backseat-signed" [2], after:
> I figured out a somewhat straight-forward way to check if a given git
> archive output is cryptographically claimed to be the source input of a
> given binary package in either Arch Linux or Debian (or both).
Elaborating more in their announcement post [3], kpcyrd writes:
> I believe this to be the "reproducible source tarball" thing some
> people have been asking about. As explained in the README, I believe
> reproducing autotools-generated tarballs isn't worth everybody's time
> and instead a distribution that claims to build from source should
> operate on VCS snapshots instead of tarballs with 25k lines of pre-
> generated shell-script.
Indeed, many distributions' packages already build from VCS snapshots,
and this trend is likely to accelerate in response to the xz incident.
The announcement led to a lengthy discussion on our mailing list [4], as
well as shorter followup thread from kpcyrd [5] about bootstrapping
Autotools [6] projects.
[2] https://github.com/kpcyrd/backseat-signed
[3]
https://lists.reproducible-builds.org/pipermail/rb-general/2024-April/003337.html
[4]
https://lists.reproducible-builds.org/pipermail/rb-general/2024-April/thread.html#3337
[5]
https://lists.reproducible-builds.org/pipermail/rb-general/2024-April/003376.html
[6] https://en.wikipedia.org/wiki/GNU_Autotools
§
'NixOS is not reproducible'
---------------------------
Morten Linderud [7] posted an post on his blog this month, provocatively
titled, "NixOS is not reproducible [8]". Although quickly admitting that
his title is indeed "clickbait", Morten goes on to clarify the precise
guarantees and promises that NixOS [9] provides its users.
Later in the most, Morten mentions that he was motivated to write the
post because:
> I have heavily invested my free-time on this topic since 2017, and met
> some of the accomplishments we have had with “Doesn’t NixOS solve this?”
> for just as long… and I thought it would be of peoples interest
> to clarify[.]
[7] https://linderud.dev/
[8] https://linderud.dev/blog/nixos-is-not-reproducible/
[9] https://nixos.org/
§
Certificate vulnerabilities in F-Droid's fdroidserver
-----------------------------------------------------
In early April, Fay Stegerman announced a certificate pinning bypass
vulnerability and Proof of Concept (PoC) [10] in the F-Droid
fdroidserver tools for "managing builds, indexes, updates, and
deployments for F-Droid repositories" to the oss-security [11]
mailing list.
> We observed that embedding a v1 (JAR) signature file in an APK with
> minSdk >= 24 will be ignored by Android/apksigner, which only checks
> v2/v3 in that case. However, since fdroidserver checks v1 first,
> regardless of minSdk, and does not verify the signature, it will accept
> a "fake" certificate and see an incorrect certificate fingerprint. […]
> We also realised that the above mentioned discrepancy between apksigner
> and androguard (which fdroidserver uses to extract the v2/v3
> certificates) can be abused here as well. […]
Later on in the month, Fay followed up with a second post detailing a
third vulnerability and a script that could be used to scan for
potentially affected .apk files [12] and mentioned that, whilst upstream
had acknowledged the vulnerability, they had not yet applied any
ameliorating fixes.
[10] https://www.openwall.com/lists/oss-security/2024/04/08/8
[11] https://www.openwall.com/lists/oss-security/
[12] https://www.openwall.com/lists/oss-security/2024/04/20/3
§
Website updates
---------------
There were a number of improvements made to our website this month,
including Chris Lamb updating the archive page [13] to recommend -X and
unzipping with TZ=UTC [14] and adding Maven, Gradle, JDK and Groovy
examples to the SOURCE_DATE_EPOCH page [15] [16]. In addition Jan
Zerebecki added a new /contribute/opensuse/ [17] page [18] and
Sertonix fixed the automatic RSS feed detection [19][20].
[13] https://reproducible-builds.org/docs/archive/
[14]
https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d15f76b8
[15] https://reproducible-builds.org/docs/source-date-epoch/
[16]
https://salsa.debian.org/reproducible-builds/reproducible-website/commit/bfcbb9a2
[17] https://reproducible-builds.org/contribute/opensuse/
[18]
https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4901c9ae
[19]
https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5f311583
[20]
https://salsa.debian.org/reproducible-builds/reproducible-website/commit/54c80767
§
"Reproducible Builds and Insights from an Independent Verifier for Arch Linux"
------------------------------------------------------------------------------
Joshua Drexel, Esther Hänggi and Iyán Méndez Veiga of the School of
Computer Science and Information Technology, Hochschule Luzern (HSLU) in
Switzerland published a paper this month entitled "Reproducible Builds
and Insights from an Independent Verifier for Arch Linux" [22]. The
paper establishes the context as follows:
> Supply chain attacks have emerged as a prominent cybersecurity threat
> in recent years. Reproducible and bootstrappable builds have the
> potential to reduce such attacks significantly. In combination with
> independent, exhaustive and periodic source code audits, these measures
> can effectively eradicate compromises in the building process. In this
> paper we introduce both concepts, we analyze the achievements over the
> last ten years and explain the remaining challenges.
What is more, the paper aims to:
> … contribute to the reproducible builds effort by setting up a
> rebuilder and verifier instance to test the reproducibility of Arch
> Linux packages. Using the results from this instance, we uncover an
> unnoticed and security-relevant packaging issue affecting 16 packages
> related to Certbot […].
A PDF [23] of the paper is available.
[22] https://doi.org/10.18420/sicherheit2024_016
[23]
https://dl.gi.de/server/api/core/bitstreams/f8685808-2e51-4a53-acc0-2b45fa240e3b/content
§
libntlm now releasing 'minimal source-only tarballs'
----------------------------------------------------
Simon Josefsson [25] wrote on his blog this month that, going forward,
the libntlm [26] project will now be releasing what they call "minimal
source-only tarballs [27]":
> The XZUtils incident [28] illustrate that tarballs with files that are
> not included in the git archive offer an opportunity to disguise
> malicious backdoors. [The] risk of hiding malware is not the only
> motivation to publish signed minimal source-only tarballs. With pre-
> generated content in tarballs, there is a risk that GNU/Linux
> distributions [ship] generated files coming from the tarball into the
> binary *.deb or *.rpm package file. Typically the person packaging the
> upstream project never realized that some installed artifacts was
> not re-built[.]
Simon's post [29] goes into further details how this was achieved, and
describes some potential caveats and counters some expected responses as
well. A shorter version can be found in the announcement for the 1.8
release of libntlm [30].
[25] https://blog.josefsson.org/
[26] https://gitlab.com/gsasl/libntlm/
[27]
https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/
[28] https://en.wikipedia.org/wiki/XZ_Utils_backdoor
[29]
https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/
[30] https://lists.nongnu.org/archive/html/libntlm/2024-04/msg00000.html
§
Distribution work
-----------------
In Debian this month, Helmut Grohne filed a bug [31] suggesting the
removal of dh-buildinfo, a tool to generate and distribute .buildinfo-
like files within binary packages. Note that this is distinct from the
.buildinfo generation performed by dpkg-genbuildinfo. By contrast, the
entirely optional dh-buildinfo generated a debian/buildinfo file that
would be shipped within binary packages as
/usr/share/doc/package/buildinfo_$arch.gz.
In addition, 21 reviews of Debian packages were added, 22 were updated
and 16 were removed this month adding to our knowledge about identified
issues [32]. A number issue types have been added, such as new
random_temporary_filenames_embedded_by_mesonpy [33] and
timestamps_added_by_librime [34] toolchain issues.
[31] https://bugs.debian.org/1068809
[32] https://tests.reproducible-builds.org/debian/index_issues.html
[33]
https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/67f129bc
[34]
https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/3675debb
In openSUSE, it was announced that their Factory distribution enabled
bit-by-bit reproducible builds [35] for almost all parts of the package.
Previously, more parts needed to be ignored when comparing package
files, but now only the signature needs to be deleted.
In addition, Bernhard M. Wiedemann published theunreproduciblepackage
[36] as a proper .rpm package which allows to better test tools
intended to debug reproducibility. Furthermore, it was announced that
Bernhard's work on a 100% reproducible openSUSE-based distribution [37]
will be funded by NLnet [38].
[35] https://news.opensuse.org/2024/04/18/factory-bit-reproducible-builds/
[36]
https://build.opensuse.org/package/show/home:bmwiedemann:reproducible/theunreproduciblepackage
[37] https://nlnet.nl/project/Reproducible-openSUSE/
[38] https://nlnet.nl
In GNU Guix, Janneke Nieuwenhuizen submitted a patch set for creating a
reproducible source tarball for Guix. That is to say, ensuring that make
dist is reproducible when run from Git. [39]
[39] https://issues.guix.gnu.org/70169/
Lastly, in Fedora, a new wiki page was created to propose a change to
the distribution. Titled "Changes/ReproduciblePackageBuilds* [40]", the
page summarises itself as a proposal whereby "A post-build cleanup is
integrated into the RPM build process so that common causes of build
irreproducibility in packages are removed, making most of Fedora
packages reproducible."
[40] https://fedoraproject.org/wiki/Changes/ReproduciblePackageBuilds
§
Mailing list news
-----------------
On our mailing list [41] this month:
* Continuing a thread started in March 2024 [42] about the Arch Linux
minimal container now being 100% reproducible [43], John Gilmore
followed up with a post [44] about the practical and philosophical
distinctions of local vs. remote storage of the various artifacts
needed to build packages.
[41] https://lists.reproducible-builds.org/listinfo/rb-general/
[42]
https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/thread.html#3301
[43]
https://reproducible-builds.org/reports/2024-03/#arch-linux-minimal-container-userland-now-100-reproducible
[44]
https://lists.reproducible-builds.org/pipermail/rb-general/2024-April/003336.html
* Chris Lamb asked the list which conferences readers are attending
these days: "After peak Covid and other industry-wide changes,
conferences are no longer the 'must attend' events they previously
were… especially in the area of software supply-chain security. In
rough, practical terms, it seems harder to justify conference travel
today than it did in mid-2019." The thread generated a number of
responses [45] which would be of interest to anyone planning travel
in Q3 and Q4 of 2024.
[45]
https://lists.reproducible-builds.org/pipermail/rb-general/2024-April/thread.html#3370
* James Addison wrote to the list about a "quirk" in Git related to its
core.autocrlf functionality [46], thus helpfully passing on a
"slightly off-topic and perhaps not of direct relevance to anyone on
the list today" note that might still be "the kind of issue that is
useful to be aware of if-and-when puzzling over unexpected git
content / checksum issues (situations that I _do_ expect people on
this list encounter from time-to-time)".
[46]
https://lists.reproducible-builds.org/pipermail/rb-general/2024-April/003385.html
§
diffoscope
----------
diffoscope [48] is our in-depth and content-aware diff utility that can
locate and diagnose reproducibility issues. This month, Chris Lamb made
a number of changes such as uploading versions 263, 264 and 265 to
Debian and made the following additional changes:
* Don't crash on invalid .zip files, even if we encounter their
'badness' halfway through the file and not at the time of their
initial opening. [49]
* Prevent odt2txt tests from always being skipped due to an
(impossibly) new version requirement. [50]
* Avoid parens-in-parens in test 'skipping' messages. [51]
* Ensure that tests with >=-style version constraints actually print
the tool name. [52]
[48] https://diffoscope.org
[49] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9c7e817c
[50] https://salsa.debian.org/reproducible-builds/diffoscope/commit/8e6f778c
[51] https://salsa.debian.org/reproducible-builds/diffoscope/commit/99afaf60
[52] https://salsa.debian.org/reproducible-builds/diffoscope/commit/e52eef5a
In addition, Fay Stegerman fixed a crash when there are (invalid)
duplicate entries in .zip which was originally reported in Debian bug
#1068705 [53]). [54] Fay also added a user-visible 'note' to a diff when
there are duplicate entries in ZIP files [55]. Lastly, Vagrant Cascadian
added an external tool pointer for the zipdetails tool under GNU Guix
[56][57].
[53] https://bugs.debian.org/1068705
[54] https://salsa.debian.org/reproducible-builds/diffoscope/commit/945fd9fa
[55] https://salsa.debian.org/reproducible-builds/diffoscope/commit/607094a5
[56] https://guix.gnu.org/
[57] https://salsa.debian.org/reproducible-builds/diffoscope/commit/90dd1883
§
Upstream patches
----------------
The Reproducible Builds project detects, dissects and attempts to fix as
many currently-unreproducible packages as possible. We endeavour to send
all of our patches upstream where appropriate. This month, we wrote a
large number of such patches, including:
* Chris Lamb:
* #1068173 [58] filed against pg-gvm [59].
* #1068176 [60] filed against goldendict-ng [61].
* #1068372 [62] filed against grokevt [63].
* #1068374 [64] filed against ttconv [65].
* #1068375 [66] filed against ludevit [67].
* #1068795 [68] filed against pympress [69].
* #1069168 [70] filed against sagemath-database-conway-polynomials [71].
* #1069169 [72] filed against gap-polymaking [73].
* #1069663 [74] filed against dub [75].
* #1069709 [76] filed against dpb [77].
* #1069784 [78] filed against python-itemloaders [79].
* #1069822 [80] filed against python-gvm [81].
[58] https://bugs.debian.org/1068173
[59] https://tracker.debian.org/pkg/pg-gvm
[60] https://bugs.debian.org/1068176
[61] https://tracker.debian.org/pkg/goldendict-ng
[62] https://bugs.debian.org/1068372
[63] https://tracker.debian.org/pkg/grokevt
[64] https://bugs.debian.org/1068374
[65] https://tracker.debian.org/pkg/ttconv
[66] https://bugs.debian.org/1068375
[67] https://tracker.debian.org/pkg/ludevit
[68] https://bugs.debian.org/1068795
[69] https://tracker.debian.org/pkg/pympress
[70] https://bugs.debian.org/1069168
[71] https://tracker.debian.org/pkg/sagemath-database-conway-polynomials
[72] https://bugs.debian.org/1069169
[73] https://tracker.debian.org/pkg/gap-polymaking
[74] https://bugs.debian.org/1069663
[75] https://tracker.debian.org/pkg/dub
[76] https://bugs.debian.org/1069709
[77] https://tracker.debian.org/pkg/dpb
[78] https://bugs.debian.org/1069784
[79] https://tracker.debian.org/pkg/python-itemloaders
[80] https://bugs.debian.org/1069822
[81] https://tracker.debian.org/pkg/python-gvm
* Jan Zerebecki:
* rpm [82] (Support reproducible automatic rebuilds, etc.)
* openSUSE-release-tools [83] (Create changelog for generated
package sources for SOURCE_DATE_EPOCH)
* pesign-obs-integration [84] (Create changelog for generated
package sources for SOURCE_DATE_EPOCH)
* openSUSE post-build-checks [85] (Set SOURCE_DATE_EPOCH)
* obs-build [86] (Fix changelog timezone handling)
* obs-service-tar_scm [87] (When generating changelog from Git,
create the file if it does not exist.)
[82] https://github.com/rpm-software-management/rpm/pull/2880
[83] https://github.com/openSUSE/openSUSE-release-tools/pull/3064
[84] https://github.com/openSUSE/pesign-obs-integration/pull/48
[85] https://github.com/openSUSE/post-build-checks/pull/62
[86] https://github.com/openSUSE/obs-build/pull/977
[87] https://github.com/openSUSE/obs-service-tar_scm/pull/484
* Thomas Goirand:
* oslo.messaging [88] (fix a hostname-related issue)
[88]
https://github.com/openstack/oslo.messaging/commit/dc55d64df989bdb5161ca8ad8d74115cc2959174
§
reprotest
---------
reprotest [90] is our tool for building the same source code twice in
different environments and then checking the binaries produced by each
build for any differences. This month, reprotest version 0.7.27 was
uploaded to Debian unstable) by Vagrant Cascadian who made the following
additional changes:
* Enable specific number of CPUs using --vary=num_cpus.cpus=X. [91]
* Consistently use 398 days for time variation, rather than choosing
randomly each time. [92]
* Disable builds of arch:any packages. [93]
* Update the description for the build_path.path option in
README.rst. [94]
* Update escape sequences for compatibility with Python 3.12. (#1068853
[95]). [96]
* Remove the generic 'upstream' signing-key [97] and update the
packages' signing key with the currently active team members [98].
* Update the packaging Standards-Version to 4.7.0. [99]
[90] https://salsa.debian.org/reproducible-builds/reprotest
[91] https://salsa.debian.org/reproducible-builds/reprotest/commit/cdabc07
[92] https://salsa.debian.org/reproducible-builds/reprotest/commit/42a53ed
[93] https://salsa.debian.org/reproducible-builds/reprotest/commit/3270c94
[94] https://salsa.debian.org/reproducible-builds/reprotest/commit/9235862
[95] https://bugs.debian.org/1068853
[96] https://salsa.debian.org/reproducible-builds/reprotest/commit/cf65735
[97] https://salsa.debian.org/reproducible-builds/reprotest/commit/7400030
[98] https://salsa.debian.org/reproducible-builds/reprotest/commit/d11398f
[99] https://salsa.debian.org/reproducible-builds/reprotest/commit/82777f9
In addition, Holger Levsen fixed some spelling errors detected by the
spellintian tool. [100]
[100] https://salsa.debian.org/reproducible-builds/reprotest/commit/96e324a
§
Reproducibility testing framework
---------------------------------
The Reproducible Builds project operates a comprehensive testing
framework running primarily at <tests.reproducible-builds.org> [101] in
order to check packages and other artifacts for reproducibility.
In April, an enormous number of changes were made by Holger Levsen:
* Debian [102]-related changes:
* Adjust for changed internal IP addresses at Codethink. [103]
* Automatically cleanup failed diffoscope user services if there
are too many failures. [104][105]
* Configure two new nodes at infomanik.cloud. [106][107]
* Schedule Debian *experimemental* even less. [108][109]
[101] https://tests.reproducible-builds.org
[102] https://debian.org/
[103] https://salsa.debian.org/qa/jenkins.debian.net/commit/d202c0449
[104] https://salsa.debian.org/qa/jenkins.debian.net/commit/e829e6e71
[105] https://salsa.debian.org/qa/jenkins.debian.net/commit/b2401650a
[106] https://salsa.debian.org/qa/jenkins.debian.net/commit/cc1ed0063
[107] https://salsa.debian.org/qa/jenkins.debian.net/commit/3709b0f1c
[108] https://salsa.debian.org/qa/jenkins.debian.net/commit/73013d6f6
[109] https://salsa.debian.org/qa/jenkins.debian.net/commit/4b5f4cb09
* Breakage detection:
* Exclude currently building packages from breakage
detection. [110]
* Be more noisy if diffoscope crashes. [111]
* Health check: provide clickable URLs in jenkins job log for
failed pkg builds due to diffoscope crashes. [112]
* Limit graph to about the last 100 days of breakages only. [113]
* Fix all found files with bad permissions. [114]
* Prepare dealing with diffoscope timeouts. [115]
* Detect more cases of failure to debootstrap base system. [116]
* Include timestamps of failed job runs. [117]
[110] https://salsa.debian.org/qa/jenkins.debian.net/commit/92078b002
[111] https://salsa.debian.org/qa/jenkins.debian.net/commit/9997af327
[112] https://salsa.debian.org/qa/jenkins.debian.net/commit/386ec0aa4
[113] https://salsa.debian.org/qa/jenkins.debian.net/commit/c88e08dfd
[114] https://salsa.debian.org/qa/jenkins.debian.net/commit/6d0c66f1e
[115] https://salsa.debian.org/qa/jenkins.debian.net/commit/98ba4fe38
[116] https://salsa.debian.org/qa/jenkins.debian.net/commit/53865a60c
[117] https://salsa.debian.org/qa/jenkins.debian.net/commit/00cca3c93
* Documentation updates:
* Document how to access arm64 nodes at Codethink. [118]
* Document how to use infomaniak.cloud. [119]
* Drop notes about long stalled LeMaker HiKey960 boards sponsored
by HPE and hosted at ETH. [120]
* Mention osuosl4 and osuosl5 and explain their usage. [121]
* Mention that some packages are built differently. [122][123]
* Improve language in a comment. [124]
* Add more notes how to query resource usage from
infomaniak.cloud. [125]
[118] https://salsa.debian.org/qa/jenkins.debian.net/commit/a247125b0
[119] https://salsa.debian.org/qa/jenkins.debian.net/commit/cd004dd6c
[120] https://salsa.debian.org/qa/jenkins.debian.net/commit/0a31a1fe9
[121] https://salsa.debian.org/qa/jenkins.debian.net/commit/3b390f7e7
[122] https://salsa.debian.org/qa/jenkins.debian.net/commit/d68086a4b
[123] https://salsa.debian.org/qa/jenkins.debian.net/commit/6067b5612
[124] https://salsa.debian.org/qa/jenkins.debian.net/commit/77dbf257b
[125] https://salsa.debian.org/qa/jenkins.debian.net/commit/ea1035e7b
* Node maintenance:
* Add ionos4 and ionos14 to THANKS. [126][127][128][129][130]
* Deprecate Squid on ionos1 and ionos10. [131]
* Drop obsolete script to powercycle arm64 architecture
nodes. [132]
* Update system_health_check for new proxy nodes. [133]
[126] https://salsa.debian.org/qa/jenkins.debian.net/commit/be7d08960
[127] https://salsa.debian.org/qa/jenkins.debian.net/commit/699b5554c
[128] https://salsa.debian.org/qa/jenkins.debian.net/commit/2e3bcbada
[129] https://salsa.debian.org/qa/jenkins.debian.net/commit/09fccba39
[130] https://salsa.debian.org/qa/jenkins.debian.net/commit/da9063ad4
[131] https://salsa.debian.org/qa/jenkins.debian.net/commit/674f55d6d
[132] https://salsa.debian.org/qa/jenkins.debian.net/commit/b4d37b5b3
[133] https://salsa.debian.org/qa/jenkins.debian.net/commit/148d252d0
* Misc changes:
* Make the update_jdn.sh script more robust. [134][135]
* Update my SSH public key. [136]
[134] https://salsa.debian.org/qa/jenkins.debian.net/commit/ef3de23bd
[135] https://salsa.debian.org/qa/jenkins.debian.net/commit/2c1d7272f
[136] https://salsa.debian.org/qa/jenkins.debian.net/commit/23ab1af4f
In addition, Mattia Rizzolo added some new host details. [137]
[137] https://salsa.debian.org/qa/jenkins.debian.net/commit/faddf9eaa
§
And finally...
--------------
If you are interested in contributing to the Reproducible Builds
project, please visit our Contribute [138] page on our website.
However, you can get in touch with us via:
* IRC: #reproducible-builds on irc.oftc.net.
* Twitter: @ReproBuilds [139]
* Mastodon: @reproducible_bui...@fosstodon.org [140]
* Mailing list: rb-general@lists.reproducible-builds.org [141]
[138] https://reproducible-builds.org/contribute/
[139] https://twitter.com/ReproBuilds
[140] https://fosstodon.org/@reproducible_builds
[141] https://lists.reproducible-builds.org/listinfo/rb-general
--
o
⬋ ⬊
o o reproducible-builds.org 💠
⬊ ⬋
o
