Hi, while preparing for our r-b world summit in Paris, it became obvious that people are interested to hear what happened in the rpm ecosystem during the last year (and sometimes even years when it was not covered before).
I think, it is mostly rather boring stuff and thus would want us to waste as little precious Paris-time as possible, so I gathered as much as I could in this email. This makes it possible to have a follow-up session where we assume the below to be known and only cover remaining open topics and questions. I included 4 main parts: 1. rpm itself 2. our Open Build Service (OBS) 3. OBS-associated osc and obs-build tools 4. build-compare 1. rpm ==================== The "RPM Package Manager" is used in openSUSE, Fedora/RedHat, Mandriva, Qubes OS and some smaller distributions - similar in spirit to dpkg. In prior years we had already added most required patches that allowed to produce bit-identical rpms. These are activated by 4 rpm macros, documented in https://en.opensuse.org/openSUSE:Reproducible_Builds : %source_date_epoch_from_changelog Y %clamp_mtime_to_source_date_epoch Y %use_source_date_epoch_as_buildtime Y %_buildhost reproducible Last year got two more patches merged to cover edge cases: - https://github.com/rpm-software-management/rpm/pull/536 build: Make sure SOURCE_DATE_EPOCH is in the past When a .changes file was updated in the morning, SOURCE_DATE_EPOCH was set to 12:00 of the same day, which broke mtime clamping in rpm. - https://github.com/rpm-software-management/rpm/pull/485 find-debuginfo.sh: sort output of find My previous tests mostly did builds without debuginfo because that increased build speed. But when wanting to verify official builds, the 'nachbau' replica-builds had to be done with debuginfo enabled and that uncovered this one small ordering issue. 2. OBS ==================== OBS is the software we use to manage and build all our SUSE and openSUSE distributions. There is a free public hosted version at https://build.opensuse.org with significant build power and storage space behind. OBS also allows to produce packages for Debian, ArchLinux and Fedora/RedHat based distributions from the same sources, making it very attractive to projects that do not want to operate their own infrastructure for building and hosting. E.g. see Debian and Ubuntu download URLs in https://www.openscad.org/downloads.html + https://lutris.net/downloads/ It is already possible to create verifiable, bit-identical binaries in OBS. https://build.opensuse.org/project/prjconf/Application:ERP:GNUHealth:Factory is already producing 23 such bit-reproducible verifiable binaries. I have written about my verification efforts in https://www.suse.com/c/?p=42014 Here, my verification efforts uncovered issues with artifacts that undergo a 2nd signing step inside of OBS. This is related to UEFI/secure-boot and affected grub2, xen and all kmp (out-of-tree kernel module) packages. https://github.com/openSUSE/pesign-obs-integration/pull/8 already fixed some issues there, but there are more open: https://github.com/openSUSE/pesign-obs-integration/issues/9 Without the private keys, the best result one could get there, would be to strip off the signatures and produce bit-identical binaries to those. TBD. In https://github.com/openSUSE/open-build-service/issues/5784 I also discovered issues with unreproducible noarch packages and implications from how OBS handles them. Internally, OBS uses arch-specific noarch packages for depending builds. But OBS only ever publishes one noarch package from the first arch listed in a project's metadata. So when a user wants to verify a build, it might be impossible, because only a different build of the noarch package is available than was originally used. This could affect Debian (arch 'all') and others as well. There have been previous discussions on this at https://lists.reproducible-builds.org/pipermail/rb-general/2018-September/001165.html continued in the following month https://lists.reproducible-builds.org/pipermail/rb-general/2018-October/001191.html Overall, this can involve issues that are hard to solve. E.g. when floating-point or fixed point arithmetics do different rounding in different architectures. But that work did also make bugs visible, so is definitely useful. 3. osc and obs-build ==================== osc is the OBS CLI tool. In r-b I use it for "osc up" and "osc build". obs-build is the component used in both OBS server-side builds and local "osc build". It sets up a disposable build-root environment for use with either kvm, lxc or chroot. It runs the build in there in a controlled way. It thereby normalizes many sources of indeterminism (user, path, umask, env). It also prevents internet access and ensures only the packages listed as "BuildRequire" are there, in order to enforce repeatable builds. https://github.com/openSUSE/obs-build/pull/424/commits/f50a8c339470323f497809349155f0262ca09809 fixed ordering indeterminism for packages with multiple .spec files and .changes files. Most effort went into the added test-suite to prevent future regressions there. https://github.com/openSUSE/obs-build/pull/469 disable ext4 dir_index proposed, but not merged yet. Would decrease filesystem-related indeterminism in official builds. This has trust-unrelated advantages in saving build power for follow-up builds and re-publishs. https://github.com/openSUSE/osc/pull/444 improve chroot makes some r-b debugging easier https://github.com/openSUSE/osc/pull/309 Speedup osc status speeds up my test-cycles of update+build https://github.com/openSUSE/obs-build/pull/328 / 326 https://github.com/openSUSE/osc/pull/290 allow to pass any option to the build command Is used for passing the varying -cpu type option and -rtc base option to kvm to build in the future without having to mess with the system clock. https://github.com/openSUSE/osc/issues/376 _multibuild bug From my side, it is still unsolved and annoying, because my stracebuild/autoprovenance and autoclassify tools rely on pulling in packages from another repo into a build environment. https://github.com/openSUSE/obs-build/pull/245 run kvm builds as non-root was a security-related contribution there, only tangentially related https://github.com/openSUSE/obs-build/pull/376 Support kvm builds on Debian To make it easier to run my reproducibleopensuse tools on Debian servers. 4. build-compare ==================== build-compare is openSUSE's chimera written in bash, combining things that strip-nondeterminism and diffoscope do. The stripping is only temporary and not shipped in packages, though. In OBS, its return value is used to decide if it is worth to publish a new binary package or to trigger builds of depending packages. I also use it in my reproducibleopensuse scripts when packages are not bit-identical to see the diffs and to prioritize my fixing work on the major diffs that cannot be auto-ignored. https://github.com/openSUSE/build-compare/pull/17 Allow to disable many content filters https://github.com/openSUSE/build-compare/pull/19 handle .egg as .zip https://github.com/openSUSE/build-compare/pull/25 fix diff returning 0 by mistake https://github.com/openSUSE/build-compare/pull/27 erroneously reported jar files as identical https://github.com/openSUSE/build-compare/pull/28 bugfix, ignored diffs in rpm metadata https://github.com/openSUSE/build-compare/pull/29 jar: always list files without date+size helps to auto-ignore more ordering issues in .jar files https://github.com/openSUSE/build-compare/pull/30 better quoting of strings to allow diffing files with spaces in its name As can be seen, there have already been 3 bugs making it report results as 'identical' even though there were significant differences. This is why the final goal is to get rid of it and have proper bit-identical build results. Somewhere along the way, we could disable or drop filters to make the remaining code more manageable. The largest blocker to that goal are build hostnames and timestamps in rpm metadata where people seem to still want the real thing for now. This write-up only covers a small part of my r-b work - e.g. not the 200+ upstream r-b patches that help making various individual packages reproducible for anyone, not just our rpm builds. Ciao Bernhard M. -- openSUSE Developer and Cloud Software Developer and Sysadmin SUSE LINUX GmbH, Nürnberg
signature.asc
Description: OpenPGP digital signature
_______________________________________________ [email protected] mailing list To change your subscription options, visit https://lists.reproducible-builds.org/listinfo/rb-general. To unsubscribe, send an email to [email protected].
