Re: [arch-projects] [PATCH] [devtools] makechrootpkg: load COMPRESS* vars from makepkg.conf
Thanks for the detailed reply Eli :) Before I go further, I should mention that this thread has sparked some interest among our teammates, and folks on IRC are researching now a possibility of switching from xz to zstd entirely. So I think we will return to this topic soon with some updates :) > And it is not a pixz issue either. For example xz -T $num will compress > with up to $num cores, as opposed to the default 1, but basic testing > shows that it produces the same output with 2, 3, 4 etc. cores but > different output from what you get when using 1 core. You cannot even > hardcode -T0, since (in addition to requiring packagers to use more > cores than they want to) that may still get interpreted as 1, which is > then unreproducible. You can verify this using taskset -c 1... > > (This has been discussed already in #archlinux-pacman.) Turns out it is possible to make xz use more threads and produce reproducible results, check out this message: https://lists.debian.org/debian-dpkg/2016/10/msg00011.html I've confirmed (with real VMs) that -T2, -T3, -T8, all produce the same result, on both single-core and multi-core CPUs, and -T0 produces the same result on multi-core CPU, but -T0 produces a different result on a single-core CPU. Therefore, the simplest workaround that produces reproducible results on all types of CPU is this: -T$([ $(nproc) -lt 2 ] && echo '2' || echo '0') Or maybe even something like this: -T$(( $(nproc) + 1 )) > For the topic of configurable implementations, there is an additional > hurdle. Since compression happens inside the chroot, even if we did > support it, it would need some way to look up the required binary and > install it in the chroot, but we have no tooling for this and in fact > makepkg does not even know how to check that the binary exists, so you > will probably just get: > > -> Compressing package... > /usr/share/makepkg/util/compress.sh: line 39: pixz: command not found > bsdtar: Write error > ==> ERROR: Failed to create package file. > > The solution would have to require either a known, embedded list of > "good" ones (if pixz is not reproducible when running the same command > line twice then we'd need to be able to blacklist that), or else assume > the user installed it when adding it to their makepkg.conf and using > pacman -Qo. It is probably overkill to use pkgfile and support > compressors that aren't installed on the host either... That is correct. One final remark, pixz does produce the same result when is run twice, I made a mistake when I was testing it. But it doesn't matter, since we'll probably settle for zstd or xz -T. -- Maxim Baz signature.asc Description: PGP signature
Re: [arch-projects] [PATCH] [devtools] makechrootpkg: load COMPRESS* vars from makepkg.conf
On Mon, Mar 18, 2019 at 10:58:57AM +0100, Levente Polyak via arch-projects wrote: > Messing with this in devtools used for building official repo packages kills > reproducible builds when not agreed on very specific flags and tools to > compress. > This is not a good idea to do and we won't kill reproducibility by going wild > west. True, sad but valid point. I've just confirmed that pixz produces different results when run twice on the same data, so I can't even propose to just switch the default to pixz without breaking reproducible builds :( Thanks for the quick feedback! -- Maxim Baz signature.asc Description: PGP signature
Re: [arch-projects] [PATCH] [devtools] makechrootpkg: load COMPRESS* vars from makepkg.conf
I did see an earlier patch proposing the same, which was rejected because it might break delta packages [1]. However, someone mentioned that pacman is dropping support for delta packages altogether [2] referring to a removal commit in pacman repo [3], so I wanted to bring this up again :) I'm creating a new thread because according to my tests the earlier patch [1] is incomplete. 1: https://lists.archlinux.org/pipermail/arch-projects/2017-May/004574.html 2: https://github.com/AladW/aurutils/issues/547 3: https://git.archlinux.org/pacman.git/commit/?id=9adb0d5b37df7ca668e23877e85431dabeea005e -- Maxim Baz signature.asc Description: PGP signature
[arch-projects] [PATCH] [devtools] makechrootpkg: load COMPRESS* vars from makepkg.conf
This allows using faster multicore compression tools like pixz Signed-off-by: Maxim Baz --- archbuild.in | 2 +- makechrootpkg.in | 18 +- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/archbuild.in b/archbuild.in index bd5706d..e3d53ec 100644 --- a/archbuild.in +++ b/archbuild.in @@ -39,7 +39,7 @@ while getopts 'hcr:' arg; do esac done -check_root SOURCE_DATE_EPOCH,SRCDEST,SRCPKGDEST,PKGDEST,LOGDEST,MAKEFLAGS,PACKAGER,GNUPGHOME +check_root SOURCE_DATE_EPOCH,SRCDEST,SRCPKGDEST,PKGDEST,LOGDEST,MAKEFLAGS,PACKAGER,GNUPGHOME,COMPRESSGZ,COMPRESSXZ,COMPRESSLRZ,COMPRESSZ,COMPRESSBZ2,COMPRESSLZO # Pass all arguments after -- right to makepkg makechrootpkg_args+=("${@:$OPTIND}") diff --git a/makechrootpkg.in b/makechrootpkg.in index 4b72a36..e4b7ca3 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -29,9 +29,9 @@ usage() { echo 'command:' echo 'mkarchroot /root base-devel' echo '' - echo 'This script reads {SRC,SRCPKG,PKG,LOG}DEST, MAKEFLAGS and PACKAGER' - echo 'from makepkg.conf(5), if those variables are not part of the' - echo 'environment.' + echo 'This script reads {SRC,SRCPKG,PKG,LOG}DEST, MAKEFLAGS, PACKAGER' + echo 'and COMPRESS{{G,X,LR,}Z,BZ2,LZO} from makepkg.conf(5),' + echo 'if those variables are not part of the environment.' echo '' echo "Default makepkg args: ${default_makepkg_args[*]}" echo '' @@ -63,12 +63,18 @@ usage() { # - LOGDEST # - MAKEFLAGS # - PACKAGER +# - COMPRESSGZ +# - COMPRESSXZ +# - COMPRESSLRZ +# - COMPRESSZ +# - COMPRESSBZ2 +# - COMPRESSLZO load_vars() { local makepkg_conf="$1" var [[ -f $makepkg_conf ]] || return 1 - for var in {SRC,SRCPKG,PKG,LOG}DEST MAKEFLAGS PACKAGER; do + for var in {SRC,SRCPKG,PKG,LOG}DEST MAKEFLAGS PACKAGER COMPRESS{{G,X,LR,}Z,BZ2,LZO}; do [[ -z ${!var:-} ]] && eval "$(grep -a "^${var}=" "$makepkg_conf")" done @@ -185,7 +191,9 @@ prepare_chroot() { sed -e '/^MAKEFLAGS=/d' -e '/^PACKAGER=/d' -i "$copydir/etc/makepkg.conf" for x in BUILDDIR=/build PKGDEST=/pkgdest SRCPKGDEST=/srcpkgdest SRCDEST=/srcdest LOGDEST=/logdest \ - "MAKEFLAGS='${MAKEFLAGS:-}'" "PACKAGER='${PACKAGER:-}'" + "MAKEFLAGS='${MAKEFLAGS:-}'" "PACKAGER='${PACKAGER:-}'" "COMPRESSGZ=(${COMPRESSGZ[*]:-})" \ + "COMPRESSXZ=(${COMPRESSXZ[*]:-})" "COMPRESSLRZ=(${COMPRESSLRZ[*]:-})" "COMPRESSZ=(${COMPRESSZ[*]:-})" \ + "COMPRESSBZ2=(${COMPRESSBZ2[*]:-})" "COMPRESSLZO=(${COMPRESSLZO[*]:-})" do grep -q "^$x" "$copydir/etc/makepkg.conf" && continue echo "$x" >>"$copydir/etc/makepkg.conf" -- 2.21.0 signature.asc Description: PGP signature