Bug#1021402: reproducible: Please force merged-/usr for build2

[Holger Levsen]

On Wed, Oct 12, 2022 at 12:05:18PM +0100, Luca Boccassi wrote:
> As per CTTE decision, buildds are still unmerged and will stay
> unmerged till at least after Bookworm as shipped.

ah right, thanks.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Where will your kids go when they become climate refugees?


signature.asc
Description: PGP signature

Bug#1021402: reproducible: Please force merged-/usr for build2

[Luca Boccassi]

On Wed, 12 Oct 2022 at 12:02, Holger Levsen  wrote:
>
> hi,
>
> On Mon, Oct 10, 2022 at 08:38:42PM +0100, Simon McVittie wrote:
> > With TC-member hat on: for at least bookworm, sid and experimental,
> > yes we would like reproducible-builds to test the non-merged-/usr
> > configuration (which will become unsupported) in one build, and the
> > merged-/usr configuration (which will become mandatory) in the other.
> > This is so that if a package is misbuilt in a merged-/usr chroot (or in
> > theory if it's misbuilt in a non-merged-/usr chroot), it will show up
> > as a non-reproducibility.
> [...]
>
> thanks for this verbose explaination. I'm still not fully convinced
> as (AIUI) by now everthing that ends up in bookworm will be build on
> Debian autobuilders, which (again, AIUI) all have usrmerged layout.

As per CTTE decision, buildds are still unmerged and will stay
unmerged till at least after Bookworm as shipped.

Kind regards,
Luca Boccassi

Bug#1021402: reproducible: Please force merged-/usr for build2

[Holger Levsen]

hi,

On Mon, Oct 10, 2022 at 08:38:42PM +0100, Simon McVittie wrote:
> With TC-member hat on: for at least bookworm, sid and experimental,
> yes we would like reproducible-builds to test the non-merged-/usr
> configuration (which will become unsupported) in one build, and the
> merged-/usr configuration (which will become mandatory) in the other.
> This is so that if a package is misbuilt in a merged-/usr chroot (or in
> theory if it's misbuilt in a non-merged-/usr chroot), it will show up
> as a non-reproducibility.
[...]

thanks for this verbose explaination. I'm still not fully convinced
as (AIUI) by now everthing that ends up in bookworm will be build on
Debian autobuilders, which (again, AIUI) all have usrmerged layout.

But fine, it doesn't really cost us anything/much to continue to test this
variation, so let's keep doing this.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

The past is over.


signature.asc
Description: PGP signature

Bug#1021402: reproducible: Please force merged-/usr for build2

[Simon McVittie]

On Mon, 10 Oct 2022 at 13:24:57 +0100, Luca Boccassi wrote:
> On Mon, 10 Oct 2022 at 13:15, Holger Levsen  wrote:
> > do we really want that? I understand this is supposed to be(come)
> > an unsupported configuration?
> 
> At least for the remainder of the Bookworm development cycle,
> definitely yes. After Bookworm is released we can re-evaluate.

With TC-member hat on: for at least bookworm, sid and experimental,
yes we would like reproducible-builds to test the non-merged-/usr
configuration (which will become unsupported) in one build, and the
merged-/usr configuration (which will become mandatory) in the other.
This is so that if a package is misbuilt in a merged-/usr chroot (or in
theory if it's misbuilt in a non-merged-/usr chroot), it will show up
as a non-reproducibility.

This lets us be more confident that building packages in a non-merged-/usr
chroot (currently being done on the official buildds) and building
packages in a merged-/usr chroot (what we should do in future) are both
valid things to do and will not break too many packages.

The most common failure mode is that if you build a package in a
merged-/usr chroot, that sometimes results in the binary package hard-coding
paths that are valid with merged-/usr but invalid for non-merged-/usr,
such as /usr/bin/sh and /bin/perl.

When everyone has merged-/usr systems, this failure mode will become
a non-issue and we can stop testing for it, but we cannot assume that
everyone running testing/unstable has merged-/usr systems until after
bookworm has been released. This is because the dist-upgrade from bullseye
to bookworm is done in an arbitrary order, so (almost) any package might
get upgraded before the new init-system-helpers pulls in usrmerge. See
 for
more details.

For trixie, and for unstable and experimental starting from the end
of the bookworm freeze, reproducible-builds should use merged-/usr for
both builds.

Thanks,
smcv

Bug#1021402: reproducible: Please force merged-/usr for build2

[Luca Boccassi]

On Mon, 10 Oct 2022 at 13:15, Holger Levsen  wrote:
>
> On Mon, Oct 10, 2022 at 12:45:24PM +0100, Luca Boccassi wrote:
> > Given we want one build as merged and one as unmerged, [...]
>
> do we really want that? I understand this is supposed to be(come)
> an unsupported configuration?

At least for the remainder of the Bookworm development cycle,
definitely yes. After Bookworm is released we can re-evaluate.

Kind regards,
Luca Boccassi

Bug#1021402: reproducible: Please force merged-/usr for build2

[Holger Levsen]

On Mon, Oct 10, 2022 at 12:45:24PM +0100, Luca Boccassi wrote:
> Given we want one build as merged and one as unmerged, [...]

do we really want that? I understand this is supposed to be(come)
an unsupported configuration?


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Der Mensch is' gut, aber die Leut' san a G'sindel!


signature.asc
Description: PGP signature

Bug#1021402: reproducible: Please force merged-/usr for build2

[Luca Boccassi]

On Mon, 10 Oct 2022 at 12:13, Holger Levsen  wrote:
>
> hi,
>
> thanks for the bug report and the patch, but wouldn't it be better
> to simple create the testing and unstable pbuilder base.tgzs *with*
> usrmerge and that's it?
>
> which means adopting bin/reproducible_setup_pbuilder.sh as needed
> too. (and bin/reproducible_common.sh too, because that's where we
> document the variations we're doing.)

Hi,

Given we want one build as merged and one as unmerged, that would mean
either storing both or having to convert back on the fly, which is
messy.
The hook patch seems simple and non-intrusive enough? Do you see any
issue with it?

Kind regards,
Luca Boccassi

Bug#1021402: reproducible: Please force merged-/usr for build2

[Holger Levsen]

hi,

thanks for the bug report and the patch, but wouldn't it be better
to simple create the testing and unstable pbuilder base.tgzs *with*
usrmerge and that's it?

which means adopting bin/reproducible_setup_pbuilder.sh as needed
too. (and bin/reproducible_common.sh too, because that's where we
document the variations we're doing.)


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

If you turn on the AC because it's too hot, you are making it worse.


signature.asc
Description: PGP signature

Bug#1021402: reproducible: Please force merged-/usr for build2

[Luca Boccassi]

Control: tags -1 patch

On Fri, 7 Oct 2022 at 16:21, Simon McVittie  wrote:
>
> Package: jenkins.debian.org
> Severity: normal
> X-Debbugs-Cc: usrme...@packages.debian.org, debian...@lists.debian.org
>
> The reproducible builds infrastructure tries to vary the merged-/usr
> status of the build system, like this:
>
> - base tarball: explicitly disable merged-/usr
> - build1: leave merged-/usr disabled
> - build2: pass "--extrapackages usrmerge" to pbuilder, to convert the
>   system to merged-/usr
>
> This has been very useful for detecting bugs of the same class as
> .
>
> However, now that new chroots are merged-/usr by default, I don't think
> this is working as intended. Disabling merged-/usr during base chroot
> creation now creates a flag file /etc/unsupported-skip-usrmerge-conversion
> representing "please don't merge /usr, contrary to the default", and
> installing the usrmerge package is not sufficient to undo the effect of
> that flag file.
>
> I believe the procedure to convert a non-merged-/usr QA chroot to
> merged-/usr is now something like this:
>
> 1. ensure the usrmerge package is installed
> 2. rm /etc/unsupported-skip-usrmerge-conversion
> 3. dpkg-reconfigure usrmerge
>
> Please do that for reproducible builds in build2, reinstating the
> previous setup where build1 is not merged-/usr but build2 is. I think
> implementing this on reproducible builds will require adding a pbuilder
> hook that does steps 2 and 3?

Turns out this is quite easy to implement, so here's a MR (only tested
in isolation):

https://salsa.debian.org/qa/jenkins.debian.net/-/merge_requests/143

Kind regards,
Luca Boccassi

Bug#1021402: reproducible: Please force merged-/usr for build2

[Luca Boccassi]

On Fri, 7 Oct 2022, 21:45 Simon McVittie,  wrote:

> On Fri, 07 Oct 2022 at 18:07:45 +0100, Luca Boccassi wrote:
> > Wouldn't it be sufficient to create the build2 chroot as merged-usr
> > from the start?
>
> That would also be fine, but I think the way the reproducible-builds
> infrastructure works is that it caches a single (non-merged-/usr)
> pbuilder tarball per suite and converts it to merged-/usr on-demand
> for build2, to avoid having to cache essentially the same content twice.
> The important thing here is that one of the two builds should be
> merged-/usr, and the other should not.
>
> Having a merged-/usr pbuilder tarball and unmerging it with
> dpkg-fsys-usrunmess for one of the two builds would also work, but I
> suspect that would be less reliable than starting from non-merged-/usr and
> merging one of the builds but not the other.


In this particular case, it would seem to me that it would be best to
optimize for correctness and safety (and less engineering cost) rather than
disk space saving?

Kind regards,
Luca Boccassi

>
>

Bug#1021402: reproducible: Please force merged-/usr for build2

[Simon McVittie]

On Fri, 07 Oct 2022 at 18:07:45 +0100, Luca Boccassi wrote:
> Wouldn't it be sufficient to create the build2 chroot as merged-usr
> from the start?

That would also be fine, but I think the way the reproducible-builds
infrastructure works is that it caches a single (non-merged-/usr)
pbuilder tarball per suite and converts it to merged-/usr on-demand
for build2, to avoid having to cache essentially the same content twice.
The important thing here is that one of the two builds should be
merged-/usr, and the other should not.

Having a merged-/usr pbuilder tarball and unmerging it with
dpkg-fsys-usrunmess for one of the two builds would also work, but I
suspect that would be less reliable than starting from non-merged-/usr and
merging one of the builds but not the other.

smcv

Bug#1021402: reproducible: Please force merged-/usr for build2

[Luca Boccassi]

On Fri, 7 Oct 2022 at 16:21, Simon McVittie  wrote:
>
> Package: jenkins.debian.org
> Severity: normal
> X-Debbugs-Cc: usrme...@packages.debian.org, debian...@lists.debian.org
>
> The reproducible builds infrastructure tries to vary the merged-/usr
> status of the build system, like this:
>
> - base tarball: explicitly disable merged-/usr
> - build1: leave merged-/usr disabled
> - build2: pass "--extrapackages usrmerge" to pbuilder, to convert the
>   system to merged-/usr
>
> This has been very useful for detecting bugs of the same class as
> .
>
> However, now that new chroots are merged-/usr by default, I don't think
> this is working as intended. Disabling merged-/usr during base chroot
> creation now creates a flag file /etc/unsupported-skip-usrmerge-conversion
> representing "please don't merge /usr, contrary to the default", and
> installing the usrmerge package is not sufficient to undo the effect of
> that flag file.
>
> I believe the procedure to convert a non-merged-/usr QA chroot to
> merged-/usr is now something like this:
>
> 1. ensure the usrmerge package is installed
> 2. rm /etc/unsupported-skip-usrmerge-conversion
> 3. dpkg-reconfigure usrmerge
>
> Please do that for reproducible builds in build2, reinstating the
> previous setup where build1 is not merged-/usr but build2 is. I think
> implementing this on reproducible builds will require adding a pbuilder
> hook that does steps 2 and 3?

Wouldn't it be sufficient to create the build2 chroot as merged-usr
from the start?

Kind regards,
Luca Boccassi

Bug#1021402: reproducible: Please force merged-/usr for build2

[Simon McVittie]

Package: jenkins.debian.org
Severity: normal
X-Debbugs-Cc: usrme...@packages.debian.org, debian...@lists.debian.org

The reproducible builds infrastructure tries to vary the merged-/usr
status of the build system, like this:

- base tarball: explicitly disable merged-/usr
- build1: leave merged-/usr disabled
- build2: pass "--extrapackages usrmerge" to pbuilder, to convert the
  system to merged-/usr

This has been very useful for detecting bugs of the same class as
.

However, now that new chroots are merged-/usr by default, I don't think
this is working as intended. Disabling merged-/usr during base chroot
creation now creates a flag file /etc/unsupported-skip-usrmerge-conversion
representing "please don't merge /usr, contrary to the default", and
installing the usrmerge package is not sufficient to undo the effect of
that flag file.

I believe the procedure to convert a non-merged-/usr QA chroot to
merged-/usr is now something like this:

1. ensure the usrmerge package is installed
2. rm /etc/unsupported-skip-usrmerge-conversion
3. dpkg-reconfigure usrmerge

Please do that for reproducible builds in build2, reinstating the
previous setup where build1 is not merged-/usr but build2 is. I think
implementing this on reproducible builds will require adding a pbuilder
hook that does steps 2 and 3?

Thanks,
smcv