Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default? /usr by default

[Simon McVittie]

Control: retitle 914897 tech-ctte: Should debootstrap disable merged /usr by 
default?

I'm retitling the bug to avoid misrepresenting the technical committee's
position on this. We have been asked to overrule the debootstrap
maintainer, but we have not yet come to a conclusion on whether we should.

On Wed, 05 Dec 2018 at 13:25:36 +0900, Hideki Yamane wrote:
>  Can we check and track this behavior in our packages?

Yes, we now do. The reproducible-builds infrastructure now uses unmerged
/usr for the first build and merged /usr for the second, since
 was fixed.

debootstrap since 1.0.111 also mitigates this class of bugs by disabling
merged /usr for --variant=buildd chroots (this was
).

Julien Cristau thinks #914208 was a sufficient/proportionate change, and
doesn't want to go further and default to --no-merged-usr for non-buildd
chroots (and in particular new debian-installer installations).

Ian Jackson thinks #914208 is not a sufficient answer (Ian, I hope I'm
not misrepresenting you here?), and escalated this bug to the technical
committee, asking us to overrule the debootstrap maintainers.

If the debootstrap/debian-installer maintainers agree with Ian on this,
then there is no need for the technical committee to consider his request
to overrule you, which is why Didier asked for your opinion on this
issue before attempting to come to a decision. If you agree with Julien,
then the technical committee still needs to consider this question.

>  Once disable merged-usr is good to prevent confusion but we detect such
>  as a bug for manage non-merged-usr and merged-usr Debian system in the end,
>  right? (or do you want to stay change in debootstrap 1.0.111 forever?)

The technical committee have not come to a conclusion on this.

My personal opinions (not overruling anyone):

If merged /usr becomes the only supported system layout at some future
time, then the change in debootstrap 1.0.111 can certainly be reverted
at that time. (If merged /usr does not become the only supported system
layout, this does not apply.)

It might also be considered appropriate to revert the change in
debootstrap 1.0.111 if data from reproducible-builds demonstrates that
bugs similar to #913226 have all been fixed or are very rare, but this
should be done cautiously, and certainly not before buster is released.

smcv

Processed: Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default? /usr by default

[Debian Bug Tracking System]

Processing control commands:

> retitle 914897 tech-ctte: Should debootstrap disable merged /usr by default?
Bug #914897 [tech-ctte] debootstrap, buster: Please disabled merged /usr by 
default
Changed Bug title to 'tech-ctte: Should debootstrap disable merged /usr by 
default?' from 'debootstrap, buster: Please disabled merged /usr by default'.

-- 
914897: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914897
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#914897: we do have a complete archive rebuild for buster and sid amd64 (Re: Bug#914897: debating the wrong thing)

[Holger Levsen]

On Tue, Dec 04, 2018 at 09:58:09PM +0100, Ansgar Burchardt wrote:
> > We later learned only a part of the archive got rebuilt since the bad
> > debootstrap backport.
> Yes, some packages were not yet rebuilt in testing, but having them
> rebuilt in unstable is enough.

looking at https://tests.reproducible-builds.org/debian/index_performance.html
and specifically at
https://tests.reproducible-builds.org/debian/unstable/amd64/stats_builds_age.png
and 
https://tests.reproducible-builds.org/debian/buster/amd64/stats_builds_age.png
it seems that all of unstable and buster has been rebuild on amd64 since we 
enabled these tests (around Nov 9th).

Looking more specifically at
https://tests.reproducible-builds.org/debian/index_amd64_oldies.html it
actually seems like the amd64 buster rebuild (testing usrmerge diffs) will only
be completed in 24h or so ;)

arm64 will be done soon as well, armhf will take a bit more time and
i386 some more.

If you only want to look at one url, look at the last one above,
index_oldies...


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Bug#914897: debating the wrong thing

[Svante Signell]

On Tue, 2018-12-04 at 21:58 +0100, Ansgar Burchardt wrote:
> 
> If we keep merged-/usr as default then we can /recommend/ people to
> install usrmerge to switch to merged-/usr; reducing the difference
> between newly-installed and existing setups is a good idea IMHO.  I
> think I filed a report for this two years ago.
> 
> Maybe we should also mention somewhere that it might be useful to not
> switch build environments (yet).

How about this case:

- Make as default non merged-/usr for all buildds.
- Make a choice of non merged-/usr or merged-/usr in the installer.
- Users wanting merged-/usr install the usrmerge package and convert to merged-
/usr.

- We have two alternatives here:
1) For those not wanting merged-/usr: 
   All is fine.

2) For those having merged-/usr installed:
   Re-run usrmerge to convert all newly installed packages to merged-/usr.
   Is this possible or is installing that package a install-once-only?

Just a thought!

Re: Bug#914897: debating the wrong thing

[Gunnar Wolf]

Svante Signell dijo [Wed, Dec 05, 2018 at 02:03:19PM +0100]:
> > If we keep merged-/usr as default then we can /recommend/ people to
> > install usrmerge to switch to merged-/usr; reducing the difference
> > between newly-installed and existing setups is a good idea IMHO.  I
> > think I filed a report for this two years ago.
> > 
> > Maybe we should also mention somewhere that it might be useful to not
> > switch build environments (yet).
> 
> How about this case:
> 
> - Make as default non merged-/usr for all buildds.
> - Make a choice of non merged-/usr or merged-/usr in the installer.
> - Users wanting merged-/usr install the usrmerge package and convert to 
> merged-
> /usr.
> 
> - We have two alternatives here:
> 1) For those not wanting merged-/usr: 
>All is fine.
> 
> 2) For those having merged-/usr installed:
>Re-run usrmerge to convert all newly installed packages to merged-/usr.
>Is this possible or is installing that package a install-once-only?

AFAICT, this would mostly cover the issues, at least as far as I
understand them. There is still one extra point: Computers used by
Debian contributors to build their packages on. If I build my package
on a merged-/usr system, it might end up breaking in non-merged
systems.

Throwaway binary uploads seems like a safe bet for this issue. Now -
Are we ready to do it?


signature.asc
Description: PGP signature

Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

[Gunnar Wolf]

Ansgar Burchardt dijo [Wed, Dec 05, 2018 at 08:17:56AM +0100]:
> The Reproducible Builds project was so kind to help and now runs one
> build in a non-merged-/usr and a second build in a merged-/usr
> environment.  Packages that hardcode the path to utilities, but would
> pick up the wrong one in a merged-/usr environment will result in a
> difference between the two builds and can thus be found.
> 
> See [1] for an overview of issues found this way; as the entire archive
> was already rebuilt in this setup, there shouldn't be many more issues
> of this type that we don't know about[2].
> (...)

Thanks for this report, Ansgar. This clearly sums up the issue. I am
amazed the whole archive managed to be rebuilt on such a short time!

I guess the next step is to file the multiple bugs pending to be
filed.

> Bug reports were already submitted for over half the packages, often
> including a simple patch (usually something like adding BASH=/bin/bash
> to dh_auto_configure).
> 
> So we look to be on a good track to address the remaining issues.

...And I guess this points towards the Technical Committee not having
to intervene in the issue. Which is, IMO, the best possible outcome.

Thanks to everybody following through with this!