Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Didier 'OdyX' Raboud]

Le lundi, 4 mars 2019, 22.28:38 h CET Margarita Manterola a écrit :
> Hi,
> 
> > - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> > 
> > The winners are:
> >  Option M `middle`
> >  Option H `hard`
> > 
> > - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> > 
> > Dear Marga, as Chair, could you please make use of your casting vote to
> > break this tie?
> 
> I'm using my casting vote to vote in favor of M `middle` (i.e. consider
> that the desirable situation at the time of bullseye is that both directory
> schemes are allowed, all packages can be built on either).
> 
> My rationale for taking this decision is as follows:
> 
> Right now we are not ready to migrate to building on merged-usr systems in
> Debian, there are still 29 known packages that are broken and need to be
> fixed.  My expectation is that those packages will be fixed in the not so
> distant future (i.e.  before bullseye) and that after that, the buildd
> profile in debootstrap will default to having a merged /usr, so that new
> buildd chroots will use that setup.
> 
> However, we have no control over how fast individual developers and
> derivative distributions will migrate to the new format. It's possible that
> more time will be required until everyone is ready to migrate.
> 
> Additionally, as of our current understanding of the issue, there are no
> known problems for building on a non-merged system. Supporting this
> setup does not imply additional work from the point of view of our
> maintainers (for now).
> 
> In the future, it would be a bug if a problem is discovered with building a
> package on a non-merged /usr system. I understand that this would mean
> additional work to the maintainers of such a package, but at least as of
> today this is a non-issue. Eventually, when fixing such bugs becomes a
> significant burden for our maintainers, it can be decided that the setup is
> no-longer supported, but my personal recommendation would be to wait at
> least until bullseye+1. That's why I'm voting "Middle" for bullseye.

I have announced this decision on debian-devel-announce:
https://lists.debian.org/debian-devel-announce/2019/03/msg1.html

I am therefore hereby closing this bug.

Thank you to all involved parties!

Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Margarita Manterola]

Hi,

> - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> 
> The winners are:
>Option M `middle`
>Option H `hard`
> 
> - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> 
> Dear Marga, as Chair, could you please make use of your casting vote to break 
> this tie?

I'm using my casting vote to vote in favor of M `middle` (i.e. consider
that the desirable situation at the time of bullseye is that both directory
schemes are allowed, all packages can be built on either).

My rationale for taking this decision is as follows:

Right now we are not ready to migrate to building on merged-usr systems in
Debian, there are still 29 known packages that are broken and need to be
fixed.  My expectation is that those packages will be fixed in the not so
distant future (i.e.  before bullseye) and that after that, the buildd
profile in debootstrap will default to having a merged /usr, so that new
buildd chroots will use that setup.

However, we have no control over how fast individual developers and
derivative distributions will migrate to the new format. It's possible that
more time will be required until everyone is ready to migrate.

Additionally, as of our current understanding of the issue, there are no
known problems for building on a non-merged system. Supporting this
setup does not imply additional work from the point of view of our
maintainers (for now).

In the future, it would be a bug if a problem is discovered with building a
package on a non-merged /usr system. I understand that this would mean
additional work to the maintainers of such a package, but at least as of
today this is a non-issue. Eventually, when fixing such bugs becomes a
significant burden for our maintainers, it can be decided that the setup is
no-longer supported, but my personal recommendation would be to wait at
least until bullseye+1. That's why I'm voting "Middle" for bullseye.

-- 
Thanks,
Marga


signature.asc
Description: PGP signature

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Didier 'OdyX' Raboud]

Le lundi, 25 février 2019, 14.58:09 h CET Didier 'OdyX' Raboud a écrit :
> I call for vote immediately on the following resolution.
> 
> === Resolution ===
> 
> The Technical Committee resolves to decline to override the debootstrap
> maintainers.
> 
> Furthermore, using its §6.1.5 "Offering advice" power, the Technical
> Committee considers that the desirable solution at the time of `bullseye`
> is:
> 
> * W: `weak`: both directory schemes are allowed, but packages should only
>  be built on hosts with classical directory schemes (or in such
>  chroots)
> 
> * M: `middle`: both directory schemes are allowed, and packages (including
>official packages) can be built on hosts with either
> classical or "merged `/usr`" directory schemes
> 
> * H: `hard`: both directory schemes are allowed, but packages should only
>  be built on hosts with "merged `/usr`" directory schemes (or in
> such chroots)
> 
> * FD: Further Discussion
> 
> === End Resolution ===

Dear all,

with 6 votes in, and one week gone, the outcome of this vote is still in 
doubt; here's an abstract of the vote calculation (a clearsigned verbose 
version is attached):

/--WMHF
V: 3214 odyx
V: 2134 bremner
V: 4213 gwolf
V: 3124 ntyni
V: 2134 marga
V: 4213 fil
  Option
  W M H F 
===   ===   ===   === 
Option W0 2 4 
Option M  6   3 6 
Option H  4 3   6 
Option F  2 0 0   

Option W Reached quorum: 4 >= 2
Option M Reached quorum: 6 >= 2
Option H Reached quorum: 6 >= 2

Option W passes Majority.   2.000 (4/2) > 1

  Option M defeats Option W by (   6 -0) =6 votes.
  Option H defeats Option W by (   4 -2) =2 votes.
  Option W defeats Option F by (   4 -2) =2 votes.
  Option M defeats Option F by (   6 -0) =6 votes.
  Option H defeats Option F by (   6 -0) =6 votes.

- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

The winners are:
 Option M `middle`
 Option H `hard`

- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=


So. We have a tie, and the voting period is over. I am quite confident we can 
neither continue discussing (FD lost to M and H unanimously, and to W by 2), 
nor allow smcv to vote (the voting period is over). We're left with the 
Chair's casting vote (as per §6.3.2). So…

Dear Marga, as Chair, could you please make use of your casting vote to break 
this tie?

Cheers,
OdyX-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Starting results calculation at Mon Mar  4 15:34:31 2019

/--WMHF
V: 3214 odyx
V: 2134 bremner
V: 4213 gwolf
V: 3124 ntyni
V: 2134 marga
V: 4213 fil
Option W "`weak`: both directory schemes are allowed, but packages should only 
be built on hosts with classical directory schemes (or in such chroots)"
Option M "`middle`: both directory schemes are allowed, and packages (including 
official packages) can be built on hosts with either classical or "merged 
`/usr`" directory schemes"
Option H "`hard`: both directory schemes are allowed, but packages should only 
be built on hosts with "merged `/usr`" directory schemes (or in such chroots)"
Option F "Further Discussion"

In the following table, tally[row x][col y] represents the votes that
option x received over option y.

  Option
  W M H F 
===   ===   ===   === 
Option W0 2 4 
Option M  6   3 6 
Option H  4 3   6 
Option F  2 0 0   



Looking at row 2, column 1, M
received 6 votes over W

Looking at row 1, column 2, W
received 0 votes over M.

Option W Reached quorum: 4 >= 2
Option M Reached quorum: 6 >= 2
Option H Reached quorum: 6 >= 2


Option W passes Majority.   2.000 (4/2) > 1


  Option M defeats Option W by (   6 -0) =6 votes.
  Option H defeats Option W by (   4 -2) =2 votes.
  Option W defeats Option F by (   4 -2) =2 votes.
  Option M defeats Option F by (   6 -0) =6 votes.
  Option H defeats Option F by (   6 -0) =6 votes.


The Schwartz Set contains:
 Option M "`middle`: both directory schemes are allowed, and packages 
(including official packages) can be built on hosts with either classical or 
"merged `/usr`" directory schemes"
 Option H "`hard`: both directory schemes are allowed, but packages 
should only be built on hosts with "merged `/usr`" directory schemes (or in 
such chroots)"



- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

The winners are:
 Option M "`middle`: both directory schemes are allowed, and packages 
(including official packages) can be built on hosts with either classical or 
"merged `/usr`" directory schemes"
 Option H "`hard`: both directory scheme

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Gunnar Wolf]

Didier 'OdyX' Raboud dijo [Mon, Feb 25, 2019 at 02:58:09PM +0100]:
> === Resolution ===
> 
> The Technical Committee resolves to decline to override the debootstrap
> maintainers.
> 
> Furthermore, using its §6.1.5 "Offering advice" power, the Technical
> Committee considers that the desirable solution at the time of `bullseye` is:
> 
> * W: `weak`: both directory schemes are allowed, but packages should only
>  be built on hosts with classical directory schemes (or in such
>  chroots)
> 
> * M: `middle`: both directory schemes are allowed, and packages (including
>official packages) can be built on hosts with either classical
>or "merged `/usr`" directory schemes
> 
> * H: `hard`: both directory schemes are allowed, but packages should only
>  be built on hosts with "merged `/usr`" directory schemes (or in
>  such chroots)
> 
> * FD: Further Discussion
> 
> === End Resolution ===

My vote is:

H > M > FD > W


signature.asc
Description: PGP signature

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Didier 'OdyX' Raboud]

Dear Technical Committee members,

I call for vote immediately on the following resolution.

=== Resolution ===

The Technical Committee resolves to decline to override the debootstrap
maintainers.

Furthermore, using its §6.1.5 "Offering advice" power, the Technical
Committee considers that the desirable solution at the time of `bullseye` is:

* W: `weak`: both directory schemes are allowed, but packages should only
 be built on hosts with classical directory schemes (or in such
 chroots)

* M: `middle`: both directory schemes are allowed, and packages (including
   official packages) can be built on hosts with either classical
   or "merged `/usr`" directory schemes

* H: `hard`: both directory schemes are allowed, but packages should only
 be built on hosts with "merged `/usr`" directory schemes (or in
 such chroots)

* FD: Further Discussion

=== End Resolution ===



=== Rationale ===

## What is "merged `/usr`"

"Merged `/usr`" describes a possible future standard directories scheme in
which the `/{bin,sbin,lib*}/` directories have been made superfluous through
replacing them by symlinks to their `/usr` equivalents
(`/usr/{bin,sbin,lib*}`).
The motivation to get Debian systems to converge towards such a scheme is
vastly documented elsewhere ([FDO's TheCaseForTheUsrMerge][0],
[wiki.d.o UsrMerge][1]) but can be summarized as the following points:

* having separate `/` and `/usr` filesystems has been useful in the past for
booting without initramfs onto a minimal root filesystem that carried just
enough to mount the `/usr` filesystem later in the boot process. Given the
evolution of physical hosts' capabilities, initramfs'es have been default in
Debian (and elsewhere) for a long time, and most systems no longer have an
intermediate state during boot in which they have only `/`, but not `/usr`,
mounted. Booting hosts through that intermediate state is not systematically
tested in Debian anymore.
* another use-case is to share system files from `/usr` between hosts (over a
network link) or containers (locally) which use different data or
configuration. Having all software under `/usr` (instead of spread between
`/` and `/usr`) makes the centralized update and the sharing easier.
* the packaging infrastructure to install files outside of `/usr` (e.g.
installing libs under `/lib` instead of `/usr/lib`) is not standard and
represents technical debt.
* given its status as remnant "folklore", the distinction between what
_needs_ to be shipped in `/` and what can stay in `/usr` is often interpreted
arbitrarily;
* allowing shipment of identically-named libraries or binaries in different
paths can confuse common understanding of paths precedence.

[0]: https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
[1]: https://wiki.debian.org/UsrMerge

The arguments against moving the base directories' scheme towards "merged
`/usr`" are as follows:

* there's no gain in disrupting something that is not inherently broken;
* `/{bin,sbin,lib*}/` → `/usr/{bin,sbin,lib*}/` symlinks create confusing
views of the system (`/bin/cat` and `/usr/bin/cat` are the same file), and
dpkg doesn't support this situation cleanly:
[#134758](https://bugs.debian.org/134758).
* it is possible for distributions to converge towards having all system
files in `/usr` in finite time instead of shortcutting this migration with
`/{bin,sbin,lib*}/` → `/usr/{bin,sbin,lib*}/` symlinks.

The compatibility symbolic links `/lib` → `/usr/lib` and `/lib64` →
`/usr/lib64` are required by the various CPUs' platform ABIs (for example
i386 requires `/lib/ld-linux.so.2` to resolve to glibc's `ld.so`, and amd64
requires `/lib64/ld-linux-x86-64.so.2`) so there are no plans to remove them
altogether. Similarly, removing `/bin` is not under consideration because it
would break the assumption that `/bin/sh` exists, and removing `/sbin` would
break the assumption that `/sbin/fsck.*` and `/sbin/mount.*` exist.

## "merged `/usr`" in Debian

In Debian buster, the current testing suite, "merged `/usr`" is only
considered for implementation with symlinks (there are no proposals for
simply dropping `/{bin,sbin,lib*}`) and is implemented in two main ways:

* existing hosts can be made to have a "merged `/usr`" by installing the
[usrmerge][2] package;
* new hosts get the `/{bin,sbin,lib*}/`→ `/usr/{bin,sbin,lib*}/` symlinks
by default when using debootstrap >= 1.0.102.

The usrmerge package contains a `/usr/lib/convert-usrmerge` perl executable
that runs in `postinst`, that will move the contents of `/{bin,sbin,lib*}/`
and replace these directories with symlinks when empty.

It is also possible to merge `/usr` in other ways, for example with
`debootstrap --merged-usr` or by bootstrapping into a chroot that already
contains the necessary symlinks.

[2]: https://tracker.debian.org/pkg/usrmerge

## Issues from "merged `/usr`"

Since the default change in debootstrap 1.0.102, some issues ha

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Didier 'OdyX' Raboud]

Le jeudi, 21 février 2019, 15.28:23 h CET Ian Jackson a écrit :
> Back in the wider world, of course many people build packages on
> Debian systems for installation `somewhere else'.  I have done it
> myself and probably many of the people reading this message have too.
> 
> What is implicitly going on here is that it is mostly-tacitly[2] being
> assumed (or declared) that `I built this .deb on my laptop[1] and gave
> it to my friend' is wrongheaded.

I don't think it is wrongheaded, but I do think that assuming that it is meant 
to work consistently under any circumstances, is.  There are _so many_ 
circumstances that make this exercise error-prone:

"I built this amd64 .deb and gave it to my friend for use on their RasbperryPi 
arm64 host"

"I built this .deb on my laptop on which I had installed a /usr/local/bin/
grep"

"I built this .deb an gave it to my friend who runs CentOS"

etc
 
> I don't think doing that is wrongheaded.  Certainly it is extremely
> common; we don't have any public pronouncements saying it is somehow
> wrong; and we have had little discussion where we as a project decided
> that this was now a use case which we feel free to just break.

Frankly, while it's not _broken_ per se, it is and has always really been 
fragile.  True, merged-/usr arguably worsens _one_ aspect of building packages 
on one's host, but in ways that are really easy to detect, and warn for.

> Personally I think that this is a very important thing to keep
> working.  It is the very core of users' collective software freedom.

You seem to be conflating "building on one's host" with "outside of any 
chroot", and equating this with "users' collective software freedom" is really 
making your point a disservice.  Being able to improve, build and share binary 
artifacts of free software is _vital_, and the whole point of building 
software distributions in the first place, but insisting that these rights are 
only "truly" exercised when builds are done outside of chroots is 
disingenuine. .deb packages already have to be built using certain tools, so 
we do set the limit (of what minimal set of tools are mandatory) somewhere, 
and my point is that this limit might not be at the right place. I'd be 
totally OK with saying "the only supported way to build .deb packages from 
buster on is using by pdebuild; however, you _can_ use dpkg-buildpackage 
alone, but be aware (and you'll get warnings) that this can lead to building 
.deb packages with undesireable properties."  The second half of the sentence 
is already true, and has always been; we probably just failed at communicating 
it sufficiently clearly.

> And that software freedom needs to be easy to exercise in practice.
> Despite a lot of excellent tooling, chroots are still not entirely
> trivial to set up and maintain.

Then that's maybe what we should be fixing.  

> I would invite the TC to read this manpage, which is trying to explain
> to a Debian user how to change the programs on their own computer
>   https://manpages.debian.org/stretch-backports/dgit/dgit-user.7.en.html
> and then consider how much even worse it would be if we had to write
> about chroots too.

Perhaps dpkg-buildpackage should be grown to build in a chroot by default? Or 
get an option to do that?

Really, I think we should stop considering that .debs built outside of 
"controlled environments" are more than temporary software transports. Our 
"standard" package building tools should build in such environments by 
default.

> To TC members who are minded to uphold the current approach, I would
> ask: can you please explicitly state your opinion on this ?  That is:
> 
>   Is it OK for a Debian user to build .deb on their laptop and give it
>   to their friend ?

Yes; provided that said .deb is built in a sane (sanitized / controlled) 
environment.  As you're talking about non-chroot builds; it is OK, provided 
that the tools warn about that. In that area, I think the "Build-Tainted-By" 
are steps in the right direction.

>   If it is OK, how will we make sure that it does not pointlessly break ?

As it is already broken in many ways, your question is biased. But I think the 
good way is to normalize "proper builds", by making sure our standard tools 
"do the right thing" by default.

> I readily admit that there are many ways that this can produce a
> result significantly different to the official Debian package,
> particularly because the package build system may configure itself
> differently in the the presence of unexpected.  The resulting package
> is probably not going to be suitable for wide distribution.
> 
> But those kind of problems are (a) not serious in practice
> (b) generally have obvious symptoms and (c) are easily worked around
> by various means.  Working around a usrmerge-generated failure is
> difficult; it usually involves doing serious violence to the upstream
> build system, or perhaps horrific rules file bodges.

Given a Build-Tainted-By flag in the binary artifact

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Didier 'OdyX' Raboud]

Le mardi, 19 février 2019, 01.59:18 h CET Steve McIntyre a écrit :
> While I'm not necessarily disagreeing with the overall point(s) here,
> some of the points in this list of arguments seem dubious at
> best. Maybe you could expand?

Sure. Sorry for publishing a new ballot before answering.

> >* booting with `/` only is not systematically tested in Debian anymore;
> 
> I'm assuming you mean "/ without /usr" here?

Yes. Clarified this point by merging it with the "separate / and /usr" one 
above.

> >* the packaging infrastructure to install files outside of `/usr` is not
> >  standard and represents technical debt:
> 
> I have no idea what you're suggesting here.

Same here (clarified in the ballot). What I'm trying to say here is that 
installing (e.g.) libs to /lib instead of /usr/lib is usually a deviation from 
standard software building (either in upstream packaging, or in debian/rules), 
that needs to be maintained on the long-term.

Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Didier 'OdyX' Raboud]

Le lundi, 18 février 2019, 08.58:53 h CET Didier 'OdyX' Raboud a écrit :
> Dear Technical Committee members,
> (CC'ed to submitter, and debootstrap maintainers for information and
> feedack)
> 
> Here's the current state of the draft resolution; which `master` is at
> https://salsa.debian.org/debian/tech-ctte/blob/master/914897_merged_usr/ball
> ot.md
> 
> I will submit it to vote on Friday 22.; the discussion period is open!

Thank you for the various comments. I have amended the ballot (which is more a
"explanation text + short ballot" incorporating the suggestions from the IRC
meeting as well as taking into accounts remarks on this bug.

I intend to answer some mails on that bug, and call for a vote on Sunday I
guess.

See: 
https://salsa.debian.org/debian/tech-ctte/blob/master/914897_merged_usr/ballot.md
for the markdown-rendered version; and below:

# #914897: tech-ctte: Should debootstrap disable merged `/usr` by default?

## What is "merged `/usr`"

"Merged `/usr`" describes a possible future standard directories scheme in
which the `/{bin,sbin,lib*}/` directories have been made superfluous through
replacing them by symlinks to their `/usr` equivalents
(`/usr/{bin,sbin,lib*}`).
The motivation to get Debian systems to converge towards such a scheme is
vastly documented elsewhere ([FDO's TheCaseForTheUsrMerge][0],
[wiki.d.o UsrMerge][1]) but can be summarized as the following points:

* having separate `/` and `/usr` filesystems has been useful in the past for
booting without initramfs onto a minimal root filesystem that carried just
enough to mount the `/usr` filesystem later in the boot process. Given the
evolution of physical hosts' capabilities, initramfs'es have been default in
Debian (and elsewhere) for a long time, and most systems no longer have an
intermediate state during boot in which they have only `/`, but not `/usr`,
mounted. Booting hosts through that intermediate state is not systematically
tested in Debian anymore.
* another use-case is to share system files from `/usr` between hosts (over a
network link) or containers (locally) which use different data or
configuration. Having all software under `/usr` (instead of spread between `/`
and `/usr`) makes the centralized update and the sharing easier.
* the packaging infrastructure to install files outside of `/usr` (e.g.
installing libs under `/lib` instead of `/usr/lib`) is not standard and
represents technical debt.
* given its status as remnant "folklore", the distinction between what _needs_
to be shipped in `/` and what can stay in `/usr` is often interpreted
arbitrarily;
* allowing shipment of identically-named libraries or binaries in different
paths can confuse common understanding of paths precedence.

[0]: https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
[1]: https://wiki.debian.org/UsrMerge

The arguments against moving the base directories' scheme towards
"merged `/usr`" are as follows:

* there's no gain in disrupting something that is not inherently broken;
* `/{bin,sbin,lib*}/` → `/usr/{bin,sbin,lib*}/` symlinks create confusing
views of the system (`/bin/cat` and `/usr/bin/cat` are the same file), and
dpkg doesn't support this situation cleanly:
[#134758](https://bugs.debian.org/134758).
* it is possible for distributions to converge towards having all system files
in `/usr` in finite time instead of shortcutting this migration with
`/{bin,sbin,lib*}/` → `/usr/{bin,sbin,lib*}/` symlinks.

The compatibility symbolic links `/lib` → `/usr/lib` and `/lib64` →
`/usr/lib64` are required by the various CPUs' platform ABIs (for example i386
requires `/lib/ld-linux.so.2` to resolve to glibc's `ld.so`, and amd64
requires `/lib64/ld-linux-x86-64.so.2`) so there are no plans to remove them
altogether. Similarly, removing `/bin` is not under consideration because it
would break the assumption that `/bin/sh` exists, and removing `/sbin` would
break the assumption that `/sbin/fsck.*` and `/sbin/mount.*` exist.

## "merged `/usr`" in Debian

In Debian buster, the current testing suite, "merged `/usr`" is only
considered for implementation with symlinks (there are no proposals for simply
dropping `/{bin,sbin,lib*}`) and is implemented in two main ways:

* existing hosts can be made to have a "merged `/usr`" by installing the
[usrmerge][2] package;
* new hosts get the `/{bin,sbin,lib*}/`→ `/usr/{bin,sbin,lib*}/` symlinks by
default when using debootstrap >= 1.0.102.

The usrmerge package contains a `/usr/lib/convert-usrmerge` perl executable
that runs in `postinst`, that will move the contents of `/{bin,sbin,lib*}/`
and replace these directories with symlinks when empty.

It is also possible to merge `/usr` in other ways, for example with
`debootstrap --merged-usr` or by bootstrapping into a chroot that already
contains the necessary symlinks.

[2]: https://tracker.debian.org/pkg/usrmerge

## Issues from "merged `/usr`"

Since the default change in debootstrap 1.0.102, some issues have arisen.
Due to the fact that s

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Ian Jackson]

Steve McIntyre writes ("Bug#914897: tech-ctte: Should debootstrap disable 
merged /usr by default?"):
> There is a deeper worry about builds that may be done against the
> "weak" background. Although buildd chroots are easily fixed up,
> there's going to be a (small, but unknown) set of maintainers who
> might be uploading binaries from merged systems. I think we're making
> good progress on source-only uploads and building in clean chroots
> etc., but...  It's also likely not easy to pick up on "wrong" binary
> packages built this way.

I think this is a valid concern but puts it far too narrowly.


There is a persistent misunderstanding embodied in your comments here
(or rather, embodied in a significant omission): the notion that the
only reason to build a package on a Debian system is for upload to
Debian.

Once I put it like that, this notion is obviously false.  What is
happening is that the conversation is focusing on our own needs,
within Debian to help us produce Debian, to the exclusion of the needs
of our users.  That is a natural tendency of course, but we must
resist it.


Back in the wider world, of course many people build packages on
Debian systems for installation `somewhere else'.  I have done it
myself and probably many of the people reading this message have too.

What is implicitly going on here is that it is mostly-tacitly[2] being
assumed (or declared) that `I built this .deb on my laptop[1] and gave
it to my friend' is wrongheaded.

I don't think doing that is wrongheaded.  Certainly it is extremely
common; we don't have any public pronouncements saying it is somehow
wrong; and we have had little discussion where we as a project decided
that this was now a use case which we feel free to just break.


Personally I think that this is a very important thing to keep
working.  It is the very core of users' collective software freedom.

And that software freedom needs to be easy to exercise in practice.
Despite a lot of excellent tooling, chroots are still not entirely
trivial to set up and maintain.

I would invite the TC to read this manpage, which is trying to explain
to a Debian user how to change the programs on their own computer
  https://manpages.debian.org/stretch-backports/dgit/dgit-user.7.en.html
and then consider how much even worse it would be if we had to write
about chroots too.


To TC members who are minded to uphold the current approach, I would
ask: can you please explicitly state your opinion on this ?  That is:

  Is it OK for a Debian user to build .deb on their laptop and give it
  to their friend ?  If it is OK, how will we make sure that it does
  not pointlessly break ?


I readily admit that there are many ways that this can produce a
result significantly different to the official Debian package,
particularly because the package build system may configure itself
differently in the the presence of unexpected.  The resulting package
is probably not going to be suitable for wide distribution.

But those kind of problems are (a) not serious in practice
(b) generally have obvious symptoms and (c) are easily worked around
by various means.  Working around a usrmerge-generated failure is
difficult; it usually involves doing serious violence to the upstream
build system, or perhaps horrific rules file bodges.


Thanks,
Ian.


[1] Implicitly, without using a chroot.
[2] IIRC some people suggested this explicitly in the thread in
d-devel.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Ansgar]

Steve McIntyre writes:
> There is a deeper worry about builds that may be done against the
> "weak" background. Although buildd chroots are easily fixed up,
> there's going to be a (small, but unknown) set of maintainers who
> might be uploading binaries from merged systems. I think we're making
> good progress on source-only uploads and building in clean chroots
> etc., but...  It's also likely not easy to pick up on "wrong" binary
> packages built this way.

That is not a new problem: maintainers can already upload packages that
refer to binaries in /usr/local/bin when they have locally installed
binaries and autodetection using $PATH is used?  /usr/local/bin is
usually before /usr/bin and /bin after all.

dpkg could add a "not-built-in-a-clean-chroot" flag to detect those.

Ansgar

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Steve McIntyre]

Hi Didier,

While I'm not necessarily disagreeing with the overall point(s) here,
some of the points in this list of arguments seem dubious at
best. Maybe you could expand?

>* having separate `/` and `/usr` filesystems has been useful in the past for
>  booting without initramfs onto a minimal root filesystem that carried just
>  enough to mount the `/usr` filesystem later in the boot process. Given the
>  evolution of physical hosts' capabilities, initramfs'es have been default in
>  Debian (and elsewhere) for a long time, and most systems no longer have an
>  intermediate state during boot in which they have only `/`, but not `/usr`,
>  mounted.
>* another use-case is to be able to share an identical `/usr` over a network
>  link; hence booting an initramfs, mounting a local `/`, then mounting `/usr`
>  over the network. It seems that an initramfs with everything needed to mount
>  a filesystem over a network link directly actually has a smaller footprint.
>* booting with `/` only is not systematically tested in Debian anymore;

I'm assuming you mean "/ without /usr" here?

>* the packaging infrastructure to install files outside of `/usr` is not
>  standard and represents technical debt:

I have no idea what you're suggesting here.

>* given its status as remnant "folklore", the distinction between what _needs_
>  to be shipped in `/` and what can stay in `/usr` is often interpreted
>  arbitrarily;
>* allowing shipment of identically-named libraries or binaries in different
>  paths can confuse common understanding of paths precedence.

...

>Various valid long-term desireable situations coexist, and while discussing
>immediate countermeasures, it is useful to keep the long-term outcome that
>those are most likely to produce.
>
>These are the five possible situations at the time of bullseye (buster + 1):
>
>* `none`: "merged `/usr`" has been reverted
>* `weak`: both directory schemes are allowed, packages only built on classical
>  hosts
>* `middle`: both directory schemes are allowed, packages can be built anywhere
>* `hard`: both directory schemes are allowed, packages only built on
>  "merged `/usr`" hosts
>* `all`: only "merged `/usr`" directory schemes are allowed, packages only
>  built on "merged `/usr`" hosts
>
>It can be summarized by the following table:
>
>```
>|  | Host types that are allowed   | Are merged `/usr` | 
>Official packages are built on| Packages built on … can break on the other 
>|
>| Codename | classical hosts | merged `/usr` hosts | symlinks allowed  | 
>classical hosts | merged `/usr` hosts |   classical hosts   |  merged `/usr` 
>hosts |
>|--|-|-|---|—|-|-|--|
>| none |   yes   |  no | no|   
>yes   |  no | yes |  yes |
>| weak |   yes   | yes |yes|   
>yes   |  no |  no |  yes |
>|   middle |   yes   | yes |yes|   
>yes   | yes |  no |   no |
>| hard |   yes   | yes |yes|   
> no   | yes |  no |   no |
>|  all |   no| yes |yes|   
> no   | yes | yes |   no |
>```
>
>The current state of buster is `weak`.
>
>=== DRAFT Resolution ===
>
>The Technical Committee resolves to:
>
>* Option A: Ask the debootstrap maintainers to disable "merged `/usr`" by
>  default
>  (Using its §6.1.4 "Overrule a Developer" power; requires a 3:1 majority)
>
>  Given that:
>  * hosts with both directory schemes already exist,
>  * the "merged `/usr`" directory scheme ought to be reserved for special
>use-cases,
>  * official packages ought to only be built on classical directory schemes,
>
>  … the Technical Committee considers that the desireable solution at the time
>  of bullseye is `weak`; and asks the debootstrap maintainers to disable
>  "merged `/usr`" by default.

There is a deeper worry about builds that may be done against the
"weak" background. Although buildd chroots are easily fixed up,
there's going to be a (small, but unknown) set of maintainers who
might be uploading binaries from merged systems. I think we're making
good progress on source-only uploads and building in clean chroots
etc., but...  It's also likely not easy to pick up on "wrong" binary
packages built this way.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
The two hard things in computing:
 * naming things
 * cache invalidation
 * off-by-one errors  -- Stig Sandbeck Mathisen

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Cyril Brulebois]

Hi,

Didier 'OdyX' Raboud  (2019-02-18):
> Dear Technical Committee members,
> (CC'ed to submitter, and debootstrap maintainers for information and feedack)
> 
> Here's the current state of the draft resolution; which `master` is at
> https://salsa.debian.org/debian/tech-ctte/blob/master/914897_merged_usr/ballot.md

Thanks, Didier.

While I haven't dived into it as deep as you did (e.g. I didn't proofread
the big table at the end), that seems like a reasonable characterization
of the current situation; many thanks for your highly detailed summary.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Cyril Brulebois]

Hi,

Simon McVittie  (2018-12-23):
> To be completely clear about the decision that Ian asked the technical
> committee to overrule:
> 
> In all debootstrap versions since 1.0.102, merged /usr is the default
> (for all variants except --variant=buildd). This means that new
> installations of Debian buster using debian-installer will have merged
> /usr.
> 
> Do the debian-installer and debootstrap maintainers think this should
> continue to be true for the buster release?

As might have been noticed, I haven't been following Debian topics very
closely lately, but on this specific topic: I'm not aware of blockers
that should prevent us from keeping this new default setting.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Hideki Yamane]

Hi,

On Sun, 23 Dec 2018 00:36:52 +
Simon McVittie  wrote:
> To be completely clear about the decision that Ian asked the technical
> committee to overrule:
> 
> In all debootstrap versions since 1.0.102, merged /usr is the default (for
> all variants except --variant=buildd). This means that new installations
> of Debian buster using debian-installer will have merged /usr.
> 
> Do the debian-installer and debootstrap maintainers think this should
> continue to be true for the buster release?

 At this time, yes. +1

 However, if it'll be a blocker for release during freeze, it should
 be reverted.


-- 
Regards,

 Hideki Yamane henrich @ debian.org/iijmio-mail.jp

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Simon McVittie]

To be completely clear about the decision that Ian asked the technical
committee to overrule:

In all debootstrap versions since 1.0.102, merged /usr is the default (for
all variants except --variant=buildd). This means that new installations
of Debian buster using debian-installer will have merged /usr.

Do the debian-installer and debootstrap maintainers think this should
continue to be true for the buster release?

(Sorry, I should have made this the only question in a message, and not
mentioned side issues in the same message.)

Thanks,
smcv

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default? /usr by default

[Hideki Yamane]

Hi,

On Wed, 5 Dec 2018 08:39:27 +
Simon McVittie  wrote:
> It might also be considered appropriate to revert the change in
> debootstrap 1.0.111 if data from reproducible-builds demonstrates that
> bugs similar to #913226 have all been fixed or are very rare, but this
> should be done cautiously, and certainly not before buster is released.

 Okay, my opinion is "Push usr-merge effort forward, fix those issues
 with it as bug that is tracked at reproducible builds(*), and turn it
 on again as default (probably after buster cycle)".

 *) 
https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html


-- 
Regards,

 Hideki Yamane henrich @ debian.org/iijmio-mail.jp

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default? /usr by default

[Simon McVittie]

Control: retitle 914897 tech-ctte: Should debootstrap disable merged /usr by 
default?

I'm retitling the bug to avoid misrepresenting the technical committee's
position on this. We have been asked to overrule the debootstrap
maintainer, but we have not yet come to a conclusion on whether we should.

On Wed, 05 Dec 2018 at 13:25:36 +0900, Hideki Yamane wrote:
>  Can we check and track this behavior in our packages?

Yes, we now do. The reproducible-builds infrastructure now uses unmerged
/usr for the first build and merged /usr for the second, since
 was fixed.

debootstrap since 1.0.111 also mitigates this class of bugs by disabling
merged /usr for --variant=buildd chroots (this was
).

Julien Cristau thinks #914208 was a sufficient/proportionate change, and
doesn't want to go further and default to --no-merged-usr for non-buildd
chroots (and in particular new debian-installer installations).

Ian Jackson thinks #914208 is not a sufficient answer (Ian, I hope I'm
not misrepresenting you here?), and escalated this bug to the technical
committee, asking us to overrule the debootstrap maintainers.

If the debootstrap/debian-installer maintainers agree with Ian on this,
then there is no need for the technical committee to consider his request
to overrule you, which is why Didier asked for your opinion on this
issue before attempting to come to a decision. If you agree with Julien,
then the technical committee still needs to consider this question.

>  Once disable merged-usr is good to prevent confusion but we detect such
>  as a bug for manage non-merged-usr and merged-usr Debian system in the end,
>  right? (or do you want to stay change in debootstrap 1.0.111 forever?)

The technical committee have not come to a conclusion on this.

My personal opinions (not overruling anyone):

If merged /usr becomes the only supported system layout at some future
time, then the change in debootstrap 1.0.111 can certainly be reverted
at that time. (If merged /usr does not become the only supported system
layout, this does not apply.)

It might also be considered appropriate to revert the change in
debootstrap 1.0.111 if data from reproducible-builds demonstrates that
bugs similar to #913226 have all been fixed or are very rare, but this
should be done cautiously, and certainly not before buster is released.

smcv