Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sean Whitton]

Hello,

On Sun 18 Sep 2022 at 03:04AM +01, Luca Boccassi wrote:

> I'd like to request the CTTE to intervene and put an end to this
> behaviour, so that the agreed plan based on your votes can be brought
> to conclusion with plenty of time to spare before Bookworm's freeze.

Intervening in this sort of way is not part of the TC's remit.

In any case, it seems that other, more relevant core teams have taken action.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Sat, 2022-09-10 at 13:37 +0100, Luca Boccassi wrote:
> On Tue, 2022-08-16 at 21:06 +0100, Luca Boccassi wrote:
> > On Thu, 2022-07-28 at 13:14 +0100, Luca Boccassi wrote:
> > > On Sun, 2022-07-17 at 00:56 +0100, Luca Boccassi wrote:
> > > > On Mon, 2021-10-18 at 12:17 -0700, Sean Whitton wrote:
> > > > > Hello,
> > > > > 
> > > > > On Wed 15 Sep 2021 at 11:46AM +01, Simon McVittie wrote:
> > > > > 
> > > > > > As discussed in our last meeting, I think we should issue
> > > > > > more
> > > > > > specific
> > > > > > advice about merged-/usr, and in particular about what
> > > > > > #978636
> > > > > > means for
> > > > > > maintainers right now.
> > > > > 
> > > > > With five votes cast in favour of the text by Simon, Niko,
> > > > > Gunnar,
> > > > > Marga
> > > > > and myself, the outcome is no longer in doubt.  Thus, in
> > > > > accordance
> > > > > with
> > > > > the Debian Constitution, the voting period has now concluded.
> > > > > 
> > > > > Therefore, using its powers under section 6.1.5 of the Debian
> > > > > Constitution, the Technical Committee issues the following
> > > > > advice:
> > > > > 
> > > > > Summary
> > > > > ===
> > > > > 
> > > > > There are currently Debian 11 installations with both the
> > > > > merged-/usr
> > > > > and non-merged-/usr filesystem layouts. All of these
> > > > > installations
> > > > > should successfully upgrade via normal upgrade paths to a
> > > > > merged-/usr
> > > > > Debian 12.  Only after the release of Debian 12 can packages
> > > > > assume
> > > > > that
> > > > > all installations will be merged-/usr.
> > > > > 
> > > > > Main points:
> > > > > 
> > > > > - We have recommended merged-/usr for Debian 12.
> > > > > - Moving individual files is not merged-/usr.
> > > > > - "Symlink farms" are not merged-/usr.
> > > > > - Upgrading a non-merged-/usr system to Debian 12 needs to
> > > > > work.
> > > > > - Upgrading a merged-/usr system to Debian 12 needs to work.
> > > > > - Packages cannot assume all systems are merged-/usr until
> > > > > the
> > > > > Debian
> > > > > 13
> > > > >   development cycle begins.
> > > > > - Upgrading via apt in the usual way should work.
> > > > > - Testing and QA systems should be able to avoid this
> > > > > transition, but
> > > > > if
> > > > >   they do, they cannot be upgraded beyond Debian 12.
> > > > > - A package building incorrectly on a non-merged-/usr system
> > > > > is
> > > > > a
> > > > > bug.
> > > > > - A package building incorrectly on a merged-/usr system is a
> > > > > bug.
> > > > > - Please stop moving individual packages' files from the root
> > > > > filesystem
> > > > >   into /usr, at least for now.
> > > > > 
> > > > > Definitions and current status
> > > > > ==
> > > > > 
> > > > > libQUAL refers to the directories described in FHS 3.0
> > > > > section
> > > > > 3.10
> > > > > [1],
> > > > > such as lib64 on the amd64 architecture.
> > > > > 
> > > > > Merged /usr is the filesystem layout in which /bin, /sbin,
> > > > > /lib
> > > > > and
> > > > > each
> > > > > /libQUAL that exists are symbolic links to the corresponding
> > > > > directories
> > > > > below /usr. This results in aliasing between /bin and
> > > > > /usr/bin,
> > > > > and
> > > > > so
> > > > > on.
> > > > > 
> > > > > In the merged-/usr layout, files whose canonical logical
> > > > > location is
> > > > > in
> > > > > one of the affected directories on the root filesystem, such
> > > > > as
> > > > > /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically
> > > > > located
> > > > > at
> > > > > the corresponding path below /usr, such as /usr/bin/bash.
> > > > > Each
> > > > > file
> > > > > in
> > > > > one of the affected directories is accessible via two paths:
> > > > > its
> > > > > canonical logical location (such as /bin/bash or
> > > > > /usr/bin/env),
> > > > > and
> > > > > the
> > > > > other path implied by the aliasing (such as /usr/bin/bash or
> > > > > /bin/env).
> > > > > 
> > > > > There are two supported categories of Debian 11 installation,
> > > > > which
> > > > > are
> > > > > currently considered equally-supported:
> > > > > 
> > > > > - Merged-/usr installations. These were installed with
> > > > > debian-
> > > > > installer
> > > > >   from Debian 10 or later, or installed with debootstrap --
> > > > > merged-
> > > > > usr,
> > > > >   or converted from the non-merged-/usr layout by installing
> > > > > the
> > > > >   usrmerge package, or installed or converted by any similar
> > > > >   procedure. They have the merged-/usr layout.
> > > > > 
> > > > > - Non-merged-/usr installations. These were installed with
> > > > >   debian-installer from Debian 9 or earlier and subsequently
> > > > > upgraded
> > > > >   without converting to merged-/usr, or installed with
> > > > > debootstrap
> > > > >   --no-merged-usr, or converted from the merged-/usr layout
> > > > > with
> > > > > dpkg's
> > > > >   "dpkg-fsys-usrunmess" utility or any similar procedure.
> 

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Tue, 2022-08-16 at 21:06 +0100, Luca Boccassi wrote:
> On Thu, 2022-07-28 at 13:14 +0100, Luca Boccassi wrote:
> > On Sun, 2022-07-17 at 00:56 +0100, Luca Boccassi wrote:
> > > On Mon, 2021-10-18 at 12:17 -0700, Sean Whitton wrote:
> > > > Hello,
> > > > 
> > > > On Wed 15 Sep 2021 at 11:46AM +01, Simon McVittie wrote:
> > > > 
> > > > > As discussed in our last meeting, I think we should issue
> > > > > more
> > > > > specific
> > > > > advice about merged-/usr, and in particular about what
> > > > > #978636
> > > > > means for
> > > > > maintainers right now.
> > > > 
> > > > With five votes cast in favour of the text by Simon, Niko,
> > > > Gunnar,
> > > > Marga
> > > > and myself, the outcome is no longer in doubt.  Thus, in
> > > > accordance
> > > > with
> > > > the Debian Constitution, the voting period has now concluded.
> > > > 
> > > > Therefore, using its powers under section 6.1.5 of the Debian
> > > > Constitution, the Technical Committee issues the following
> > > > advice:
> > > > 
> > > > Summary
> > > > ===
> > > > 
> > > > There are currently Debian 11 installations with both the
> > > > merged-/usr
> > > > and non-merged-/usr filesystem layouts. All of these
> > > > installations
> > > > should successfully upgrade via normal upgrade paths to a
> > > > merged-/usr
> > > > Debian 12.  Only after the release of Debian 12 can packages
> > > > assume
> > > > that
> > > > all installations will be merged-/usr.
> > > > 
> > > > Main points:
> > > > 
> > > > - We have recommended merged-/usr for Debian 12.
> > > > - Moving individual files is not merged-/usr.
> > > > - "Symlink farms" are not merged-/usr.
> > > > - Upgrading a non-merged-/usr system to Debian 12 needs to
> > > > work.
> > > > - Upgrading a merged-/usr system to Debian 12 needs to work.
> > > > - Packages cannot assume all systems are merged-/usr until the
> > > > Debian
> > > > 13
> > > >   development cycle begins.
> > > > - Upgrading via apt in the usual way should work.
> > > > - Testing and QA systems should be able to avoid this
> > > > transition, but
> > > > if
> > > >   they do, they cannot be upgraded beyond Debian 12.
> > > > - A package building incorrectly on a non-merged-/usr system is
> > > > a
> > > > bug.
> > > > - A package building incorrectly on a merged-/usr system is a
> > > > bug.
> > > > - Please stop moving individual packages' files from the root
> > > > filesystem
> > > >   into /usr, at least for now.
> > > > 
> > > > Definitions and current status
> > > > ==
> > > > 
> > > > libQUAL refers to the directories described in FHS 3.0 section
> > > > 3.10
> > > > [1],
> > > > such as lib64 on the amd64 architecture.
> > > > 
> > > > Merged /usr is the filesystem layout in which /bin, /sbin, /lib
> > > > and
> > > > each
> > > > /libQUAL that exists are symbolic links to the corresponding
> > > > directories
> > > > below /usr. This results in aliasing between /bin and /usr/bin,
> > > > and
> > > > so
> > > > on.
> > > > 
> > > > In the merged-/usr layout, files whose canonical logical
> > > > location is
> > > > in
> > > > one of the affected directories on the root filesystem, such as
> > > > /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically
> > > > located
> > > > at
> > > > the corresponding path below /usr, such as /usr/bin/bash. Each
> > > > file
> > > > in
> > > > one of the affected directories is accessible via two paths:
> > > > its
> > > > canonical logical location (such as /bin/bash or /usr/bin/env),
> > > > and
> > > > the
> > > > other path implied by the aliasing (such as /usr/bin/bash or
> > > > /bin/env).
> > > > 
> > > > There are two supported categories of Debian 11 installation,
> > > > which
> > > > are
> > > > currently considered equally-supported:
> > > > 
> > > > - Merged-/usr installations. These were installed with debian-
> > > > installer
> > > >   from Debian 10 or later, or installed with debootstrap --
> > > > merged-
> > > > usr,
> > > >   or converted from the non-merged-/usr layout by installing
> > > > the
> > > >   usrmerge package, or installed or converted by any similar
> > > >   procedure. They have the merged-/usr layout.
> > > > 
> > > > - Non-merged-/usr installations. These were installed with
> > > >   debian-installer from Debian 9 or earlier and subsequently
> > > > upgraded
> > > >   without converting to merged-/usr, or installed with
> > > > debootstrap
> > > >   --no-merged-usr, or converted from the merged-/usr layout
> > > > with
> > > > dpkg's
> > > >   "dpkg-fsys-usrunmess" utility or any similar procedure. They
> > > > have
> > > > the
> > > >   traditional, non-merged-/usr layout in which /bin/bash and
> > > >   /usr/bin/env have exactly those physical paths, and
> > > > /usr/bin/bash
> > > > and
> > > >   /bin/env do not exist.
> > > > 
> > > > Merged-/usr is not the only filesystem layout that has been
> > > > proposed
> > > > for
> > > > unifying the root filesystem with /usr. For avoidance of 

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Thu, 2022-07-28 at 13:14 +0100, Luca Boccassi wrote:
> On Sun, 2022-07-17 at 00:56 +0100, Luca Boccassi wrote:
> > On Mon, 2021-10-18 at 12:17 -0700, Sean Whitton wrote:
> > > Hello,
> > > 
> > > On Wed 15 Sep 2021 at 11:46AM +01, Simon McVittie wrote:
> > > 
> > > > As discussed in our last meeting, I think we should issue more
> > > > specific
> > > > advice about merged-/usr, and in particular about what #978636
> > > > means for
> > > > maintainers right now.
> > > 
> > > With five votes cast in favour of the text by Simon, Niko, Gunnar,
> > > Marga
> > > and myself, the outcome is no longer in doubt.  Thus, in accordance
> > > with
> > > the Debian Constitution, the voting period has now concluded.
> > > 
> > > Therefore, using its powers under section 6.1.5 of the Debian
> > > Constitution, the Technical Committee issues the following advice:
> > > 
> > > Summary
> > > ===
> > > 
> > > There are currently Debian 11 installations with both the merged-/usr
> > > and non-merged-/usr filesystem layouts. All of these installations
> > > should successfully upgrade via normal upgrade paths to a merged-/usr
> > > Debian 12.  Only after the release of Debian 12 can packages assume
> > > that
> > > all installations will be merged-/usr.
> > > 
> > > Main points:
> > > 
> > > - We have recommended merged-/usr for Debian 12.
> > > - Moving individual files is not merged-/usr.
> > > - "Symlink farms" are not merged-/usr.
> > > - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> > > - Upgrading a merged-/usr system to Debian 12 needs to work.
> > > - Packages cannot assume all systems are merged-/usr until the Debian
> > > 13
> > >   development cycle begins.
> > > - Upgrading via apt in the usual way should work.
> > > - Testing and QA systems should be able to avoid this transition, but
> > > if
> > >   they do, they cannot be upgraded beyond Debian 12.
> > > - A package building incorrectly on a non-merged-/usr system is a
> > > bug.
> > > - A package building incorrectly on a merged-/usr system is a bug.
> > > - Please stop moving individual packages' files from the root
> > > filesystem
> > >   into /usr, at least for now.
> > > 
> > > Definitions and current status
> > > ==
> > > 
> > > libQUAL refers to the directories described in FHS 3.0 section 3.10
> > > [1],
> > > such as lib64 on the amd64 architecture.
> > > 
> > > Merged /usr is the filesystem layout in which /bin, /sbin, /lib and
> > > each
> > > /libQUAL that exists are symbolic links to the corresponding
> > > directories
> > > below /usr. This results in aliasing between /bin and /usr/bin, and
> > > so
> > > on.
> > > 
> > > In the merged-/usr layout, files whose canonical logical location is
> > > in
> > > one of the affected directories on the root filesystem, such as
> > > /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located
> > > at
> > > the corresponding path below /usr, such as /usr/bin/bash. Each file
> > > in
> > > one of the affected directories is accessible via two paths: its
> > > canonical logical location (such as /bin/bash or /usr/bin/env), and
> > > the
> > > other path implied by the aliasing (such as /usr/bin/bash or
> > > /bin/env).
> > > 
> > > There are two supported categories of Debian 11 installation, which
> > > are
> > > currently considered equally-supported:
> > > 
> > > - Merged-/usr installations. These were installed with debian-
> > > installer
> > >   from Debian 10 or later, or installed with debootstrap --merged-
> > > usr,
> > >   or converted from the non-merged-/usr layout by installing the
> > >   usrmerge package, or installed or converted by any similar
> > >   procedure. They have the merged-/usr layout.
> > > 
> > > - Non-merged-/usr installations. These were installed with
> > >   debian-installer from Debian 9 or earlier and subsequently upgraded
> > >   without converting to merged-/usr, or installed with debootstrap
> > >   --no-merged-usr, or converted from the merged-/usr layout with
> > > dpkg's
> > >   "dpkg-fsys-usrunmess" utility or any similar procedure. They have
> > > the
> > >   traditional, non-merged-/usr layout in which /bin/bash and
> > >   /usr/bin/env have exactly those physical paths, and /usr/bin/bash
> > > and
> > >   /bin/env do not exist.
> > > 
> > > Merged-/usr is not the only filesystem layout that has been proposed
> > > for
> > > unifying the root filesystem with /usr. For avoidance of doubt, we do
> > > not consider other filesystem layouts to be implementations of
> > > merged-/usr.  In particular, we do not consider these to be
> > > implementations of merged-/usr:
> > > 
> > > - all affected files physically located in /bin, /sbin, /lib and
> > >   /libQUAL, with /usr/bin as a symlink to /bin, etc. (this is the
> > >   reverse of merged-/usr, and was historically used in the hurd-i386
> > >   port)
> > > 
> > > - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with
> > >   individual symbolic 

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Sun, 2022-07-17 at 00:56 +0100, Luca Boccassi wrote:
> On Mon, 2021-10-18 at 12:17 -0700, Sean Whitton wrote:
> > Hello,
> > 
> > On Wed 15 Sep 2021 at 11:46AM +01, Simon McVittie wrote:
> > 
> > > As discussed in our last meeting, I think we should issue more
> > > specific
> > > advice about merged-/usr, and in particular about what #978636
> > > means for
> > > maintainers right now.
> > 
> > With five votes cast in favour of the text by Simon, Niko, Gunnar,
> > Marga
> > and myself, the outcome is no longer in doubt.  Thus, in accordance
> > with
> > the Debian Constitution, the voting period has now concluded.
> > 
> > Therefore, using its powers under section 6.1.5 of the Debian
> > Constitution, the Technical Committee issues the following advice:
> > 
> > Summary
> > ===
> > 
> > There are currently Debian 11 installations with both the merged-/usr
> > and non-merged-/usr filesystem layouts. All of these installations
> > should successfully upgrade via normal upgrade paths to a merged-/usr
> > Debian 12.  Only after the release of Debian 12 can packages assume
> > that
> > all installations will be merged-/usr.
> > 
> > Main points:
> > 
> > - We have recommended merged-/usr for Debian 12.
> > - Moving individual files is not merged-/usr.
> > - "Symlink farms" are not merged-/usr.
> > - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> > - Upgrading a merged-/usr system to Debian 12 needs to work.
> > - Packages cannot assume all systems are merged-/usr until the Debian
> > 13
> >   development cycle begins.
> > - Upgrading via apt in the usual way should work.
> > - Testing and QA systems should be able to avoid this transition, but
> > if
> >   they do, they cannot be upgraded beyond Debian 12.
> > - A package building incorrectly on a non-merged-/usr system is a
> > bug.
> > - A package building incorrectly on a merged-/usr system is a bug.
> > - Please stop moving individual packages' files from the root
> > filesystem
> >   into /usr, at least for now.
> > 
> > Definitions and current status
> > ==
> > 
> > libQUAL refers to the directories described in FHS 3.0 section 3.10
> > [1],
> > such as lib64 on the amd64 architecture.
> > 
> > Merged /usr is the filesystem layout in which /bin, /sbin, /lib and
> > each
> > /libQUAL that exists are symbolic links to the corresponding
> > directories
> > below /usr. This results in aliasing between /bin and /usr/bin, and
> > so
> > on.
> > 
> > In the merged-/usr layout, files whose canonical logical location is
> > in
> > one of the affected directories on the root filesystem, such as
> > /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located
> > at
> > the corresponding path below /usr, such as /usr/bin/bash. Each file
> > in
> > one of the affected directories is accessible via two paths: its
> > canonical logical location (such as /bin/bash or /usr/bin/env), and
> > the
> > other path implied by the aliasing (such as /usr/bin/bash or
> > /bin/env).
> > 
> > There are two supported categories of Debian 11 installation, which
> > are
> > currently considered equally-supported:
> > 
> > - Merged-/usr installations. These were installed with debian-
> > installer
> >   from Debian 10 or later, or installed with debootstrap --merged-
> > usr,
> >   or converted from the non-merged-/usr layout by installing the
> >   usrmerge package, or installed or converted by any similar
> >   procedure. They have the merged-/usr layout.
> > 
> > - Non-merged-/usr installations. These were installed with
> >   debian-installer from Debian 9 or earlier and subsequently upgraded
> >   without converting to merged-/usr, or installed with debootstrap
> >   --no-merged-usr, or converted from the merged-/usr layout with
> > dpkg's
> >   "dpkg-fsys-usrunmess" utility or any similar procedure. They have
> > the
> >   traditional, non-merged-/usr layout in which /bin/bash and
> >   /usr/bin/env have exactly those physical paths, and /usr/bin/bash
> > and
> >   /bin/env do not exist.
> > 
> > Merged-/usr is not the only filesystem layout that has been proposed
> > for
> > unifying the root filesystem with /usr. For avoidance of doubt, we do
> > not consider other filesystem layouts to be implementations of
> > merged-/usr.  In particular, we do not consider these to be
> > implementations of merged-/usr:
> > 
> > - all affected files physically located in /bin, /sbin, /lib and
> >   /libQUAL, with /usr/bin as a symlink to /bin, etc. (this is the
> >   reverse of merged-/usr, and was historically used in the hurd-i386
> >   port)
> > 
> > - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with
> >   individual symbolic links such as /bin/bash -> /usr/bin/bash for
> > only
> >   those files that historically had their canonical logical location
> > on
> >   the root filesystem
> > 
> > - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with
> >   individual symbolic links such as /bin/env -> 

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Josh Triplett]

Sam Hartman wrote:
> that looks like you are proposing doing it in an expedited manner *as
> part of usrmerge*

I am not proposing doing *anything* in an expedited manner. I don't
think it'd be feasible to make such a change in an expedited manner even
if we *wanted* to, and in any case I wouldn't expect that. And I'm
certainly not proposing making a change "as part of usrmerge".

(I realize the subject of this message is about merged /usr. However, the
point of this was not "let's change it because merged /usr"; the point
was "perhaps we should take a look at this and evaluate it".)

I was attempting to look at the system as a system, ask a question that
I've seen raised many past times, and ask if there may be value in
*having a discussion about it*. I would expect such a discussion to
happen in parallel, and any potential change that arose as a result of
it to happen in parallel, uncoupled to anything else.

It seems unfortunate when *bringing up* the possibility of *discussing*
a change is viewed as breaking things. I suggested "evaluating the net
value" of /etc/shells. Part of the point of such an evaluation would be
to discover, in fact, how people use it, what value it provides, what
people's use cases are, what detriments it has, what other ways the same
problems could be solved, and similar. That, to me, seems like precisely
the kind of process by which people's input *can* be heard and
respected.

I don't think there are any areas of functionality that we shouldn't
*evaluate* and consider possibilities in. That doesn't mean we should
ever gratuitously change things without discussion and without taking
input and use cases into account.

- Josh Triplett

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sean Whitton]

Hello,

On Tue 26 Jul 2022 at 03:31pm +02, Johannes Schauer Marin Rodrigues wrote:

> Hi tech-ctte
>
> Quoting Luca Boccassi (2022-07-25 15:22:10)
>> There's also a pending MR for reportbug that would be good to have, but
>> doesn't need to block other progress:
>>
>> https://salsa.debian.org/reportbug-team/reportbug/-/merge_requests/77
>
> I just remembered a pending MR that I'd say is a little more important than
> "good to have":
>
> https://salsa.debian.org/debian/debianutils/-/merge_requests/21
>
> Without this one-line change, /etc/shells on merged-/usr systems will have 
> some
> shells listed twice and some shells (like /usr/bin/bash) missing. The reason
> is, that dpkg-realpath doesn't support merged-/usr and thus a workaround is
> needed.
>
> Given the recent CTTE ruling over debianutils (#994275) and since the MR from
> five months ago has no maintainer activity, I wanted to ask you for informal
> advice on this. Should I just file a bug and NMU with maximum delay? Or is
> application of this fix deemed more important?

Informally, I suggest an NMU with a delay of five days, based on the
idea that the bug is of at least Severity: important.

-- 
Sean Whitton

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Russ Allbery]

Sam Hartman  writes:
>> "Josh" == Josh Triplett  writes:

> Josh> Over the years, I've seen a few proposals floated to consider
> Josh> dropping /etc/shells; this would just require dropping
> Josh> pam_shells.so from /etc/pam.d/chsh. That would also have the
> Josh> side effect of solving this problem, and making one less thing
> Josh> requiring maintainer scripts.

> I think that would be a really bad idea.
> The issue is not on the chsh side, but more that membership in
> /etc/shells is a really good (but not perfect) indicator about whether
> this is an account that supports normal logins.

I agree with Sam on this: I would not couple discussion of dropping this
mechanism with usrmerge, and I would be very cautious here.

There are a lot of facilities in Debian that are mostly internal plumbing
and that only a few administrators are likely to fiddle with (and those
often being sophisticated users who follow Debian closely).  This is not
one of them.  /etc/shells is a very old UNIX security mechanism, and while
I would not design it today the way that it was designed, and it has a lot
of caveats and weird edge cases, it is a security mechanism that predates
the existence of Linux and that was (and probably, to a lesser extent, is)
used in a wide variety of older environments and configurations.

This is the sort of operating system facility that may be a load-bearing
security control for systems where everyone has forgotten that it is
security-critical.  It is possible, even likely, that there exist
production Debian systems in the wild where the /etc/shells mechanism is
the primary control standing in the way of an obvious privilege escalation
vulnerability.  To be clear, that's not a great situation for those
systems to be in, since this mechanism is a bit fragile and probably not
as strong as one would like!  But nonetheless we should be very careful
about taking any action that might break its historical properties.

-- 
Russ Allbery (r...@debian.org)  

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sam Hartman]

> "Josh" == Josh Triplett  writes:
Josh> Over the years, I've seen a few proposals floated to consider
Josh> dropping /etc/shells; this would just require dropping
Josh> pam_shells.so from /etc/pam.d/chsh. That would also have the
Josh> side effect of solving this problem, and making one less thing
Josh> requiring maintainer scripts.

I think that would be a really bad idea.
The issue is not on the chsh side, but more that membership in
/etc/shells is a really good (but not perfect) indicator about whether
this is an account that supports normal logins.

I can see the arguments for change, but:

1) /etc/shells does have some value

2) It's something existing admins do expect, and there actually is a lot
of value in continuing to meet expectations

3) The negative political consequences to making that sort of change as
part of usrmerge  cannot be over stated.

There actually are legitimate concerns that  the people favoring
systemd, usrmerge, and the like are disregarding legitimate concerns
about stability of interfaces and expectations.
Having you as a significant member of that camp propose this change in a
manner that looks like you are proposing doing it in an expedited manner
*as part of usrmerge* smells really bad and adds legitimacy to those
concerns.
Even when you disagree with people, I'd ask you to consider these sort
of implications.
I think it's a significant part of helping people feel that their input
is respected as the world changes.

--Sam

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Johannes Schauer Marin Rodrigues]

Quoting Josh Triplett (2022-07-26 17:29:43)
> Johannes Schauer Marin Rodrigues wrote:
> > I just remembered a pending MR that I'd say is a little more important than
> > "good to have":
> > 
> > https://salsa.debian.org/debian/debianutils/-/merge_requests/21
> > 
> > Without this one-line change, /etc/shells on merged-/usr systems will have 
> > some
> > shells listed twice and some shells (like /usr/bin/bash) missing. The reason
> > is, that dpkg-realpath doesn't support merged-/usr and thus a workaround is
> > needed.
> 
> Over the years, I've seen a few proposals floated to consider dropping
> /etc/shells; this would just require dropping pam_shells.so from
> /etc/pam.d/chsh. That would also have the side effect of solving this
> problem, and making one less thing requiring maintainer scripts.
> 
> Would it be worth reopening that discussion, and evaluating the net
> value of continuing to maintain /etc/shells?

"one less thing requiring maintainer scripts" is not an argument anymore
because update-shells from debianutils is precisely the mechanism that avoids
maintainer scripts for updating /etc/shells. It does so by using a dpkg trigger
on /usr/share/debianutils/shells.d instead. We were able to remove maintainer
scripts updating /etc/shells (partly or completely) from bash as well as dash
thanks to update-shells in debianutils.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Josh Triplett]

Johannes Schauer Marin Rodrigues wrote:
> I just remembered a pending MR that I'd say is a little more important than
> "good to have":
> 
> https://salsa.debian.org/debian/debianutils/-/merge_requests/21
> 
> Without this one-line change, /etc/shells on merged-/usr systems will have 
> some
> shells listed twice and some shells (like /usr/bin/bash) missing. The reason
> is, that dpkg-realpath doesn't support merged-/usr and thus a workaround is
> needed.

Over the years, I've seen a few proposals floated to consider dropping
/etc/shells; this would just require dropping pam_shells.so from
/etc/pam.d/chsh. That would also have the side effect of solving this
problem, and making one less thing requiring maintainer scripts.

Would it be worth reopening that discussion, and evaluating the net
value of continuing to maintain /etc/shells?

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Johannes Schauer Marin Rodrigues]

Hi tech-ctte

Quoting Luca Boccassi (2022-07-25 15:22:10)
> There's also a pending MR for reportbug that would be good to have, but
> doesn't need to block other progress:
> 
> https://salsa.debian.org/reportbug-team/reportbug/-/merge_requests/77

I just remembered a pending MR that I'd say is a little more important than
"good to have":

https://salsa.debian.org/debian/debianutils/-/merge_requests/21

Without this one-line change, /etc/shells on merged-/usr systems will have some
shells listed twice and some shells (like /usr/bin/bash) missing. The reason
is, that dpkg-realpath doesn't support merged-/usr and thus a workaround is
needed.

Given the recent CTTE ruling over debianutils (#994275) and since the MR from
five months ago has no maintainer activity, I wanted to ask you for informal
advice on this. Should I just file a bug and NMU with maximum delay? Or is
application of this fix deemed more important?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Johannes Schauer Marin Rodrigues]

Hi Luca,

Quoting Luca Boccassi (2022-07-25 15:22:10)
> usrmerge has migrated to testing, and the deboostrap MR has been merged and
> just uploaded to unstable. Do you want to do the equivalent upload for
> mmdebstrap now, so that it's ready as well?

thank you for the heads-up!

Yes, the mmdebstrap upload will not be optional because the upload of
debootstrap 1.0.127 breaks the mmdebstrap autopkgtest and to make sure that
debootstrap migrates, I'll have to do an mmdebstrap upload adapting its
autopkgtest to the new debootstrap behaviour.

> Once debootstrap 1.0.127 has migrated to testing we can arrange an upload to
> bullseye-backports, and then ensure all the buildds are using that new
> version. Then i-s-h can be updated.

At the point that i-s-h is uploaded, I'll have to do yet another mmdebstrap
upload because that again will break the expectations I encoded in mmdebstrap's
autopkgtests. I am running the mmdebstrap autopkgtest daily on jenkins so I'll
definitely get notified once the i-s-h upload breaks the autopkgtest once
again. :)

Thanks!

cheers, josch

signature.asc
Description: signature

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Mon, 2022-07-18 at 21:34 +0100, Luca Boccassi wrote:
> On Mon, 2022-07-18 at 22:10 +0200, Johannes Schauer Marin Rodrigues
> wrote:
> > Hi,
> > 
> > Quoting Luca Boccassi (2022-07-18 21:03:14)
> > > It was renamed following a request on #debian-ftp while it was in
> > > NEW, as the
> > > feedback was that 'usrmerge' and 'usrmerged' were too similar and
> > > thus easily
> > > confused. The 'usrmerged' one can be disregarded and will be de-
> > > crufted.
> > 
> > I think that's very sensible.
> > 
> > > > Does that sound okay to you and does the patch look like it does
> > > > the right
> > > > thing?
> > > 
> > > Yes, without knowing much about mmdebstrap, the diff looks good to
> > > me.
> > > I'd only ask that in the comment of the script hooks/no-merged-
> > > usr/setup00.sh if you could please mention explicitly that it
> > > creates an
> > > unsupported system. Maybe even print a warning when it's called.
> > 
> > Running mmdebstrap with --hook-dir=hooks/no-merged-usr will now print
> > the
> > following to stderr:
> > 
> > Warning: starting with Debian 12 (Bookworm), systems without merged-
> > /usr are not supported anymore
> > 
> > > Also, I assume it is creating the metapackage on-the-fly because it
> > > doesn't
> > > have the downloaded packages available at that point? Not a
> > > problem, just
> > > double checking.
> > 
> > That is correct. But I think it's not ideal if mmdebstrap creates a
> > chroot
> > containing a package that doesn't come from the archive. I now
> > extended the
> > hook script such that calling mmdebstrap with --hook-
> > dir=hooks/merged-usr will
> > first install the custom built metapackage, then install the
> > essential packages
> > and then upgrade to the real usr-is-merged package.
> 
> Nice!
> 
> > I think then the roadmap is to release debootstrap with #71 merged,
> > then upload
> > init-system-helpers that depends "usrmerge | usr-is-merged" and then
> > I test and
> > upload mmdebstrap including those hook scripts. Since those are just
> > hooks,
> > nothing stops people from using them with the mmdebstrap version
> > currently in
> > unstable and testing, so nothing should be blocked by this in case I
> > should
> > need longer to release an mmdebstrap version shipping these hook
> > scripts.
> 
> That's great. Currently waiting on usrmerge=29 to migrate to testing,
> then the deboostrap change becomes mergeable. Also waiting on Simon to
> clarify a few things regarding buildds and the plan around that -
> depending on the outcome, the plan w.r.t. debootstrap might change
> slightly or not.

Hello Josch,

usrmerge has migrated to testing, and the deboostrap MR has been merged
and just uploaded to unstable. Do you want to do the equivalent upload
for mmdebstrap now, so that it's ready as well?

Once debootstrap 1.0.127 has migrated to testing we can arrange an
upload to bullseye-backports, and then ensure all the buildds are using
that new version. Then i-s-h can be updated.

There's also a pending MR for reportbug that would be good to have, but
doesn't need to block other progress:

https://salsa.debian.org/reportbug-team/reportbug/-/merge_requests/77

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sean Whitton]

Hello,

On Tue 19 Jul 2022 at 08:01am +01, Simon McVittie wrote:

> It's unfortunate that when a package like usrmerge has more than one
> version waiting in NEW, the two operations available to the ftp team seem
> to be "reject all versions of usrmerge" and "accept all versions of
> usrmerge": if the result of NEW review is that the new binary package
> needs to be renamed, as it was in this case, then the ideal result would
> have been to reject usrmerge/28 from the queue and immediately accept
> usrmerge/29.

Just fyi, that's not right, we can reject individual uploads, but accept
means accepting all of them because of how overrides work.

-- 
Sean Whitton

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Christoph Berg]

Re: Luca Boccassi
> for now we'll stay the course.

Thanks for working on this and the well-written summary mails!

Christoph

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Mon, 2022-07-18 at 16:52 +0100, Luca Boccassi wrote:
> On Mon, 2022-07-18 at 14:32 +0200, Helmut Grohne wrote:
> > > Once deboostrap is updated and deployed on the buildds, a "usrmerge |
> > > usr-is-merged" dependency will be added to the Priority: essential
> > > init-system-helpers package, and uploaded to unstable to complete the
> > > transitions for all installations that are older than buster or that
> > > have been manually installed as unmerged-usr. Any system not upgraded
> > 
> > I think this setup is prone to breakage. While apt prefers the first
> > alternative, it doesn't have to pick it and other resolvers are even
> > more prone to picking non-default alternatives. Imagine one of the perl
> > modules being uninstallable. Even apt will pick the usr-is-merged
> > package in that scenario. Are you ware of this potential problem? Do you
> > have any ideas on how to handle it? To be clear: I do not think this
> > aspect to be a show-stopper. Let's just talk about it on a technical
> > level.
> > 
> > Let me propose something strange without having thought through the
> > implications in depth (i.e. it may be a totally bad idea):
> > 
> > What would happen if we were to replace the usr-is-merged package with a
> > "trivial-usr-merge" package? This package would actually perform a
> > /usr-merge in simplified conditions. Like usr-is-merged, it would refuse
> > to perform a merge on actual systems. However in the case of
> > non-container chroots (those not running init), it would perform the
> > merge in a simpler way that doesn't require additional perl modules and
> > doesn't come with the same amount of safety-guarantees.
> > 
> > A benefit of this approach would be that I could say mmebstrap
> > --include=trivial-usr-merge. Possibly though, this is a bad idea for
> > reasons I am not presently understanding well. Thank you for
> > considering.
> 
> From what I understand there is really no difference between a full
> system and a chroot from the implementation point of view. The things
> it deals with are various situations the directories can be in, but
> whether the system was booted or not doesn't seem to be handled
> differently at all. In other words, if a 'simplified' version with
> fewer dependencies was possible, it could be just used everywhere, I
> think?
> 
> Regarding the failure mode, I think it is theorethically possible, but
> very unlikely? The package has 3 individual perl modules dependencies
> (one direct, libfile-find-rule-perl, and two transitive, libfile-find-
> rule-perl and libnumber-compare-perl) and that's it, they don't depend
> nor conflict or break with anything else, just perl:any. So you'd need
> a headless system, that is dist-upgrading without using apt, and that
> has somehow made perl:any uninstallable.
> Do we know if other resolvers have coverage via autopkgtest/piuparts,
> that would pick on these issues?

Summarizing what we talked about on IRC yesterday evening:

- both apt and aptitude appear to do the expected thing when resolving,
even when the extra dependencies are not already installed
- apt with non-standard external resolver aspcud does the wrong thing,
even when the extra dependencies are already installed
- apt with other non-standard external resolvers can't seem to be
tested as resolution does not work at all and it errors out earlier

So there is a very small chance that the dependency 'usrmerge | usr-is-
merged' will do the wrong thing for some users, if aspcud is in use.
perl:any being uninstallable and thus making apt/aptitude fall back the
other way does not seem like a realistic situation that can happen
without anyone noticing much bigger issues. The failure mode in either
case is that a clear error and instruction to install usrmerge will be
printed.

As agreed, if this turns out to be a problem once the upload is in
unstable, we can switch the dependency in i-s-h to be strictly on
'usrmerge' and vendor the dozen of Perl modules that are required by
it, like Ubuntu did, but for now we'll stay the course.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Mon, 18 Jul 2022 at 20:03:14 +0100, Luca Boccassi wrote:
> On Mon, 2022-07-18 at 17:53 +0200, Johannes Schauer Marin Rodrigues
> > Also, what is the relationship between the usr-is-merged package and the
> > usrmerged package? Both were/are built by src:usrmerge but its changelog
> > doesn't indicate when or why the naming changed so I'm left a bit confused
> > about the relationship between these two packages. Since your deboootstrap
> > merge request changed from usrmerged to usr-is-merged I guess the latter is 
> > the
> > correct choice?
> 
> It was renamed following a request on #debian-ftp while it was in NEW,
> as the feedback was that 'usrmerge' and 'usrmerged' were too similar
> and thus easily confused. The 'usrmerged' one can be disregarded and
> will be de-crufted.

It's unfortunate that when a package like usrmerge has more than one
version waiting in NEW, the two operations available to the ftp team seem
to be "reject all versions of usrmerge" and "accept all versions of
usrmerge": if the result of NEW review is that the new binary package
needs to be renamed, as it was in this case, then the ideal result would
have been to reject usrmerge/28 from the queue and immediately accept
usrmerge/29.

I agree that usrmerge and usrmerged are confusingly similar names, and
usr-is-merged is a lot more clearly different.

smcv

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Mon, 2022-07-18 at 22:10 +0200, Johannes Schauer Marin Rodrigues
wrote:
> Hi,
> 
> Quoting Luca Boccassi (2022-07-18 21:03:14)
> > It was renamed following a request on #debian-ftp while it was in
> > NEW, as the
> > feedback was that 'usrmerge' and 'usrmerged' were too similar and
> > thus easily
> > confused. The 'usrmerged' one can be disregarded and will be de-
> > crufted.
> 
> I think that's very sensible.
> 
> > > Does that sound okay to you and does the patch look like it does
> > > the right
> > > thing?
> > 
> > Yes, without knowing much about mmdebstrap, the diff looks good to
> > me.
> > I'd only ask that in the comment of the script hooks/no-merged-
> > usr/setup00.sh if you could please mention explicitly that it
> > creates an
> > unsupported system. Maybe even print a warning when it's called.
> 
> Running mmdebstrap with --hook-dir=hooks/no-merged-usr will now print
> the
> following to stderr:
> 
> Warning: starting with Debian 12 (Bookworm), systems without merged-
> /usr are not supported anymore
> 
> > Also, I assume it is creating the metapackage on-the-fly because it
> > doesn't
> > have the downloaded packages available at that point? Not a
> > problem, just
> > double checking.
> 
> That is correct. But I think it's not ideal if mmdebstrap creates a
> chroot
> containing a package that doesn't come from the archive. I now
> extended the
> hook script such that calling mmdebstrap with --hook-
> dir=hooks/merged-usr will
> first install the custom built metapackage, then install the
> essential packages
> and then upgrade to the real usr-is-merged package.

Nice!

> I think then the roadmap is to release debootstrap with #71 merged,
> then upload
> init-system-helpers that depends "usrmerge | usr-is-merged" and then
> I test and
> upload mmdebstrap including those hook scripts. Since those are just
> hooks,
> nothing stops people from using them with the mmdebstrap version
> currently in
> unstable and testing, so nothing should be blocked by this in case I
> should
> need longer to release an mmdebstrap version shipping these hook
> scripts.

That's great. Currently waiting on usrmerge=29 to migrate to testing,
then the deboostrap change becomes mergeable. Also waiting on Simon to
clarify a few things regarding buildds and the plan around that -
depending on the outcome, the plan w.r.t. debootstrap might change
slightly or not.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2022-07-18 21:03:14)
> It was renamed following a request on #debian-ftp while it was in NEW, as the
> feedback was that 'usrmerge' and 'usrmerged' were too similar and thus easily
> confused. The 'usrmerged' one can be disregarded and will be de-crufted.

I think that's very sensible.

> > Does that sound okay to you and does the patch look like it does the right
> > thing?
> 
> Yes, without knowing much about mmdebstrap, the diff looks good to me.
> I'd only ask that in the comment of the script hooks/no-merged-
> usr/setup00.sh if you could please mention explicitly that it creates an
> unsupported system. Maybe even print a warning when it's called.

Running mmdebstrap with --hook-dir=hooks/no-merged-usr will now print the
following to stderr:

Warning: starting with Debian 12 (Bookworm), systems without merged-/usr are 
not supported anymore

> Also, I assume it is creating the metapackage on-the-fly because it doesn't
> have the downloaded packages available at that point? Not a problem, just
> double checking.

That is correct. But I think it's not ideal if mmdebstrap creates a chroot
containing a package that doesn't come from the archive. I now extended the
hook script such that calling mmdebstrap with --hook-dir=hooks/merged-usr will
first install the custom built metapackage, then install the essential packages
and then upgrade to the real usr-is-merged package.

I think then the roadmap is to release debootstrap with #71 merged, then upload
init-system-helpers that depends "usrmerge | usr-is-merged" and then I test and
upload mmdebstrap including those hook scripts. Since those are just hooks,
nothing stops people from using them with the mmdebstrap version currently in
unstable and testing, so nothing should be blocked by this in case I should
need longer to release an mmdebstrap version shipping these hook scripts.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Mon, 2022-07-18 at 17:53 +0200, Johannes Schauer Marin Rodrigues
wrote:
> Hi all,
> 
> Quoting Luca Boccassi (2022-07-18 16:42:03)
> > On Mon, 2022-07-18 at 16:04 +0200, Johannes Schauer Marin Rodrigues
> > wrote:
> > > Hi Luca & Helmut,
> > > 
> > > Quoting Helmut Grohne (2022-07-18 14:32:31)
> > > > On Sun, Jul 17, 2022 at 12:56:14AM +0100, Luca Boccassi wrote:
> > > > > A MR is pending for debootstrap [1] to make use of this new package 
> > > > > and
> > > > > file flag when --variant=buildd is passed, so that buildds can 
> > > > > continue
> > > > > to be unmerged-usr until the CTTE says otherwise, as per decision
> > > > > above.
> > > 
> > > I don't think I have access to the mail to which Helmut replied but I 
> > > guess [1]
> > > is this merge request?
> > > 
> > > https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/71
> > 
> > Yes - the thread with the first mail is on the CTTE mailing list:
> > https://lists.debian.org/debian-ctte/2022/07/threads.html
> > 
> > btw, any reason you dropped the list from CC?
> 
> I just didn't think that mmdebstrap-specific stuff would be on-topic on
> tech-ctte. Luca agreed for me to send my full reply to their mail back to the
> list in public.
> 
> > > > Can I ask you to also work with the mmdebstrap maintainer to make sure
> > > > that merged-/usr and opting out of it works well there? Specifically, 
> > > > I'd
> > > > like to have it work without pulling the usrmerge package (and perl
> > > > module dependencies) both in a merged and unmerged (buildd) layout.
> > > > mmdebstrap used to have switches, but now mostly ignores /usr-merge (and
> > > > thus doesn't merge).  If we do nothing, it'll pull in usrmerge and its
> > > > perl module dependencies with no easy way to opt out. I'm not sure what
> > > > the solution is, but I'm positive that you'll find a reasonable
> > > > compromise once talking to one another. mmdebstrap has tried to stay 
> > > > away
> > > > from custom set-up code, which makes this slightly non-trivial.
> > > 
> > > [SNIP RANT]
> > > 
> > > Anyways, I think what makes the situation a bit better as far as 
> > > mmdebstrap is
> > > concerned is, that we only need to be able to build an unmerged chroot 
> > > until
> > > the transition is complete, correct? Does this mean that the following 
> > > can be a
> > > solution for everybody that wants to build an unmerged buildd chroot?
> > > 
> > > mmdebstrap --variant=buildd \
> > > --setup-hook='echo "this system will not be supported in the future" 
> > > > "$1/etc/unsupported-skip-usrmerge-conversion"' \
> > > unstable chroot.tar
> > > 
> > > I can throw this into one of the hook scripts in 
> > > /usr/share/mmdebstrap/hooks
> > > and then the CLI invocation would be shorter if desired but I guess this 
> > > will
> > > only be run from scripts anyways? This will still install the usrmerge 
> > > package
> > > and its dependencies though, more on that below.
> > 
> > If you can also add set it up so that usr-is-merged gets installed,
> > then you avoid the usrmerge pkg and its extra dependencies as well. But
> > yes that should work.
> 
> Yes, that seems useful. But then usr-is-merged being installed would be a lie.
> I guess we accept that hack?

I think so, as long as it is understood that the system is unsupported
from Bookworm onward. The flag file was added specifically to allow
CIs/etc to stay back as indicated by the CTTE, so it's fine to use it
as such.

> Also, what is the relationship between the usr-is-merged package and the
> usrmerged package? Both were/are built by src:usrmerge but its changelog
> doesn't indicate when or why the naming changed so I'm left a bit confused
> about the relationship between these two packages. Since your deboootstrap
> merge request changed from usrmerged to usr-is-merged I guess the latter is 
> the
> correct choice?

It was renamed following a request on #debian-ftp while it was in NEW,
as the feedback was that 'usrmerge' and 'usrmerged' were too similar
and thus easily confused. The 'usrmerged' one can be disregarded and
will be de-crufted.

> > > > > Once deboostrap is updated and deployed on the buildds, a "usrmerge |
> > > > > usr-is-merged" dependency will be added to the Priority: essential
> > > > > init-system-helpers package, and uploaded to unstable to complete the
> > > > > transitions for all installations that are older than buster or that
> > > > > have been manually installed as unmerged-usr. Any system not upgraded
> > > > 
> > > > I think this setup is prone to breakage. While apt prefers the first
> > > > alternative, it doesn't have to pick it and other resolvers are even
> > > > more prone to picking non-default alternatives. Imagine one of the perl
> > > > modules being uninstallable. Even apt will pick the usr-is-merged
> > > > package in that scenario. Are you ware of this potential problem? Do you
> > > > have any ideas on how to handle it? To be clear: I do not think this
> > > > aspect to 

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Johannes Schauer Marin Rodrigues]

Hi all,

Quoting Luca Boccassi (2022-07-18 16:42:03)
> On Mon, 2022-07-18 at 16:04 +0200, Johannes Schauer Marin Rodrigues
> wrote:
> > Hi Luca & Helmut,
> > 
> > Quoting Helmut Grohne (2022-07-18 14:32:31)
> > > On Sun, Jul 17, 2022 at 12:56:14AM +0100, Luca Boccassi wrote:
> > > > A MR is pending for debootstrap [1] to make use of this new package and
> > > > file flag when --variant=buildd is passed, so that buildds can continue
> > > > to be unmerged-usr until the CTTE says otherwise, as per decision
> > > > above.
> > 
> > I don't think I have access to the mail to which Helmut replied but I guess 
> > [1]
> > is this merge request?
> > 
> > https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/71
> 
> Yes - the thread with the first mail is on the CTTE mailing list:
> https://lists.debian.org/debian-ctte/2022/07/threads.html
> 
> btw, any reason you dropped the list from CC?

I just didn't think that mmdebstrap-specific stuff would be on-topic on
tech-ctte. Luca agreed for me to send my full reply to their mail back to the
list in public.

> > > Can I ask you to also work with the mmdebstrap maintainer to make sure
> > > that merged-/usr and opting out of it works well there? Specifically, I'd
> > > like to have it work without pulling the usrmerge package (and perl
> > > module dependencies) both in a merged and unmerged (buildd) layout.
> > > mmdebstrap used to have switches, but now mostly ignores /usr-merge (and
> > > thus doesn't merge).  If we do nothing, it'll pull in usrmerge and its
> > > perl module dependencies with no easy way to opt out. I'm not sure what
> > > the solution is, but I'm positive that you'll find a reasonable
> > > compromise once talking to one another. mmdebstrap has tried to stay away
> > > from custom set-up code, which makes this slightly non-trivial.
> > 
> > [SNIP RANT]
> > 
> > Anyways, I think what makes the situation a bit better as far as mmdebstrap 
> > is
> > concerned is, that we only need to be able to build an unmerged chroot until
> > the transition is complete, correct? Does this mean that the following can 
> > be a
> > solution for everybody that wants to build an unmerged buildd chroot?
> > 
> > mmdebstrap --variant=buildd \
> > --setup-hook='echo "this system will not be supported in the future" > 
> > "$1/etc/unsupported-skip-usrmerge-conversion"' \
> > unstable chroot.tar
> > 
> > I can throw this into one of the hook scripts in /usr/share/mmdebstrap/hooks
> > and then the CLI invocation would be shorter if desired but I guess this 
> > will
> > only be run from scripts anyways? This will still install the usrmerge 
> > package
> > and its dependencies though, more on that below.
> 
> If you can also add set it up so that usr-is-merged gets installed,
> then you avoid the usrmerge pkg and its extra dependencies as well. But
> yes that should work.

Yes, that seems useful. But then usr-is-merged being installed would be a lie.
I guess we accept that hack?

Also, what is the relationship between the usr-is-merged package and the
usrmerged package? Both were/are built by src:usrmerge but its changelog
doesn't indicate when or why the naming changed so I'm left a bit confused
about the relationship between these two packages. Since your deboootstrap
merge request changed from usrmerged to usr-is-merged I guess the latter is the
correct choice?

> > > > Once deboostrap is updated and deployed on the buildds, a "usrmerge |
> > > > usr-is-merged" dependency will be added to the Priority: essential
> > > > init-system-helpers package, and uploaded to unstable to complete the
> > > > transitions for all installations that are older than buster or that
> > > > have been manually installed as unmerged-usr. Any system not upgraded
> > > 
> > > I think this setup is prone to breakage. While apt prefers the first
> > > alternative, it doesn't have to pick it and other resolvers are even
> > > more prone to picking non-default alternatives. Imagine one of the perl
> > > modules being uninstallable. Even apt will pick the usr-is-merged
> > > package in that scenario. Are you ware of this potential problem? Do you
> > > have any ideas on how to handle it? To be clear: I do not think this
> > > aspect to be a show-stopper. Let's just talk about it on a technical
> > > level.
> > > 
> > > Let me propose something strange without having thought through the
> > > implications in depth (i.e. it may be a totally bad idea):
> > > 
> > > What would happen if we were to replace the usr-is-merged package with a
> > > "trivial-usr-merge" package? This package would actually perform a
> > > /usr-merge in simplified conditions. Like usr-is-merged, it would refuse
> > > to perform a merge on actual systems. However in the case of
> > > non-container chroots (those not running init), it would perform the
> > > merge in a simpler way that doesn't require additional perl modules and
> > > doesn't come with the same amount of safety-guarantees.
> > > 
> > > A 

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Mon, 2022-07-18 at 14:32 +0200, Helmut Grohne wrote:
> Hi Luca,
> 
> Thank you for providing such a detailed and well-rationalized plan for
> moving forward. It removes quite an amount of uncertainty about where
> we're heading. I've missed this kind of clarity in previous
> conversations.
> 
> On Sun, Jul 17, 2022 at 12:56:14AM +0100, Luca Boccassi wrote:
> > A MR is pending for debootstrap [1] to make use of this new package and
> > file flag when --variant=buildd is passed, so that buildds can continue
> > to be unmerged-usr until the CTTE says otherwise, as per decision
> > above.
> 
> Can I ask you to also work with the mmdebstrap maintainer to make sure
> that merged-/usr and opting out of it works well there? Specifically,
> I'd like to have it work without pulling the usrmerge package (and perl
> module dependencies) both in a merged and unmerged (buildd) layout.
> mmdebstrap used to have switches, but now mostly ignores /usr-merge (and
> thus doesn't merge). If we do nothing, it'll pull in usrmerge and its
> perl module dependencies with no easy way to opt out. I'm not sure what
> the solution is, but I'm positive that you'll find a reasonable
> compromise once talking to one another. mmdebstrap has tried to stay
> away from custom set-up code, which makes this slightly non-trivial.

Is mmdebstrap used on buildds/piuparts/autopkgtest or similar
infrastructure that is to be held back? AFAIK they all use deboostrap,
hence I only looked at it, but I might be wrong. I ask because the
answer has implications on the urgency of any solution, not because it
shouldn't be handled.

We can certainly have a discussion in parallel to see what's the best
solution there.

> > Once deboostrap is updated and deployed on the buildds, a "usrmerge |
> > usr-is-merged" dependency will be added to the Priority: essential
> > init-system-helpers package, and uploaded to unstable to complete the
> > transitions for all installations that are older than buster or that
> > have been manually installed as unmerged-usr. Any system not upgraded
> 
> I think this setup is prone to breakage. While apt prefers the first
> alternative, it doesn't have to pick it and other resolvers are even
> more prone to picking non-default alternatives. Imagine one of the perl
> modules being uninstallable. Even apt will pick the usr-is-merged
> package in that scenario. Are you ware of this potential problem? Do you
> have any ideas on how to handle it? To be clear: I do not think this
> aspect to be a show-stopper. Let's just talk about it on a technical
> level.
> 
> Let me propose something strange without having thought through the
> implications in depth (i.e. it may be a totally bad idea):
> 
> What would happen if we were to replace the usr-is-merged package with a
> "trivial-usr-merge" package? This package would actually perform a
> /usr-merge in simplified conditions. Like usr-is-merged, it would refuse
> to perform a merge on actual systems. However in the case of
> non-container chroots (those not running init), it would perform the
> merge in a simpler way that doesn't require additional perl modules and
> doesn't come with the same amount of safety-guarantees.
> 
> A benefit of this approach would be that I could say mmebstrap
> --include=trivial-usr-merge. Possibly though, this is a bad idea for
> reasons I am not presently understanding well. Thank you for
> considering.

From what I understand there is really no difference between a full
system and a chroot from the implementation point of view. The things
it deals with are various situations the directories can be in, but
whether the system was booted or not doesn't seem to be handled
differently at all. In other words, if a 'simplified' version with
fewer dependencies was possible, it could be just used everywhere, I
think?

Regarding the failure mode, I think it is theorethically possible, but
very unlikely? The package has 3 individual perl modules dependencies
(one direct, libfile-find-rule-perl, and two transitive, libfile-find-
rule-perl and libnumber-compare-perl) and that's it, they don't depend
nor conflict or break with anything else, just perl:any. So you'd need
a headless system, that is dist-upgrading without using apt, and that
has somehow made perl:any uninstallable.
Do we know if other resolvers have coverage via autopkgtest/piuparts,
that would pick on these issues?

> > The patch from user uau that the dpkg maintainer rejected in the past
> > has been submitted to the existing bug [2] for completeness (with
> > permission from the author).
> > It will not be necessary to initiate or complete the migration to
> > merged-usr, and we will not hold back waiting for a resolution on it,
> > given the maintainer has made his intentions abundantly clear on the
> > matter as recently as four months ago [3]. The patch is presented
> > simply because it will likely be necessary, in that or any other form,
> > to lift the moratorium on moving files between packages 

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Helmut Grohne]

Hi Luca,

Thank you for providing such a detailed and well-rationalized plan for
moving forward. It removes quite an amount of uncertainty about where
we're heading. I've missed this kind of clarity in previous
conversations.

On Sun, Jul 17, 2022 at 12:56:14AM +0100, Luca Boccassi wrote:
> A MR is pending for debootstrap [1] to make use of this new package and
> file flag when --variant=buildd is passed, so that buildds can continue
> to be unmerged-usr until the CTTE says otherwise, as per decision
> above.

Can I ask you to also work with the mmdebstrap maintainer to make sure
that merged-/usr and opting out of it works well there? Specifically,
I'd like to have it work without pulling the usrmerge package (and perl
module dependencies) both in a merged and unmerged (buildd) layout.
mmdebstrap used to have switches, but now mostly ignores /usr-merge (and
thus doesn't merge). If we do nothing, it'll pull in usrmerge and its
perl module dependencies with no easy way to opt out. I'm not sure what
the solution is, but I'm positive that you'll find a reasonable
compromise once talking to one another. mmdebstrap has tried to stay
away from custom set-up code, which makes this slightly non-trivial.

> Once deboostrap is updated and deployed on the buildds, a "usrmerge |
> usr-is-merged" dependency will be added to the Priority: essential
> init-system-helpers package, and uploaded to unstable to complete the
> transitions for all installations that are older than buster or that
> have been manually installed as unmerged-usr. Any system not upgraded

I think this setup is prone to breakage. While apt prefers the first
alternative, it doesn't have to pick it and other resolvers are even
more prone to picking non-default alternatives. Imagine one of the perl
modules being uninstallable. Even apt will pick the usr-is-merged
package in that scenario. Are you ware of this potential problem? Do you
have any ideas on how to handle it? To be clear: I do not think this
aspect to be a show-stopper. Let's just talk about it on a technical
level.

Let me propose something strange without having thought through the
implications in depth (i.e. it may be a totally bad idea):

What would happen if we were to replace the usr-is-merged package with a
"trivial-usr-merge" package? This package would actually perform a
/usr-merge in simplified conditions. Like usr-is-merged, it would refuse
to perform a merge on actual systems. However in the case of
non-container chroots (those not running init), it would perform the
merge in a simpler way that doesn't require additional perl modules and
doesn't come with the same amount of safety-guarantees.

A benefit of this approach would be that I could say mmebstrap
--include=trivial-usr-merge. Possibly though, this is a bad idea for
reasons I am not presently understanding well. Thank you for
considering.

> The patch from user uau that the dpkg maintainer rejected in the past
> has been submitted to the existing bug [2] for completeness (with
> permission from the author).
> It will not be necessary to initiate or complete the migration to
> merged-usr, and we will not hold back waiting for a resolution on it,
> given the maintainer has made his intentions abundantly clear on the
> matter as recently as four months ago [3]. The patch is presented
> simply because it will likely be necessary, in that or any other form,
> to lift the moratorium on moving files between packages manually,
> whenever the CTTE decides that should happen. Whatever happens (or
> doesn't) to that patch/bug will not hold back the plan discussed here.

It seems as if the ball on this is being dropped on procedural grounds.
Yes, we need to figure that part out and it does take too long.
However, I think that the technical bits should be solved in parallel.
Solving the procedural issues is much easier if the technical work is
ready. I ask you to resume work there. This was effectively requested by
RT as well. I think that many recognize that unmerged will go away
before too long regardless of whether one likes it. But we also want a
dpkg that can handle the layout in a less fragile way even when our dpkg
maintainer disagrees with that vision.

> Do any of the Technical Committee members believe that this plan does
> not comply fully with their decision quoted above?

The remarks concerning mmdebstrap (including trivial-usr-merge) are to
be understood without a ctte-hat. The alternative picking and dpkg patch
concerns are mild concerns raised with my ctte-hat.

Helmut

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Luca Boccassi]

On Mon, 2021-10-18 at 12:17 -0700, Sean Whitton wrote:
> Hello,
> 
> On Wed 15 Sep 2021 at 11:46AM +01, Simon McVittie wrote:
> 
> > As discussed in our last meeting, I think we should issue more
> > specific
> > advice about merged-/usr, and in particular about what #978636
> > means for
> > maintainers right now.
> 
> With five votes cast in favour of the text by Simon, Niko, Gunnar,
> Marga
> and myself, the outcome is no longer in doubt.  Thus, in accordance
> with
> the Debian Constitution, the voting period has now concluded.
> 
> Therefore, using its powers under section 6.1.5 of the Debian
> Constitution, the Technical Committee issues the following advice:
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr
> and non-merged-/usr filesystem layouts. All of these installations
> should successfully upgrade via normal upgrade paths to a merged-/usr
> Debian 12.  Only after the release of Debian 12 can packages assume
> that
> all installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian
> 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but
> if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a
> bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root
> filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10
> [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and
> each
> /libQUAL that exists are symbolic links to the corresponding
> directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so
> on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in
> one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located
> at
> the corresponding path below /usr, such as /usr/bin/bash. Each file
> in
> one of the affected directories is accessible via two paths: its
> canonical logical location (such as /bin/bash or /usr/bin/env), and
> the
> other path implied by the aliasing (such as /usr/bin/bash or
> /bin/env).
> 
> There are two supported categories of Debian 11 installation, which
> are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-
> installer
>   from Debian 10 or later, or installed with debootstrap --merged-
> usr,
>   or converted from the non-merged-/usr layout by installing the
>   usrmerge package, or installed or converted by any similar
>   procedure. They have the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with
>   debian-installer from Debian 9 or earlier and subsequently upgraded
>   without converting to merged-/usr, or installed with debootstrap
>   --no-merged-usr, or converted from the merged-/usr layout with
> dpkg's
>   "dpkg-fsys-usrunmess" utility or any similar procedure. They have
> the
>   traditional, non-merged-/usr layout in which /bin/bash and
>   /usr/bin/env have exactly those physical paths, and /usr/bin/bash
> and
>   /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed
> for
> unifying the root filesystem with /usr. For avoidance of doubt, we do
> not consider other filesystem layouts to be implementations of
> merged-/usr.  In particular, we do not consider these to be
> implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and
>   /libQUAL, with /usr/bin as a symlink to /bin, etc. (this is the
>   reverse of merged-/usr, and was historically used in the hurd-i386
>   port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with
>   individual symbolic links such as /bin/bash -> /usr/bin/bash for
> only
>   those files that historically had their canonical logical location
> on
>   the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with
>   individual symbolic links such as /bin/env -> /usr/bin/env for all
>   files in the affected directories, regardless of whether they
>   historically had their canonical logical location on the root
>   filesystem
> 
> [1]:
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[David Bremner]

Simon McVittie  writes:

> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".
>
>  begin text to be voted on 
>
> Summary
> ===
>
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
>
> Main points:
>
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
>
> Definitions and current status
> ==
>
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
>
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
>
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
>
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
>
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
>
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
>
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
>
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
>
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
>
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
>
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
>
> Upgrade path from Debian 11 to Debian 12
> 
>
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
>
> Debian installations have traditionally had a straightforward upgrade
> path between consecutive releases, and the technical committee expects
> maintainers to continue this. In the case of #978636, the 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Elana Hashman]

On Wed, Oct 13, 2021 at 08:13:08PM +0100, Simon McVittie wrote:
> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

I vote (belatedly, for the record):

yes > FD

- e

> 
>  begin text to be voted on 
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> 
> 
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
> 
> Debian installations have traditionally had a straightforward upgrade
> path between 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Christoph Berg]

Re: Simon McVittie
> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

I vote yes > further discussion.

>  begin text to be voted on 
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> 
> 
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
> 
> Debian installations have traditionally had a straightforward upgrade
> path between consecutive releases, and the technical committee expects
> maintainers to 

Bug#994388: marked as done (tech-ctte: More specific advice regarding merged-/usr and implications of #978636)

[Debian Bug Tracking System]

Your message dated Mon, 18 Oct 2021 12:17:57 -0700
with message-id <87sfwyqh0a@melete.silentflame.com>
and subject line tech-ctte: More specific advice regarding merged-/usr and 
implications of #978636
has caused the Debian Bug report #994388,
regarding tech-ctte: More specific advice regarding merged-/usr and 
implications of #978636
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)


-- 
994388: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: tech-ctte
Severity: normal

As discussed in our last meeting, I think we should issue more specific
advice about merged-/usr, and in particular about what #978636 means for
maintainers right now.

Constitutionally, I'm asking the TC to use its power to offer formal
advice (Debian constitution, §6.1.5).

As for what that advice is, I'm open to suggestions, but I'm drafting
some possible wording, which I'll upload to
https://salsa.debian.org/debian/tech-ctte/ when I have a bug number
for it.

Some questions that I think we need to answer explicitly:

- What do we mean by merged-/usr? (We already said this in #914897, but
  I think it's worth reiterating that symlink farms are not it.)

- Is it OK for packages in testing/unstable during Debian 12 development
  to assume/require a merged-/usr system? (tl;dr: some maintainers think
  the answer is yes, but I think the answer is no.)

- When will it be OK for packages in testing/unstable to assume/require
  a merged-/usr system? (tl;dr: some maintainers think the answer is
  "right now", but I think the answer is "when the Debian 13 cycle begins"
  and not before.)

- Should maintainers proactively move files in packages from the root
  filesystem into /usr? (tl;dr: I think they should not, at least until
  the implications are better-understood.)

- Should maintainers of tools, e.g. debootstrap, proactively move files
  in packages from the root filesystem into /usr, e.g. systemd system units?
  (tl;dr: I think they should not.)

- Are packages built on a merged-/usr system required to work correctly
  on a non-merged-/usr system? If they are not, is it RC?
  (I think they are required to work, and it should usually be RC if
  they don't; constitutionally I don't think we can unilaterally declare
  a class of bugs to be RC, at least not while using our "advice" power,
  but we can certainly recommend that maintainers and the release team
  should consider them to be RC.)

smcv
--- End Message ---
--- Begin Message ---
Hello,

On Wed 15 Sep 2021 at 11:46AM +01, Simon McVittie wrote:

> As discussed in our last meeting, I think we should issue more specific
> advice about merged-/usr, and in particular about what #978636 means for
> maintainers right now.

With five votes cast in favour of the text by Simon, Niko, Gunnar, Marga
and myself, the outcome is no longer in doubt.  Thus, in accordance with
the Debian Constitution, the voting period has now concluded.

Therefore, using its powers under section 6.1.5 of the Debian
Constitution, the Technical Committee issues the following advice:

Summary
===

There are currently Debian 11 installations with both the merged-/usr
and non-merged-/usr filesystem layouts. All of these installations
should successfully upgrade via normal upgrade paths to a merged-/usr
Debian 12.  Only after the release of Debian 12 can packages assume that
all installations will be merged-/usr.

Main points:

- We have recommended merged-/usr for Debian 12.
- Moving individual files is not merged-/usr.
- "Symlink farms" are not merged-/usr.
- Upgrading a non-merged-/usr system to Debian 12 needs to work.
- Upgrading a merged-/usr system to Debian 12 needs to work.
- Packages cannot assume all systems are merged-/usr until the Debian 13
  development cycle begins.
- Upgrading via apt in the usual way should work.
- Testing and QA systems should be able to avoid this transition, but if
  they do, they cannot be upgraded beyond Debian 12.
- A package building incorrectly on a non-merged-/usr system is a bug.
- A package building incorrectly on a merged-/usr system is a bug.
- Please stop moving individual packages' files from the root filesystem
  into /usr, at least for now.

Definitions and current status
==

libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
such as lib64 on the amd64 architecture.

Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
/libQUAL that exists are sym

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Margarita Manterola]

> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

My vote on the text quoted below:
  yes > further discussion

> 
>  begin text to be voted on 
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> 
> 
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
> 
> Debian installations have traditionally had a straightforward upgrade
> path between consecutive releases, and the technical committee expects
> 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sean Whitton]

Hello,

On Wed 13 Oct 2021 at 08:13PM +01, Simon McVittie wrote:

> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

I vote

yes > further discussion

>  begin text to be voted on 
>
> Summary
> ===
>
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
>
> Main points:
>
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
>
> Definitions and current status
> ==
>
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
>
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
>
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
>
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
>
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
>
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
>
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
>
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
>
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
>
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
>
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
>
> Upgrade path from Debian 11 to Debian 12
> 
>
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
>
> Debian installations have traditionally had a straightforward upgrade
> path between consecutive releases, and the technical 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Gunnar Wolf]

Simon McVittie dijo [Wed, Oct 13, 2021 at 08:13:08PM +0100]:
> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

My vote on the text quoted below is:

yes > further discussion

>  begin text to be voted on 
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> 
> 
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
> 
> Debian installations have traditionally had a straightforward upgrade
> path between 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Niko Tyni]

On Wed, Oct 13, 2021 at 08:13:08PM +0100, Simon McVittie wrote:
> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

I vote: "yes" > "further discussion".

>  begin text to be voted on 
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> 
> 
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
> 
> Debian installations have traditionally had a straightforward upgrade
> path between consecutive releases, and 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 13 Oct 2021 at 20:13:08 +0100, Simon McVittie wrote:
> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

I vote yes > further discussion.

>  begin text to be voted on 
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> 
> 
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
> 
> Debian installations have traditionally had a straightforward upgrade
> path between consecutive releases, and the 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

I'm calling for votes on the following resolution as formal advice from
the Technical Committee (Constitution §6.1.5). There are no non-accepted
amendments, so the options to vote on are "yes" or "further discussion".

 begin text to be voted on 

Summary
===

There are currently Debian 11 installations with both the merged-/usr and
non-merged-/usr filesystem layouts. All of these installations should
successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
Only after the release of Debian 12 can packages assume that all
installations will be merged-/usr.

Main points:

- We have recommended merged-/usr for Debian 12.
- Moving individual files is not merged-/usr.
- "Symlink farms" are not merged-/usr.
- Upgrading a non-merged-/usr system to Debian 12 needs to work.
- Upgrading a merged-/usr system to Debian 12 needs to work.
- Packages cannot assume all systems are merged-/usr until the Debian 13
  development cycle begins.
- Upgrading via apt in the usual way should work.
- Testing and QA systems should be able to avoid this transition, but if
  they do, they cannot be upgraded beyond Debian 12.
- A package building incorrectly on a non-merged-/usr system is a bug.
- A package building incorrectly on a merged-/usr system is a bug.
- Please stop moving individual packages' files from the root filesystem
  into /usr, at least for now.

Definitions and current status
==

libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
such as lib64 on the amd64 architecture.

Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
/libQUAL that exists are symbolic links to the corresponding directories
below /usr. This results in aliasing between /bin and /usr/bin, and
so on.

In the merged-/usr layout, files whose canonical logical location is
in one of the affected directories on the root filesystem, such as
/bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
the corresponding path below /usr, such as /usr/bin/bash. Each file in
one of the affected directories is accessible via two paths: its canonical
logical location (such as /bin/bash or /usr/bin/env), and the other path
implied by the aliasing (such as /usr/bin/bash or /bin/env).

There are two supported categories of Debian 11 installation, which are
currently considered equally-supported:

- Merged-/usr installations. These were installed with debian-installer
  from Debian 10 or later, or installed with debootstrap --merged-usr,
  or converted from the non-merged-/usr layout by installing the usrmerge
  package, or installed or converted by any similar procedure. They have
  the merged-/usr layout.

- Non-merged-/usr installations. These were installed with debian-installer
  from Debian 9 or earlier and subsequently upgraded without converting
  to merged-/usr, or installed with debootstrap --no-merged-usr, or
  converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
  utility or any similar procedure. They have the traditional,
  non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
  those physical paths, and /usr/bin/bash and /bin/env do not exist.

Merged-/usr is not the only filesystem layout that has been proposed for
unifying the root filesystem with /usr. For avoidance of doubt, we do not
consider other filesystem layouts to be implementations of merged-/usr.
In particular, we do not consider these to be implementations of merged-/usr:

- all affected files physically located in /bin, /sbin, /lib and /libQUAL,
  with /usr/bin as a symlink to /bin, etc. (this is the reverse of
  merged-/usr, and was historically used in the hurd-i386 port)

- a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
  symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
  historically had their canonical logical location on the root filesystem

- a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
  symbolic links such as /bin/env -> /usr/bin/env for all files in the
  affected directories, regardless of whether they historically had
  their canonical logical location on the root filesystem

[1]: 
https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential

Upgrade path from Debian 11 to Debian 12


The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
should only support the merged-/usr layout. This resolution describes
the implications of that decision in more detail.

Debian installations have traditionally had a straightforward upgrade
path between consecutive releases, and the technical committee expects
maintainers to continue this. In the case of #978636, the upgrades we
are interested in are:

- Upgrading from Debian 11 (stable release) to Debian 12 (stable release)

- Upgrading from Debian 11 (stable release) to testing/unstable during the
  development cycle 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sean Whitton]

Hello Simon,

On Tue 05 Oct 2021 at 07:48PM +01, Simon McVittie wrote:

> On Sun, 03 Oct 2021 at 16:52:15 -0700, Sean Whitton wrote:
>> On Mon 27 Sep 2021 at 10:59AM +01, Simon McVittie wrote:
>> > On Thu, 16 Sep 2021 at 15:35:11 -0700, Sean Whitton wrote:
>> >> (1) The reason for this, to put it a bit simplistically, is that we
>> >> don't require apt to perform the upgrade between stable releases in any
>> >> particular order, right?  Or are there other reasons distinct from this
>> >> one that I'm missing?  I think it would be good to state the thing about
>> >> apt (in better language than mine) in the text.
>> >
>> > I think that's the main reason. We have not traditionally mandated
>> > the use of a special upgrade tool like Ubuntu's do-release-upgrade(8),
>> > so the upgrade happens in whatever order apt chooses, which can vary
>> > between machines.
>> >
>> > Another reason why I think we want Debian 12 packages to be installable
>> > onto non-merged-/usr systems is that to be able to do our development work,
>> > they need to be installable onto testing/unstable systems, which (again)
>> > means that the upgrade order is undefined.
>>
>> Right, we're on the same page then, but would you agree with me that the
>> resolution should state this justification explicitly?
>
> I hope that
> 
> implements this to your satisfaction. If not, suggestions for better
> wording welcome - I would prefer not to be the only one writing this
> document!

Thanks, yes, this addresses my feedback.

I've pushed some wording tweaks.  Hope you don't mind me not filing a
MR -- seemed uncontroversial so thought I'd just push.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Niko Tyni]

On Wed, Sep 15, 2021 at 12:35:38PM +0100, Simon McVittie wrote:

> https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md

Many thanks for your work, and sorry for not participating earlier.

I agree with you that it would be good to send advice out sooner
rather than later.

I also agree with pretty much all you're saying in the text.

In particular:

> Questions for the committee:
> 
> - §(Definitions and current status): Does everyone agree with my
>   characterization of where we are now?

Ack from me.

> - §(Upgrade path from Debian 11 to Debian 12): Does everyone agree
>   with what I've written here about the implications of #978636?

I think I can follow the logic, and I agree with it.

> - §(Upgrade path from Debian 11 to Debian 12): Is the last paragraph
>   "If a suitable transition mechanism is not available by the time the
>   Debian 12 freeze is reached..." necessary, or is it implicit that
>   *obviously* we won't demand that the project carries on with merged-/usr
>   if it turns out not to be possible?

I think it should be included.

> - §(Testing and QA): Do we want to recommend this?

It does feel a bit like micro-managing to me. But the reasoning
makes sense. I'm fine with recommending it.
 
> - §(Building packages): Does everyone agree with my interpretation of
>   what we agreed in #914897 and its implications? Do we need to make a
>   recommendation for the Debian 12 development cycle, or is it enough
>   to assume that the "middle" option that we resolved in #914897
>   continues to be our opinion?

I agree and I don't think we need a new recommendation.

> - §(Building packages): I almost wrote an extra paragraph about how
>   this class of bugs becomes a non-issue when merged-/usr is the only
>   supported layout - but actually it doesn't! If we consider building
>   packages while having /usr/local/bin/sed to be a supported thing you
>   can do, then we need to ensure that /usr/local/bin/sed doesn't get
>   hard-coded into the resulting package, and the steps you take to
>   make that happen are the same as the steps you take to fix this class
>   of bugs.

FWIW I think we've traditionally considered such /usr/local -related
issues as bugs in the build setup rather than in the packages.

> - §(Moratorium on moving files' logical locations into /usr):
>   I think we should stop moving files into /usr on an individual basis,
>   at least until the consequences are fully understood, and perhaps for
>   the whole Debian 12 release cycle (after which, assuming merged-/usr
>   goes as we have recommended, maintainers can swap their logical location
>   without needing a transitional mechanism any more). Implementing
>   merged-/usr as the only supported layout means that moving files into
>   /usr on an individual basis is mostly unnecessary, because it has no
>   practical effect any more.

Agreed on the moratorium, mainly because of the unnecessity and the
error-prone nature of the moves. Sam brought up concerns about the
Replaces failure mode mentioned in the text, that part might need
changing.

On Mon, Sep 27, 2021 at 10:59:37AM +0100, Simon McVittie wrote:

> I am a little concerned that usrmerge is doing more intrusive surgery on
> a running system than even what's normal for an apt/dpkg-based upgrade,
> so I would prefer not to rule out designs that defer this action to a
> later time.

I share this concern.

-- 
Niko

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Sun, 03 Oct 2021 at 16:52:15 -0700, Sean Whitton wrote:
> On Mon 27 Sep 2021 at 10:59AM +01, Simon McVittie wrote:
> > On Thu, 16 Sep 2021 at 15:35:11 -0700, Sean Whitton wrote:
> >> (1) The reason for this, to put it a bit simplistically, is that we
> >> don't require apt to perform the upgrade between stable releases in any
> >> particular order, right?  Or are there other reasons distinct from this
> >> one that I'm missing?  I think it would be good to state the thing about
> >> apt (in better language than mine) in the text.
> >
> > I think that's the main reason. We have not traditionally mandated
> > the use of a special upgrade tool like Ubuntu's do-release-upgrade(8),
> > so the upgrade happens in whatever order apt chooses, which can vary
> > between machines.
> >
> > Another reason why I think we want Debian 12 packages to be installable
> > onto non-merged-/usr systems is that to be able to do our development work,
> > they need to be installable onto testing/unstable systems, which (again)
> > means that the upgrade order is undefined.
> 
> Right, we're on the same page then, but would you agree with me that the
> resolution should state this justification explicitly?

I hope that

implements this to your satisfaction. If not, suggestions for better
wording welcome - I would prefer not to be the only one writing this
document!

> > [Reasoning in a previous mail] makes me reluctant to require a special
> > upgrade procedure if it is not strictly necessary.
> 
> This is persuasive.  What do you think about including it in the text?

This is also in
.

> > I'm honestly not sure which of these is "the" reason why I'm recommending
> > the conservative approach. Some combination of your second and third
> > points, I suppose?
> 
> Based on what you say above I think it's the second and third, indeed.
> If we add to the text the things I'm suggesting we add, I think this
> sort of query about our motivations will not arise in the minds of
> readers.

Does 
cover this? If not, please suggest something that would.

Thanks,
smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Mon, 27 Sep 2021 at 09:18:29 -0600, Sam Hartman wrote:
> [smcv wrote]
> >On merged-/usr systems, there is a possible failure mode involving files
> >being moved between packages (with Replaces) during the same release
> >cycle that their logical location is changed from the root filesystem
> >to the corresponding aliased directory in /usr, which can result in
> >the affected file disappearing. This can be avoided by not changing
> >the file's logical location until the beginning of the Debian 13
> >development cycle, after the transition to merged-/usr is complete.
>
> I don't understand how transitioning files in the Debian 13 cycle is
> going to work any better than in the Debian 12 cycle unless dpkg happens
> to change.

I think you might be right about this.

It might not be possible to do these changes of logical location without
first enhancing dpkg to understand that certain directory hierarchies
are aliases for each other.

Or, perhaps it can work in a subset of cases, and people with an interest
in merged-/usr can identify patterns that are safe, and have guidance
that recommends sticking to those patterns?

(For example we might need to require that if files that were historically
in /bin, /sbin, /lib* are moved between packages, then either there's some
sequencing requirement enforced by Pre-Depends/Conflicts to make sure the
/usr move is done before the ownership change, or the files do not move to
/usr until the next release cycle? I haven't thought this through, I'm just
saying these as examples of things that people who do the detailed design
might try.)

Do we want to specifically say in our advice that proponents of merged-/usr
are invited to design solutions for this?

The Technical Committee does not do detailed design, so I think identifying
specific patterns that are safe is not our job - but if domain experts
identify good patterns, and ask us to check their working and then issue
advice recommending those patterns, we can do that.

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sean Whitton]

Hello,

On Mon 27 Sep 2021 at 10:59AM +01, Simon McVittie wrote:

> On Thu, 16 Sep 2021 at 15:35:11 -0700, Sean Whitton wrote:
>> You write:
>>
>> >> - Because Debian 11 installations with the non-merged-/usr layout already
>> >>   exist, all packages in Debian 12 should be installable onto a 
>> >> non-merged-/usr
>> >>   system along with their dependencies, and work correctly on the 
>> >> resulting
>> >>   system.
>>
>> (1) The reason for this, to put it a bit simplistically, is that we
>> don't require apt to perform the upgrade between stable releases in any
>> particular order, right?  Or are there other reasons distinct from this
>> one that I'm missing?  I think it would be good to state the thing about
>> apt (in better language than mine) in the text.
>
> I think that's the main reason. We have not traditionally mandated
> the use of a special upgrade tool like Ubuntu's do-release-upgrade(8),
> so the upgrade happens in whatever order apt chooses, which can vary
> between machines.
>
> Another reason why I think we want Debian 12 packages to be installable
> onto non-merged-/usr systems is that to be able to do our development work,
> they need to be installable onto testing/unstable systems, which (again)
> means that the upgrade order is undefined.

Right, we're on the same page then, but would you agree with me that the
resolution should state this justification explicitly?

> We also have best-effort support for partial upgrades, either
> oldstable -> stable or stable -> testing/unstable: upgrading only a
> subset of the available packages, plus their mandatory dependencies,
> but without also upgrading apparently-unrelated packages. This can
> only ever be partially supported, because it leads to a combinatorial
> explosion of possible partially-upgraded systems, and realistically we
> can never test them all; but I think it's best if we try to avoid
> knowingly making this worse.

I'd suggest we don't mention this as it's much more obscure and the
above is sufficient justification.

>> (2) Some people on -devel would seem to think that we can have smooth
>> upgrades from bullseye to bookworm without requiring support for
>> non-merged-/usr in every single package in bookworm, i.e., without
>> supporting non-merged-/usr for testing/unstable installations during the
>> current release cycle.
>
> Some people on -devel have argued that it would be OK to require use of
> a special upgrade tool analogous to Ubuntu's do-release-upgrade just this
> once, or that it would be OK to require a particular upgrade sequence. For
> example, we could ask users to install the usrmerge package first, and
> then upgrade; or if dpkg is modified to special-case the merged-/usr
> aliasing symlinks, then we might ask users to upgrade dpkg first, and
> then upgrade the rest of the system.
>
> I think there is considerable anecdotal evidence that many Debian users
> do not read the release notes, and if we ask them to carry out an upgrade
> procedure more complicated than
>
> $editor /etc/apt/sources.list && apt update && apt full-upgrade
>
> they will often ignore that request and still expect their upgrade to
> work (because in practice it often does, and we've been training users
> to expect that for years). That makes me reluctant to require a special
> upgrade procedure if it is not strictly necessary.

This is persuasive.  What do you think about including it in the text?
Specifically, mentioning that we are deciding, for the project, that
we are not going to do this using something like do-release-upgrade(8).

>> What smcv has been arguing for is the most conservative option.  But
>> which of the following is it the TC wants to say:
>>
>> - we are sure that this is the only way to ensure smooth upgrades, such
>>   that it pretty much follows from our previous decision on merged-/usr
>>
>> - we think there might be viable alternatives to requiring every package
>>   in bookworm to work on both layouts, but we aren't sure they are safe
>>   enough, so we're recommending a simple and conservative approach
>>
>> - we think that ensuring non-merged-/usr testing/unstable installations
>>   work during this release cycle is reason enough to take this highly
>>   conservative approach
>
> I'm honestly not sure which of these is "the" reason why I'm recommending
> the conservative approach. Some combination of your second and third
> points, I suppose?

Based on what you say above I think it's the second and third, indeed.
If we add to the text the things I'm suggesting we add, I think this
sort of query about our motivations will not arise in the minds of
readers.

Thanks!

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sam Hartman]

>On merged-/usr systems, there is a possible failure mode involving files
>being moved between packages (with Replaces) during the same release
>cycle that their logical location is changed from the root filesystem
>to the corresponding aliased directory in /usr, which can result in
>the affected file disappearing. This can be avoided by not changing
>the file's logical location until the beginning of the Debian 13
>development cycle, after the transition to merged-/usr is complete.



Simon, I've reviewed your draft, and with the exception of the above
text, it all seems consistent with my understanding of the technical
details.

I don't understand how transitioning files in the Debian 13 cycle is
going to work any better than in the Debian 12 cycle unless dpkg happens
to change.
So I don't see how the above text is correct.


If I'm missing something, perhaps we could clarify the text, because I
suspect I have a better than average understanding of the issues.  If I
don't get it, others probably will misunderstand too.

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 15 Sep 2021 at 12:35:38 +0100, Simon McVittie wrote:
> https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md

I have updated this draft to incorporate my edits from !3, and feedback
from bremner on IRC.

I'd like to keep this moving, because it's somewhat time-sensitive: people
are already taking action based on our earlier resolution, some of which is
not necessarily the action we would have wanted them to take.

On Wed, 15 Sep 2021 at 11:46:02 +0100, Simon McVittie wrote:
> Some questions that I think we need to answer explicitly:
> 
> - What do we mean by merged-/usr? (We already said this in #914897, but
>   I think it's worth reiterating that symlink farms are not it.)

This is defined (again) by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#definitions-and-current-status

> - Is it OK for packages in testing/unstable during Debian 12 development
>   to assume/require a merged-/usr system? (tl;dr: some maintainers think
>   the answer is yes, but I think the answer is no.)
>
> - When will it be OK for packages in testing/unstable to assume/require
>   a merged-/usr system? (tl;dr: some maintainers think the answer is
>   "right now", but I think the answer is "when the Debian 13 cycle begins"
>   and not before.)

Both of these are addressed by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#upgrade-path-from-debian-11-to-debian-12

> - Should maintainers proactively move files in packages from the root
>   filesystem into /usr? (tl;dr: I think they should not, at least until
>   the implications are better-understood.)
> 
> - Should maintainers of tools, e.g. debootstrap, proactively move files
>   in packages from the root filesystem into /usr, e.g. systemd system units?
>   (tl;dr: I think they should not.)

Both of these are addressed by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#moratorium-on-moving-files-logical-locations-into-usr

> - Are packages built on a merged-/usr system required to work correctly
>   on a non-merged-/usr system? If they are not, is it RC?
>   (I think they are required to work, and it should usually be RC if
>   they don't)

https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#building-packages

On Thu, 16 Sep 2021 at 15:35:11 -0700, Sean Whitton wrote:
> You write:
> 
> >> - Because Debian 11 installations with the non-merged-/usr layout already
> >>   exist, all packages in Debian 12 should be installable onto a 
> >> non-merged-/usr
> >>   system along with their dependencies, and work correctly on the resulting
> >>   system.
> 
> (1) The reason for this, to put it a bit simplistically, is that we
> don't require apt to perform the upgrade between stable releases in any
> particular order, right?  Or are there other reasons distinct from this
> one that I'm missing?  I think it would be good to state the thing about
> apt (in better language than mine) in the text.

I think that's the main reason. We have not traditionally mandated
the use of a special upgrade tool like Ubuntu's do-release-upgrade(8),
so the upgrade happens in whatever order apt chooses, which can vary
between machines.

Another reason why I think we want Debian 12 packages to be installable
onto non-merged-/usr systems is that to be able to do our development work,
they need to be installable onto testing/unstable systems, which (again)
means that the upgrade order is undefined.

We also have best-effort support for partial upgrades, either oldstable
-> stable or stable -> testing/unstable: upgrading only a subset of the
available packages, plus their mandatory dependencies, but without also
upgrading apparently-unrelated packages. This can only ever be partially
supported, because it leads to a combinatorial explosion of possible
partially-upgraded systems, and realistically we can never test them all;
but I think it's best if we try to avoid knowingly making this worse.

Finally, there is a reason to want this that is circular in nature: if we
want Debian 12 packages to be installable onto non-merged-/usr systems,
and we want to be able to say with reasonable confidence that we think
they work in that situation, then we need to be able to test that they
do, in fact, work. As mentioned under the "Testing and QA" heading,
if we do not keep it possible to force autopkgtest VMs/containers,
piuparts chroots, reproducible-builds chroots and manual-test VMs (or
even scratch installations on real hardware) to be non-merged-/usr,
then we will have trouble testing that. However, I have tried to make it
clear that if a developer sets up such a system, they cannot expect to
be able to upgrade it cleanly to Debian 13, or to a post-Debian-12
state of testing/unstable.

> (2) Some people on -devel would seem to think that we can have smooth
> upgrades from bullseye to bookworm without requiring 

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Sam Hartman]

> "Simon" == Simon McVittie  writes:

Simon> Package: tech-ctte Severity: normal

Simon> As discussed in our last meeting, I think we should issue
Simon> more specific advice about merged-/usr, and in particular
Simon> about what #978636 means for maintainers right now.

I wasn't at this meeting, but as someone who has followed the merged
/usr discussions reasonably closely, I'd like to second the idea of
giving this advice.


I think we had a very productive discussion on debian-devel.
I think with a round or two more we could confirm project consensus on
some of the areas where you propose to give advice.

Honestly, though, I think  that having the TC give a decision in this
instance will be more effective and will take up less time overall.

I think the advice you propose to give is consistent with project
consensus where there is a consensus.  I think there are a couple of
areas, like gradual migration of paths where there is no rough consensus.  In
particular, I think Neils is not the only one who is uncomfortable
assuming that dpkg changes will eventually come along to better support
aliasing.
In those areas, advice is needed, and the TC is a body that can give
that advice.
The advice you are proposing to give seems technically sound to me.

So, thanks very much for your hard work on this issue!


signature.asc
Description: PGP signature

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Zack Weinberg]

As a Debian user I'm pleased to see the ctte taking proactive steps to
ensure that the merged-/usr transition will still allow smooth
upgrades from Debian 11 to 12 and 12 to 13.

As an upstream contributor to several pieces of software included in
Debian, and as someone with an interest in ensuring that software
developed on Linux is not *accidentally* unportable to other OSes
under the 'Unix' umbrella, I'd like to stick an oar in regarding:

On Wed, 15 Sep 2021 at 12:39:53 +0100, Simon McVittie wrote:
> On Wed, 15 Sep 2021 at 12:35:38 +0100, Simon McVittie wrote:
> > - §(Building packages): I almost wrote an extra paragraph about
> >   how this class of bugs becomes a non-issue when merged-/usr is
> >   the only supported layout - but actually it doesn't! If we
> >   consider building packages while having /usr/local/bin/sed to be
> >   a supported thing you can do, then we need to ensure that
> >   /usr/local/bin/sed doesn't get hard-coded into the resulting
> >   package, and the steps you take to make that happen are the same
> >   as the steps you take to fix this class of bugs.
>
> I think that class of bugs probably does become non-RC when the
> merged-/usr transition is complete, though.

After the transition is complete, /usr/local is not the only possible
source of problems.  For the various files with a "canonical" location
either in /usr or in /, either specified by some standard or by
convention, and regularly referred to by absolute pathname, all
software should continue to refer to those files by their "canonical"
name, so that it remains portable to Unixes that have not and may
never undergo a /usr merge.  The most common class of such files is
those used in #! directives: /bin/sh not /usr/bin/sh, /usr/bin/perl
not /bin/perl, etc.  I would ideally like this to be spelled out in
Policy, as an explicit list of files that MUST be referred to without
/usr, all others to be referred to with /usr.

[I agree that from Debian's perspective it makes sense for these bugs
to be RC until the transition is complete, and non-RC afterward.]

zw

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 15 Sep 2021 at 12:35:38 +0100, Simon McVittie wrote:
> - §(Building packages): I almost wrote an extra paragraph about how
>   this class of bugs becomes a non-issue when merged-/usr is the only
>   supported layout - but actually it doesn't! If we consider building
>   packages while having /usr/local/bin/sed to be a supported thing you
>   can do, then we need to ensure that /usr/local/bin/sed doesn't get
>   hard-coded into the resulting package, and the steps you take to
>   make that happen are the same as the steps you take to fix this class
>   of bugs.

I think that class of bugs probably does become non-RC when the
merged-/usr transition is complete, though.

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 15 Sep 2021 at 11:46:11 +0100, Simon McVittie wrote:
> As for what that advice is, I'm open to suggestions, but I'm drafting
> some possible wording, which I'll upload to
> https://salsa.debian.org/debian/tech-ctte/ when I have a bug number
> for it.

Here it is:

https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md

Edits and merge requests welcome. I suggest we use merge requests for
substantive changes.

This is a collection of various pieces of advice. I hope that this is
sufficiently uncontroversial among the TC that we can improve the wording
where necessary, then take a unanimous or close-to-unanimous vote
"yes > further discussion" when we feel that it reflects consensus?

If there are individual bits that are more contentious, then I suggest
we vote on them as formally or informally as we are comfortable
with, then make the formal resolution reflect the results of those
votes; alternatively, we could kick out anything more controversial into
a separate resolution if we need to.

Because this is advice to maintainers about decisions that, in some cases,
they are making right now, I would like to keep this moving and try to
reach a consensus we can announce to debian-devel-announce soon.

Questions for the committee:

- §(Definitions and current status): Does everyone agree with my
  characterization of where we are now?

- §(Upgrade path from Debian 11 to Debian 12): Does everyone agree
  with what I've written here about the implications of #978636?

  I've tried to be careful to distinguish between the Debian 12
  stable release and the state of testing/unstable during the development
  cycle; better wordsmithing welcome.

- §(Upgrade path from Debian 11 to Debian 12): Is the last paragraph
  "If a suitable transition mechanism is not available by the time the
  Debian 12 freeze is reached..." necessary, or is it implicit that
  *obviously* we won't demand that the project carries on with merged-/usr
  if it turns out not to be possible?

- §(Testing and QA): Do we want to recommend this?

- §(Building packages): Does everyone agree with my interpretation of
  what we agreed in #914897 and its implications? Do we need to make a
  recommendation for the Debian 12 development cycle, or is it enough
  to assume that the "middle" option that we resolved in #914897
  continues to be our opinion?

  I assume our advice power doesn't extend to unilaterally declaring
  a class of bugs to be RC, but we can certainly advise the release team
  and package maintainers that they *should* consider a class of bugs
  to be RC; if they follow our advice, great, and if they don't, we can
  consider whether we need to overrule in individual cases.

- §(Building packages): I almost wrote an extra paragraph about how
  this class of bugs becomes a non-issue when merged-/usr is the only
  supported layout - but actually it doesn't! If we consider building
  packages while having /usr/local/bin/sed to be a supported thing you
  can do, then we need to ensure that /usr/local/bin/sed doesn't get
  hard-coded into the resulting package, and the steps you take to
  make that happen are the same as the steps you take to fix this class
  of bugs.

- §(Moratorium on moving files' logical locations into /usr):
  I think we should stop moving files into /usr on an individual basis,
  at least until the consequences are fully understood, and perhaps for
  the whole Debian 12 release cycle (after which, assuming merged-/usr
  goes as we have recommended, maintainers can swap their logical location
  without needing a transitional mechanism any more). Implementing
  merged-/usr as the only supported layout means that moving files into
  /usr on an individual basis is mostly unnecessary, because it has no
  practical effect any more.

  Note that my opinion on this is consistent with Adrian's request
  to overrule the debianutils maintainer and require installkernel
  and run-parts to be moved back to the rootfs. We should make sure
  that whatever we decide here is consistent with the way in which we
  overrule or decline to overrule the debianutils maintainer.

> - Should maintainers of tools, e.g. debootstrap, proactively move files
>   in packages from the root filesystem into /usr, e.g. systemd system units?
>   (tl;dr: I think they should not.)

Ansgar pointed out on IRC that of course I meant to say debhelper here, not
debootstrap. The version of debhelper in git reverts the change that
previously moved systemd units from /lib/systemd/system into
/usr/lib/systemd/system, but at the time of writing that revert is not
in a release.

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

Package: tech-ctte
Severity: normal

As discussed in our last meeting, I think we should issue more specific
advice about merged-/usr, and in particular about what #978636 means for
maintainers right now.

Constitutionally, I'm asking the TC to use its power to offer formal
advice (Debian constitution, §6.1.5).

As for what that advice is, I'm open to suggestions, but I'm drafting
some possible wording, which I'll upload to
https://salsa.debian.org/debian/tech-ctte/ when I have a bug number
for it.

Some questions that I think we need to answer explicitly:

- What do we mean by merged-/usr? (We already said this in #914897, but
  I think it's worth reiterating that symlink farms are not it.)

- Is it OK for packages in testing/unstable during Debian 12 development
  to assume/require a merged-/usr system? (tl;dr: some maintainers think
  the answer is yes, but I think the answer is no.)

- When will it be OK for packages in testing/unstable to assume/require
  a merged-/usr system? (tl;dr: some maintainers think the answer is
  "right now", but I think the answer is "when the Debian 13 cycle begins"
  and not before.)

- Should maintainers proactively move files in packages from the root
  filesystem into /usr? (tl;dr: I think they should not, at least until
  the implications are better-understood.)

- Should maintainers of tools, e.g. debootstrap, proactively move files
  in packages from the root filesystem into /usr, e.g. systemd system units?
  (tl;dr: I think they should not.)

- Are packages built on a merged-/usr system required to work correctly
  on a non-merged-/usr system? If they are not, is it RC?
  (I think they are required to work, and it should usually be RC if
  they don't; constitutionally I don't think we can unilaterally declare
  a class of bugs to be RC, at least not while using our "advice" power,
  but we can certainly recommend that maintainers and the release team
  should consider them to be RC.)

smcv