Bug#1078598: piuparts.debian.org: oldstable22testing, oldstable222sid should not fail on /usr moves

[Simon McVittie]

Package: piuparts.debian.org
Severity: normal
X-Debbugs-Cc: debian-ctte@lists.debian.org

piuparts.debian.org currently reports test failures for many packages
in the "oldstable22testing" upgrade path (oldstable → stable → testing)
and the "oldstable222sid" upgrade path (oldstable → stable → testing → sid)
as a result of files having been moved from /foo to /usr/foo.
For example, libaccountsservice-doc:all reports these:

0m48.6s ERROR: FAIL: File(s) moved between /{bin|sbin|lib*} and 
/usr/{bin|sbin|lib*}:   /lib/x86_64-linux-gnu/libblkid.so.1.1.0 
['libblkid1:amd64'] => /usr/lib/x86_64-linux-gnu/libblkid.so.1.1.0 
['libblkid1:amd64']
  /lib/x86_64-linux-gnu/libblkid.so.1 ['libblkid1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libblkid.so.1 ['libblkid1:amd64']
  /lib/x86_64-linux-gnu/libfdisk.so.1.1.0 ['libfdisk1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libfdisk.so.1.1.0 ['libfdisk1:amd64']
  /lib/x86_64-linux-gnu/libfdisk.so.1 ['libfdisk1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libfdisk.so.1 ['libfdisk1:amd64']
  /lib/x86_64-linux-gnu/libgcrypt.so.20 ['libgcrypt20:amd64'] => 
/usr/lib/x86_64-linux-gnu/libgcrypt.so.20 ['libgcrypt20:amd64']
  /lib/x86_64-linux-gnu/libmount.so.1.1.0 ['libmount1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libmount.so.1.1.0 ['libmount1:amd64']
  /lib/x86_64-linux-gnu/libmount.so.1 ['libmount1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libmount.so.1 ['libmount1:amd64']
  /lib/x86_64-linux-gnu/libsmartcols.so.1.1.0 ['libsmartcols1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libsmartcols.so.1.1.0 ['libsmartcols1:amd64']
  /lib/x86_64-linux-gnu/libsmartcols.so.1 ['libsmartcols1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libsmartcols.so.1 ['libsmartcols1:amd64']
  /lib/x86_64-linux-gnu/libsystemd.so.0 ['libsystemd0:amd64'] => 
/usr/lib/x86_64-linux-gnu/libsystemd.so.0 ['libsystemd0:amd64']
  /lib/x86_64-linux-gnu/libudev.so.1 ['libudev1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libudev.so.1 ['libudev1:amd64']
  /lib/x86_64-linux-gnu/libuuid.so.1.3.0 ['libuuid1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libuuid.so.1.3.0 ['libuuid1:amd64']
  /lib/x86_64-linux-gnu/libuuid.so.1 ['libuuid1:amd64'] => 
/usr/lib/x86_64-linux-gnu/libuuid.so.1 ['libuuid1:amd64']
  /lib/udev/hwclock-set ['util-linux'] => /usr/lib/udev/hwclock-set 
['util-linux-extra']
  /lib/udev/rules.d/85-hwclock.rules ['util-linux'] => 
/usr/lib/udev/rules.d/85-hwclock.rules ['util-linux-extra']

Clearly none of those are actually acccountsservice's fault, and my
understanding is that none are even considered to be a problem.

The Debian Technical Committee issued advice that files should not be moved
from /foo to /usr/foo during the Debian 12 release cycle (#994388) and
the early part of the Debian 13 release cycle (#1035831), but this
moratorium was lifted in late 2023 (#1053901) and, instead, there has been
an ongoing effort to relocate files from /foo to /usr/foo. I believe the
goal is that the only package owning paths in /{bin,sbin,lib*} in trixie
will be base-files.

As a result, I think piuparts should stop reporting these moves as errors
in upgrade paths whose end state is Debian 13 'trixie' or later.
It would still be appropriate to report /foo → /usr/foo moves as errors if
the end state is Debian 12 'bookworm' or older, as per #994388.

If I'm reading the configuration in
https://salsa.debian.org/debian/piuparts/-/blob/develop/instances/piuparts.conf-template.pejacevic
correctly, it might be appropriate to keep "--warn-on-usr-move fail" in
"flags-end-bookworm", but remove it from "flags-start-bullseye"?
That way, the moratorium on /foo → /usr/foo moves would still be enforced
for bullseye → bookworm, but not for bullseye → bookworm → trixie, which I
think is what we want.

Or, it might be better to remove "--warn-on-usr-move fail"
from both "flags-end-bookworm" and "flags-start-bullseye", and instead
add it to each of the finite number of test scenarios where the end state
is strictly older than trixie (including bookworm, bookworm-pu, etc.).

Thanks,
smcv

Bug#1077764: Ruling request on os-release specification implementation

[Simon McVittie]

On Mon, 05 Aug 2024 at 09:02:51 +0200, Marc Haber wrote:
> On Mon, Aug 05, 2024 at 02:07:54AM +0200, Timo Röhling wrote:
> > > If trixie was identified as trixie, and sid was identified as unstable,
> > > what compromise would be, er, compromised, precisely?
> > Unstable would become less useful at weeding out bugs in packages before
> > they reach testing,
> 
> Can you elaborate on that? Why does providing more and better
> information make it harder to identify bugs?

I think the concern is somewhere along these lines:

* Some package, let's call it foobar, reads os-release and changes its
  behaviour according to whether it sees trixie/testing or unstable

* foobar_1.2-3 is in unstable and works correctly there

* The testing migration scripts let it migrate

* trixie's os-release is different from unstable's (this is the essence
  of what Luca is asking for)

* Unfortunately, when it sees trixie's os-release, foobar_1.2-3 behaves
  incorrectly

* Now our mechanisms to avoid regressions in testing have failed to
  prevent a regression, because the regression was never visible to users
  of unstable, and in fact didn't exist until foobar migrated

(All of this assumes the current state, where trixie = testing. The
same would be true, with appropriate name changes, during development
of forky or later.)

That's my understanding of why we try to make unstable as similar as
possible to a hypothetical future state of testing, and ideally
programmatically indistinguishable - unless you specifically go looking
at the apt configuration, at which point you're responsible for looking
at the *whole* apt configuration, and not just trying to derive a single
suite name from it.

A mitigation is that when autopkgtest tests a proposed migration, what
it is actually testing is a hypothetical future state of testing where
we pretend that this one package and its mandatory dependencies have
already migrated, in order to answer the question: if we really allowed
this package to migrate, would it break anything?

So if foobar has good enough autopkgtest coverage, *that* would stop it
migrating - but this is also why it's important that autopkgtests work
with the apt configuration that they're given (as Helmut described in more
detail in )
instead of figuring out whether they're on testing or unstable and
then using that information (and only that information) to construct a
test fixture.

There is a completely valid trade-off to be made about whether the
concern that I've described here (or similar reasons why testing and
unstable should be hard to distinguish) is more or less important than
the reasons why Luca and others want testing and unstable to be easy to
distinguish. That's fine! We make decisions all the time about which
option is better, or about which option is less-bad. But I think it's
an over-simplification to imply that this is a decision with only valid
arguments in one direction, and no valid arguments in the other. When
I talk about seeing both sides of a dispute, this is part of what I mean.

smcv
not a TC member

Bug#1077764: Ruling request on os-release specification implementation

[Simon McVittie]

On Fri, 02 Aug 2024 at 09:07:12 -0700, Russ Allbery wrote:
> Luca Boccassi  writes:
> > It is correct as-is. VERSION_ID is meant to identify a release, not
> > updates or point releases. A release as in, Debian Bookworm, or Fedora
> > 40, or Ubuntu Noble, and so on.
> 
> Why would you not want to identify point releases?  This really surprises
> me.

I think the idea is that two releases have a different VERSION_ID if and
only if they can both be fully up to date, but still remain different. If
we make the analogy of git, VERSION_ID labels a branch, not a tag.

If Debian 12 is like a git branch (and unstable is like git main),
then Debian 12.0 and 12.6 are more like tags on the Debian 12 branch.
Debian 12 is still maintained and changing, but Debian 12.6 was a point
in time, it already happened, and now it's never going to change.

So, 11 vs. 12 vs. 13 are different VERSION_IDs, because a fully updated
Debian 11 is not the same as a fully updated Debian 12 or 13.

But 12.5 vs. 12.6 don't have different VERSION_IDs, because if you upgrade
Debian 12.5, it *becomes* 12.6: there is no separate 12.5.x branch that
anyone is maintaining as something distinct from 12.6.

(It might make sense for there to be a defined field in os-release for
"what's the most recent point release we've caught up with?" *as well*,
but VERSION_ID is not that.)

smcv

Bug#1077764: Ruling request on os-release specification implementation

[Simon McVittie]

On Fri, 02 Aug 2024 at 12:19:20 +0100, Luca Boccassi wrote:
> To further clarify why the status quo with VERSION_CODENAME=trixie in
> sid is really bad: it used to be that if you had "debian" mentioned in
> os-release but no other version identifying fields, you knew you were
> on testing OR unstable and you'd have to deploy horrendous hacks to
> attempt and figure out which of the two it was really.

OK, I think this is progress:

What is the scenario / use-case in which it becomes necessary to
distinguish between those two suites?

To put that another way, what external piece of software needs to
change its behaviour, dependent on whether you are running testing
(of an unspecified datestamp) or unstable (of an unspecified datestamp)?

Or perhaps you are thinking of a scenario in which a *person* needs to
change their behaviour, dependent on whether they are running testing
or unstable?

> Sorry, but there's no other way to define this than a bug.

I would personally characterize it as a potential root cause for bugs,
more than as a bug itself. To me, an example of one of those bugs might
look more like "I run ansible/Puppet/etc. against my test VM, and I
expect it to do a useful thing, but actually it..." (with the buggy result
perhaps being to crash, or to install the wrong package, or something).

smcv

Bug#1077764: Ruling request on os-release specification implementation

[Simon McVittie]

On Fri, 02 Aug 2024 at 11:35:54 +0100, Luca Boccassi wrote:
> VERSION_CODENAME=trixie was added, and the problem as explained is
> that it's present in sid too. So the only identifier we have in sid,
> identifies it as trixie, which is categorically and unequivocally
> wrong.

When involved in a disagreement, please could you try to make an effort
to understand the point of view of the other side, rather than declaring
them to be categorically and unequivocally wrong? I think that would
lead to fewer people feeling that they're being shouted at and refusing
to engage with you, which is likely to result in more of the changes
you want to see actually being adopted.

Dismissing the other side's point of view as "just wrong" is a pattern
that I keep seeing in Debian, from multiple people, and frankly it's
exhausting. Even when I completely agree with you about the technical
content of what you're saying, it makes every disagreement into an
unpleasant interaction that drains my energy and motivation, and I know
that I'm not the only DD who is put off by the way relatively minor
disagreements escalate.

Normally when there's a non-obvious technical decision to be made,
"both sides" have a valid design principle that they're trying to follow,
and the essence of the decision is a value-judgement about which of the
two contradictory design principles is more important than the other
one (better cost/benefit ratio, defining cost as broadly as possible).
These are decisions that we need to make, but they shouldn't be a fight:
sometimes we have to agree to disagree, and sometimes we have to accept
that our reasons to want option A are outweighed by someone else's
reasons to want option B.

In this case, I think the two design principles might be:

- you think of sid as an independent rolling-release distribution in
  its own right, an alternative to e.g. Arch Linux, and so you want to
  label sid images/containers/chroots/etc. in a way that is consistent
  with that point of view;

- Santiago thinks of sid as a tool to be used as part of the process of
  making our next stable release, an alternative to e.g. Fedora Rawhide,
  and wants to label sid images/etc. in a way that is consistent with
  *that* point of view

and each of you is arguing in favour of the metadata that makes most
sense when you start from that point of view?

Of course, the real answer to "is sid a rolling release distribution or
is it a tool for making the next stable?" is "yes", so neither of those
points of view is completely wrong, but neither of them is the whole
story either.

I think that, as a project, we need a lot more willingness to frame
disagreements in terms of "I see your point, but I think my point is
more important", and a lot less "you're just wrong". All of us are
doing our best to make Debian the best possible Free operating system,
even if we don't always agree on precisely what that means.

smcv
not a TC member

Bug#1077764: Ruling request on os-release specification implementation

[Simon McVittie]

On Fri, 02 Aug 2024 at 10:31:29 +0200, Marc Haber wrote:
> On Thu, Aug 01, 2024 at 06:58:09PM +0100, Luca Boccassi wrote:
> > I could echo "ID=windows 3.1" into my local
> > /etc/os-release and nothing would stop me or fix it until the next
> > stable release.
> 
> Not even automatically. /etc/os-release is a conffile, isnt it?

No, it's canonically a symlink to /usr/lib/os-release which is "owned"
by the packaging system (specifically base-files.deb, although Luca's
proposal in this thread involves moving it to another package). It's
technically possible to delete the symlink, replace it with a regular
file and edit the regular file, for example to add a VARIANT or IMAGE_ID
- although I'm not sure how that would interact with upgrades, and in
practice the situation where you'd be most likely to do that would be
an image-based environment where you throw away the image and bootstrap
a new one instead of upgrading.

smcv

Bug#1077764: Ruling request on os-release specification implementation

[Simon McVittie]

On Thu, 01 Aug 2024 at 20:00:40 -0700, Russ Allbery wrote:
> I just know that I've seen a lot of code that uses version
> numbers or code names this way, mostly in things like Puppet rules.  Most
> of the time people will probably get this right, but there are some
> obvious potential mistakes such as coding a condition that says 13 behaves
> the same as 12 (but the eventual release won't) or that 13 always behaves
> in some new way (but testing systems that weren't upgraded to the released
> version 13 don't and instead behave like 12).

I think the only right ways to handle version-dependent conditions
are either feature detection (looking for the feature of interest, e.g.
dpkg --assert-versioned-provides), or looking at the version number of the
(possibly backported) package of interest (e.g. dpkg-query -W systemd),
or if neither of those is feasible, something like this pattern:

if major_version <= 11:
do old things
else if major_version == 12:
do things suitable for bookworm
else:
assume testing/unstable/future and do new things

with more or fewer "else if" depending on the number of major versions
with divergent behaviour that you aim to support. All of those patterns
would work without a VERSION_ID in testing/unstable, or even without a
VERSION_CODENAME.

But, obviously, third-party software doesn't always get this right, and
will sometimes say things like "all Debian systems except testing/unstable
have 32-bit time_t on armhf" - which happens to be true *today*, but will
immediately become false when we release trixie.

> The most mistake-resistant approach would be to give
> testing some *other* code name that isn't trixie, and only give the code
> name of trixie at the time of release, but that's also weirdly different
> from how we use codenames in the archive structure

That also breaks some of the design principles of the
unstable -> testing -> stable flow, which are that testing should be
approximately equivalent to a previous state of unstable (one where
everything worked acceptably together), the next stable should be
functionally equivalent to a late-freeze version of testing, and as
little as possible should change as we cross release milestones, to
avoid perturbing packages' behaviour.

We don't 100% stick to those design principles, because getting all of
unstable into a state where it works acceptably together and is ready
to migrate is not actually feasible most of the time, so instead we
migrate a package or a group of packages at a time - but the design goal
is that there are no surprises when testing is updated, because every
behaviour change has already been seen before, in unstable. So I can
understand why Santiago is reluctant to have testing and unstable be
obviously, identifiably different.

I think the release team needs to be involved in the decision of how
testing and unstable should be labelled, because it's the release team
that is responsible for the flow of packages from unstable into testing
and the flag-day transition from "trixie is testing" to "trixie is stable",
and it's also the release team that would have to deal with the fallout
if packages changed their behaviour (possibly during hard freeze!) when
they saw /etc/os-release change from unstable to trixie.

In other distributions like Fedora and Ubuntu, the branch that is going
to become the next stable (Fedora rawhide, Ubuntu devel) is already
labelled as though it *was* the next stable. The Debian equivalent of
this would be to label testing as Debian 13, trixie, starting as soon
as the branch opens (I often call it something like "a Debian 13 alpha"
when I need to communicate its status to developers outside the Debian
bubble) and I think that would be a reasonable thing to do.

The unusual thing about Debian is that we have not one but two "future"
branches, unstable and testing, together with the unstable -> testing
migration, *and people actually use unstable*. In Fedora, I believe
new packages go directly to rawhide (the Debian equivalent would be not
having unstable, and uploading directly to testing), and the VERSION_ID
in rawhide is the major version that the *next* Fedora release will
have (so, yes, the Debian equivalent would be to have had VERSION_ID=13
starting from the beginning of trixie).

In Ubuntu, they do technically have an equivalent of unstable (they call
it "proposed"), but my understanding is that nobody outside the Ubuntu
infrastructure actually runs it on their computers - the ability to
install it on systems other than buildds isn't advertised. So if it's
mis-labelled, there's no actual practical impact, because it's very
close to their equivalent of testing, and nobody is running third-party
software on it anyway. In terms of how it's used, it's more like Debian's
buildd-unstable branch, used on buildds, QA/test infrastructure and
nowhere else.

The closest equivalent of what Fedora and Ubuntu do would be to label
both testing and unstable as tho

Bug#1077764: Ruling request on os-release specification implementation

[Simon McVittie]

On Thu, 01 Aug 2024 at 16:54:20 +0100, Luca Boccassi wrote:
> The second [objection from the base-files maintainer] is pushing forward a
> philosophical explanation according to which testing and unstable are
> not actually different images, but they are one and the same (two sides
> of the same coin is the expression used). While that might be true in a
> philosophical sense, this is a practical matter, and it concerns only
> the os-release specification and its implementation. In such context,
> testing and unstable are different and independent images, built from
> different and independent archives of packages. For example, at any
> point in time you can do:
> 
> deboostrap testing /tmp/a
> deboostrap unstable /tmp/b
> 
> and if your shell history is lost, you have no reliable, stable,
> distro-agnostic way to identify what exactly /tmp/a and /tmp/b are.

But, let's continue your example: suppose you ran those commands back in
January. Now, 6 months later, run similar commands again:

debootstrap testing /tmp/c
debootstrap unstable /tmp/d

With your proposed labelling of testing and unstable as being fundamentally
different, you would have:

/tmp/a: testing from January, labelled as testing (or trixie or Debian 13)
/tmp/b: unstable from January, labelled as unstable
/tmp/c: testing from July, labelled as testing (or trixie or Debian 13)
/tmp/d: unstable from July, labelled as unstable

But, contrary to that, the differences between a and b will be relatively
small, and so will the differences between c and d; whereas the differences
between a and c will be large (even though they're both labelled as
testing) and so will the differences between b and d.

For example, a and b will have glibc 2.37, but c and d will have 2.39,
with whatever behaviour changes have happened in the last 6 months.
Similarly, neither a nor b has been through the t64 transition, but
both c and d have.

I think the most important fact about all of these chroots is
"it's strictly newer than Debian 12, so expect some behaviour
changes". Precisely how much newer, and precisely which behaviour changes,
is not such a simple question to answer: it depends which package's
behaviour you're interested in, and if you want to answer that question,
the only way is to look at individual packages, so that you can tell
whether they were last updated in January or whether they have been
updated to July's version.

smcv
(not a TC member)

Bug#1065170: tech-ctte: Requesting advice on glib2.0 #1065022, file deletion by postrm during t64 transition

[Simon McVittie]

On Sat, 02 Mar 2024 at 20:19:47 +, Simon McVittie wrote:
> I have proposed a change for glib2.0/experimental at
> https://salsa.debian.org/gnome-team/glib/-/merge_requests/32 which
> implements the "delete the old postrm" approach

An equivalent glib2.0 change is now in unstable.

Unfortunately, GTK 2 and 3 both have an equivalent issue for their input
method modules (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1065493,
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1065494 respectively).
I'm proposing to address this in the same way as for glib2.0 with
https://salsa.debian.org/gnome-team/gtk2/-/merge_requests/3 and
https://salsa.debian.org/gnome-team/gtk3/-/merge_requests/22 respectively
(untested, test-builds are in progress). As with glib2.0, I apologise for
not having caught this at nmudiff time.

gdk-pixbuf and GTK 4 probably have a latent version of the same issue,
but have avoided a practical problem during this particular transition by
not having any time_t in their ABIs and therefore not needing a rename,
so they are presumably not immediately urgent to address.

I have not attempted to audit other libraries that have plugins, either
inside or outside the GNOME team, for an equivalent problem.

smcv

Bug#1065170: tech-ctte: Requesting advice on glib2.0 #1065022, file deletion by postrm during t64 transition

[Simon McVittie]

Copying context from elsewhere in the thread, Sam Hartman wrote:
> Are there solutions in the space of having glib2.0-0 continue to exist
> as a package depended on by glib2.0-0t64 or depending on the new library
> allowing you to replace the postrm?

to which I replied:

If libglib2.0-0 continues to exist, then packages that expect the affected
entry points to have 32-bit time_t will still have their dependencies
satisfied, but then when they call the affected entry points, they will
crash, because their time_t is not the same size as GLib's. So as far
as I can see, this is functionally equivalent to reverting the rename:
to be correct, it would have to be accompanied by versioned Breaks on
every package that calls into the affected entry points.

On Mon, 04 Mar 2024 at 08:46:52 -0700, Sam Hartman wrote:
> > "Matthew" == Matthew Garrett  writes:
> Matthew> I would like some further
> Matthew> analysis of Sam's proposal, though - I don't think there's
> Matthew> any advantage in undoing the existing solution, but if it
> Matthew> would work then it's maybe a more straightforward solution
> Matthew> for any similar issues in future?
> 
> Unfortunately, I think Simon's response to me is definitive.
> Ultimately if the old package exists, it will continue to satisfy
> dependencies.
> That's exactly what we don't want in the time_t transition.

I think the one thing we could do in this direction would be to mitigate
problems like this one *on architectures unaffected by the transition*
(in this case, 64-bit plus i386) by having this situation:

Package: libglib2.0-0
Architecture: amd64 arm64 i386 s390x (etc.)
Depends: libglib2.0-0t64 (>= ${binary:Version})
Description: empty transitional package

Package: libglib2.0-0t64
Architecture: any
Breaks: libglib2.0-0 (<< ${binary:Version})
Replaces: libglib2.0-0 (<< ${binary:Version})
Description: GLib library

I think this would have made upgrades significantly more smooth on the
non-armel, non-armhf architectures, as well - but perhaps I'm missing an
important reason why the 64-bit time_t transition team have chosen not to
do this (and perhaps it's not even possible).

Nothing we do in this direction would help armel and armhf, so there would
still be a problem here to be solved (and probably we would not have found
out about this particular bug until much, much later, when an armel or
armhf GUI user did the upgrade and encountered the bug).

This is obviously only going to help if "most" architectures are
unaffected by the transition. If our next such transition involves
genuinely breaking ABI on the more widely-used architectures, then I
don't see any way to make it less painful. Perhaps this is a sign that
we should try hard to only have transitions shaped like this one if
there is no alternative.

smcv

Bug#1065170: tech-ctte: Requesting advice on glib2.0 #1065022, file deletion by postrm during t64 transition

[Simon McVittie]

On Sat, 02 Mar 2024 at 20:19:47 +, Simon McVittie wrote:
> I have proposed a change for glib2.0/experimental at
> https://salsa.debian.org/gnome-team/glib/-/merge_requests/32 which
> implements the "delete the old postrm" approach.

I've uploaded this to experimental, incorporating feedback from Helmut.
(Thank you for your thorough review!)

Due to the time required to put each version through build/testing,
I am not able to propose a corresponding upload for unstable tonight
(and having this version in experimental for at least a few hours is
probably not a bad plan in any case).

smcv

Bug#1065170: tech-ctte: Requesting advice on glib2.0 #1065022, file deletion by postrm during t64 transition

[Simon McVittie]

On Sat, 02 Mar 2024 at 20:19:50 +, Simon McVittie wrote:
> I am currently downloading all versions of libglib2.0-0 that have
> existed on amd64 as tracked by snapshot.d.o. My plan is to extract their
> DEBIAN/postrm, import them into a git branch and go back through the
> history that way.

https://salsa.debian.org/smcv/glib2.0-control-files has the full history.
`git log -p postrm` confirms that the only things the postrm has ever
done in versions known to snapshot.d.o are:

- deleting gschemas.compiled, which is what we want to avoid here;
- deleting giomodule.cache, which is what we want to avoid here;
- running ldconfig, which is handled by triggers now

So I think removing the postrm is viable.

smcv

Bug#1065170: tech-ctte: Requesting advice on glib2.0 #1065022, file deletion by postrm during t64 transition

[Simon McVittie]

I have proposed a change for glib2.0/experimental at
https://salsa.debian.org/gnome-team/glib/-/merge_requests/32 which
implements the "delete the old postrm" approach. I would appreciate
review and comments on whether this is something that would be acceptable
to upload to experimental, and/or to cherry-pick to the debian/trixie
branch that is currently being used for unstable uploads.

I believe it is ready for review, and it includes semi-automated
tests for #1065022, which do pass for a test-build; but at the time of
writing I am waiting for the systematic testing that is appropriate for
a production-quality upload, because it takes a long time (around 1 to
2 hours of sbuild, autopkgtest and piuparts in the case of glib2.0). I
hope that I am not wasting reviewers' time by asking for review of
only-partially-tested changes.

As well as the "delete the old postrm" part, the proposed change also
includes an attempt to prevent this situation from happening again, by
changing the postrm logic so the cleanup is not done if a future package
like libglib2.0-0xyz takes over /usr/lib/*/glib-2.0.

Because my upload pipeline is time-consuming and I only get a finite
number of attempts in a day, I would prefer it if reviewers could make
it clear whether any deficiencies are considered to be experimental
upload blockers, unstable upload blockers, or merely nice-to-have. If
only nice-to-have changes are requested, then I will not necessarily
restart the release process for them.

If I do not receive any positive or negative feedback by the time the
build finishes, I will probably upload it to DELAYED in an attempt to take
myself off the critical path, and I might also prepare a DELAYED upload
for unstable. This is an attempt to balance the two competing factors
that result in there being no acceptable thing to do in this situation:
because this is a critical bug that will eventually become critical-path
for a project-wide transition, the expectation is that I must upload a
fix as soon as possible, but because maintainer scripts run as root on
hundreds of thousands of systems, the expectation is that I must not
upload without sufficient testing. So, my intention is to do my best,
and then invite other developers to take responsibility for a better,
higher-version-numbered upload with different timing if they prefer.

On Fri, 01 Mar 2024 at 17:44:57 +, Simon McVittie wrote:
> Perhaps giomodules.cache should *also* only be removed
> during purge, but fixing that has never been anyone's highest priority.

That is also implemented in
https://salsa.debian.org/gnome-team/glib/-/merge_requests/32.

> On Fri, 01 Mar 2024 at 15:58:10 +0100, Helmut Grohne wrote:
> > I'm not sure anyone of us wants to look into multiple old
> > libglib2.0-0.postrm scripts
> 
> If my choice is between spending O(hours) on reading old postrm scripts
> or O(days) on NMUing GLib-dependent packages, then I'd prefer to read
> the postrm scripts. I can't say that either is something I would look
> forward to, but if it's what the project requires from me then it will
> have to happen.

I am currently downloading all versions of libglib2.0-0 that have
existed on amd64 as tracked by snapshot.d.o. My plan is to extract their
DEBIAN/postrm, import them into a git branch and go back through the
history that way. I am doing this in parallel with building a release
candidate for experimental rather than doing this analysis first, because
as stated above my release pipeline is very long, and I am optimistically
assuming that the code added by debhelper when it replaces the #DEBHELPER#
token will not be functionally significant. If that assumption is wrong
then I will presumably need to to start again and rethink my approach.

smcv

Bug#1065170: tech-ctte: Requesting advice on glib2.0 #1065022, file deletion by postrm during t64 transition

[Simon McVittie]

On Fri, 01 Mar 2024 at 13:21:51 +0100, Christoph Berg wrote:
> > Possible solution: other ideas?
> 
> Make glib2.0-t64 use a different cache filename?

I'm not happy about the idea of introducing long-term divergence in
the upstream code as a result of a Debian-specific packaging problem;
I think we should be going in the direction of fewer, not more,
"weird Debianisms" that break reasonable upstream assumptions.

If there was a finite time window after which we could revert this, that
would be less bad, but as far as I can see it would be possible to keep
libglib2.0-0 installed for multiple release cycles and then purge it, and
it would have just as much impact whenever it happens to get purged.

On Fri, 01 Mar 2024 at 15:58:10 +0100, Helmut Grohne wrote:
> For this one (but not gschemas.compiled), I am wondering whether
> installing a trigger-interest in /usr/share/doc/libglib2.0-0/copyright
> might work. That is a file that is going away and therefore, removal of
> libglib2.0-0 should cause libglib2.0-0t64.postinst to be invoked with a
> triggered argument after libglib2.0-0.postrm having done the damage.
> 
> Do you agree that this would solve giomodule.cache but not
> gschemas.compiled?

Probably? Sorry, I don't have a sufficiently in-depth understanding of
the finer points of triggers to know whether it is guaranteed that the
trigger would go off after libglib2.0-0.postrm has already run.

(Also, as said elsewhere, sysadmins are allowed to remove/exclude
/usr/share/doc, so I wouldn't want to make that load-bearing.)

> Of course, not solving the other one makes this somewhat theoretical.

Right; the schemas problem is the more serious one anyway. So I don't
think it's necessarily worth spending time on bringing my understanding
of triggers up to a point where I would be able to answer your previous
question.

> > - for gschemas.compiled (shared by all architectures), if the multiarch
> >   refcount of the library reaches 0, then the file is deleted during the
> >   next postrm purge
> 
> Do you happen to remember why gschemas.compiled is being removed in
> purge rather than remove?

I don't remember, as such (it was before my involvement) but thankfully
we have version control to answer questions like this for us. According
to the git history, it was originally removed in remove, but that caused
breakage during upgrades (which run the old postrm and then, eventually,
the new postinst): there was a window of time in which the file did
not exist.

For GIO modules, having a window of breakage is not great but not a
disaster, because "most" GLib programs are at least partially functional
without them. Perhaps giomodules.cache should *also* only be removed
during purge, but fixing that has never been anyone's highest priority.

For GSettings schemas, missing schemas are usually treated as a
programming error (assertion failure and hard crash) which caused more
extensive brokenness:

debian/libglib2.0-0.postrm.in: Only remove the compiled schemas on purge,
not during upgrades. Otherwise we have no schemas available until the new
postinst is run, which leads to applications aborting on missing schemas.
— c8b28bc30b4a5ca7ec7bf643ebaeae88bf21c588, 2012-03-30

> > libglib2.0-0t64 could gain a preinst that deletes
> > /var/lib/dpkg/info/libglib2.0-0:${DEB_HOST_ARCH}.postrm
> 
> Consider a variant of this. We don't really know how old libglib2.0-0
> is

The postrm was introduced in 2010 and hasn't changed a whole lot (14
commits between the beginning of time and current experimental, according
to git log -p --follow debian/libglib2.0-0t64.postrm in the experimental
packaging).

> and I'm not sure anyone of us wants to look into multiple old
> libglib2.0-0.postrm scripts

We don't support skipping an upgrade, so probably only the one in bookworm
(and Ubuntu 22.04 I suppose) is relevant here?

If my choice is between spending O(hours) on reading old postrm scripts
or O(days) on NMUing GLib-dependent packages, then I'd prefer to read
the postrm scripts. I can't say that either is something I would look
forward to, but if it's what the project requires from me then it will
have to happen.

> Both of my thoughts leave a window of time where users of these caches
> will fail.

Yes, and in the case of GSettings schemas that does seem particularly
high-impact.

> > Possible solution: revert t64 rename for glib2.0
> 
> Do you see this as a realistic solution for trixie?

Which is "this"?

If you mean reverting the rename and adding Breaks, then yes, it's a
simple matter of doing an extensive amount of of routine work (rename
back, no-changes-NMU all packages that mention those symbols if necessary
so that they will have a higher version number, and add versioned Breaks).
I can't say that I'm looking forward to that work, particularly if the
project expects me to do all of it personally, but it's routine.
I don't know how it would fit into Ubuntu's schedules and deadlines, but
if the alternative is "most G

Bug#1065170: tech-ctte: Requesting advice on glib2.0 #1065022, file deletion by postrm during t64 transition

[Simon McVittie]

Package: tech-ctte
Severity: normal
X-Debbugs-Cc: debian-d...@lists.debian.org, debian-gtk-gn...@lists.debian.org, 
vor...@debian.org

I'm requesting advice from the tech-ctte (or anyone else with relevant
knowledge, e.g. the dpkg team or the drivers of the time64 transition)
on how to resolve glib2.0 bug #1065022. This is time-sensitive,
because it is a RC bug (temporarily breaking many applications across
this transition) and will hold up the time64 transition.

Background
==

glib2.0 has two similar patterns where files that are managed by dpkg
are summarized in a non-dpkg-managed file, maintained by triggers and the
library postinst/postrm.

The first of these patterns is GSettings schemas. There is a tool that
loads GSettings schemas in XML format from /usr/share/glib-2.0/schemas
and aggregates them into a single binary blob in a more efficient format,
/usr/share/glib-2.0/schemas/gschemas.compiled. For performance reasons,
applications only load gschemas.compiled: there is no support for loading
the more authorable but less efficient XML files directly.

The second of these patterns is GIO modules, a plugin architecture which
loads .so files from /usr/lib/${DEB_HOST_MULTIARCH}/gio/modules
and summarizes their functionality
in /usr/lib/${DEB_HOST_MULTIARCH}/gio/modules/giomodule.cache.
Applications that want to load plugins parse giomodule.cache, and only
dlopen the plugins that provide the desired functionality (for example
an application that doesn't do any networking will not load plugins that
only implement gio-proxy-resolver).

This is implemented with dpkg file-based triggers: when a package adds
or removes GSettings schemas or GIO modules, it triggers processing by
the libglib2.0-0{,t64} postinst. The implementation has been approximately
the same shape for 10 years, and has worked well until now.

Because dpkg doesn't have an equivalent of RPM %ghost files, the two
generated summary files need to be deleted by the library's postrm.
As of bookworm (and still true in trixie), the implementation is:

- for giomodule.cache (per-architecture), the file is simply deleted by
  postrm remove

- for gschemas.compiled (shared by all architectures), if the multiarch
  refcount of the library reaches 0, then the file is deleted during the
  next postrm purge

The bug
===

When we transition from libglib2.0-0 to libglib2.0-0t64, this involves
the removal of libglib2.0-0. In the postrm of libglib2.0-0, removing
libglib2.0-0:amd64 deletes
/usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache, and so on for
all the other architectures.

The result is that until the postinst of libglib2.0-0t64:amd64 is run,
amd64 applications will be unable to load GIO plugins, causing
functionality loss (for example, inability to use https, because the
TLS plugin is not loaded).

Similarly, either during or after the transition from libglib2.0-0
to libglib2.0-0t64, users will want to purge libglib2.0-0. In
the postrm of libglib2.0-0, if there are no multiarch
instances of libglib2.0-0 remaining, purging the package deletes
/usr/share/glib-2.0/schemas/gschemas.compiled. The result is that
applications that want to load GSettings schemas will not find their
required schemas, which is normally treated as a programming error
(incorrect installation) that causes a crash with an assertion failure.

The workaround is: after removal or purging of libglib2.0-0, reinstall
either libglib2.0-0t64 or any package that will trigger libglib2.0-0t64.
On multiarch systems, this must be done for the architecture that matches
the instance of libglib2.0-0 that was removed.

During upgrade, I am unsure what ordering guarantees we have about
the postrm of libglib2.0-0 running before or after the postinst of
libglib2.0-0t64 - perhaps we avoid the giomodule.cache bug in practice,
because the postrm runs before the postinst? But purge can happen at any
later time, so we certainly cannot guarantee that libglib2.0-0t64.postinst
will run after purging libglib2.0-0.

I apologise for not having foreseen this.

Non-solutions
=

I am not interested in solutions that would require a use of a time
machine to change the postrm that was shipped in bookworm: bookworm was
already released, and now we are stuck with it. *After* the time-sensitive
part of this issue has been solved, I plan to look into making the postrm
robust against future transitions similar to this one by adding some way
for the new package to take over responsibility for giomodule.cache and
gschemas.compiled, but for this particular transition it's too late: the
first time at which we could rely on that functionality is trixie -> forky.

I am also not interested in solutions that require design changes in GLib,
for example adding a fallback slow-path that ignores the absence of the
summary files and loads the individual GSettings schemas and GIO modules
directly. This is because upstream would not accept such a change, and it
would introduce significant delta into Debian,

Re: /usr-move: Do we support upgrades without apt?

[Simon McVittie]

On Thu, 21 Dec 2023 at 15:31:55 +0100, Marc Haber wrote:
> Do those GUI frontends that work via packagekit or other frameworks
> count as "using apt"?

Managing apt/dpkg packages via packagekit uses libapt-pkg6.0 (via
/usr/lib/*/packagekit-backend/libpk_backend_apt.so). I don't know whether
that's enough to give it the specific desirable behaviour around Conflicts
that Helmut is referring to, but I hope it is.

Other non-CLI package management like unattended-upgrades is generally
in a similar situation, using libapt-pkg or its language bindings,
but not the apt(8) or apt-get(8) CLIs specifically.

smcv

Bug#1053901: tech-ctte: Repeal merged-/usr file movement moratorium

[Simon McVittie]

On Fri, 13 Oct 2023 at 21:59:56 +0100, Sean Whitton wrote:
> OPTION A:
> 
> The Technical Committee formally repeals its moratorium recommending
> that maintainers of individual packages should not proactively move
> files from the root filesystem to corresponding locations under /usr in
> the data.tar.* of packages (see #1035831).
> 
> Under Constitution 6.1.5, the Technical Committee now recommends that
> maintainers consult with those driving the merged-/usr transition before
> moving files from the root filesystem to corresponding locations under
> /usr in the data.tar.* of packages.
> 
> The transition driver, which at the time of writing is Helmut Grohne, is
> using a phased approach, in which the moratorium is rolled back for only
> certain classes of packages, and changes, at a time.  In addition,
> restructuring uploads should be targeted at experimental, and left for
> three days.  This is in order that automated testing by dumat can occur.
> 
> We recommend following , as edited by
> the transition driver(s).  If there is any doubt as to whether a change
> you wish to make is appropriate, please seek explicit approval from the
> transition driver(s).
> 
> OPTION N:
> 
> None of the above.

I vote A > N.

Thanks,
smcv


signature.asc
Description: PGP signature

Bug#1052697: tech-ctte: proposed change to apt-listchanges algorithm needs expert consideration

[Simon McVittie]

On Tue, 26 Sep 2023 at 06:11:01 -0400, Jonathan Kamens wrote:
> This approach was based on two assumptions, neither of which is always
> true:
> 
> 1. Assume that the version numbering for all packages that come from
>the same source package are in the same series.
> 2. Assume that the version numbering of entries always matches the
>aforementioned version numbering.
> 
> For an example of where these assumptions break down, look at the
> dmsetup package:
> 
> - The source package for dmsetup is lvm2.
> - The version number for the dmsetup package is lower, but with a
>   higher epoch than, the version number for the lvm2 package.

It is possible to ask dpkg for the name *and version* of the source
package from which a binary package was built. For example:

$ dpkg-query -W dbus
dbus1.14.10-1
$ dpkg-query -W -f '${Source}\n' dbus

$ dpkg-query -W dbus-daemon
dbus-daemon 1.14.10-1
$ dpkg-query -W -f '${Source}\n' dbus-daemon
dbus
$ dpkg-query -W dmsetup
dmsetup 2:1.02.185-2
$ dpkg-query -W -f '${Source}\n' dmsetup
lvm2 (2.03.16-2)

The source package as defaults to the binary package and the source
version defaults to the binary version, so in this example we have:

- binary: dbus_1.14.10-1 -> source: dbus_1.14.10-1
- binary: dbus-daemon_1.14.10-1 -> source: dbus_1.14.10-1
- binary: dmsetup_2:1.02.185-2 -> source: lvm2_2.03.16-2

If we disregard non-standardized files like changelog.Debian.devmapper.gz,
I think we can model changelog entries as being keyed by (source package,
source version) pairs.

(One exception to that: binNMUs generate a special binNMU changelog entry,
which doesn't correspond to a source package as-is, but apt-listchanges
would presumably want to behave as though that source package had existed
for the purposes of changelog entry display.)

smcv

Re: Bug#1050001: Unwinding directory aliasing

[Simon McVittie]

I would very much prefer it if we can keep this conversation respectful,
because mediating between angry people is exhausting (and the more
spoons I have to put into that, the less effectively I can advocate for
the technical opinions that I think you and I mostly agree on, which I
assume is not what you want).

I'm keen to avoid the anti-pattern where a valid technical point gets
disregarded or even opposed because the way it was expressed puts other
project members on the defensive.

If you're frustrated, I understand that, but please can we take that
off-list?

On Wed, 23 Aug 2023 at 10:35:42 +0100, Luca Boccassi wrote:
> Suse [...] tried [symlink farms similar to those that Ian proposes]
> for a few years, then had to backtrack and
> go the way everyone else successfully went instead, which quickly wrapped up 
> as
> expected.

I alluded to that in a previous mail to this thread, but I don't have
first-hand knowledge of the specifics of how that went, and it sounds
as though you might. Do you have references that you can point us to?
(To the list or as private email for me to summarize on-list later,
whichever seems more appropriate.

Thanks,
smcv

Bug#1050001: Unwinding directory aliasing

[Simon McVittie]

On Wed, 23 Aug 2023 at 17:04:36 +0100, Ian Jackson wrote:
> Simon McVittie writes ("Bug#1050001: Unwinding directory aliasing"):
> > What do you consider to be the end goal of this proposal?
> 
> My idea of a desired end state is as follows:
> 
> /bin and /lib etc. remain directories (so there is no aliasing).  All
> actual files are shipped in /usr.  / contains compatibility symlinks
> pointing into /usr, for those files/APIs/programs where this is needed
> (which is far from all of them).  Eventualloy, over time, the set of
> compatibility links is reduced to a mere handful.

This is not merged-/usr with the meaning used by the technical committee's
past resolutions, and by most (all?) non-Debian distributions (among
which Fedora and Arch were among prominent early adopters).

I recognise that you don't want merged-/usr, and instead you want
this non-merged-/usr layout, which shares some of the properties of
merged-/usr; but it isn't the same thing, and it makes discussion
and reasoning unnecessarily difficult if we use the same name for two
different things, so please could you avoid the term "merged /usr"
for this?

> I think this is a more desirable situation than the current planned
> end state, which is that /bin and /lib are symlinks.

The meaning ascribed to "merged /usr" or "the /usr merge" by previous
TC resolutions is exactly the layout where /bin and /lib (and so on)
are symlinks. Outside Debian, that's also the layout described in
documents like "The Case for the /usr Merge".

I acknowledge that, whatever we choose to call it, you would prefer not
to end at that state, and this is a point on which our opinions differ.

> The current plan, as I understand it, is that we will fix these
> problems by arranging to *always* name files by their canonical paths,
> ie the ones in /usr.

Using the word "canonical" is not necessarily helpful here, because
there are two reasonable-but-contradictory things you might mean by
it. Depending who you ask, the canonical path of /bin/sh on a system
with some sort of unified /usr might be:

* /usr/bin/sh, because that's the physical path as returned by realpath()
  (I believe this is what you mean when you say "canonical", because
  you're thinking in terms of the path canonicalization operation done
  by realpath() and similar things);

* or /bin/sh, because that's the interoperable path that has worked on
  all Linux distributions since time immemorial (even though this might
  not have anything to do with the physical path)

May I suggest we avoid saying "canonical" as ambiguous, and use something
like "physical path" and "traditional path" for these two concepts?

If by "name files" you mean references to filenames from elsewhere, then
that is not the plan as I understand it (see below).

If by "name files" you mean "name files in dpkg's database", then yes,
I believe the current plan is that we end up with dpkg's idea of the list
of installed files referring to every file by its physical path, so that
for example the dpkg database contains the physical path /usr/bin/sh,
even though the traditional path is /bin/sh. Another way to express
this is that if you install a Debian chroot and pack it into an archive
in the obvious way, what you should get (in the absence of any uses of
dpkg-divert, etc.) is the union of the data.tar of all the installed
packages, plus additional non-dpkg-managed files like /etc/passwd.

> Naming files by their canonical names will have to be done everywhere.

I would dispute that. We routinely name system-critical files by
non-physical paths - for example /bin/sh is really /{usr/,}bin/dash,
and /lib64/ld-linux-x86-64.so.2 is really
/{usr/,}lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 - and have done so
for a long time.

> violations of the "use only [physical] names" rule are not only
> expected, they are *necessary*:

Right, and that's why that is not a rule we are following.

One of the reasons that merged-/usr appeals to me personally is
that it takes an entire equivalence class of bugs, and turns them
into non-bugs. If merged-/usr is ubiquitous, then we don't need to
expect third-party software developers to "just know" that /bin/sh and
/usr/bin/perl are the traditionally "correct" paths, because /usr/bin/sh
and /bin/perl become equally valid and interoperable things to put at
the beginning of a script.

In the state we had before bookworm, where merged-/usr was supported but
not mandatory, we required Debian maintainers to be careful to refer to
files by their traditional name, even though on a newly-installed Debian
system with merged-/usr, the "other" name would have worked equally well;
and we were also implicitly expecting upstream and thi

Re: Bug#1050001: Unwinding directory aliasing

[Simon McVittie]

On Sun, 20 Aug 2023 at 19:26:34 +0200, Adam Borowski wrote:
> On Fri, Aug 18, 2023 at 02:39:48PM +0100, Simon McVittie wrote:
> > we would still potentially have other issues triggered by
> > the directories being distinct from one another (like the one discussed
> > by the tech committee in #911225, which was exactly a regression caused
> > by having moved a library in the traditional Debian way).
> 
> Such a move would be 100% safe as dpkg has full knowledge of the filesystem
> layout.  The regression in #911225 applies only to usrmerge-aliased layout.

No, #911225 only affected *non*-aliased systems. It happened before the
usrmerge package was widely used, and appears to have been triggered by
moving GLib from /lib into /usr/lib in the way proposed in #1050001.

On systems where /lib and /usr/lib were aliased, it would have been a
non-issue, because ldconfig would compare the two library versions (with
strverscmp() or equivalent) in the single aliased directory, and make
the SONAME symlink point to the newer version number. (Probably it would
redundantly do that twice, once for the alias and once for the canonical
directory; but that's harmless, as long as it gets the right answer both
times, which it should, because the version comparison is deterministic.)

On systems where those directories were not aliased, ldconfig created
a SONAME symlink in each of /lib/MULTIARCH and /usr/lib/MULTIARCH
separately, and then stored the one in /lib/MULTIARCH in ld.so.cache (even
though it pointed to an older library) because /lib/MULTIARCH appears in
ld.so.conf.d with a higher precedence than /usr/lib/MULTIARCH, leading to
unintended use of an older library version.

We were never able to figure out why dpkg's full knowledge of the
filesystem layout didn't ensure that the older GLib was removed from
/lib/MULTIARCH on the bug reporters' systems, the same way it was
correctly removed for the majority of Debian users, but the absence of
aliasing in older releases clearly didn't protect us from that failure
mode.

smcv

Bug#1050001: Unwinding directory aliasing

[Simon McVittie]

On Fri, 18 Aug 2023 at 07:57:14 +0100, Ian Jackson wrote:
> I think we can probably fix it by backing out this change, and then
> doing usrmerge the traditional Debian way by making changes to
> debhelper, so that we move the files package by package, in the .debs.

What do you consider to be the end goal of this proposal?

If I understand your proposal correctly, it is to return all Debian
systems to their traditional (let's say circa 2010) layout, and then
restart the transition differently. Imagine for a moment that the
usrmerge package, debootstrap --merged-usr, etc. had never existed,
and all Debian systems were in their 2010 state (or equivalently, that
your proposed revert has happened and we are now ready for the second
stage of your plan).

If we carry out this transition package-by-package without central
coordination ("the traditional Debian way"), it seems to me that the
best we can achieve is for /bin, /sbin, /lib* to be symlink farms,
consisting of symlinks that are either owned by the same package that
owns the symlink target, or are unowned from dpkg's perspective and are
created by maintainer scripts.

Here's a simplified view of the resulting system:

/bin/sh -> /usr/bin/sh
/bin/rm -> /usr/bin/rm
/usr/bin/env (regular file)
/usr/bin/sh (regular file)
/usr/bin/perl (regular file)
/usr/bin/rm (regular file)

(Obviously in real life, /bin and /usr/bin are a lot larger than that,
and /sbin and /lib* are also necessary for a bootable system, but
considering a small number of /{usr/,}bin files is enough to illustrate
my point.)

This does some but not all of what merged-/usr does: calling /usr/bin/sh
would become a non-bug, but calling /bin/env would still be an error,
/bin would still represent non-trivial on-disk and/or in-dpkg-database
state, and we would still potentially have other issues triggered by
the directories being distinct from one another (like the one discussed
by the tech committee in #911225, which was exactly a regression caused
by having moved a library in the traditional Debian way).

The merged-/usr version of that same simplified system is this:

/bin -> usr/bin
/usr/bin/env (regular file)
/usr/bin/sh (regular file)
/usr/bin/perl (regular file)
/usr/bin/rm (regular file)

(Again, in real life we also have to consider /sbin and /lib*, but
showing /bin is enough to illustrate the difference.)

As was discussed in 2019 and repeatedly since then, some of the reasons
to want merged-/usr are only available with that layout, and are not
provided by the symlink farm.

If I remember correctly, openSUSE tried to get from unmerged /usr to
merged /usr by essentially the route you propose, successfully reached
the symlink-farm state, but then got stuck without a way to get from the
symlink farm to the single symbolic link. Do you have a plan for how that
would be achieved without breaking upgrades or going behind dpkg's back?

If your plan was a longer, slower, arguably more robust way to achieve
the same end goal, that would be more appealing than a longer, slower,
arguably more robust way to get halfway to the same end goal and then
become unable to go further.

smcv

Re: debootstrap: bad NMU produces buildds not supported by dpkg _and_ CTTE

[Simon McVittie]

On Sun, 16 Jul 2023 at 12:42:11 +0100, Luca Boccassi wrote:
> In the meanwhile, I'll immediately revert the sabotage.

Both of you, please don't turn this into an NMU war in the archive:
that doesn't benefit anyone. I would have preferred it if Adam had not
immediately uploaded a 0-day revert, but preserving the status quo from
bookworm is not the worst state to be in while we discuss this.

If Adam's concerns about this change are valid, then we should address
them, either by doing something different in debootstrap or by reporting
bugs against affected packages.

If Adam's concerns about this change turn out to be unfounded, then we
should reinstate my change (as NMU'd by Luca) in a calm and considered
way, and all we will have lost is a few days of progress and a few bytes
of changelog.

smcv
(speaking as an individual developer, not on behalf of the TC, but I
hope this would be TC consensus if we had had a chance to discuss it)

Re: Bug#1041207: debootstrap: bad NMU produces buildds not supported by dpkg _and_ CTTE

[Simon McVittie]

On Sat, 15 Jul 2023 at 18:27:24 +0200, Adam Borowski wrote:
> bluca's NMU on 2023-07-15 makes debootstrap produce chroots using the
> aliased-dirs scheme.

My intention in the MR that was included in the NMU[1] was to default to
merged-/usr chroots in all cases for trixie and up, but continue to
produce unmerged-/usr chroots for bookworm. When I tested the MR, that's
the result I got[2]. Have you observed something different?

There was consensus among the TC[3] that this change was appropriate for
trixie/sid at this time.

To the best of my knowledge, the official Debian buildds run on stable
(possibly oldstable in some cases) and use debootstrap from there, so any
changes here would not affect official buildds until/unless they are
backported into (old)stable point releases via -proposed-updates.

smcv

[1] https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/93
[2] 
https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/93#note_410656
[3] 
http://meetbot.debian.net/debian-ctte/2023/debian-ctte.2023-07-11-17.58.log.html
18:39 to 18:50 in the timestamps' time zone

Re: Bug#1041207: debootstrap: bad NMU produces buildds not supported by dpkg _and_ CTTE

[Simon McVittie]

On Sat, 15 Jul 2023 at 18:27:24 +0200, Adam Borowski wrote:
> But, what matters here is the CTTE ruling in #1035831 -- for the time being,
> packages must not move files between locations affected by the aliasing.

If that happens in reality, then yes, that's bad, and reverting the change
is a mitigation. What packages have this behaviour?

We are going to need to bring back this change relatively early in the
trixie cycle in any case, for the reasons given in the commit message.
I have not yet analyzed whether we need this change before we can lift
the moratorium on file moves, but I suspect we might.

> Packages built in an usrmerged chroot place such files under /usr while
> built without usrmerge into whatever place they were installed to -- which
> is a direct breach of the ruling.

Do you have examples of packages that differ in this way when built in
a merged- or unmerged-/usr environment? I think we should treat this
as a RC-for-trixie bug in those packages (and in fact I would have been
tempted to call it RC for bookworm as well, again for the reasons given
by the TC, even though during the trixie cycle it was mitigated by using
unmerged-/usr fro buildds).

During most of the bookworm cycle, https://reproducible-builds.org/ has
been doing "build1" in unmerged-/usr and "build2" in merged-/usr, with
differences tracked in

(that list is not necessarily complete, there can also be unidentified
differences in
).

I did some bug-reporting for this during the bookworm cycle and I don't
remember reporting any bugs where a package changed whether it installed
/bin/foo or /usr/bin/foo (or sbin or lib* equivalents) according to
whether it was built in a merged-/usr chroot or not: the bugs I was
reporting were generally about file contents, not the file list.

Most of the remaining merged- vs. unmerged-/usr differences
have been packages that install an Autotools-generated
Makefile into /usr/share/doc/PACKAGE/examples, like for example
 which says "EGREP = /bin/grep -E"
when built in an unmerged-/usr chroot, but /usr/bin/grep in a merged-/usr
chroot. That's annoying but not really a serious issue, because it's only
an example file, and in my opinion should not be RC (but I would consider
it potentially RC if the path was in a functionally-necessary file).

If a package installs in the typical multiple-binary-package way:

- make
- make install DESTDIR=$(pwd)/debian/tmp
- dh_install splits up debian/tmp using debian/*.install
- dh_builddeb

then a mismatch between the expected /bin/foo and a new /usr/bin/foo
will generally cause FTBFS rather than a misbuilt package, because
dh_install will fail to find /bin/foo. So I think the highest risks
for this failure mode are going to be packages that use the simplified
single-binary-package code path:

- make
- make install DESTDIR=$(pwd)/debian/my-package
- dh_builddeb

and packages that do not use debhelper in the conventional way.

smcv

Bug#1037563: tech-ctte: Call for votes on TC membership of Timo Röhling

[Simon McVittie]

On Wed, 14 Jun 2023 at 10:04:54 +0100, Sean Whitton wrote:
> ===BEGIN
> The Technical Committee recommends that Timo Röhling  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> R: Recommend to appoint Timo Röhling 
> F: Further discussion
> ===END

I vote R > F.


signature.asc
Description: PGP signature

Bug#1037562: tech-ctte: Call for votes on TC membership of Stefano Rivera

[Simon McVittie]

On Wed, 14 Jun 2023 at 10:03:19 +0100, Sean Whitton wrote:
> ===BEGIN
> The Technical Committee recommends that Stefano Rivera  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> R: Recommend to appoint Stefano Rivera 
> F: Further discussion
> ===END

I vote R > F.

smcv


signature.asc
Description: PGP signature

Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> This sounds like a very interesting use case, and the first real one
> mentioned, which is great to see - but I do not fully follow yet, from
> what you are saying it seems to me that what you need is for your
> binaries to use the usual pt_interp, that bit is clear. But why does
> it matter if /usr/bin/ls on the host uses a different one?

We don't need to run the ls from the host, but we do need to run
glibc-related executables like ldconfig and localedef from either the
host or the container runtime, whichever is newer. Because glibc is
a single source package, executables and libraries within the glibc
bubble sometimes make use of private symbols in libraries that are also
within the glibc bubble (and IMO they have a right to do so), even though
executables from outside glibc would be discouraged or disallowed from
doing so. This means that when we have chosen a particular version of
glibc (which, again, must be whichever one is newer), we try to use its
matching version for *everything* originating in the glibc source package.

In principle we could get exactly the same situation if we've imported a
library from the host system (as a dependency of the graphics stack) that
calls an executable as a subprocess and expects it to be >= the version
it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

The wider point than my specific use-case, though, is that when there's a
standard, you can't predict what other software authors have looked at the
statement "you can rely on this" and ... relied on it. See also Russ's
(as ever, excellent) mails to the same thread.

I appreciate that you are trying to explore the edges of the
problem/constraint space and say "what if we did this, could that work?",
and it's good that you are doing that; but part of that process is
working with the other people on this list when they say "no, we can't
do that because...", and respecting their input.

Thanks,
smcv

Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Mon, 15 May 2023 at 06:48:04 +0200, Johannes Schauer Marin Rodrigues wrote:
> Obviously, with Luca's proposal, binaries from packages built with a different
> dynamic linker path in them would not work on distributions without 
> merged-/usr
> symlinks. But if the property of stuff from Debian being able to run on
> non-Debian non-merged-/usr systems is an important one, then why was it okay 
> to
> have merged-/usr as the default? Because with merged-/usr we already changed
> the interface contract for a lot of things because now binaries and libraries
> can also be found at other locations than on non-merged-/usr systems. A script
> with a /usr/bin/bash shebang built on and for Debian will not work on a system
> without the symlinks.

pre-bookworm gcc writes /lib64/ld-linux-x86-64.so.2 into the ELF header
of amd64 binaries and I think post-bookworm gcc should continue to do so
even though that has never been the physical path to the ELF interpreter
on Debian (it's really /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, or
historically a version-numbered path). People who are only testing on
Debian systems, even in pre-merged-/usr releases like Debian 9, could
already have been relying on /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
existing; and now that we're saying merged-/usr is mandatory,
people who are only testing on Debian >= 12 could also start
relying on /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 or
/usr/lib64/ld-linux-x86-64.so.2; but we arrange for both/all paths
to exist, and we advise developers that they should only rely on the
interoperable /lib64/ld-linux-x86-64.so.2, treating all other paths that
lead to the same binary as an implementation detail.

I don't think it's any different to say that both /usr/bin/sh and /bin/sh
exist and will work, but ask that everyone should continue to use /bin/sh
and treat /usr/bin/sh as an implementation detail.

The way I think about merged /usr is a variant of Postel's principle:
be conservative in what you require, but be liberal about what will work.
A merged-/usr system can run software that assumes merged-/usr, and can
also run software that doesn't (because of the compatibility symlinks);
a non-merged-/usr system can run software that doesn't assume merged-/usr,
but software that makes that assumption will sometimes fail to work on it.

Consistent with that, despite being in favour of merged-/usr myself,
I didn't switch my development laptop or my autopkgtest VM images
to the merged-/usr setup until it became effectively mandatory
during the bookworm cycle - because if a package was doing something
not-maximally-portable, I wanted my development laptop or my test VM to
be one of the places it would fail to work, so that I would notice and
report that bug.

Conversely, I *did* switch non-development computers (servers and family
laptops) to merged-/usr, because on those systems the important thing is
for software to work, even if in theory it "doesn't deserve" to work.

On post-bookworm, merged-/usr Debian systems, if we keep using #!/bin/sh
and #!/usr/bin/perl scripts, and avoiding #!/usr/bin/sh and #!/bin/perl
scripts, then I think that's a positive thing for cross-distro interop -
and using the interoperable path to the ELF interpreter (dynamic linker)
is completely consistent with that.

As far as I can see, post-bookworm Lintian will continue to warn about
the non-interoperable spellings like #!/usr/bin/sh and #!/bin/perl,
unless changed to special-case them.

Note that the paths that are canonical from the point of view
of cross-distro interoperability (like #!/bin/sh) are not always
the same as the paths that are canonical from the point of view of
realpath(). *At the moment*, they are usually canonical from dpkg's
point of view, but that won't be the case forever, and I think that's
fine. /lib64/ld-linux-x86-64.so.2 isn't considered to be canonical by
either realpath() or dpkg either (dpkg knows it's a symlink, even on
non-merged-/usr systems where /lib64 is a real directory), but it's
canonical for the x86_64-linux-gnu ABI and that's what we say is most
important.

> With merged-/usr we already *did* "change fundamental things around" for
> reasons that are really not clear to me (but which i do not want to discuss
> here) and as a result did not "care about interoperability" (with those who do
> not also adopt it).

Didn't we? With merged /usr, the "theoretically correct" #!/bin/sh
scripts that have always worked will continue to work, and additionally,
"incorrect" #!/usr/bin/sh scripts will *also* work. That sounds like
caring about interoperability to me!

The one piece of interop we lose with merged-/usr becoming mandatory is
that if a developer has only tested on Debian (and other merged-/usr distros
like Fedora), if they have used a less-interoperable spelling of the path
like #!/usr/bin/sh or #!/bin/perl, their testing will not highlight that
source of potential non-interop with others.

However, I don't think it's credible to claim 

Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.

*raises hand*

Hello, I represent an example of those people. With my $work hat on,
I'm involved in maintaining a family of Debian derivatives (the Steam
Runtime) that is used to run Steam games on arbitrary distributions,
including but not limited to Debian. Some of these binaries are built
on a Debian derivative, but run on an arbitrary other distribution,
using a RPATH[1] to find their non-glibc dependencies.

At the moment those binaries are built in ancient environments (based
on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
ubiquitous, we'll want to start relying on newer versions of Debian
(which will still be very old versions *at the time*, but I'm sure that
by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
are not released yet). In this use-case, yes we do need to be using the
interoperable ELF interpreter path.

We were able to use (Ubuntu and) Debian for this *because* Debian is
a relatively "ordinary" distribution that tends to follow cross-distro
standards. The major counterexample is multiarch library paths, but
multiarch library paths have some compelling technical advantages, and
because there's no ambiguity about which architecture uses which directory,
they're actually better for interop in some ways than /usr/lib (which
is ambiguous, because the Red Hat family of distros expects to find 32-bit
libraries there, but the Arch family expects 64-bit libraries, and it's
not possible to construct a tree where both get what they want).

I've spent a lot of the last 5 years working on putting Steam games in
containers so that they'll work more reliably on all distros, including
Debian, with less reliance on weird library search paths (although we
still have to use binaries built on an ancient Debian derivative with a
non-trivial RPATH to bootstrap the container environment). Because of
constraints around recent GPUs needing current graphics drivers even
if running on an otherwise ancient library stack, Steam's container
runtime has to construct a hybrid environment where glibc is either the
one from the host or the one from the container runtime environment,
whichever is newer; and while doing that, we have experienced some
broken situations that were caused by distributions diverging from the
interoperable ELF interpreter. One concrete example is that Arch Linux
uses the interoperable ELF interpreter for *almost* all executables,
but uses a different ELF interpreter for executables from the glibc
source package, for whatever reason; we were able to work around this,
but for a while it caused Arch to fail to launch these containers where
Debian/Fedora/etc. could.

This is not something that any of the distributions involved is
going to formally support, so the overall system consists of various
unsupported-but-works-in-practice things happening; but there is
absolutely no chance of Valve/Steam shipping separate binaries for
each distro (there are too many distros for that, even if you don't
count source-based distros like Gentoo that have an almost unlimited
number of concrete instantiations as binaries). However loudly we
might insist that our small subset-of-a-subset is the one they should
be targeting, what we're going to get is binaries that work on "mostly
interoperable" distros. As a result, if we want games on Linux (and
anything else requiring binary interop) to continue to be a thing,
pragmatically we should aim to be one of those "mostly interoperable"
distros, and encourage our friends and/or rivals in other distros to
do the same. Otherwise, the most likely outcome is for developers to go
back to an attitude of "I'm not going to support Linux, too many random
differences" and releasing for Windows only, and we all lose out.

As a result of all this, I would strongly prefer our compiler to
default to hard-coding the same interoperable ELF interpreter that
glibc upstream and the various major distros have agreed on (for example
/lib64/ld-linux-x86-64.so.2 on amd64), and I would also prefer it if we
can use that interoperable path in all the binaries we ship, including
src:glibc and the rest of the transitively Essential set.

One way that I like to think about this sort of thing is that we have a
finite number of "weird Debianism" tokens available, and we should aim
to spend them on the things that give us the best cost/benefit ratio.
We've never considered changing the ELF interpreter to be one of those,
to the extent of having a /lib64 solely for its benefit (on amd64)
even though by policy we don't generally use lib64.

(Incidentally, Steam's container runtime is always a merged-/usr
environment, to provide an environment that maximizes the probability
of any given script or binary working correctly; but it 

Bug#1026104: longstanding problem with dependencies of python3-numpy in testing

[Simon McVittie]

On Tue, 10 Jan 2023 at 20:04:08 +0100, Helmut Grohne wrote:
> We do not see a simple change or
> patch that would be readily applicable to improve the situation. Sandro
> indicated that he does not want to work on such a patch, which is his
> right granted by the Debian Constitution. On the other hand, he
> indicated that he would be [open to] reviewing a patch.

I've sent a followup to #945824 with two versions of a non-trivial but
reasonably simple change that improves the situation. One is conservative
and shouldn't need ongoing maintenance, the other has a smaller diffstat
but has to make some assumptions which could conceivably become false
in future. Hopefully one or the other will be considered suitable.

Please take any further discussion of those proposed changes to #945824.

The changes I proposed in #945824 were prepared in my role as an ordinary
Debian developer and do not reflect a position of the TC.

smcv

Bug#1024823: tech-ctte: Call for votes on TC membership of Matthew Garrett

[Simon McVittie]

On Fri, 25 Nov 2022 at 15:39:12 -0700, Sean Whitton wrote:
> ===BEGIN
> The Technical Committee recommends that Matthew Garrett  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> H: Recommend to Appoint Matthew Garrett 
> F: Further Discussion
> ===END

I vote H > F.

smcv


signature.asc
Description: PGP signature

Re: merged-usr

[Simon McVittie]

On Tue, 13 Sep 2022 at 12:57:28 +0200, Adam Borowski wrote:
> # debootstrap --no-merged-usr unstable foo ...
> # chroot foo dpkg -l usr-is-merged|grep ^ii
> ii  usr-is-merged  30   all  Transitional package to assert a 
> merged-/usr system
...
> On the other hand: WTH?

I assume your "WTH" was at a non-merged-/usr chroot having the usr-is-merged
metapackage?

As I pointed out when I was asked to review the debootstrap change[1],
this is actually correct, but it's not at all obvious that it's correct.
The usr-is-merged metapackage is arguably misnamed: its actual semantics
are that the system is on the "new" side of the merged-/usr transition
(either the /usr-like directories have been merged into /usr, or that
merge has deliberately been avoided in a way that will become unsupported
with the release of bookworm, either of which makes it unnecessary to
install the usrmerge package and its non-trivial dependencies).

But that's quite a lot to express in a package name, renaming it would
require another trip through NEW, and in the only scenario that is
intended to be supported beyond bookworm release day, its name is
accurate; so its current name seems close enough to the truth.

smcv

[1] 
https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/71/#note_324399

Re: tech-ctte: more on merged-/usr

[Simon McVittie]

On Tue, 19 Jul 2022 at 13:42:21 +0100, Luca Boccassi wrote:
> And before release day, there's no such thing as a bookworm buildd,
> right? Everything is built in unstable?

As far as I know, bookworm buildd chroots already exist, and are used
for uploads to testing-proposed-updates and bookworm-security (which the
release team strongly discourages, but sometimes building in testing
is the only option for a critical fix if a package in unstable is
sufficiently broken).

smcv

Re: tech-ctte: more on merged-/usr

[Simon McVittie]

On Sun, 17 Jul 2022 at 14:21:30 +0100, Luca Boccassi wrote:
> So the reasoning behind adding the override to debootstrap in --
> variant=buildd mode is twofold.
> 
> From a very practical perspective, it means it's supereasy to allow all
> those builds and tools you mentioned to run in unmerged-usr, there's no
> extra config or code changes required anywhere else, it just works.

Shouldn't `debootstrap --no-merged-usr` also have this behaviour for
other variants? If a "larger" component like autopkgtest or piuparts
explicitly asks for non-merged /usr in some other variant (in practice
it would usually be minbase or default for those tools), it seems like
debootstrap should do as it says, even if that means going into the
unsupported mode.

--variant=buildd currently implies --no-merged-usr, so this would still
provide the behaviour you want for buildds.

For the older suites where absence of an explicit option is equivalent
to --no-merged-usr, this flag-file wouldn't be created anyway, because
it's an older suite that doesn't need it?

(I've commented as such on debootstrap!71.)

> From a strategic point of view, catching all those class of bugs you
> mentioned is not trivial

It'll be even less trivial if we remove the ability for autopkgtest and
piuparts to (create chroots that will) test this code path, which is
why I asked for an opt-out mechanism in the first place...

> By keeping
> the buildds in umerged-usr state for Bookworm, and switching them as
> soon as it is released, it seems to me we can avoid caring about
> [packages which are misbuilt on merged-/usr] at all.

So when do you plan to start operating bookworm and sid buildds as
merged-/usr? Toolchain freeze day? Transition freeze day? Full freeze
day? Release day?

I don't really like the idea of post-release bookworm security updates,
etc. being built in chroots that are explicitly unsupported.

> Depending on the answer, I
> might need to add a new flag to debootstrap to force the unmerged-usr
> setup. Right now in the current diff it is implied by --variant=buildd.

Isn't that flag just --no-merged-usr, which we've had since Marco added
it in 2016?

smcv

Re: tech-ctte: more on merged-/usr

[Simon McVittie]

On Mon, 18 Jul 2022 at 13:03:36 -0700, Tianon Gravi wrote:
> Hopefully I'm not too OT (please feel free to tell me if so! [1] would
> be an appropriate place to continue with more discussion on the topic)
> but another angle to this is the container images.
> 
> [1]: 
> https://github.com/debuerreotype/docker-debian-artifacts/issues/131#issuecomment-900591334
> 
> They're currently *not* merged-/usr specifically because their usage
> is in a gray area between user and build and we didn't want to break
> the latter -- I'm guessing this means that for Bookworm, they should
> technically stay in the "unsupported" configuration for the lifetime
> of the release? /o\

That would seem pretty bad to me. Official or pseudo-official Debian
containers should be set up to be in a supported configuration.

However, I do see your point about these containers being in a grey area
between user and build, and there is a risk (a relatively small risk at
this point) that if people are building packages in a debian:bookworm
Docker container, the resulting packages will only work on merged-/usr
systems.

At the time I commented on docker-debian-artifacts#131, I think it would
have been premature to /usr-merge these images, but now that I've raised
the known misbuilt-on-merged-/usr bugs to RC, I think it's time to consider
applying merged-/usr anyway.

I'd appreciate input from the rest of the TC, but I think this is what I'd
do if it was up to me:

1. apply merged-/usr to suites > bookworm in debuerreotype, perhaps
   something like this:
   - in general stop passing --merged-usr *or* --no-merged-usr to
 debootstrap by default, so debootstrap will decide automatically based
 on suite
   - as a special case, add --no-merged-usr for buster and bullseye, for
 continuity (debootstrap defaults to merged-/usr for those suites, but
 their Docker containers have traditionally been unmerged), and
 also bookworm in the short term
   - `debuerreotype --[no-]merged-usr` continues to be an override

2. wait for the proposed migration step (init-system-helper Depends
   usrmerge) to reach testing

3. extend merged-/usr to debuerreotype suites >= bookworm by removing
   bookworm's special case in debuerreotype

Does that sound reasonable to everyone? Starting to /usr-merge debian:sid
before debian:bookworm would give you a chance to spot any showstopper
issues.

smcv

Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Mon, 18 Jul 2022 at 20:03:14 +0100, Luca Boccassi wrote:
> On Mon, 2022-07-18 at 17:53 +0200, Johannes Schauer Marin Rodrigues
> > Also, what is the relationship between the usr-is-merged package and the
> > usrmerged package? Both were/are built by src:usrmerge but its changelog
> > doesn't indicate when or why the naming changed so I'm left a bit confused
> > about the relationship between these two packages. Since your deboootstrap
> > merge request changed from usrmerged to usr-is-merged I guess the latter is 
> > the
> > correct choice?
> 
> It was renamed following a request on #debian-ftp while it was in NEW,
> as the feedback was that 'usrmerge' and 'usrmerged' were too similar
> and thus easily confused. The 'usrmerged' one can be disregarded and
> will be de-crufted.

It's unfortunate that when a package like usrmerge has more than one
version waiting in NEW, the two operations available to the ftp team seem
to be "reject all versions of usrmerge" and "accept all versions of
usrmerge": if the result of NEW review is that the new binary package
needs to be renamed, as it was in this case, then the ideal result would
have been to reject usrmerge/28 from the queue and immediately accept
usrmerge/29.

I agree that usrmerge and usrmerged are confusingly similar names, and
usr-is-merged is a lot more clearly different.

smcv

Re: tech-ctte: more on merged-/usr

[Simon McVittie]

On Sun, 17 Jul 2022 at 11:34:36 +0100, Simon McVittie wrote:
> I personally agree that buildds should stay unmerged-/usr until the known
> bugs that would be triggered by merged-/usr buildds have been raised to
> RC severity, though. #992645 is a typical example. I'll try to have a look
> at those bugs today.

I've now gone through
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org&tag=usrmerge,
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=md%40linux.it&tag=usrmerge
and
https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
and raised severity as follows:

* most bugs: escalated to serious with #994388 as justification
* bugs only affecting example files: left as non-RC
* ypserv #983138, pkg-config #992620: left as non-RC for now, with a
  comment that I am unsure whether they should be RC or not
  (better-informed opinions welcome)

I haven't investigated whether the unreproducibility in gcc-arm-none-eabi,
gcc-mingw-w64 and gcc-sh-elf is serious: these packages are too large for
the reproducible-builds infrastructure to run diffoscope successfully.

It would be great if someone with the necessary computational resources
to inspect these packages, or someone with better knowledge of gcc
internals, could do so. I suspect they will be in the same situation as
gcc-riscv64-unknown-elf and gcc-xtensa-lx106 (RC-buggy).

Thanks,
smcv

Re: tech-ctte: more on merged-/usr

[Simon McVittie]

On Sun, 17 Jul 2022 at 11:34:36 +0100, Simon McVittie wrote:
> On Sun, 17 Jul 2022 at 00:56:14 +0100, Luca Boccassi wrote:
[some discussion of the transition to merged /usr]

Please note that #994388 has been closed and archived, and does not
accept new messages, so I've removed it from Cc.

If your intention is to keep the TC updated on the progress of merged-/usr
for our information, and maybe get some informal advice from individual
TC members, then debian-ctte@lists.debian.org seems appropriate.

If you would like the technical committee to make use of any of its
constitutional powers (https://www.debian.org/devel/constitution#item-6)
regarding merged-/usr or otherwise, please open a new bug with the
request. In particular, requesting advice from the TC as a whole (as I
did in #911225) would be an appropriate reason to open a new bug. If it
becomes necessary to overrule a maintainer, that would also be a topic
for a bug report.

Thanks,
smcv

Re: tech-ctte: more on merged-/usr

[Simon McVittie]

On Sun, 17 Jul 2022 at 00:56:14 +0100, Luca Boccassi wrote:
> Piuparts has been enhanced with a new test case that covers the
> moratorium on moving files manually between their location in the root
> directories and /usr.

Thanks for doing this, it will help a lot.

> A MR is pending for debootstrap [1] to make use of this new [...]
> file flag when --variant=buildd is passed, so that buildds can continue
> to be unmerged-usr until the CTTE says otherwise, as per decision
> above.

I don't think we actually ever said that buildds must continue to be
unmerged-/usr. In #914897 we resolved that for Debian 11, build systems
can validly be either merged-/usr or unmerged-/usr, implicitly meaning
that packages that are mis-built on either merged-/usr or unmerged-/usr
buildds have a bug (#994388 recommended that this bug is treated as RC).
In #994388 we confirmed that this continues to apply during the
bookworm-is-testing cycle.

In #914897 this narrowly defeated the alternative of mandating that build
systems must be merged-/usr for Debian 11 (which, with hindsight, would
have been too disruptive at that time, and I'm glad we didn't choose it).

I personally agree that buildds should stay unmerged-/usr until the known
bugs that would be triggered by merged-/usr buildds have been raised to
RC severity, though. #992645 is a typical example. I'll try to have a look
at those bugs today.

The QA systems that I had in mind when I suggested that we need an
opt-out were things like autopkgtest and piuparts, rather than buildds,
to ensure that we can detect this scenario:

* a package (let's say ncftp, #992645) is built on a merged-/usr buildd
* it builds successfully, but the contents are incompatible with
  unmerged-/usr systems
  - for example in #992645 the binary gets /usr/bin/tar hard-coded into it,
but that path only exists on merged-/usr
* a QA test (autopkgtest, piuparts or similar) installs the package into
  an unmerged-/usr environment
* the package fails to install, or installs but fails to work
  - I don't think ncftp has a test that exercises its tar integration,
but please imagine it did...
* as per #994388, the TC think this should be treated as a RC bug

The opt-out is needed in order to allow autopkgtest and piuparts to create
the necessary unmerged-/usr environment to be able to run that test. If
they can only create merged-/usr bookworm environments, then that test
would not be implementable.

> Once deboostrap is updated and deployed on the buildds, a "usrmerge |
> usr-is-merged" dependency will be added to the Priority: essential
> init-system-helpers package, and uploaded to unstable to complete the
> transitions for all installations that are older than buster or that
> have been manually installed as unmerged-usr. Any system not upgraded
> will be considered unsupported, and any package that doesn't work on
> the only supported layout will be considered RC-buggy, as per decision
> above. No special installation or upgrade path will be necessary, the
> normal apt upgrade/install process works as expected, and the
> transition happens from a maintainer script and it does not require to
> be ordered before or after any other package installation or upgrade.

This all seems like a reasonable course of action to me. The usr-is-merged
package avoids the dependency impact of the Perl modules that usrmerge
needs when it does the actual transition.

After usrmerge has been installed successfully, is it straightforward to
replace it with usr-is-merged so that usrmerge's Perl dependencies can be
autoremoved?

> The patch from user uau that the dpkg maintainer rejected in the past
> has been submitted to the existing bug [2] for completeness (with
> permission from the author).

If I remember correctly, Julian Andres Klode was developing a version
of this patch that replaced the hard-coding of the /usr merge with a
more general way to declare "directory A is an alias for directory B"
and have it stored in the dpkg database, so that dpkg has the mechanism
and some Essential package like init-system-helpers or base-files could
have the policy, in the hope that this would be more acceptable to the
dpkg maintainer. Is that code available?

smcv

Bug#1007717: Ballot and call for votes

[Simon McVittie]

On Mon, 20 Jun 2022 at 17:31:16 -0700, Sean Whitton wrote:
> Using its powers under constitution 6.1.5, the Technical Committee
> issues the following advice:
> 
>   1. It is not a bug of any severity for a package with a non-native
>  version number to use a native source package format.
> 
>   2. Thus, we think that dpkg shouldn't issue warnings, or otherwise
>  complain, when a non-native version number is used w/ 3.0 (native).
> 
>   3. We suggest that the wontfix tag on #737634 be reconsidered.
> 
>   4a. We believe that there are indeed circumstances in which
>   1.0-with-diff is the best choice for a particular source package,
>   including, but not limited to, git-first packaging workflows.
> 
>   This is because there is currently no other source format which is
>   such that avoid both (i) uploading the whole source, including
>   upstream, for every upload; and (ii) having to maintain
>   debian/patches/ inside the package tree.
> 
>   4c. We believe that there are indeed circumstances in which
>   1.0-with-diff is the best choice for a particular source package,
>   including, but not limited to, git-first packaging workflows.
>   However, we recommend discontinuing use of 1.0-with-diff in other
>   circumstances, to simplify the contents of the archive.
> 
>   This is because ... [second paragraph as in 4a].
> 
>   5. We decline to comment on the recent source package format MBF.
> 
> Option A -- issue items 1-3, 4a and 5
> 
> Option C -- issue items 1-3, 4c and 5
> 
> Option X -- issue only items 1, 2, 3 and 5
> 
> Option N -- none of the above.

I vote C > A > X > N.

smcv


signature.asc
Description: PGP signature

Bug#1003653: Revision of removal of rename.ul from package util-linux

[Simon McVittie]

On Wed, 20 Apr 2022 at 15:31:13 +0100, Matthew Vernon wrote:
> ===Begin Resolution A
> The Technical Committee overrides the util-linux maintainer, and requires
> that util-linux's rename should be shipped as /usr/bin/rename.ul in a binary
> package built from src:util-linux. The package containing rename.ul must not
> conflict with the rename package nor divert /usr/bin/rename.
> ===End Resolution A
> 
> ===Begin Resolution B
> The Technical Committee overrides the util-linux maintainer, and requires
> that util-linux's rename should be shipped in a binary package built from
> src:util-linux. If this package Conflicts with the rename package, then it
> must not contain any other binaries.
> ===End Resolution B
> 
> ===Begin Resolution N
> None of the above
> ===End Resolution N

I vote A > B > N.

smcv


signature.asc
Description: PGP signature

Bug#1004611: Resignation & call for votes to elect the Chair

[Simon McVittie]

On Sun, 30 Jan 2022 at 14:10:08 -0700, Sean Whitton wrote:
> As the membership of the committee has changed, according to convention
> I hereby resign as Chair, effective one week from now, and call for
> votes to elect the Chair of the Technicall Committee.
...
> The ballot:
> 
> ===BEGIN
> 
> A: Christoph Berg 
> B: Helmut Grohne 
> C: Elana Hashman 
> D: Simon McVittie 
> E: Niko Tyni 
> F: Matthew Vernon 
> G: Sean Whitton 
> H: Gunnar Wolf 
> 
> ===END

I vote G > A = C = E = H > D > B = F.

smcv


signature.asc
Description: PGP signature

Bug#1003653: Revision of removal of rename.ul from package util-linux

[Simon McVittie]

On Mon, 31 Jan 2022 at 10:11:19 +, Matthew Vernon wrote:
> There are two "rename" programs, one part of upstream util-linux "rename.ul"
> and one provided by the rename package "rename.pl"[0]

Almost!

The one in src:rename is installed as file-rename(1p), aka prename(1p)
via a symlink (you noted that this is the case on stable, but it's also
the case in unstable). /usr/share/doc/rename/examples/rename.pl is just
the core functionality of file-rename(1p), excluding CLI boilerplate
like displaying help in order to be a better/clearer example. There's
no rename.pl in PATH.

Other than that, you're correct.

Unfortunately, neither implementation has an upstream name other than
"rename". The file-rename naming is a Debian invention, with the rationale
that it comes from the File::Rename CPAN distribution (what we'd call
a source package). Similarly, the rename.ul naming is a Debian invention
to avoid colliding with File::Rename's rename.

> For a long time, Debian's "/usr/bin/rename" has been [file-rename] (via the
> alternatives system).

The history here is that perl.deb used to ship rename(1) as a Debian-specific
addition, based on a script eg/rename that was included in the Perl source
code before 2000.

https://bugs.debian.org/304705 was an earlier attempt to give users a
choice between util-linux rename and Perl rename via alternatives, which
is the reason the alternatives are there in the first place. However,
this was a wrong use of alternatives, leading to
https://bugs.debian.org/439935 and removal of the alternative.

When the Perl maintainers removed its implementation of rename(1),
since the alternatives were already there (left over from #304705), they
used the alternatives mechanism as a way to transition to the separate
implementation in src:rename.

Requests to ship util-linux rename(1) go back as far as at least
2004 (https://bugs.debian.org/228737).

> Assuming that's all correct, my feeling is that there is no particular
> reason for Debian's rename to stop being [file-rename], but that we should
> make rename.ul available to users who want it. I think the maintainer would
> be happy to move rename.ul into bsdextrautils (as /usr/bin/rename.ul)? Taking
> it out of essential, not considering it an alternative to [file-rename], and
> keeping it available for people who want it.

If the util-linux maintainer is OK with that, then I think that's probably
going to be the least-bad solution. It's unfortunate that Debian and other
distros have settled on incompatible rename implementations, but it has
happened, and now neither group of distributions can be compatible with
the other one without breaking compatibility with older versions of itself.

smcv

Bug#1003738: tech-ctte: Call for votes on TC membership of Matthew Vernon

[Simon McVittie]

On Fri, 14 Jan 2022 at 11:56:20 -0700, Sean Whitton wrote:
> ===BEGIN
> The Technical Committee recommends that Matthew Vernon  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> H: Recommend to Appoint Matthew Vernon 
> F: Further Discussion
> ===END

I vote H > F.

smcv


signature.asc
Description: PGP signature

Bug#1003737: tech-ctte: Call for votes on TC membership of Helmut Grohne

[Simon McVittie]

On Fri, 14 Jan 2022 at 11:55:17 -0700, Sean Whitton wrote:
> ===BEGIN
> The Technical Committee recommends that Helmut Grohne  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> H: Recommend to Appoint Helmut Grohne 
> F: Further Discussion
> ===END

I vote H > F.

smcv


signature.asc
Description: PGP signature

Re: Bug#1000239: Rescue system won't find root partition, but insists on /usr

[Simon McVittie]

(Speaking only on my own behalf, not on behalf of the TC, here)

On Fri, 03 Dec 2021 at 16:08:24 -0500, Nicholas D Steeves wrote:
> 1. Debian isn't yet ready for usrmerge

Merged /usr is not actually the problem here, although it exacerbates what
appears to be a pre-existing bug in the rescue mode[0]. The root cause is
that since Debian 9 [1][2], the "/usr-like" parts of the root filesystem
are no longer guaranteed to be self-contained: important shared libraries[3]
and executables have been gradually moving into /usr for a while, and
stretch was the point at which this was declared to be officially allowed.

Merged /usr makes this very noticeable because it's the most extreme
version of that process, where *literally everything* "/usr-like" (most
notably /bin/sh) moves into /usr. When /usr is a separate filesystem, that
leaves a root filesystem that potentially contains only /etc, symlinks and
mount points, which is certainly not enough to be useful to chroot into.

> 2. Reassign to src:rescue, and fix the rescue system.

I think this will have to be the answer. See also [0].

> 3. Disallow configuring of a mount for "/usr"

That would be unfortunate, since one of the things enabled by merged
/usr is the ability to keep all "/usr-like" content on a separate /usr
filesystem that is mounted read-only during normal operation, remounting
it rw only for system updates[4], while leaving /etc and /var rw (and
potentially offering the ability to reformat the partition with /etc
and /var, and then repopulate it via systemd-tmpfiles or similar, as a
"factory reset" mechanism).

smcv

[0] https://bugs.debian.org/769738 (opened in 2014)
[1] 
https://www.debian.org/releases/stretch/amd64/release-notes/ch-information.en.html#late-mounting-usr
[2] and possibly earlier: that release note was documenting current practice
rather than describing a new change
[3] for example about half the libraries in `ldd /sbin/cryptsetup`
[4] which potentially happen atomically across a reboot, as seen in ostree,
or while rebooted into a special boot mode, as with
systemd.offline-updates(7)

Bug#994275: Call for votes on "Reverting breaking changes in debianutils"

[Simon McVittie]

On Wed, 20 Oct 2021 at 12:30:54 -0700, Sean Whitton wrote:
> I hereby call for votes on the following ballot to resolve #994275.

I vote A > B > FD (which I believe means the outcome is no longer in doubt
and we resolve A).

> === Resolution A ===
> 
> The Technical Committee resolves:
> 
> 1. The debianutils package must continue to provide the which(1) program
>until a compatible utility is available in a package that is at least
>transitively essential in Debian 12.
> 
>For the Debian 12 release, we expect which(1) to be in either an
>Essential package or a transitively Essential package (that is, a
>package that is depended on by an Essential package).
> 
> 2. The which(1) program must not print any deprecation warnings.
> 
> 3. We decline to overrule the maintainer of debianutils regarding the
>use of alternatives.  If another package takes over responsibility
>for which(1), then the debianutils maintainers and the other
>package's maintainers should coordinate to choose a suitable
>mechanism, which might be either versioned Depends/Breaks/Replaces,
>dpkg-divert, alternatives or something else.
> 
> 4. The debianutils package must continue to provide the tempfile(1)
>program until a compatible utility is available in a package that is
>at least transitively essential in Debian 12.
> 
>For the Debian 12 release, we expect tempfile(1) to be in either an
>Essential package or a transitively Essential package.
> 
> 5. Programs in debianutils must not be moved to /usr until we have a
>project-wide consensus on going ahead with such a move, and any
>programs that have already been moved must be moved back.  In
>particular, this means debianutils must contain /bin/run-parts and
>/sbin/installkernel for the time being.
> 
> === Resolution B ===
> 
> As Resolution A, except strike point (2) and renumber succeeding items.
> 
> === End Resolutions ===


signature.asc
Description: PGP signature

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 13 Oct 2021 at 20:13:08 +0100, Simon McVittie wrote:
> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

I vote yes > further discussion.

>  begin text to be voted on 
> 
> Summary
> ===
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: 
> https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> =

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

I'm calling for votes on the following resolution as formal advice from
the Technical Committee (Constitution §6.1.5). There are no non-accepted
amendments, so the options to vote on are "yes" or "further discussion".

 begin text to be voted on 

Summary
===

There are currently Debian 11 installations with both the merged-/usr and
non-merged-/usr filesystem layouts. All of these installations should
successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
Only after the release of Debian 12 can packages assume that all
installations will be merged-/usr.

Main points:

- We have recommended merged-/usr for Debian 12.
- Moving individual files is not merged-/usr.
- "Symlink farms" are not merged-/usr.
- Upgrading a non-merged-/usr system to Debian 12 needs to work.
- Upgrading a merged-/usr system to Debian 12 needs to work.
- Packages cannot assume all systems are merged-/usr until the Debian 13
  development cycle begins.
- Upgrading via apt in the usual way should work.
- Testing and QA systems should be able to avoid this transition, but if
  they do, they cannot be upgraded beyond Debian 12.
- A package building incorrectly on a non-merged-/usr system is a bug.
- A package building incorrectly on a merged-/usr system is a bug.
- Please stop moving individual packages' files from the root filesystem
  into /usr, at least for now.

Definitions and current status
==

libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
such as lib64 on the amd64 architecture.

Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
/libQUAL that exists are symbolic links to the corresponding directories
below /usr. This results in aliasing between /bin and /usr/bin, and
so on.

In the merged-/usr layout, files whose canonical logical location is
in one of the affected directories on the root filesystem, such as
/bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
the corresponding path below /usr, such as /usr/bin/bash. Each file in
one of the affected directories is accessible via two paths: its canonical
logical location (such as /bin/bash or /usr/bin/env), and the other path
implied by the aliasing (such as /usr/bin/bash or /bin/env).

There are two supported categories of Debian 11 installation, which are
currently considered equally-supported:

- Merged-/usr installations. These were installed with debian-installer
  from Debian 10 or later, or installed with debootstrap --merged-usr,
  or converted from the non-merged-/usr layout by installing the usrmerge
  package, or installed or converted by any similar procedure. They have
  the merged-/usr layout.

- Non-merged-/usr installations. These were installed with debian-installer
  from Debian 9 or earlier and subsequently upgraded without converting
  to merged-/usr, or installed with debootstrap --no-merged-usr, or
  converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
  utility or any similar procedure. They have the traditional,
  non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
  those physical paths, and /usr/bin/bash and /bin/env do not exist.

Merged-/usr is not the only filesystem layout that has been proposed for
unifying the root filesystem with /usr. For avoidance of doubt, we do not
consider other filesystem layouts to be implementations of merged-/usr.
In particular, we do not consider these to be implementations of merged-/usr:

- all affected files physically located in /bin, /sbin, /lib and /libQUAL,
  with /usr/bin as a symlink to /bin, etc. (this is the reverse of
  merged-/usr, and was historically used in the hurd-i386 port)

- a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
  symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
  historically had their canonical logical location on the root filesystem

- a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
  symbolic links such as /bin/env -> /usr/bin/env for all files in the
  affected directories, regardless of whether they historically had
  their canonical logical location on the root filesystem

[1]: 
https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential

Upgrade path from Debian 11 to Debian 12


The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
should only support the merged-/usr layout. This resolution describes
the implications of that decision in more detail.

Debian installations have traditionally had a straightforward upgrade
path between consecutive releases, and the technical committee expects
maintainers to continue this. In the case of #978636, the upgrades we
are interested in are:

- Upgrading from Debian 11 (stable release) to Debian 12 (stable release)

- Upgrading from Debian 11 (stable release) to testing/unstable during the
  development cycle that

Bug#994275: Reverting breaking changes in debianutils

[Simon McVittie]

On Wed, 15 Sep 2021 at 01:36:26 +0300, Adrian Bunk wrote:
> The release team has so far protected users of testing from the
> problem by blocking testing migration, but this is not a long-term
> solution.

Adrian asked in #994275 for changes in several topics to be reverted:

- which(1) deprecation
  - deprecation warnings on stderr
  - management via alternatives
  - possible future removal
- tempfile(1) removal
- installkernel(8) moving from /sbin to /usr/sbin
- run-parts(8) moving from /bin to /usr/bin

Which of those topics were your reason for adding a "block debianutils"
hint? Are there any of those topics whose resolution you would consider
to be a prerequisite for letting debianutils migrate to testing again?

The hint references #992399, which is related to tempfile(1), but
#992399 has been closed (via a merge with #992396, which was resolved by
debianutils adding a versioned Breaks on x11-common versions that needed
tempfile). Do the release team consider #992399 to have been adequately
resolved, or do you consider debianutils to still have a RC bug?

If debianutils reinstated tempfile(1) in bookworm, and removed it in
testing/unstable after the release of bookworm, would the release team
consider that to be an adequate transitional period?

Thanks,
smcv

Bug#994275: Reverting breaking changes in debianutils

[Simon McVittie]

On Wed, 15 Sep 2021 at 01:36:26 +0300, Adrian Bunk wrote:
> More specifically, I am asking the Technical Committee to decide that:

I think this is really 5 separate (but related) requests, each of which
we could either uphold or decline, separately. Do you agree?

> 1. The "which" program must be provided by an essential package.

Constitutionally, I think this would have to be the TC formally offering
advice (under 6.1.5 in the Debian constitution), because the maintainers
of debianutils haven't attempted to remove "which", so there is not
yet any maintainer action that you want us to overrule.

For example, that advice could take the form of saying that if which(1)
is removed from debianutils in future, then we would expect that to be
accompanied by a suitable versioned dependency on a transitively-Essential
package that provides which(1) (either for bookworm, or indefinitely).

Does that sound right to you?

Or are you asking us to go further than that, and say that which(1)
can only be removed from debianutils if it is picked up by a different
Essential package?

> 2. The "which" program must not print any deprecation warnings.

I think that's a request to overrule the maintainer of debianutils
(under 6.1.4). I assume that was your intention?

> 3. The "which" program must not be provided through alternatives.

Also 6.1.4, I think.

> 4. The debianutils package must continue to provide the "tempfile" program.

Also 6.1.4.

Would you be equally happy with a TC resolution that keeps tempfile(1) in
the transitively-Essential set, but not necessarily via debianutils? For
example, we could consider saying that debianutils must either reinstate
tempfile(1), or add a dependency on some other (transitively Essential)
package that provides tempfile(1).

> 5. Programs in debianutils must not be moved to /usr unless there is
>project-wide consensus on packages doing such a move, and premature
>moving must be reverted.

Also 6.1.4.

smcv

Bug#994275: Reverting breaking changes in debianutils

[Simon McVittie]

On Sun, 03 Oct 2021 at 22:09:31 +, Clint Adams wrote:
> The package description uses the phrases "specific to Debian" and
> "installation scripts of Debian packages".  The fact that
> debianutils is used on non-deb operating systems suggests a failure
> at the former.

Given its package description, I too am confused by other distributions
having picked up debianutils (as it seems at least Arch and Gentoo have),
but the packages found in non-Debian-derived distributions are not under
our control.

> The fact that 95% of my inbox consists of hatemail
> about the interactive usage of `which` suggests a failure at the
> latter.

I'm sorry you're receiving hatemail about this package, and that's not
OK, but it's orthogonal to whether the changes discussed in this bug
were the right thing for Debian.

> debianutils should consist solely of utilities that are actually
> deserving of Essential status, that are maintained by debianutils
> upstream, and do not needlessly duplicate functionality provided by
> other sources.

I was under the impression that debianutils is (intended to be)
a Debian-specific package with no separate upstream existence. Does
it have releases other than "whatever is in unstable"? Is there an
upstream maintainer distinct from its Maintainer and Uploaders, namely
you and Manoj?

"debianutils should consist of utilities maintained in debianutils"
seems slightly circular. Is there an intended scope for what's in
debianutils, beyond "things that need to be Essential because they
always used to be Essential"?

I see that CONTRIBUTING documents some of the utilities in debianutils
as being unsupported/pending deprecation, such that if new features are
desired, contributors are asked to instead fork the utility and coordinate
removal from debianutils (and presumably takeover by a new package, given
their Essential status). Can we infer the utilities you want to continue to
maintain are everything not on that list?

debianutils currently has:

- add-shell, remove-shell, update-shells: Debian-specific(?)
  maintainer-script infrastructure for maintaining /etc/shells,
  analogous to how init-system-helpers maintains system services.
  Clearly at least transitively Essential, because if they were a
  separate package, the Essential dash and bash would depend on them.

- ischroot, run-parts, savelog, which (and historically tempfile):
  these seem more like general-purpose utilities than anything else,
  and Essential from their wide use without an explicit dependency?
  Of these, CONTRIBUTING describes ischroot, savelog and which as being
  unsupported (deprecated or pending deprecation) but presumably
  run-parts is not in that category?

  CONTRIBUTING calls out logrotate as being better than savelog, but
  savelog gets used by Essential packages like dpkg, and I don't think
  logrotate is necessarily suitable to be transitively Essential.

- installkernel: the Debian implementation of an integration hook that
  is expected to be invoked by the upstream Linux kernel build
  system, analogous to how LSB init scripts can expect to source
  /lib/lsb/init-functions and we provide a Debian implementation of that
  interface in lsb-base?

> Divestiture of mktemp, readlink, sensible-editor, sensible-pager,
> sensible-browser, tempfile, and possibly mkboot were adjustments
> toward a better debianutils.

Yes, and in general I think that direction is fine, as long as the
transition away from these utilities being in debianutils is done carefully.

mktemp and readlink were taken over by Essential coreutils, with
a transitional Pre-Depends; sensible-* were moved to non-Essential
sensible-utils, with a transitional Depends (making sensible-utils
transitively Essential for one release) to ensure existing systems didn't
lose them, and presumably some reasonable plan to ensure that dropping
them from the Essential set wouldn't break existing packages.

mkboot doesn't have a replacement that I know of, but it seems like the
sort of utility that had a very limited use by the time it was removed,
so I'm happy to trust your judgement on what the appropriate time was to
remove it ("your" referring to you and Manoj as a team here - I know it
was actually Manoj who removed that one).

However, I don't think which(1) and tempfile(1) are really in the same
situation as all those. From my point of view, debianutils which(1) has a
lot in common with sysvinit-utils pidof(8): if we were defining Essential
today, those programs likely wouldn't be in it, and they should probably
be used a lot less than they are (in favour of the command builtin and
pgrep, respectively); but the fact remains that they *are* in Essential,
and too widely-used (by packages with no dependency, because Essential)
for a mass-bug-filing to be straightforward.

which(1) and tempfile(1) suffer from this somewhat worse than pidof,
because "which" is a common English word and "tempfile" is a common
name for variables, so locating users of those utilit

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Sun, 03 Oct 2021 at 16:52:15 -0700, Sean Whitton wrote:
> On Mon 27 Sep 2021 at 10:59AM +01, Simon McVittie wrote:
> > On Thu, 16 Sep 2021 at 15:35:11 -0700, Sean Whitton wrote:
> >> (1) The reason for this, to put it a bit simplistically, is that we
> >> don't require apt to perform the upgrade between stable releases in any
> >> particular order, right?  Or are there other reasons distinct from this
> >> one that I'm missing?  I think it would be good to state the thing about
> >> apt (in better language than mine) in the text.
> >
> > I think that's the main reason. We have not traditionally mandated
> > the use of a special upgrade tool like Ubuntu's do-release-upgrade(8),
> > so the upgrade happens in whatever order apt chooses, which can vary
> > between machines.
> >
> > Another reason why I think we want Debian 12 packages to be installable
> > onto non-merged-/usr systems is that to be able to do our development work,
> > they need to be installable onto testing/unstable systems, which (again)
> > means that the upgrade order is undefined.
> 
> Right, we're on the same page then, but would you agree with me that the
> resolution should state this justification explicitly?

I hope that
<https://salsa.debian.org/debian/tech-ctte/-/merge_requests/4>
implements this to your satisfaction. If not, suggestions for better
wording welcome - I would prefer not to be the only one writing this
document!

> > [Reasoning in a previous mail] makes me reluctant to require a special
> > upgrade procedure if it is not strictly necessary.
> 
> This is persuasive.  What do you think about including it in the text?

This is also in
<https://salsa.debian.org/debian/tech-ctte/-/merge_requests/4>.

> > I'm honestly not sure which of these is "the" reason why I'm recommending
> > the conservative approach. Some combination of your second and third
> > points, I suppose?
> 
> Based on what you say above I think it's the second and third, indeed.
> If we add to the text the things I'm suggesting we add, I think this
> sort of query about our motivations will not arise in the minds of
> readers.

Does <https://salsa.debian.org/debian/tech-ctte/-/merge_requests/4>
cover this? If not, please suggest something that would.

Thanks,
smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Mon, 27 Sep 2021 at 09:18:29 -0600, Sam Hartman wrote:
> [smcv wrote]
> >On merged-/usr systems, there is a possible failure mode involving files
> >being moved between packages (with Replaces) during the same release
> >cycle that their logical location is changed from the root filesystem
> >to the corresponding aliased directory in /usr, which can result in
> >the affected file disappearing. This can be avoided by not changing
> >the file's logical location until the beginning of the Debian 13
> >development cycle, after the transition to merged-/usr is complete.
>
> I don't understand how transitioning files in the Debian 13 cycle is
> going to work any better than in the Debian 12 cycle unless dpkg happens
> to change.

I think you might be right about this.

It might not be possible to do these changes of logical location without
first enhancing dpkg to understand that certain directory hierarchies
are aliases for each other.

Or, perhaps it can work in a subset of cases, and people with an interest
in merged-/usr can identify patterns that are safe, and have guidance
that recommends sticking to those patterns?

(For example we might need to require that if files that were historically
in /bin, /sbin, /lib* are moved between packages, then either there's some
sequencing requirement enforced by Pre-Depends/Conflicts to make sure the
/usr move is done before the ownership change, or the files do not move to
/usr until the next release cycle? I haven't thought this through, I'm just
saying these as examples of things that people who do the detailed design
might try.)

Do we want to specifically say in our advice that proponents of merged-/usr
are invited to design solutions for this?

The Technical Committee does not do detailed design, so I think identifying
specific patterns that are safe is not our job - but if domain experts
identify good patterns, and ask us to check their working and then issue
advice recommending those patterns, we can do that.

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 15 Sep 2021 at 12:35:38 +0100, Simon McVittie wrote:
> https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md

I have updated this draft to incorporate my edits from !3, and feedback
from bremner on IRC.

I'd like to keep this moving, because it's somewhat time-sensitive: people
are already taking action based on our earlier resolution, some of which is
not necessarily the action we would have wanted them to take.

On Wed, 15 Sep 2021 at 11:46:02 +0100, Simon McVittie wrote:
> Some questions that I think we need to answer explicitly:
> 
> - What do we mean by merged-/usr? (We already said this in #914897, but
>   I think it's worth reiterating that symlink farms are not it.)

This is defined (again) by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#definitions-and-current-status

> - Is it OK for packages in testing/unstable during Debian 12 development
>   to assume/require a merged-/usr system? (tl;dr: some maintainers think
>   the answer is yes, but I think the answer is no.)
>
> - When will it be OK for packages in testing/unstable to assume/require
>   a merged-/usr system? (tl;dr: some maintainers think the answer is
>   "right now", but I think the answer is "when the Debian 13 cycle begins"
>   and not before.)

Both of these are addressed by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#upgrade-path-from-debian-11-to-debian-12

> - Should maintainers proactively move files in packages from the root
>   filesystem into /usr? (tl;dr: I think they should not, at least until
>   the implications are better-understood.)
> 
> - Should maintainers of tools, e.g. debootstrap, proactively move files
>   in packages from the root filesystem into /usr, e.g. systemd system units?
>   (tl;dr: I think they should not.)

Both of these are addressed by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#moratorium-on-moving-files-logical-locations-into-usr

> - Are packages built on a merged-/usr system required to work correctly
>   on a non-merged-/usr system? If they are not, is it RC?
>   (I think they are required to work, and it should usually be RC if
>   they don't)

https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#building-packages

On Thu, 16 Sep 2021 at 15:35:11 -0700, Sean Whitton wrote:
> You write:
> 
> >> - Because Debian 11 installations with the non-merged-/usr layout already
> >>   exist, all packages in Debian 12 should be installable onto a 
> >> non-merged-/usr
> >>   system along with their dependencies, and work correctly on the resulting
> >>   system.
> 
> (1) The reason for this, to put it a bit simplistically, is that we
> don't require apt to perform the upgrade between stable releases in any
> particular order, right?  Or are there other reasons distinct from this
> one that I'm missing?  I think it would be good to state the thing about
> apt (in better language than mine) in the text.

I think that's the main reason. We have not traditionally mandated
the use of a special upgrade tool like Ubuntu's do-release-upgrade(8),
so the upgrade happens in whatever order apt chooses, which can vary
between machines.

Another reason why I think we want Debian 12 packages to be installable
onto non-merged-/usr systems is that to be able to do our development work,
they need to be installable onto testing/unstable systems, which (again)
means that the upgrade order is undefined.

We also have best-effort support for partial upgrades, either oldstable
-> stable or stable -> testing/unstable: upgrading only a subset of the
available packages, plus their mandatory dependencies, but without also
upgrading apparently-unrelated packages. This can only ever be partially
supported, because it leads to a combinatorial explosion of possible
partially-upgraded systems, and realistically we can never test them all;
but I think it's best if we try to avoid knowingly making this worse.

Finally, there is a reason to want this that is circular in nature: if we
want Debian 12 packages to be installable onto non-merged-/usr systems,
and we want to be able to say with reasonable confidence that we think
they work in that situation, then we need to be able to test that they
do, in fact, work. As mentioned under the "Testing and QA" heading,
if we do not keep it possible to force autopkgtest VMs/containers,
piuparts chroots, reproducible-builds chroots and manual-test VMs (or
even scratch installations on real hardware) to be non-merged-/usr,
then we will have trouble testing that. However, I have tried to make it
clear that if a developer sets up such a system, they cannot expect to

Bug#994275: Reverting breaking changes in debianutils

[Simon McVittie]

On Sat, 25 Sep 2021 at 02:22:44 +, Clint Adams wrote:
>  * debianutils gets closer to achieving its mission, by having
>one fewer irrelevant utility that does not belong

This seems a good opportunity to ask what I think is a key question here:
what do you consider debianutils' mission to be?

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 15 Sep 2021 at 11:34:33 -0400, Zack Weinberg wrote:
> For the various files with a "canonical" location
> either in /usr or in /, either specified by some standard or by
> convention, and regularly referred to by absolute pathname, all
> software should continue to refer to those files by their "canonical"
> name

There are two sorts of hard-coding that are common for absolute paths,
which we should not conflate.

I think you might be primarily talking about the situation where source
code refers to an executable by its hard-coded absolute path? (As in
"#!" of a script that is installed unmodified, as opposed to being
generated from a template.)

The situation I was talking about is where a build system auto-detects
the "correct" location for some tool, such as sed or perl, and writes
that detected location into an installed file - for example reading a
template that starts with #!@PERL@ and outputting an executable script
that starts with #!/usr/bin/perl, or similar. When this happens in a
binary package system like .deb, we need to make sure that the location
that will be detected on the build system is correct for all systems
where the .deb could get installed, even if the /usr-merge status of
the build system and the installed system are different (and we need to
continue to do this as long as both merged-/usr and non-merged-/usr are
supported). Upstreams that do this auto-detection are often trying to
help the use-case of portability of a single set of source code between
OSs, but unfortunately the auto-detection is sometimes unhelpful when a
single binary package is expected to support being installed on multiple
dissimilar but equally-supported OS configurations.

In both of these cases, during the transitional period that ends when
merged-/usr becomes required, package maintainers need to make sure
(by patching the source code, or configuring the build system, or
whatever other method is most suitable) that the package they maintain
will continue to work on non-merged-/usr Debian systems. There is no
requirement that Debian packages are portable to non-Debian systems,
however.

> The most common class of such files is
> those used in #! directives: /bin/sh not /usr/bin/sh, /usr/bin/perl
> not /bin/perl, etc.  I would ideally like this to be spelled out in
> Policy, as an explicit list of files that MUST be referred to without
> /usr, all others to be referred to with /usr.

A comprehensive list of executables whose paths in /{s,}bin can/should
be hard-coded seems at first glance as though it would be too large to
be particularly useful: on my x86_64 system, `apt-file search` tells
me it can see 245 files in /bin and 549 in /sbin. I'm confident that
many (most?) of those cannot be relied on to be in /bin,/sbin in all
non-Debian OSs.

As a 90% solution, it might be worthwhile to have a non-comprehensive
list of common interpreters and their canonical paths, but I think
Lintian is better-placed than Policy to do that, and it already has a list.

> As an upstream contributor to several pieces of software included in
> Debian, and as someone with an interest in ensuring that software
> developed on Linux is not *accidentally* unportable to other OSes
> under the 'Unix' umbrella, I'd like to stick an oar in

Sorry, I think portability to other Unixes is outside Debian's scope. One
of the advantages of merged /usr is that this is no longer something we
have to argue with each upstream that does not use (our idea of) the
canonical paths for everything, because both /bin and /usr/bin paths
will work.

With upstream hat on, I think this is going to be a losing battle in
general, because outside a few narrow areas that are fixed by Unix
folk history (mainly /bin/sh and /usr/bin/env), there is no portable
canonical path. For example, I regularly see merge requests to various
upstream projects saying that #!/bin/bash is not sufficiently portable
and they need to use #!/usr/bin/env bash on some OS - but in Debian,
we often prefer #!/bin/bash, to avoid packaged scripts being broken by
a locally-installed and potentially incompatible /usr/local/bin/bash.

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 15 Sep 2021 at 12:35:38 +0100, Simon McVittie wrote:
> - §(Building packages): I almost wrote an extra paragraph about how
>   this class of bugs becomes a non-issue when merged-/usr is the only
>   supported layout - but actually it doesn't! If we consider building
>   packages while having /usr/local/bin/sed to be a supported thing you
>   can do, then we need to ensure that /usr/local/bin/sed doesn't get
>   hard-coded into the resulting package, and the steps you take to
>   make that happen are the same as the steps you take to fix this class
>   of bugs.

I think that class of bugs probably does become non-RC when the
merged-/usr transition is complete, though.

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

On Wed, 15 Sep 2021 at 11:46:11 +0100, Simon McVittie wrote:
> As for what that advice is, I'm open to suggestions, but I'm drafting
> some possible wording, which I'll upload to
> https://salsa.debian.org/debian/tech-ctte/ when I have a bug number
> for it.

Here it is:

https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md

Edits and merge requests welcome. I suggest we use merge requests for
substantive changes.

This is a collection of various pieces of advice. I hope that this is
sufficiently uncontroversial among the TC that we can improve the wording
where necessary, then take a unanimous or close-to-unanimous vote
"yes > further discussion" when we feel that it reflects consensus?

If there are individual bits that are more contentious, then I suggest
we vote on them as formally or informally as we are comfortable
with, then make the formal resolution reflect the results of those
votes; alternatively, we could kick out anything more controversial into
a separate resolution if we need to.

Because this is advice to maintainers about decisions that, in some cases,
they are making right now, I would like to keep this moving and try to
reach a consensus we can announce to debian-devel-announce soon.

Questions for the committee:

- §(Definitions and current status): Does everyone agree with my
  characterization of where we are now?

- §(Upgrade path from Debian 11 to Debian 12): Does everyone agree
  with what I've written here about the implications of #978636?

  I've tried to be careful to distinguish between the Debian 12
  stable release and the state of testing/unstable during the development
  cycle; better wordsmithing welcome.

- §(Upgrade path from Debian 11 to Debian 12): Is the last paragraph
  "If a suitable transition mechanism is not available by the time the
  Debian 12 freeze is reached..." necessary, or is it implicit that
  *obviously* we won't demand that the project carries on with merged-/usr
  if it turns out not to be possible?

- §(Testing and QA): Do we want to recommend this?

- §(Building packages): Does everyone agree with my interpretation of
  what we agreed in #914897 and its implications? Do we need to make a
  recommendation for the Debian 12 development cycle, or is it enough
  to assume that the "middle" option that we resolved in #914897
  continues to be our opinion?

  I assume our advice power doesn't extend to unilaterally declaring
  a class of bugs to be RC, but we can certainly advise the release team
  and package maintainers that they *should* consider a class of bugs
  to be RC; if they follow our advice, great, and if they don't, we can
  consider whether we need to overrule in individual cases.

- §(Building packages): I almost wrote an extra paragraph about how
  this class of bugs becomes a non-issue when merged-/usr is the only
  supported layout - but actually it doesn't! If we consider building
  packages while having /usr/local/bin/sed to be a supported thing you
  can do, then we need to ensure that /usr/local/bin/sed doesn't get
  hard-coded into the resulting package, and the steps you take to
  make that happen are the same as the steps you take to fix this class
  of bugs.

- §(Moratorium on moving files' logical locations into /usr):
  I think we should stop moving files into /usr on an individual basis,
  at least until the consequences are fully understood, and perhaps for
  the whole Debian 12 release cycle (after which, assuming merged-/usr
  goes as we have recommended, maintainers can swap their logical location
  without needing a transitional mechanism any more). Implementing
  merged-/usr as the only supported layout means that moving files into
  /usr on an individual basis is mostly unnecessary, because it has no
  practical effect any more.

  Note that my opinion on this is consistent with Adrian's request
  to overrule the debianutils maintainer and require installkernel
  and run-parts to be moved back to the rootfs. We should make sure
  that whatever we decide here is consistent with the way in which we
  overrule or decline to overrule the debianutils maintainer.

> - Should maintainers of tools, e.g. debootstrap, proactively move files
>   in packages from the root filesystem into /usr, e.g. systemd system units?
>   (tl;dr: I think they should not.)

Ansgar pointed out on IRC that of course I meant to say debhelper here, not
debootstrap. The version of debhelper in git reverts the change that
previously moved systemd units from /lib/systemd/system into
/usr/lib/systemd/system, but at the time of writing that revert is not
in a release.

smcv

Bug#994275: Reverting breaking changes in debianutils

[Simon McVittie]

On Wed, 15 Sep 2021 at 01:36:26 +0300, Adrian Bunk wrote:
> 5. Programs in debianutils must not be moved to /usr unless there is
>project-wide consensus on packages doing such a move, and premature
>moving must be reverted.

This part touches on an issue we are looking at in parallel to this
(#994388: more specific advice on merged-/usr).

I don't think it's a problem that there is an overlap between the two
tech-ctte bugs, because in #994275 you're asking for a maintainer to be
overruled, whereas in #994388 I am only asking for the technical committee
to offer advice; we should vote on those separately. Obviously it would
be unhelpful if the way in which we overruled or declined to overrule the
debianutils maintainer was inconsistent with our advice to the project as
a whole, so we need to be consistent, but that shouldn't be a problem :-)

I personally think we should have a moratorium on moving the logical
location of files from the directories affected by merged-/usr into /usr,
at least until we fully understand the implications, and perhaps for as
long as the whole Debian 12 development cycle.

smcv

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

[Simon McVittie]

Package: tech-ctte
Severity: normal

As discussed in our last meeting, I think we should issue more specific
advice about merged-/usr, and in particular about what #978636 means for
maintainers right now.

Constitutionally, I'm asking the TC to use its power to offer formal
advice (Debian constitution, §6.1.5).

As for what that advice is, I'm open to suggestions, but I'm drafting
some possible wording, which I'll upload to
https://salsa.debian.org/debian/tech-ctte/ when I have a bug number
for it.

Some questions that I think we need to answer explicitly:

- What do we mean by merged-/usr? (We already said this in #914897, but
  I think it's worth reiterating that symlink farms are not it.)

- Is it OK for packages in testing/unstable during Debian 12 development
  to assume/require a merged-/usr system? (tl;dr: some maintainers think
  the answer is yes, but I think the answer is no.)

- When will it be OK for packages in testing/unstable to assume/require
  a merged-/usr system? (tl;dr: some maintainers think the answer is
  "right now", but I think the answer is "when the Debian 13 cycle begins"
  and not before.)

- Should maintainers proactively move files in packages from the root
  filesystem into /usr? (tl;dr: I think they should not, at least until
  the implications are better-understood.)

- Should maintainers of tools, e.g. debootstrap, proactively move files
  in packages from the root filesystem into /usr, e.g. systemd system units?
  (tl;dr: I think they should not.)

- Are packages built on a merged-/usr system required to work correctly
  on a non-merged-/usr system? If they are not, is it RC?
  (I think they are required to work, and it should usually be RC if
  they don't; constitutionally I don't think we can unilaterally declare
  a class of bugs to be RC, at least not while using our "advice" power,
  but we can certainly recommend that maintainers and the release team
  should consider them to be RC.)

smcv

Bug#985270: Resignation + Call for votes for new Chair

[Simon McVittie]

On Mon, 15 Mar 2021 at 10:30:59 +0100, Margarita Manterola wrote:
> ===BEGIN===
> 
> The chair of the Debian Technical Committee will be:
> 
>  A : Margarita Manterola
>  B : David Bremner
>  C : Niko Tyni
>  D : Gunnar Wolf
>  E : Simon McVittie
>  F : Sean Whitton
>  G : Elana Hashman
>  H : Christoph Berg
> 
> ===END===

I vote:

F > A = B = C = D = G > E > H


signature.asc
Description: PGP signature

Bug#982987: tech-ctte: Call for votes for new CTTE member

[Simon McVittie]

On Wed, 17 Feb 2021 at 12:45:05 -0600, Gunnar Wolf wrote:
> ===BEGIN
> 
> The Technical Committee recommends that Christoph Berg  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> A: Recommend to appoint Christoph Berg 
> B: Further Discussion
> 
> ===END

I vote A > B.

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Mon, 01 Feb 2021 at 13:35:28 +0100, Adam Borowski wrote:
> On Sun, Jan 31, 2021 at 03:34:27PM -0600, Gunnar Wolf wrote:
> > I want to state (and not as part of the vote, but just as
> > yet another DD) that the only way I feel makes sense to continue now
> > is via Simon's ① option: Symlinking /bin → /usr/bin, /lib → /usr/lib,
> > /sbin → /usr/sbin, etc.
> > 
> > That will allow us to take a step later on to mandate packages not
> > shipping files in the no-longer-root-level-directories, but that
> > should be at least one further release cycle down the lane.
> 
> I'd strongly urge the opposite order: FIRST decree that no package may
> ship any file to non-canonical path (ie, have dpkg extract anything over
> a symlink), and only then flip the switch.

I don't want to repeat myself, so please see
 for one
reason why that doesn't work (unless you have a solution that I've
missed).

smcv

Bug#975075: analogies

[Simon McVittie]

Sorry for the delay in responding to this, but I wanted to wait until
the technical question before the committee had arrived at a result
before replying to it.

On Thu, 19 Nov 2020 at 06:49:37 -0800, Felix Lechner wrote:
> In Debian, users of 'sysvinit' strike me as such a "disfavored class".

This sounds as though you're saying that Debian maintainers making
technical decisions about the packages they maintain, affecting people
who have chosen to use or contribute to Debian, are ethically or morally
equivalent to a government whose citizens have little or no choice whose
jurisdiction they live under, making laws that discriminate or persecute
on the basis of characteristics like sexuality.

I don't think that's a reasonable analogy to make, and I suspect that
members of genuinely persecuted groups would feel that it's trivializing
their situation.

I hope that wasn't your intention, but please be careful about analogies
that relate to sensitive topics.

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Mon, 25 Jan 2021 at 11:45:55 -0700, Sean Whitton wrote:
> ===BEGIN
> The Technical Committee resolves that Debian 'bookworm' should support
> only the merged-usr root filesystem layout, dropping support for the
> non-merged-usr layout.
> 
> Until after the release of 'bullseye', any implementation of this
> resolution must be done in the 'experimental' distribution, or otherwise
> kept out of the critical paths for the release of 'bullseye'.
> 
> We do not recommend any particular implementation of the migration.
> 
> Y: Yes, support only merged-usr in the 'bookworm' release.
> N: No, continue to support both layouts in 'bookworm'.
> F: Further Discussion
> ===END

I vote Y > F > N.

This is on the assumption that we are defining merged /usr the
same way we did when we resolved #914897 in

(what Guillem Jover calls "merged /usr via aliased directories", with
symlinks like /bin -> usr/bin).

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Tue, 26 Jan 2021 at 12:17:37 -0600, Gunnar Wolf wrote:
> Wouter Verhelst dijo [Tue, Jan 26, 2021 at 01:17:38PM +0200]:
> > We can (and should, IMO) declare *today* that for bookworm, shipping
> > files in / (as opposed to /usr) that are not compatibility symlinks will
> > be RC.
> 
> I agree with you

For at least the Essential packages like bash and libc6, I don't think
we can go in that order. Given that implementations of merged /usr via
aliasing symlinks already exist, I think Essential packages that ship
part of their Essential functionality in /bin, /sbin, /lib* must stay
as they are now, until after we have had a flag day (a release)
at which either:

a. those aliasing symlinks are guaranteed to be in place (Ansgar's request
   in this bug)
b. those aliasing symlinks are forbidden and must be rolled back (Guillem's
   preferred answer to this, but judging by #914897 and discussion of this
   bug, very unlikely to be the technical committee's preferred answer)

That's because these two axioms collide:

* Essential packages must provide their Essential functionality while
  unpacked but not configured
  - e.g. unpacking libc6:i386 creates /lib/ld-linux.so.2, which is Essential
functionality that other packages rely on

* There are existing installations with merged /usr via aliasing symlinks
  - Therefore compatibility symlinks (in either direction!) must be created
by a maintainer script rather than directly shipped in the
dpkg --fsys-tarfile, otherwise they will fail to unpack on those existing
systems
- e.g. if unpacking libc6:i386 creates /usr/lib/ld-linux.so.2, then it
  cannot also create /lib/ld-linux.so.2; that would have to happen
  later, when it's configured

I don't see a way to have a compromise position in Essential packages,
other than the one we have right now (where the name in the
dpkg --fsys-tarfile must match other packages' expectations, whether
that means with or without /usr), without breaking the Essential
property.

When thinking about this transition (and any contentious issue, really),
please distinguish between:

- things that (in someone's opinion, maybe yours) we *shouldn't* do;
- things that (according to properties we take as axiomatic, like the
  Essential property and the requirement that upgrades work) we *can't* do

I have been trying hard to consider all the options that are available -
even the ones that I personally think we *shouldn't* do - but immediately
dismiss the ones that we *can't* do.

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Tue, 26 Jan 2021 at 11:27:07 +0100, Bastian Blank wrote:
> Before unpacking those packages, both /bin and /lib symlinks must
> already exist, because it's past the cutoff date of non-aliased support.

I would like that to become true, but the cutoff date of non-aliased
support has not yet happened, and this bug is about making it
happen. *After* we have done that, we can consider mandating your 1b.

> Option 2 doesn't provide us any benefit.  The world already implemented
> option 1.

I personally agree, and I hope the technical committee as a whole will too,
but I wanted to describe option 2 so that we can be clear about what we're
resolving and whether option 2 is it.

> We are already effectively at 1a.  So we need to decide if we want 1b to
> fix the fallout.

We are *partially* at 1a: systems that were installed since the release
of buster are (unless steps were taken to avoid that), and systems that
have had usrmerge installed are, but other upgraded systems are not.

I'm typing this on a laptop that is still not merged-/usr, because I think
it's more likely that packages will work correctly on merged-/usr, and
if packages that I care about fail to work on a non-merged-/usr system,
I want it to happen to me, not to someone who is in a worse position to
debug it.

> > The only other option that I can see is for dpkg to gain the ability to
> > populate what we might call the "mergeable" directories (/bin, /sbin,
> > /lib*) in a purely declarative way, during unpack.
> 
> No, because we want to support /usr only systems or snapshots, where /
> is populated on first boot.

Sure, and I'm not saying that this is the option that we should take
(in fact I think we should not do this), but it's an option that exists
and is possible. I think the reasons to reject it outweigh the reasons
to take it, but I want to be aware of all the options that exist, not
just the ones that are convenient for my own plans.

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Mon, 25 Jan 2021 at 21:22:29 +, Simon McVittie wrote:
> On Mon, 25 Jan 2021 at 11:46:35 -0700, Sean Whitton wrote:
> > > The Technical Committee resolves that Debian 'bookworm' should support
> > > only the merged-usr root filesystem layout, dropping support for the
> > > non-merged-usr layout.
> 
> Should we be more specific than this in what we vote on, to avoid
> later having to adjudicate between developers who say that a particular
> implementation is or isn't merged-usr?

Sorry, I had missed that we have prior art for this. When we
resolved #914897 [1] we wrote:

## What is "merged `/usr`"

"Merged `/usr`" describes a possible future standard directories
scheme in which the `/{bin,sbin,lib*}/` directories have been made
superfluous through replacing them by symlinks to their `/usr`
equivalents (`/usr/{bin,sbin,lib*}`).

That's exactly what Guillem calls "merged /usr via aliasing", or the
"layout 1" from my previous mail.

I still think our resolution for #978636 should be clear on what we mean
by merged-usr (like the resolution for #914897 was), but this gives me
more confidence that we did indeed all intend to be voting on mandating
merged /usr via aliasing, vs. not mandating that, vs. further discussion.

smcv

[1] https://lists.debian.org/debian-devel-announce/2019/03/msg1.html

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Tue, 26 Jan 2021 at 10:30:34 +0100, Bastian Blank wrote:
> On Tue, Jan 26, 2021 at 09:01:12AM +0000, Simon McVittie wrote:
> > On Tue, 26 Jan 2021 at 08:02:06 +0100, Bastian Blank wrote:
> > > Aren't there two sub-solutions?
> > > 
> > > 1a. with packages shipping files both in /bin und /usr/bin.
> > > 1b. with packages shipping files only in /usr/bin.
> > 
> > What precisely do you mean by "shipping files"?
> 
> "We allow packages to ship files".  So either we force all packages to
> only ship files in /usr/*, instead of e.g. /bin.  Or we continue with
> status quo, where packes may use either /bin or /usr/bin, which is part
> of the current mess.

Oh, I see. So when you say "both" in 1a, you're referring to the overall
system - like the fact that we have both /bin/bash and /usr/bin/perl.

I don't see how we can force all packages to only ship files in /usr/*
(your 1b), except by either *first* moving to merged-/usr-via-aliasing
(layout 1, which in this case would have to be your 1a because it has to
happen before we can have 1b), or introducing new functionality in dpkg
and waiting another release cycle or two to make it guaranteed-to-exist.

Rationale:

* bash and libc6 are Essential
  (so are other packages, but these two are enough to demonstrate my point)
* bash has traditionally shipped /bin/bash, and this is part of its
  Essential interface (other packages ship #!/bin/bash scripts)
* libc6 ships /lib/ld-linux.so.2 and other architectures' equivalents,
  and this is part of its Essential interface
  (other i386 packages have ELF interpreter /lib/ld-linux.so.2)
* Essential packages are required to be functional after being merely
  unpacked, not configured (otherwise debootstrap can't work)

So if we unpack bash and libc6:i386, but do not configure them, /bin/bash
and /lib/ld-linux.so.2 must exist in the resulting chroot.

If that chroot is *already* mandated to be merged-/usr-via-aliasing
(layout 1), then it would be fine for bash's filesystem tarball
to contain ./usr/bin/bash and libc6's filesystem tarball to contain
./usr/lib/ld-linux.so.2, because the /bin -> usr/bin and /lib -> usr/lib
symlinks take care of keeping those packages' Essential interfaces intact.

However, if that chroot is in layout 2a or 2b, and bash's filesystem tarball
contains ./usr/bin/bash, then its Essential interface is incomplete:
there will be no /bin/bash symlink until the package is configured
(maintainer scripts are run).

I agree that your 1b is preferable *eventually*, but I think your 1a is
necessary as a stepping-stone to get there.

The only other option that I can see is for dpkg to gain the ability to
populate what we might call the "mergeable" directories (/bin, /sbin,
/lib*) in a purely declarative way, during unpack. If that isn't introduced
until Debian 12, then the packages in Debian 13 would be the first that
can rely on that feature existing.

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Mon, 25 Jan 2021 at 14:47:56 -0800, Keith Packard wrote:
> I think that and a transition plan are both key to this project. I
> recently installed Debian from scratch on my HiFive unmatched board and
> it got merged / and /usr.

That ship has already sailed: on #914897 in 2019, before Debian 10, the
TC resolved (unanimously) not to overrule the debootstrap maintainers
after they made merged /usr via aliasing symlinks the default for
new installations. We also voted on what we considered the desired
situation for Debian 11 to be, with option M "middle" (merged /usr not
mandatory, packages can be built on either merged-/usr or non-merged-/usr)
narrowly winning over option H "hard" (packages should only be built in
a merged-/usr environment).

Option W "weak" (packages should only be built in a non-merged-/usr
environment) was defeated by both those options.

> When I built packages on this device, they
> created invalid packages as the shared library dependencies all listed
> /lib instead of /usr/lib.

That shouldn't normally be the case, because shared library dependencies
are done in terms of SONAMEs and package names rather than in terms of
absolute paths (unless you are using RPATHs or RUNPATHs). One of the
variations tested by reproducible-builds.org is that it builds a package
on merged and unmerged /usr, so for any package that is reported to be
reproducible on that infrastructure (there are a lot of them), it
doesn't matter.

This sometimes requires forcing a canonical path via something like
--with-dbus-daemon=/usr/bin/dbus-daemon if the package's configure scripts
would normally detect an absolute path via PATH search and hard-code the
result; in my experience it has normally been executables rather than
libraries that have this problem, because ld.so and shlibs/symbols files
already provide an abstraction layer over the precise physical location
of shared libraries. (#913729 is a typical bug of this class.)

If you are doing lower-level things with libraries, or if you are shipping
libtool .la files or something else that hard-codes the absolute path to
a library, then, yes, this could be a problem with that specific package.

The resolution of #914897 says this is currently considered a bug in
those packages, although if we complete a transition to all supported
Debian systems being merged /usr via aliased directories (layout 1)
then these bugs cease to be bugs.

> Any plan where a transitioning system cannot be used for ongoing Debian
> development seems unworkable to me -- it leaves all active Debian
> developers working on un-transitioned systems, which greatly reduces
> test coverage from people in the best position to help fix things.

The resolution of #914897 was that we consider both transitioned and
un-transitioned systems equally valid for ongoing Debian development, and
any package that builds differently in those circumstances has a bug.

I for one have been continuing to use un-transitioned systems for the
opposite of the reason you are: to give that configuration more test
coverage, because it is *more* likely to exhibit bugs!

The classes of bugs "a file is /usr/bin/x but another package refers to
/bin/x" and "a file is /bin/y but another package refers to /usr/bin/y"
cease to be visible with merged /usr via aliasing, and can only affect
systems where /bin and /usr/bin are distinct. One of the motivations for
merged /usr is to have those classes of avoidable bugs cease to be bugs
at all: if /bin literally *is* /usr/bin on every supported Debian system,
then it doesn't matter which name you use for it, because both paths
are equivalent anyway.

However, if we want that (which I think we do), we need to get there
from here.

> I don't understand the value of 2b; it's functionally similar to layout
> 1, but makes for additional work to create the shadow links, along with
> consuming space for them.

Right. I wanted to describe this layout so that we can be clear
about whether the TC considers it to be a valid implementation of our
recommendation. If we don't, then hopefully we can avoid anyone arguing
that we have mandated this additional work.

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Tue, 26 Jan 2021 at 08:02:06 +0100, Bastian Blank wrote:
> On Mon, Jan 25, 2021 at 09:22:29PM +0000, Simon McVittie wrote:
> > Some developers seem to be using "merged /usr" to refer to multiple
> > concrete layouts:
> > 1. an arrangement where all regular files that have traditionally been
> >in /bin, /sbin, /lib and /lib64 are physically located in /usr,
> >with symlinks /bin -> usr/bin, /sbin -> usr/sbin, /lib -> usr/lib,
> >/lib64 -> usr/lib64
> >(this is what usrmerge, debootstrap --merged-usr and Fedora do; dpkg
> >maintainer Guillem Jover refers to this as "merged /usr via aliased
> >directories" which seems like a good unambiguous term)
> 
> Aren't there two sub-solutions?
> 
> 1a. with packages shipping files both in /bin und /usr/bin.
> 1b. with packages shipping files only in /usr/bin.

What precisely do you mean by "shipping files"?

If the content of the .deb (as in dpkg --fsys-tarfile) included both
/bin/bash and /usr/bin/bash, then the package would fail to install on
systems where the aliasing symlinks already exist (unless there was a
special case for this situation in dpkg, which would require at least one
more release cycle before we could do it). Given that Debian 10 systems
with the aliasing symlinks already exist, I think we can probably
rule that out as not a practically available solution for Debian 12.

If the content of the .deb only included /usr/bin/bash, relying on an
externally-constructed /bin -> usr/bin symlink to ensure that the path
/bin/bash is resolvable, then that's the logical conclusion of layout 1,
but is not something we can do until *after* there has been a release
in which layout 1 is the only thing we support (with systems not already
in that layout undergoing a migration step whose precise implementation
is outside the TC's scope - usrmerge.deb is one implementation of that
migration step, but perhaps someone will design something better).
I believe Ansgar's goal in opening this bug was to make Debian 12 the
first release in which layout 1 is the only thing supported, allowing
the transition to this form of .deb content to start (and maybe finish)
during the Debian 13 cycle.

If the content of the .deb only included /usr/bin/bash, with a maintainer
script to create /bin/bash if the aliasing symlink /bin -> usr/bin does
not already exist, then that's compatible with either 1, 2a or 2b - but
in the case of Essential packages that have traditionally installed files
to /bin, /sbin or /lib, it's incompatible with the traditional layout
with a distinct /bin and /usr/bin (etc.), due to the extra requirements
placed on Essential packages.

I think at least the Essential subset needs to continue to have files in
/bin, /sbin, /lib in their dpkg --fsys-tarfile until *after* a transition
to (some form of) merged /usr has been completed, otherwise Essential
packages that have merely been unpacked and not configured will not
be providing their Essential functionality at paths like /bin/bash and
/lib/ld-linux.so.2 that other packages rely on.

But perhaps I misunderstood you and you mean something else?

smcv

Bug#978636: move to merged-usr-only?

[Simon McVittie]

On Mon, 25 Jan 2021 at 11:46:35 -0700, Sean Whitton wrote:
> > The Technical Committee resolves that Debian 'bookworm' should support
> > only the merged-usr root filesystem layout, dropping support for the
> > non-merged-usr layout.

Should we be more specific than this in what we vote on, to avoid
later having to adjudicate between developers who say that a particular
implementation is or isn't merged-usr?

Some developers seem to be using "merged /usr" to refer to multiple
concrete layouts:

1. an arrangement where all regular files that have traditionally been
   in /bin, /sbin, /lib and /lib64 are physically located in /usr,
   with symlinks /bin -> usr/bin, /sbin -> usr/sbin, /lib -> usr/lib,
   /lib64 -> usr/lib64
   (this is what usrmerge, debootstrap --merged-usr and Fedora do; dpkg
   maintainer Guillem Jover refers to this as "merged /usr via aliased
   directories" which seems like a good unambiguous term)

2. an arrangement where all regular files that have traditionally been
   in /bin, /sbin, /lib and /lib64 are physically located in /usr,
   with /bin etc. becoming "symlink farms" containing symlinks like
   /bin/sh -> /usr/bin/sh, /lib/ld-linux.so.2 -> /usr/lib/ld-linux.so.2
   and so on
   2a. in one version of this, only those files that traditionally
   existed in the rootfs have symlinks, so that /bin/sh -> /usr/bin/sh
   exists but /bin/perl -> /usr/bin/perl does not
   2b. in another version of this, every file in /usr/bin, /usr/sbin,
   /usr/lib is represented in the symlink farm, so that
   /bin/perl -> /usr/bin/perl exists

I think we should choose wording to vote on such that if we vote yes,
it is clear which of these layouts are consistent with that resolution,
and which are not.

Guillem considers layout 1 to be broken and a mess. I don't agree,
but I'm not a dpkg maintainer. We should be aware that mandating this
implementation is likely to require some overruling.

I don't consider 2a to be merged /usr: it only does half the job of making
the directories in the rootfs equivalent to the directories in /usr.
For example, we would be able to run (for example) /usr/bin/sh scripts
developed on distros that don't distinguish, but not /bin/perl scripts.
However, it does do half the job, and I think Guillem might be considering
2a to be an implementation of the general concept of merged /usr?

I'm not sure whether *I* consider 2b to be merged /usr or not. It makes
the directories on the rootfs exactly equivalent to those in /usr, but
retains a lot of complexity. I would personally be willing to tolerate
that as a stepping-stone to reach layout 1, if the people who do the
work consider it to be less bad than doing the equivalent of usrmerge,
but I don't like it as a post-bullseye end goal in its own right.

If we're voting for layout 1 / status quo / FD, then that could be worded as:

The Technical Committee resolves that Debian 'bookworm' should support
only the "merged-usr via aliased directories" root filesystem layout
in which /bin, /sbin and /lib are symbolic links to the corresponding
directories in /usr, dropping support for the non-merged-usr layout.

Or if we're saying "either 1 or 2b":

The Technical Committee resolves that Debian 'bookworm' should support
only a merged-usr root filesystem layout in which /bin, /sbin and
/lib are either symbolic links to the corresponding files in /usr,
or directories containing such symbolic links, with every file
/usr/bin/foo also available via the path /bin/foo and so on.

Or if we're saying "either 1 or 2a or 2b":

The Technical Committee resolves that Debian 'bookworm' should support
only a merged-usr root filesystem layout in which /bin, /sbin and
/lib are either symbolic links to the corresponding files in /usr,
or directories containing such symbolic links.

(I'm assuming that nobody will try to argue that /lib64, /libx32 and other
libQUAL directories should be treated differently from /lib, even if I
don't explicitly enumerate them.)

Sorry to derail this but I think we should be clear about what we're
resolving.

I think this is consistent with not recommending any particular
implementation of the migration path, which I'm taking to mean we don't
mind how existing unmerged installations reach the desired state - e.g.
if we recommend layout 1, you could get there by installing usrmerge.deb,
or via a special-case in dpkg, or any other mechanism that arrives at
layout 1.

smcv

Bug#975075: tech-ctte: non-systemd dependencies in non-NM packages

[Simon McVittie]

On Sat, 02 Jan 2021 at 18:36:23 +, Matthew Vernon wrote:
> While [lowering NM's dependency on libpam-systemd from Depends to
> Recommends] does address the co-installability of network-manager with
> elogind, I would like you to still say something officially about the issue,
> please, since this is not the only affected package.
> 
> As an example, bug 923387 (which Michael is also maintainer of) for udisks2,
> where the dependency must be a Depends not a Recommends (so the workaround
> used for network-manager won't help).

If you intend the scope of this bug to involve overruling maintainers'
decisions in packages other than NM, what other packages/bugs did you
have in mind? Is it just udisks2/#923387, or are there more?

Speaking only for myself as an individual TC member, rather than speaking
for the whole TC, but I suspect others feel similarly:

I wouldn't want to give a ruling that would be interpreted as precedent to
(effectively) overrule multiple maintainer decisions (whether they're
decisions by a single maintainer in multiple packages, or multiple
maintainers' decisions in multiple packages) without at least being aware
of how many packages and decisions we are affecting - similar to the
principle that when the Policy editors add a new normative requirement,
they usually want to know how many packages were compliant with the old
Policy but are considered non-compliant under the new Policy.

>From the original request, it seemed to me that network-manager was
considerably more important to you than other packages with systemd
dependencies, and so you were prioritizing a request to overrule that
particular maintainer decision, so we focused on network-manager. I'm
sorry if that was a misinterpretation.

Perhaps you were hoping we would give you a very general ruling like
"dependencies on libpam-systemd (must|should) always be replaced by
default-logind | logind" that is immediately applicable to all the
other packages you are interested in, and would have an effect similar
to overruling decisions in the other affected packages too, without us
having to specifically vote (with a 3:1 supermajority) to do so.

However, I think we would be reluctant to give a general ruling like that,
because it would not always be correct. systemd is quite featureful and
its interface is quite large (if I understand correctly, this is one of
the major things that people who would prefer a different init system
dislike about it). Different packages use different subsets of that
interface, sometimes larger[1] than the subset that can be provided by
elogind (because elogind closely resembles systemd-logind, and by design
the other parts of systemd's interface are outside its scope). The extent
to which the various parts of the interface are required by the dependent
package (whether they would be Depends, Recommends, Suggests or nothing,
if they were represented by separate dependencies) also varies.

dbus-user-session is one concrete example of a package that needs a
working `systemd --user` instance per uid (a per-uid user-service
manager), and so will not do what its (informal) specification
says if it is forced to be installed on non-systemd systems -
so replacing its libpam-systemd dependency with
"default-logind | logind" would be incorrect. If that dependency
was replaced, the replacement would have to include something like
"libpam-systemd | libpam-start-dbus-daemon-per-uid", where the latter
doesn't currently exist.

smcv

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

On Mon, 14 Dec 2020 at 22:56:57 +0100, Philip Hands wrote:
> Could I just check if there's a point of common acceptability which both
> sides of this discussion could live with?
>
>   libpam-systemd | network-manager-nonsystemd

Presumably this is an optimized form of what we perhaps conceptually mean:

default-logind | logind,
systemd-sysv | network-manager-nonsystemd,

i.e. it needs a logind implementation (preferably our default, which we
happen to know is systemd's), plus either systemd or some glue to support
running the daemon without systemd?

(I agree that the optimized form is clearer and probably less work for
apt's resolver than the unoptimized form, as long as systemd being our
default is not in doubt - but I'm mentioning the unoptimized form because
I think it might illustrate the motivation better.)

smcv

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

On Sun, 13 Dec 2020 at 14:38:24 -0700, Sean Whitton wrote:
> The dependency issue is more challenging.

As I said earlier in the thread, if we don't want to overrule on the
logind dependency, then I don't think overruling on the init script
would make any sense (whereas overruling on the logind dependency but
not the init script maybe would - that would effectively mean we were
asking sysvinit users who want to use NM to find a way to maintain an
init script out-of-band).

However, I think it's still worth talking about the init script, because
most of what I'm saying is non-NM-specific.

> Participants in the thread who have argued on that side of the
> discussion seem to implicitly rely on the idea that a package maintainer
> is responsible for applying an equally high level of quality assurance
> to every file or feature in their package, as that which they would
> apply to its most basic or flagship features.

Ordinarily, perhaps yes. However, for whatever reason, people are
particularly emotionally attached to particular init systems (or perhaps
to the absence of systemd). We can see this here already: I don't think a
dependency on a particular httpd or email server would have been brought
to the technical committee asking for a maintainer to be overruled,
and it seems unlikely to have had participants describing a maintainer
declining an NMU as an outrage.

If NM's LSB init script was present, but stopped working - perhaps
because upstream deliberately made more use of systemd facilities, or
because upstream accidentally relied on systemd facilities due to none of
the upstream developers using anything else, or because the command-line
syntax changed and the upstream-provided systemd unit was updated but the
downstream init script wasn't - what do we expect to happen?

In the cases where the regression was accidental, ideally, the answer
would be someone calmly and politely offering a tested patch, but it
sadly seems equally likely to result in hostility, and I think it's
reasonable for a maintainer to try to avoid that preemptively by making
it clear that the LSB init script is someone else's responsibility.
Our volunteers are not automata, and I think maintainers need to be able
to set boundaries for their responsibilities to protect their mental state.

Similarly, in the case where the dependency is deliberate, I don't
think we want the responsibilities of a Debian maintainer to include
interceding between angry sysvinit users and upstream.

> We want maintainers to
> perform testing which is adequate to ensure that the core features of a
> package won't regress if they upload a new version, but we do not take
> maintainers to be responsible for testing every optional feature and we
> accept that such things might be temporarily broken until someone other
> than the maintainer can provide a patch.

I think perhaps you have higher expectations of bug reporters than I do
- perhaps because I'm involved in triaging/maintenance for user-facing
desktop packages. Bug reports are not always accompanied by patches,
and somewhat frequently come with the implication (or even an explicit
demand) that the maintainer must stop whatever it is they are doing and
fix the bug immediately.

In the case of init system integration, it isn't completely clear what
the severity of "NM doesn't work with sysvinit" would be, and proponents
of sysvinit might argue for critical because losing network connectivity
effectively breaks the whole system in some cases, or serious because
the package should be able to work without its non-mandatory dependencies.
RC bugs are one of the few places where the project puts specific
expectations on maintainers.

Conversely, there's also a reasonable argument for important, normal
or wishlist, because sysvinit is one of multiple options - but getting
into an argument over bug severity is still getting into an argument,
and for developers who don't enjoy conflict, takes significant "spoons"
to deal with.

For what it's worth, if we look at the results of GR 2019-002, and
ignore the winning option B because it did not specify the severity
of bugs about requiring a non-default init, we can see that option F
(non-default init support is wishlist) was voted ahead of options A
(non-default init support is important), D and H (non-default init support
is RC if a patch is available), and E (LSB init scripts are required,
presumably meaning RC). The design of our voting system is (meant to be)
such that we can draw conclusions from how options other than the winning
option were ranked, not just from the winning option vs. all the rest.

> We do not expect maintainers to maintain
> an environment to test their package on ports architectures before
> uploading, but we do expect them to apply patches provided by porters
> who discover regressions.

I don't think that's completely true. If a maintainer considers a
proposed patch to be unsuitable, we normally accept their judgement;
and if a proposed pa

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

On Wed, 18 Nov 2020 at 17:33:26 +, Matthew Vernon wrote:
> #921012 is about changing network-manager to Depend upon "default-logind |
> logind" rather than libpam-systemd

This is a change that is *sometimes* appropriate, but sometimes not,
depending on what facility the dependent package intends to receive
from the dependency. It needs to be assessed on a case-by-case basis,
and cannot be done mechanically across the distribution.

default-logind | logind is appropriate if the package's needs are
limited to "something that looks sufficiently similar to systemd-logind"
(like for example policykit-1, where I applied exactly that change),
but is not appropriate if it needs other systemd-specific facilities
(like for example dbus-user-session, which currently needs a working
`systemd --user` and has no way to function correctly otherwise).

I haven't fully investigated what facilities NM requires from
systemd. According to #921012 it requires hostnamed in its default
configuration, and according to the Gentoo wiki it expects to see
/etc/machine-id for DHCPv6. There might be others.

> #964139 is about restoring the removed network-manager init script which was
> removed from the package.

There's some good discussion about this elsewhere in the thread, in
particular around putting init-system integration in a place where the
maintenance responsibility and effort rests with those who are interested
in supporting the relevant init system.

I think we have three options:

* overrule the NM maintainer on both #921012 (logind dependency) and
  #964139 (init script inclusion) as you ask;
* decline to overrule the NM maintainer on either point;
* overrule the NM maintainer on #921012 but do not overrule on #964139, and
  instead expect the init script to be provided by some other package that
  is maintained by people who are interested in non-systemd init systems

I don't think it would make any sense to overrule on #964139 but
not on #921012, because if NM depends on libpam-systemd (which
requires/assumes systemd as pid 1), then people who want to use non-default
init systems will remain equally unable to use NM. Do you agree?

If NM is only compatible with non-systemd init systems when
placed in a non-default configuration, then it might make sense
for it to have a dependency on something like
"systemd-sysv | initscripts-network-manager", where
initscripts-network-manager provides an LSB init script and
also provides configuration fragments that configure NM
in a non-default way that does not require systemd (for example providing
the configuration fragment mentioned in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921012#72).

> Both changes are necessary for it to be possible to install network-manager
> on a sysvinit system; it is going to be hard to use Debian without
> network-manager.

It's a popular package, but to put this in perspective, this is the same
network-manager for which a previous technical committee overruled the
GNOME maintainers to require that the GNOME metapackages must not have
a hard dependency on it. Now, a lot can change in 8 years, but even so
it seems like a stretch to argue that Debian is sufficiently hard to
use without NM that having non-default init systems exclude use of NM
is necessarily a showstopper for the ability to experiment with those
non-default init systems?

We can't expect non-default init systems to be as closely integrated as
the default, and diverging from defaults is always going to involve some
compromises and reconfiguration, particularly defaults that are low down
in the dependency stack.

Alternatives include at least connman, wicd (not in testing because it
requires Python 2, but I'm sure interested developers could constructively
help to push it forward), and ifupdown. (Also systemd-networkd, but that
expects systemd as pid 1, and given its upstream maintenance model it
would seem unreasonable to expect a downstream maintainer to change that.)

smcv

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

On Sat, 21 Nov 2020 at 10:47:17 +0200, Wouter Verhelst wrote:
> On Thu, Nov 19, 2020 at 09:04:07AM +, Matthew Vernon wrote:
> > 1) poor user experience - a package of initscripts would clutter
> > /etc/init.d/ with a huge number of files (most of which would be no use to
> > the user)
>
> This could be fairly easily fixed.
> 
> Option one:
> - Don't install the init scripts in /etc/init.d, but install them in
>   /usr/share somewhere.
> - Install a dpkg trigger that uses ucf to copy the relevant init scripts
>   /etc/init.d (and goes through the rest of the dance to enable them)
>   when the relevant package(s) is/are installed.

Or someone could teach sysv-rc or insserv to look for init scripts in
/usr/share/init.d, with an init script in /etc/init.d of the same name
taking precedence; or the init script in /etc/init.d could be a symlink
to one in /usr/share by default, with the sysadmin able to replace it with
a regular file if they want to modify it; or something along those lines.
I'm sure people who prefer to use LSB init scripts can work something out.

It occurs to me that something along these lines would largely solve one
of the ongoing design issues with LSB init scripts (and cron jobs, and
other interfaces based on dropping an arbitrary executable script into a
designated place in /etc): the fact that they are conffiles, and therefore
remain in /etc when their package is removed but not purged. This can
break other related or unrelated packages if the script is not correctly
guarded by "if [ -x /usr/sbin/something-relevant ]", and is difficult
to fix reliably with a package update: script maintainers can of course
fix their script to have that guard at any time, but users affected by
the bug will not receive the fixed script, because the script's package
is no longer installed and so will never be upgraded to a fixed version.

I think I've seen several occasions on which an obsolete init script
(or cron job or hook or other integration script) remaining on the system
triggered RC bugs during upgrade, leading to some related package having
to have (non-Policy-compliant!) glue code to remove or otherwise defang
the integration script so that the upgrade could proceed.

Having that guard also means the script still has to be executed in
order to figure out that it doesn't need to do anything, which is a
shell invocation that didn't need to happen.

Now, this is clearly not the way LSB init scripts have traditionally
worked in Debian. However, I note that the winning option in GR 2019-002
says we will "support exploring alternatives". It does not say that we
will preserve sysvinit, LSB init scripts and sysv-rc in precisely the way
they have traditionally worked in Debian - after a couple of decades I
would hope that we have already adequately explored sysv-rc, and have
a reasonably clear picture of its advantages and disadvantages. If
people want to continue to use it, addressing or mitigating the
disadvantages seems like part of the task of maintaining it and keeping
it fit-for-purpose.

> This puts the burden of maintaining said init script on the maintainer.
> In my reading of the GR, that's not expected of maintainers anymore.
> 
> I think it's fair for a maintainer to be expected to file a bug on an
> "init scripts" package if their CLI changes. The rest should then be the
> job of the maintainers of that init scripts package.

That's my understanding too, and Wouter's mail continues by making good
points about this topic.

If we are unable to use particular system services (in this case NM) with
a particular non-default init system without putting extra responsibility
on the maintainers of those system services, then I think that's a
limitation of that init system that its maintainers could usefully
address, and addressing that limitation would certainly be within the
scope of exploring alternatives to systemd.

smcv

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

On Thu, 19 Nov 2020 at 21:04:00 -0800, Elana Hashman wrote:
> It would be much appreciated if Michael Biebl or another maintainer from
> the Utopia team could add some context here.

Most of the people in the pkg-utopia team are not active contributors to
most of its packages, so I don't think soliciting comment from members of
the pkg-utopia team is necessarily going to be particularly enlightening.
Looking at the Uploaders and commit history, Michael is very much the
primary maintainer of NetworkManager and its adjacent packages.

I am technically a member of that team myself, but I don't feel that I
can comment on Michael's decisions or the reasons for them, and the one
time I contributed a patch to NM, I behaved the same way I would have
done for an entirely unrelated package. The "utopia" name is a relic of
the team's origins in packaging "Project Utopia", a loose umbrella for
projects aiming to improve device management and hotplugging in Linux
generally and GNOME specifically, most of which were subsequently
superseded (in particular HAL, the central Project Utopia package,
which was replaced by udev improvements for the lower-level parts and
multiple more-focused higher-level projects like upower, udisks and
arguably parts of systemd for the higher-level parts).

If the utopia team didn't already exist and a team was needed for what it
maintains, it would probably be the freedesktop team (which also exists,
with a significant overlap in membership).

smcv

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

On Thu, 19 Nov 2020 at 11:40:20 +, Ian Jackson wrote:
> My view is
> that the behaviour seen in #921012 and #964139 is an outrage

I don't think this is constructive, and if a package's maintainer doesn't
want to enter into conversations, this doesn't seem like an approach
that will change their mind.

Matthew opened this thread with a message that focused on technical points
and avoided emotive language, and I would like to keep the discussion on
that basis. The technical committee's responsibility is to make decisions
on their technical merits, and I'm sure you don't think that we are or
should be more likely to take your side because you have expressed outrage.

If you feel that the community team or DAM needs to be involved, they're
available to be contacted (perhaps they have been already), but this
pseudo-package is not their issue tracker and this bug is not theirs
to resolve.

Thanks,
smcv

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

Some procedural points, without going into the merits of the technical
committee doing as you ask or not:

On Wed, 18 Nov 2020 at 17:33:26 +, Matthew Vernon wrote:
> I invite the technical committee to rule that:
> * The network-manager init script should be restored
> * Network-manager should Depend: on default-logind | logind rather than
> libpam-systemd

This looks like a request to use the technical committee's power to
overrule the maintainer of network-manager under section 6.1.4 of the
Debian constitution. Is that what you are asking for?

This is clearly something that the technical committee is empowered to
do, if we choose to do so (which requires a 3:1 majority in favour of
overruling the maintainer).

> * Similar init-compatibility changes should not be blocked by package
> maintainers unless they are defective on their own merits

This seems like a broader request, and it's less clear which of the
technical committee's powers you are asking us to use. Please could
you either clarify what you are asking us to do, or reduce the scope
of this request to the issue that you are immediately concerned with,
namely network-manager?

If what you want on a relatively short timescale is for the maintainer
of network-manager to be overruled, and the timing is important to you
because of the imminent bullseye freeze, then I would suggest that it
might be more expedient to turn this tech-ctte bug report into a request
to overrule the network-manager maintainer, and put aside the broader point
for now. If that's the case, retitling it to something more like

tech-ctte: Overrule network-manager maintainer to apply non-systemd init 
patches

would seem appropriate.

As Josh has said elsewhere in the thread, we can give advice (6.1.6),
we can decide matters of technical policy (6.1.1), and we can tie-break
on technical matters where developers' jurisdictions overlap (6.1.2);
but we cannot overrule the views of Debian's membership as a whole, as
expressed by GRs. Elsewhere in this thread there has been some dispute
over whether it is the TC's place to make this resolution. If we can
stop thinking about that question, then it seems to me that we are more
likely to be able to answer the other request on the timescale you need.

Obviously, when we decide to overrule or not overrule the network-manager
maintainer, that can certainly give maintainers of other packages
in a similar position a hint as to what the likely outcome would be
if there was a request to overrule *them* - but different packages'
circumstances are different, so our decision would not necessarily always
go the same way.

For example, if the maintainers of dbus and gnome-initial-setup chose
to make them depend on systemd and you asked the technical committee to
overrule both decisions, I suspect we would be more likely to overrule
on dbus (because losing dbus would make a lot of packages, uses and
high-level features unavailable to those who want to experiment with
other init systems), and less likely to overrule on gnome-initial-setup
(because that's just one feature).

(With dbus maintainer hat on, I do not intend to remove its LSB init
script, and I certainly don't intend to remove the init script and then
use TC powers to require myself to put it back! :-) It's just an example
of a package for which losing non-systemd init support would have a wider
impact on users of non-systemd init.)

smcv

Bug#975075: tech-ctte: Should maintainers be able to block init compatibility changes?

[Simon McVittie]

On Sat, 21 Nov 2020 at 10:42:04 +0100, Vincent Bernat wrote:
> I don't think this is very common. Init scripts are very specific to a
> distribution. A Debian init script cannot be used for Redhat. A SUSE
> init script does not work with Redhat. I find doubtful that the
> compatibility would be better with the BSD init scripts (this may not be
> what you meaned). AFAIK, OpenBSD does not use initscripts. A FreeBSD
> initscript is unlikely to work on any Linux as it sources /etc/rc.subr.
> 
> From my experience, when upstream ships an init script, this is usually
> unsable by most distributions (not to the standard), so it has to be
> rewritten. Init scripts are not contributed upstream as upstream doesn't
> want to handle all this complexity.

For what it's worth, as an upstream and Debian maintainer of dbus, I
removed the LSB-style init scripts for Red Hat and Slackware from the
upstream source code of dbus, because nobody was testing them and so I
had no confidence that they continued to work (Red Hat moved to Upstart
and then to systemd, and Slackware used their own init script instead
of the one provided upstream, so the relevant downstreams were not even
using the versions that were part of dbus stable releases, and they
certainly weren't testing development releases).

dbus in Debian continues to have a LSB init script that was never sent
upstream; Debian is better-placed to maintain that than upstream is (even
though at the moment it would be me making changes either way!), and it
can continue to take advantage of Debianisms like start-stop-daemon.

The LSB/sysv-rc interface (execute arbitrary code, which is going to be
a lot simpler if it relies on distro-specific facilities like
start-stop-daemon) does not really lend itself to writing portable init
scripts. The few portable init scripts that I have seen have generally
had robustness issues, precisely because they bend over backwards to
avoid using distro-specific facilities.

Echoing what Vincent said, my experience has been that systemd units *can*
typically be used unmodified across distributions - partly because they
are declarative, partly because the upstream part of systemd provides
facilities to do typical system service things without having to open-code
them or rely on a helper like start-stop-daemon (for example tracking
lifetime, dropping privileges, and sending SIGTERM to terminate gracefully
or SIGHUP to reload), and partly because the "drop-in" mechanism gives
us a way to add distro overrides where needed without necessarily having
to patch the unit.

smcv

Review/testing requested for gnome-team/glib!9: cleaning up old GLib from /lib/

[Simon McVittie]

Now that GNOME 3.38 is (mostly) in testing, I've finally implemented a
fix for the GLib upgrade issues referred to the technical committee in
#911225 "Advice on stale libraries in a higher-precedence path entry"
(see #896019, #955331, #954960). Sorry for the very long delay in
implementing this.

There's an autopkgtest, which passes on both unmerged-/usr and merged-/usr
systems, but because this is deleting important libraries at a system
level, I would very much appreciate review and testing from other
developers before I release this. See:
https://salsa.debian.org/gnome-team/glib/-/merge_requests/9

Briefly, GLib in stretch was installed in /lib/, which is
higher-priority for the runtime dynamic linker than /usr/lib/.
In buster, we moved it to /usr/lib/. For reasons we still don't
understand, on a minority of systems a stale copy remains in /lib/
and is not deleted, causing versioned dependencies to break.

I have now implemented approximately what Didier suggested in the technical
committee discussion: GLib's postinst looks for copies of GLib in
/lib/. If they exist and are not managed by dpkg, and
/usr/lib//libglib-2.0.so.0 exists and *is* managed by dpkg, then
we move the stale copies into a new directory
/lib//removed-by-upgrade-bug896019 to stop them breaking
higher-level libraries.

One difference between Didier's recommendation and what I have actually
implemented is that I'm not checking the stale libraries against known
md5sums of the version of GLib in jessie. See the MR for rationale.

This bug does not affect merged-/usr systems, where the stale version can
still be present but is harmless. As a result, my workaround skips itself
on merged-/usr systems (unless run manually with --force).

This is mostly cleanup from stretch -> buster upgrades, since we don't
support skipping a version (although we could conceivably put it in a
buster point release after a *lot* of testing).

Please send any comments to the MR (preferred) or to debian-gtk-gnome. I'm
cross-posting to the technical committee to keep them in the loop, but not
opening a new tech-ctte bug for this.

Thanks,
smcv

Bug#961156: tech-ctte: Call for votes on TC membership of Elana Hashman

[Simon McVittie]

On Wed, 20 May 2020 at 17:10:26 -0300, David Bremner wrote:
> I call for votes on the following ballot to fill a vacant seat in the
> Debian Technical Committee. The voting period starts immediately and
> lasts for up to one week, or until the outcome is no longer in doubt
> (§6.3.1).
> 
> ===BEGIN
> The Technical Committee recommends that Elana Hashman  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> S: Recommend to Appoint Elana Hashman 
> F: Further Discussion
> ===END

I vote S > F.

smcv


signature.asc
Description: PGP signature

Bug#961153: tech-ctte: Call for votes on TC membership of Sean Whitton

[Simon McVittie]

On Wed, 20 May 2020 at 17:03:16 -0300, David Bremner wrote:
> I call for votes on the following ballot to fill a vacant seat in the
> Debian Technical Committee. The voting period starts immediately and
> lasts for up to one week, or until the outcome is no longer in doubt
> (§6.3.1).
> 
> ===BEGIN
> The Technical Committee recommends that Sean Whitton  be
> appointed by the Debian Project Leader to the Technical Committee.
> 
> S: Recommend to Appoint Sean Whitton 
> F: Further Discussion
> ===END

I vote S > F.

smcv


signature.asc
Description: PGP signature

Bug#947847: please install systemd-sysusers using update-alternatives

[Simon McVittie]

On Mon, 27 Jan 2020 at 11:18:53 +0100, Thomas Goirand wrote:
> you don't seem to agree that it'd be ok for one to use
> opensysuser instead of the systemd implementation if systemd is running.
> I do not agree with this, and I believe it is up to the users to decide
> what to do, even if we, as an operating system, must provide sensible
> defaults

I hope everyone involved can agree that:

- we should provide choices where their benefit exceeds their cost,
  because more choice results in a more widely applicable system;

- we should not provide choices where their cost exceeds their benefit,
  because more guarantees result in a more robust system

(By "cost" I mean developer time spent, avoidable bugs caused, etc. -
not money.)

To take some clear examples: at one end of the scale, we provide a way to
configure per-system whether /usr/bin/editor is emacs or vi or whatever,
because that's primarily about subjective, personal UI preference. At the
other end of the scale, we *don't* provide a way to configure per-system
whether /bin/tar is GNU or BSD tar: we want packages that run tar to be
able to make the simplifying assumption that on any Debian system, it's
going to be GNU tar, and we do not want to deal with bug reports from
users who have switched to an unexpected tar implementation.

So this is at least partially a question of whether making
opensysusers vs. systemd-sysusers easily sysadmin-configurable has a
cost greater or less than its benefit, or to put it another way, where
it falls on a scale between /usr/bin/editor and /bin/tar.

With that in mind: Thomas, or anyone else asking for this change, what
benefits do you expect users of Debian to obtain from being able to
install both opensysusers' implementation of the sysusers.d interface
and systemd's, and choose which one is invoked from the boot procedure
and packages' postinsts via configuration, as opposed to via package
installation/removal?

I think there are three categories of systems that it might make sense
to consider separately here. In ascending order of "amount of systemd":

- non-Linux ports to which systemd does not intend to be portable (and
  I think we can treat non-systemd-by-policy derivatives like Devuan as
  basically equivalent to these), where the systemd implementation of
  -sysusers simply does not exist

- Linux systems not booted with systemd
  (either no init system at all, like a typical schroot or Docker
  container, or a non-systemd init system like sysvinit)

- Linux systems booted using systemd

It might be helpful to consider which benefits apply to which of these
categories.

For example, the first category certainly benefits from opensysusers
*existing*, but does not directly benefit from co-installability or from
having a choice, because there is no such choice: systemd's -sysusers
is not available there, so using something non-systemd is the only option.

I think we have a fairly good picture of the costs that would be
incurred from using alternatives: more interacting code paths to test,
potentially more configurations that are technically possible but are
not considered supported, and packages with "Depends: systemd (>= 321)"
(or indeed systemd itself, in the case of systems using it to boot)
not being able to rely on having access to all systemd 321 features,
which doesn't seem like a least-astonishment situation to be in. However,
Michael, or anyone else opposing this change: if you have anything to
add to those, please do.

Thanks,
smcv

Bug#932795: How to handle FTBFS bugs in release architectures

[Simon McVittie]

On Fri, 26 Jul 2019 at 19:05:50 +0200, Santiago Vila wrote:
> The practical implications of this is that we are currently forcing
> users to spend extra money if they want *assurance* that all the
> packages (and not just "most" of them) will build, which is a pity.

I have two counterpoints to that. Please bear in mind that I'm speaking
here as someone who has rebuilt a subset of a Debian derivative on
single-core "cloud" worker machines, and encountered what I think is
the same gcc build-system bug you encountered - so I'm not denying that
a bug exists, or claiming that it is only a theoretical concern.

The first is that we aren't really discussing whether it's a bug for
a package to fail to build on a single-CPU machine - I think everyone
involved agrees that it is. We're discussing whether it is or should be a
*release-critical* bug. Making bugs RC is a big hammer: it's saying that
if the bug cannot be fixed before the next Debian release, then removing
the package is preferable to keeping it with the bug still open. This
is not something we should do lightly: I for one would prefer to have a
version of gcc that can only be built on multi-core machines rather than
no version of gcc at all. The build system bug that prevented gcc from
compiling successfully on a single-core system appears to be something
to do with Ada (in the derivative I work on, I was able to avoid it by
disabling Ada, because the derivative doesn't need that language), but I
think Debian as a project would also prefer to have a gcc with Ada support
that cannot compile on single-core systems, instead of a gcc with no Ada
support that can.

Analogously, Policy says packages should build reproducibly, and I think
everyone also agrees that it should be possible to cross-compile packages.
People open bugs (ideally with patches) when they find violations of
those desirable properties - but those bugs aren't RC, because we would
prefer to have a non-reproducible or a non-cross-buildable package than
a missing package.

Sometimes a bug is prioritized lower than its reporter would like,
for example because the package's maintainer doesn't consider it to be
very important, because no solution is known and finding a solution is
expected to be difficult relative to the severity of the bug, or because
a solution is known but its level of regression risk or release-team
distraction isn't appropriate for the current stage of a freeze. In these
situations, escalating the bug to RC in the hope that it will force the
maintainer or release team to re-prioritize it is generally considered
to be unconstructive.

My second counterpoint is that I don't think we can claim to be treating
"minimize the money spent by users who rebuild the archive" as an
important goal in general, and indeed I think it would be harmful if we
did. You could equally say that lots of other things we do are "forcing"
such users to spend extra money, including:

- building documentation at all
- building documentation from its real source code, rather than accepting
  prebuilt documentation from upstream tarballs
- running autoreconf, rather than accepting prebuilt Autotools goo from
  upstream tarballs
- building optimized code
- running build-time tests
- building various optional features
- having debug symbols
- having Haskell, Java, Ada, etc. toolchains and ecosystems
- having packages whose builds won't fit on a 20G or 50G disk
- having packages whose builds won't fit in 2G or 4G of RAM
- having more than one version of gcc
- having more than one version of LLVM
- updating packages regularly
- encouraging our users to exercise their Free Software rights, rather
  than just taking our binaries as-is
- exercising our Free Software right to modify and recompile, rather
  than just taking upstream binaries as-is

... but we do all of those things anyway, because we consider their
benefits to be greater than their costs.

smcv

Bug#934948: Unnecessary dependencies vs multiple binary packages

[Simon McVittie]

On Wed, 21 Aug 2019 at 22:59:50 +0530, Pirate Praveen wrote:
> On 2019, ഓഗസ്റ്റ് 21 7:06:34 PM IST, Simon McVittie  wrote:
> >* As far as I can tell, the command-line executable
> >`.../bin/autoprefixer`
> >  is not in the PATH. I don't know whether it is run automatically in
> >  some other way, analogous to a program in /usr/libexec.
> >  - Please fact-check: is it in the PATH? Is it run as an executable
> >in some other way?
> 
> I was not using this command personally so I was OK to remove nodejs
> dependency if it was rejected.

That doesn't answer my question.

I looked at this executable again and it seems as though its purpose is
to query metadata about the autoprefixer nodejs library, analogous to
the -config programs sometimes found in -dev packages (sdl2-config and
so on). Is that correct?

Is it run automatically by some piece of Node infrastructure (like the
way the build systems of some SDL games automatically run sdl2-config
to learn which compiler and linker options they should use for SDL),
or is it intended to be run manually by a developer, or what?

Is there any situation in which it would be run by a developer who does
not already know that they have nodejs installed?

> So in general there is two cases I want to be able to create multiple 
> binaries.
> 
> 1. Cases like node-autoprefixer if the executable is useful (in specific case 
> of node-autoprefixer , I can live without the executable for now).
> 2. Cases like ruby-task-list where there is more dependencies than the 
> interpreter.

I think these are different, and each should be considered on its own
merits, so this bug might make more sense if it is cloned (split) to offer
advice on the two different situations.

For the node-autoprefixer case, if we decide that the executable is not
sufficiently important to the overall function of the package to merit a
dependency, it's probably useful to compare and contrast with cleancss.deb
from the node-clean-css source package (which *is* a separate binary
package, and it looks to me as though that was the correct decision).

smcv

Bug#934948: Unnecessary dependencies vs multiple binary packages

[Simon McVittie]

On Sat, 17 Aug 2019 at 11:20:22 +0530, Pirate Praveen wrote:
> I'd like to bring to your notice a disagreement with ftp masters about adding
> multiple binary packages when the same source package has code targeting
> multiple environments.

While attempting to construct a summary of the situation and constraints
I realised you were talking about two separate situations, which are not
necessarily particularly similar. The correct answer might be different
in each case.

Please correct anything that I've got wrong here:

Background
==

* When I say "library" below I mean the equivalent of libfoo.so.0 or
  libyaml-perl: reusable code that can be imported into a program or another
  library but is not directly executable in its own right.

* When I say "executable" below I mean the equivalent of
  /usr/bin/dpkg-architecture: a script that is intended to be executed
  directly.

* All correctly-packaged Ruby libraries have a dependency on the Ruby
  interpreter, regardless of whether they also contain a #!/usr/bin/ruby
  executable script. This is similar to what is done for Perl and Python.
  - Please fact-check: is this true?

* Packages that contain an executable #!/usr/bin/ruby script in the PATH
  or in some location from which it will be run as an executable (like
  /usr/libexec) must have a dependency on the Ruby interpreter, otherwise
  they cannot work, which would be a grave bug.

* Correctly-packaged node.js libraries (that do not also contain an
  executable #!/usr/bin/nodejs script) *do not* normally have a dependency
  on the node.js interpreter, unlike Ruby, Perl or Python.
  - Please fact-check: is this true?

* Packages that contain a #!/usr/bin/nodejs script in the PATH or in some
  location from which it will be run as an executable must have a
  dependency on the node.js interpreter, otherwise they cannot work,
  which would be a grave bug.

* Packages that contain a #!/usr/bin/nodejs or #!/usr/bin/ruby script in
  /usr/share/doc/*/examples do not necessarily have a dependency on the
  appropriate interpreter, because it's only an example.

* Some, but not all, JavaScript libraries are suitable for use in both
  node.js and web browsers, similar to the way some, but not all, Python
  code is suitable for both Python 2 and Python 3.

* Correctly-packaged libraries of JavaScript for use in web browsers
  normally depend on javascript-common, but do not depend on an interpreter.
  - Please fact-check: is this true?

ruby-task-list
==

* The ruby-task-list source package contains (among other things)
  a Ruby library, `task_list`, and a Node.JS library, `deckar01-task_list`.

* You want to put the Ruby library in the package dictated by Ruby policy,
  ruby-task-list.deb, and the Node.JS library in the package dictated by
  Node.JS policy, node-deckar01-task-list.deb.
  - The ftp team objection to this is that both libraries are very small
(a few K each) so the resulting data:metadata ratio is unfavourable.

* The ftp team want you to combine those two into one binary package,
  ruby-task-list.deb, and give it
  Provides: node-deckar01-task-list (= ${binary:Version}).
  - Your objection to this is that users who install the combined package
have to install Ruby, even if they only want the node.js library.
This is because it contains a Ruby library, and Ruby libraries have
Depends on the Ruby interpreter.
  - You said that users who install the combined package would also have
to install node.js even if they only want the Ruby library, but I'm
not sure whether this is true? node-normalize.css, for example,
doesn't seem to have Depends on node.js?

node-autoprefixer
=

* The node-autoprefixer source package contains a "pure Javascript" library
  named `autoprefixer` that can be used from either node.js or any other
  Javascript environment, including web browsers.

* It also contains a command-line executable `.../bin/autoprefixer` which
  has #!/usr/bin/nodejs.

* As far as I can tell, the command-line executable `.../bin/autoprefixer`
  is not in the PATH. I don't know whether it is run automatically in
  some other way, analogous to a program in /usr/libexec.
  - Please fact-check: is it in the PATH? Is it run as an executable
in some other way?

* You wanted to put a copy of the library and command-line executable in
  a binary package named node-autoprefixer for use in the node.js
  interpreter (which would have Depends: nodejs because of the executable),
  and a second copy of the library for use in web browsers in a binary
  package named libjs-autoprefixer (which would not have any particular
  dependencies except javascript-common).
  - The ftp team objection to this is that, again, both libraries are
rather small.

* The ftp team recommendation was to have a node-autoprefixer package with
  Provides: libjs-autoprefixer (= ${binary:Version}).
  - Your objection to this is that users who install the combined package
  

Bug#932795: How to handle FTBFS bugs in release architectures

[Simon McVittie]

On Thu, 25 Jul 2019 at 16:39:38 +0300, Adrian Bunk wrote:
> On Thu, Jul 25, 2019 at 01:22:42PM +0200, Santiago Vila wrote:
> > The only thing it did not have was more than one CPU, but AFAIK that's
> > not something that may be considered as a misconfiguration.
> 
> For CPU-bound tasks like package building I would consider this build 
> environment a misconfiguration.

I can see your point, but some cloud-based automated build environments
that I've used at work only provide one vCPU per build. I believe this
is done because builds typically don't parallelize perfectly (a build
that takes 10 minutes on one core will usually take more than 5 minutes
on two cores, because there are single-threaded bottlenecks like the
linker or documentation toolchains), so if you have a lot of packages
to build, it's a more efficient use of limited resources to build two
packages with parallel=1 on one core each than to build a single package
with parallel=2 on two cores. This seems like it would apply equally well
whether your limited resource is the money you pay for cloud VM instances,
or the number of physical CPUs available in your own non-cloudy VM host.

(Of course, if you're optimizing for latency rather than throughput, to
get each new version deployed and tested as quickly as possible, then the
opposite is true and it's better to sprint through each package as fast as
possible on as many cores as you can get - it's a question of priorities.)

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924325 does look like
something I've encountered in my day job, although we worked around it
by disabling Ada to bypass the problematic part of the build (we don't
actually have any need for Ada in that environment, so speeding up the
build was welcome) rather than by adding more cores.

smcv

Bug#932795: How to handle FTBFS bugs in release architectures

[Simon McVittie]

On Thu, 25 Jul 2019 at 13:22:42 +0200, Santiago Vila wrote:
> The only thing it did not have was more than one CPU, but AFAIK that's
> not something that may be considered as a misconfiguration.

Roughly what proportion of Debian packages are failing to build in
this environment?

Roughly how many of the failures are failure to compile (like #924325),
and how many are failing build-time tests and would likely have built
successfully if you had been using DEB_BUILD_OPTIONS=nocheck
(like #907829)?

I'm trying to get a picture of how much of a practical problem this is,
and what the more common bug is: race conditions etc. in the actual build,
or faulty assumptions (race conditions or otherwise) in build-time tests.

smcv

Bug#932795: Ethics of FTBFS bug reporting

[Simon McVittie]

On Tue, 23 Jul 2019 at 13:54:10 +0200, Santiago Vila wrote:
> Ethics of FTBFS bug reporting

I don't think framing this as a question of ethics is necessarily
helpful. When people disagree on a technical question, a recurring
problem is that both "sides" end up increasingly defensive, arguing
from an entrenched position, and unwilling to be persuaded. Using terms
that the other "side" is likely to interpret as an accusation of being
unethical seems likely to exacerbate this.

> I reported this bug:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=907829
> 
> and it was downgraded on the basis that the official autobuilders
> are multi-core.

Do I understand correctly that you are asking the TC to exercise our
power to overrule developers, in order to overrule the maintainer's
and/or the release team's judgement about the severity of (bugs like)
#907829?

Or are you asking the TC for advice, or are you asking us to use a
different one of the TC's powers?

> * The informal guideline which is being used, "FTBFS are serious if
> and only if they happen on buildd.debian.org", is not written anywhere
> and it's contradictory with Debian Policy, which says "it must be
> possible to build the package when build-essential and the
> build-dependencies are installed".

I had always interpreted the informal guideline as: FTBFS on the official
buildds of release architectures are always serious, because they mean we
can't release the package; FTFBS anywhere else (non-release architectures,
unofficial buildds, pbuilder, etc.) *might* be serious, but might not,
depending on how "reasonable" the build environment is.

There are many aspects of a build environment that might be considered
reasonable and might not, and they are generally evaluated on a
case-by-case basis. A working build environment needs "enough" RAM (a
lot more for gcc or WebKit than for hello); it needs "enough" disk space
(likewise); it needs a writeable /tmp; it needs a correctly-installed
Debian toolchain (I hope you wouldn't argue that it's RC if a package
FTFBS with a patched gcc in /usr/local/bin); it needs to be on a
case-sensitive filesystem (I hope you wouldn't argue that FTBFS on
FAT/NTFS/SMB would be release-critical); it needs to not have weird
LD_PRELOAD hacks subverting its expectations; and so on.

We also have packages that FTBFS (usually as a result of test failures)
when built as uid 0, when built as gid 0, when built with not enough
CPUs, when built with too many CPUs (a lot of race conditions become more
obvious with make -j32), when built in a time zone 13 hours away from UTC,
when built on filesystems that don't provide the FIEMAP ioctl, when built
on filesystems that don't have sub-second timestamp resolution, and many
other failure modes. Clearly these are bugs. However, not all bugs are
equally serious. For example, we've managed to release the glib2.0 package
for years, despite it failing to build when you're uid 0 (a test fails
because it doesn't expect to be able to exercise CAP_DAC_ADMIN), because
we consider building as uid 0 to be at least somewhat unreasonable.

If build-time test failures are always RC, however usual or unusual the
build environment, then one rational response would be for all maintainers
to disable build-time tests (if they are particularly conscientious,
they might open a wishlist bug, "should run tests", at the same time
as closing a RC bug like "FTBFS due to test failure when gid == 0"). I
don't think that is a desirable outcome. We are not building packages for
the sake of building them, but so that they can be used - which means we
should welcome efforts like build-time tests that improve our confidence
that the package is actually usable in practice, and not just buildable,
and try to avoid creating incentives to remove them.

For the specific question of whether a single CPU core is a "reasonable"
build environment, my answer at the moment is "I don't know".

> * Because this is a violation of a Policy "must" directive, I consider
> the downgrade to be a tricky way to modify Debian Policy without
> following the usual Policy decision-making procedure.

The wording of the serious severity is that it is a "severe" violation
of Debian Policy, which is qualified with "(*roughly*, it violates a
"must" or "required" directive)" (my emphasis). This suggests that there
can exist Policy "must" violations that are not RC.

The release team are the authority on what is and isn't RC: the fact that
serious bugs are normally RC is merely convention. However, I suspect
that the release team would not welcome being asked to add -ignore
tags to serious bugs that describe non-severe Policy "must" violations,
and would ask the package's maintainer to downgrade the bug instead.

> To illustrate why I think this guideline can't be universal, let's
> consider the case (as a "thought experiment") where we have a package
> which builds ok with "dpkg-buildpackage -A" and "dpkg-buildpackage -B"
> but FTBFS when built

Bug#923450: tech-ctte: requirements for being pre-dependency of bin:init

[Simon McVittie]

On Thu, 28 Feb 2019 at 12:06:15 +, Dmitry Bogatov wrote:
>   currently, inclusion of new init system into Debian requires inclusion into
> pre-dependency of bin:init package, which is unsatisfactory.
> 
> As can be followed in #838480, current practice puts maintainer of any
> new init system to be included into Debian at disadvantage, essentially
> granting maintainer of bin:init (which happens to be `systemd' team now)
> veto right. According to Constitution 3.1, individual developer poses
> power of descision making only with regard their /own/ work.
> 
> Hereby, I ask TCCE to provide requirements init system must satisfy that
> /warrants/ inclusion into pre-dependency of essential bin:init.
> Alternatively, I ask to consider introducion of virtual package
> 'init-system'.

We discussed this at last month's technical committee meeting (sorry for
the delay in replying - I've been busy IRL, and so was the person previously
asked to reply).

Committee consensus
---

Yes, we agree that the maintainers of the init-system-helpers package
(which builds the init binary) have a gatekeeper role for making init
systems as straightforward to install to /sbin/init as e.g. sysvinit is.
We don't consider this to be a problem. They seem to be applying due
scrutiny and care to this process, based in part on experience that they
gained during systemd's time as an unsupported/non-default init system.

We would recommend that the init-system-helpers maintainers should
document their requirements for being an alternative pre-dependency of
init, for instance in init's README.Debian. I'll mention some possible
criteria further down this mail, but those are not part of the committee
consensus (and detailed design work is, constitutionally, not a function
of the technical committee, although some of us might be able to provide
useful suggestions in our personal/non-TC roles, and I have tried to
do so).

It is entirely possible to include init systems in Debian
without being pre-dependencies of init: for instance, at the time
systemd was made available as a non-default init during the wheezy
release cycle, the recommended mechanism to use it was to boot with
init=/bin/systemd. People who are interested in trying new init systems
that are not pre-dependencies of init can do the same, and we would
recommend structuring runit packaging to make this possible if it isn't
already. (For example, systemd was available and worked well in wheezy,
but was most straightforward to test by using init=/bin/systemd; the
systemd-sysv package, which makes systemd provide /sbin/init, couldn't
be installed without applying force until part way through the jessie
release cycle.)

Another way to use an init system that is not a pre-dependency of init
is to override the Important flag and remove init. I think we have a
consensus that we consider this to be a feature, not a bug: it provides an
indication that this particular non-default init system is unlikely to
be suitable for people who do not know for sure that they want it.

We do not consider an init-system virtual package to be a suitable
solution here.

My personal suggestions
---

As a general operating principle, I think adding a new init system to
the pre-dependencies of init should only be done when the init system is
definitely entirely ready for general use, and has already been tested
on a variety of systems by multiple people (by alternative mechanisms
like booting with init=/sbin/my-init or overriding the Important flag to
uninstall the init package); so modifying init should be the last step in
enabling a new init system. I don't think there should be a presumption
that Debian should consider every init system that has been packaged to be
equally important or equally strongly supported, and if it isn't obvious
that a new init system is ready for wide use, then it probably isn't.

Some criteria that the init-system-helpers maintainers might like to
consider, which I believe are all satisfied by systemd and sysv-rc,
and were satisfied by upstart before it was removed from the archive:

* If the init system is booted in the most obvious way on a simple TUI
  system (similar to a default Debian install with Priority: standard
  packages), it should start at least one console getty on a VT for
  interactive login, without the sysadmin needing to set this up manually.

* It should be possible to configure the init system to provide a getty
  on a serial console, for testability on headless servers and VMs,
  and it should be documented how to do this.
  (systemd makes this very convenient, by starting a getty on the kernel
  console by default, and I would encourage this approach. sysvinit is
  less convenient here, but does at least have a relatively well-known
  way to enable a getty on a serial console.)

* If the init system is booted in the most obvious way on a simple GUI
  system, it should be possible to start an X11 display manager and log
  

Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Simon McVittie]

To be completely clear about the decision that Ian asked the technical
committee to overrule:

In all debootstrap versions since 1.0.102, merged /usr is the default (for
all variants except --variant=buildd). This means that new installations
of Debian buster using debian-installer will have merged /usr.

Do the debian-installer and debootstrap maintainers think this should
continue to be true for the buster release?

(Sorry, I should have made this the only question in a message, and not
mentioned side issues in the same message.)

Thanks,
smcv

Bug#914897: debootstrap, buster: Please disabled merged /usr by default

[Simon McVittie]

On Mon, 03 Dec 2018 at 17:45:11 +0100, Svante Signell wrote:
> On Sun, 2018-12-02 at 21:04 +0100, Marc Haber wrote:
> > moving hundreds of megabytes from /usr to / over time.
> 
> This solution was proposed by GNU/Hurd several years ago, and was scrapped due
> to not being big enough player in the *NIX world:
> https://www.gnu.org/software/hurd/faq/slash_usr_symlink.html

This did make more sense than the arbitrary split, but with hindsight I'm
glad it did't take off, because the /usr merge does have an important
advantage over the old Hurd proposal: it takes all the static OS files
(which are more similar than they are different) and groups them
together. Unifying those files into / as Hurd used to do would have
made it harder to disentangle them from the sysadmin-modifiable /etc
directory, which needs to be on the root filesystem anyway because it's
where we find /etc/fstab.

If I remember correctly, the people who implemented the /usr merge in
Fedora actually started by proposing that the static files were unified
into / (as in older Hurd versions), and later switched their design
around when they realised that grouping those directories together would
be better.

The name /usr is indeed an unfortunate historical accident.

> We think that we have found a more
> flexible solution called union filesystems, which allow to create virtual
> filesystems which are the union of several other filesystems.  However, 
> support
> for union filesystems is still in early development.

I get the impression that union mounts in Linux aren't completely reliable
either (and have some awkward corner-cases), so solutions that don't
require them seem likely to be more robust.

smcv

Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default?

[Simon McVittie]

On Wed, 05 Dec 2018 at 14:03:19 +0100, Svante Signell wrote:
> How about this case:
> 
> - Make as default non merged-/usr for all buildds.

This has been done: I sent patches, which have been applied.
(This is actually implemented in two different places, either of which
would have been sufficient to make this happen on all official buildds:
the Debian sysadmins' script to refresh sbuild chroots explicitly
requests unmerged /usr, and debootstrap --variant=buildd also defaults
to unmerged /usr.)

> - Make a choice of non merged-/usr or merged-/usr in the installer.

If you mean "the installer maintainers make a choice": this is effectively
what we have now. The debian-installer maintainers are the maintainers
of debootstrap, they have chosen to merge /usr in new installations,
and this bug report is a request for the technical committee to decide
whether to overrule them.

If you mean "users who run the installer are asked whether they want
merged or unmerged /usr": this seems worse than either merged /usr or
unmerged /usr, because it's asking new Debian users to make a choice in
which they likely have no way to know which answer is most appropriate,
because they aren't using Debian yet. If even experienced Debian users
disagree over what the answer should be (which we do, as you can see in
this thread), then we should not expect new users to be able to make
an informed decision.

> - Users wanting merged-/usr install the usrmerge package and convert to 
> merged-
> /usr.

This is one of several ways to achieve that result: either install
usrmerge, or install into a sysroot that already contains symlinks
/lib -> /usr/lib etc., or use debootstrap --merged-usr (or a recent
debootstrap version in which merged /usr is the default for recent
suites), or do the equivalent of usrmerge by hand.

Merging /usr on an existing system isn't actually difficult to implement
if you are manipulating the system from the outside: the hard parts of
the task of the usrmerge package are making it work on a running system,
and avoiding the system being left in a broken state if the transition
to merged /usr is left half-complete.

An alternative to the usrmerge package might be to do this transition
in an initramfs hook or something similar, which would guarantee that
nothing else is concurrently altering /usr or the directories that are
meant to be merged into it.

> 2) For those having merged-/usr installed:
>Re-run usrmerge to convert all newly installed packages to merged-/usr.
>Is this possible or is installing that package a install-once-only?

If you already have the merged-/usr filesystem layout (either by
installing usrmerge, by using debootstrap with --merged-usr, or by
creating the symlinks some other way), then all newly installed packages
effectively already behave as though they had been converted to merged
/usr: dpkg sees that the .deb contains a file like /bin/sed, but the
root filesystem contains a symlink /bin -> /usr/bin, so it dereferences
the symlink and unpacks sed into /usr/bin. This is part of dpkg's more
general support for relocating directories elsewhere by replacing them
with a symlink.

If that wasn't true, then the usrmerge package would not have been
feasible to implement. It would clearly have been unacceptable to make it
impossible to install unmodified packages later.

This remains true even if the usrmerge package is subsequently removed,
if it was ever installed. The symlinks /bin, /sbin, /lib* remain present,
because removing them would break the system: paths like /bin/sh,
/sbin/fsck, /lib/ld-linux.so.2 and /lib64/ld-linux-x86-64.so.2 must
continue to work indefinitely (even if the files are physically located
at /usr/bin/sh, /usr/sbin/fsck and so on).

smcv

Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default? /usr by default

[Simon McVittie]

Control: retitle 914897 tech-ctte: Should debootstrap disable merged /usr by 
default?

I'm retitling the bug to avoid misrepresenting the technical committee's
position on this. We have been asked to overrule the debootstrap
maintainer, but we have not yet come to a conclusion on whether we should.

On Wed, 05 Dec 2018 at 13:25:36 +0900, Hideki Yamane wrote:
>  Can we check and track this behavior in our packages?

Yes, we now do. The reproducible-builds infrastructure now uses unmerged
/usr for the first build and merged /usr for the second, since
 was fixed.

debootstrap since 1.0.111 also mitigates this class of bugs by disabling
merged /usr for --variant=buildd chroots (this was
).

Julien Cristau thinks #914208 was a sufficient/proportionate change, and
doesn't want to go further and default to --no-merged-usr for non-buildd
chroots (and in particular new debian-installer installations).

Ian Jackson thinks #914208 is not a sufficient answer (Ian, I hope I'm
not misrepresenting you here?), and escalated this bug to the technical
committee, asking us to overrule the debootstrap maintainers.

If the debootstrap/debian-installer maintainers agree with Ian on this,
then there is no need for the technical committee to consider his request
to overrule you, which is why Didier asked for your opinion on this
issue before attempting to come to a decision. If you agree with Julien,
then the technical committee still needs to consider this question.

>  Once disable merged-usr is good to prevent confusion but we detect such
>  as a bug for manage non-merged-usr and merged-usr Debian system in the end,
>  right? (or do you want to stay change in debootstrap 1.0.111 forever?)

The technical committee have not come to a conclusion on this.

My personal opinions (not overruling anyone):

If merged /usr becomes the only supported system layout at some future
time, then the change in debootstrap 1.0.111 can certainly be reverted
at that time. (If merged /usr does not become the only supported system
layout, this does not apply.)

It might also be considered appropriate to revert the change in
debootstrap 1.0.111 if data from reproducible-builds demonstrates that
bugs similar to #913226 have all been fixed or are very rare, but this
should be done cautiously, and certainly not before buster is released.

smcv

Bug#914897: debootstrap, buster: Please disabled merged /usr by default

[Simon McVittie]

On Sun, 02 Dec 2018 at 21:04:55 +0100, Marc Haber wrote:
> The next debhelper change might choose to give / instead of /usr as a
> target directory by default, moving hundreds of megabytes from /usr to /
> over time.

I don't think anyone is proposing that. There's no reason why it would be
preferred over the merged /usr arrangement implemented in both usrmerge
and debhelper --merged-usr, which is the same as is implemented in some
other Linux distributions (e.g. Fedora): /usr is a real directory, and
/bin, /sbin, /lib* are symlinks to the corresponding directories in /usr.

Unifying /usr with / by making /usr a symlink to / is the opposite of
the merged /usr arrangement that is currently implemented. The Debian
hurd-i386 port did try having a /usr -> / symlink a few years ago as a
simplification, but it has all the same transitional issues as merged
/usr, with fewer advantages, which makes it unappealing.

The purpose of merged /usr is to group together all the static files
(/bin, /sbin, /lib* and the current /usr), which are more similar than
they are diferent, into one place that can be a (maybe read-only) mount
point (/usr). If you do the opposite, your root filesystem is still a
mixture of mutable files in /etc and static files in /bin, /sbin, /lib*,
so you haven't gained as much simplification as you would have had with
merged /usr.

(You talk about "default target directories", but there is nothing
so elaborate: debootstrap --merged-usr simply unpacks packages into a
chroot that already contains the symbolic links like /bin -> /usr/bin,
instead of into an empty chroot.)

smcv

Bug#914897: debootstrap, buster: Please disabled merged /usr by default

[Simon McVittie]

On Sun, 02 Dec 2018 at 19:54:39 +, Simon McVittie wrote:
> When installed on a merged /usr system:

Oops, of course this should have said:

.deb contains -> /bin/grep/usr/bin/perl
/bin/fooexists thanks to /bin symlink   exists thanks to /bin symlink
/usr/bin/foo   physical locationphysical location