Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

2021-09-27 Thread Sam Hartman


>On merged-/usr systems, there is a possible failure mode involving files
>being moved between packages (with Replaces) during the same release
>cycle that their logical location is changed from the root filesystem
>to the corresponding aliased directory in /usr, which can result in
>the affected file disappearing. This can be avoided by not changing
>the file's logical location until the beginning of the Debian 13
>development cycle, after the transition to merged-/usr is complete.



Simon, I've reviewed your draft, and with the exception of the above
text, it all seems consistent with my understanding of the technical
details.

I don't understand how transitioning files in the Debian 13 cycle is
going to work any better than in the Debian 12 cycle unless dpkg happens
to change.
So I don't see how the above text is correct.


If I'm missing something, perhaps we could clarify the text, because I
suspect I have a better than average understanding of the issues.  If I
don't get it, others probably will misunderstand too.



Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636

2021-09-27 Thread Simon McVittie
On Wed, 15 Sep 2021 at 12:35:38 +0100, Simon McVittie wrote:
> https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md

I have updated this draft to incorporate my edits from !3, and feedback
from bremner on IRC.

I'd like to keep this moving, because it's somewhat time-sensitive: people
are already taking action based on our earlier resolution, some of which is
not necessarily the action we would have wanted them to take.

On Wed, 15 Sep 2021 at 11:46:02 +0100, Simon McVittie wrote:
> Some questions that I think we need to answer explicitly:
> 
> - What do we mean by merged-/usr? (We already said this in #914897, but
>   I think it's worth reiterating that symlink farms are not it.)

This is defined (again) by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#definitions-and-current-status

> - Is it OK for packages in testing/unstable during Debian 12 development
>   to assume/require a merged-/usr system? (tl;dr: some maintainers think
>   the answer is yes, but I think the answer is no.)
>
> - When will it be OK for packages in testing/unstable to assume/require
>   a merged-/usr system? (tl;dr: some maintainers think the answer is
>   "right now", but I think the answer is "when the Debian 13 cycle begins"
>   and not before.)

Both of these are addressed by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#upgrade-path-from-debian-11-to-debian-12

> - Should maintainers proactively move files in packages from the root
>   filesystem into /usr? (tl;dr: I think they should not, at least until
>   the implications are better-understood.)
> 
> - Should maintainers of tools, e.g. debootstrap, proactively move files
>   in packages from the root filesystem into /usr, e.g. systemd system units?
>   (tl;dr: I think they should not.)

Both of these are addressed by
https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#moratorium-on-moving-files-logical-locations-into-usr

> - Are packages built on a merged-/usr system required to work correctly
>   on a non-merged-/usr system? If they are not, is it RC?
>   (I think they are required to work, and it should usually be RC if
>   they don't)

https://salsa.debian.org/debian/tech-ctte/-/blob/master/994388_merged_usr_advice/draft.md#building-packages

On Thu, 16 Sep 2021 at 15:35:11 -0700, Sean Whitton wrote:
> You write:
> 
> >> - Because Debian 11 installations with the non-merged-/usr layout already
> >>   exist, all packages in Debian 12 should be installable onto a 
> >> non-merged-/usr
> >>   system along with their dependencies, and work correctly on the resulting
> >>   system.
> 
> (1) The reason for this, to put it a bit simplistically, is that we
> don't require apt to perform the upgrade between stable releases in any
> particular order, right?  Or are there other reasons distinct from this
> one that I'm missing?  I think it would be good to state the thing about
> apt (in better language than mine) in the text.

I think that's the main reason. We have not traditionally mandated
the use of a special upgrade tool like Ubuntu's do-release-upgrade(8),
so the upgrade happens in whatever order apt chooses, which can vary
between machines.

Another reason why I think we want Debian 12 packages to be installable
onto non-merged-/usr systems is that to be able to do our development work,
they need to be installable onto testing/unstable systems, which (again)
means that the upgrade order is undefined.

We also have best-effort support for partial upgrades, either oldstable
-> stable or stable -> testing/unstable: upgrading only a subset of the
available packages, plus their mandatory dependencies, but without also
upgrading apparently-unrelated packages. This can only ever be partially
supported, because it leads to a combinatorial explosion of possible
partially-upgraded systems, and realistically we can never test them all;
but I think it's best if we try to avoid knowingly making this worse.

Finally, there is a reason to want this that is circular in nature: if we
want Debian 12 packages to be installable onto non-merged-/usr systems,
and we want to be able to say with reasonable confidence that we think
they work in that situation, then we need to be able to test that they
do, in fact, work. As mentioned under the "Testing and QA" heading,
if we do not keep it possible to force autopkgtest VMs/containers,
piuparts chroots, reproducible-builds chroots and manual-test VMs (or
even scratch installations on real hardware) to be non-merged-/usr,
then we will have trouble testing that. However, I have tried to make it
clear that if a developer sets up such a system, they cannot expect to
be able to upgrade it cleanly to Debian 13, or to a post-Debian-12
state of testing/unstable.

> (2) Some people on -devel would seem to think that we can have smooth
> upgrades from bullseye to bookworm without requiring