Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Zack" == Zack Weinberg  writes:
Zack> Maybe it was not obvious to anyone in 2016 that the package
Zack> database being inconsistent with the filesystem could cause
Zack> actual data loss.  However, I think it was the responsibility
Zack> of the developers of usrmerge to find out how bad that problem
Zack> actually was, rather than dismissing it.  And, as it has
Zack> proven to be a genuinely critical problem, it is also their
Zack> responsibility to fix it, per the 'no one can be forced to do
Zack> anything' rule.

I agree with the above, but very definitely not with the further
conclusions you draw.

I think that the usrmerge proponents need to fix the problem before
encouraging moving  files from / to /usr in packages.
They can do that either by:

* proposing fixes to dpkg

or

* As Michael has pointed out, proposing mechanisms to guarantee that no
  Debian release both includes file migrations and package migrations.

We disagree about the rollback.
If we're ever communicating in a more realtime context I'd love to chat
about the balances around making sure we do not reward bad behavior.

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Ansgar  writes:

> So I do not think there should be a command-line option for this; unless
> it would behave like `--add-architecture` and register the setting
> somewhere.

Agreed.

I also think it needs to be a two-phase thing, because dpkg has to convert
its internal database to respect the new aliases.  So there should be a
desired configuration and then an applied configuration, and the latter is
only updated once the database conversion is complete, with dpkg refusing
to perform package operations if the desired configuration doesn't match
the applied configuration.

Even if the command that changes the configuration also does the database
conversion, I think we need this, since I don't believe it's possible to
do the database conversion atomically (or at least it would involve extra
complexity that I don't think we want), so we have to handle the database
conversion being interrupted.

> I do not think this should be a configuration file (unless you suggest
> supporting other values or removing individual lines; after all a
> configuration file exists to be changed).

Are you thinking of perhaps shipping this configuratino somewhere in
/usr/lib instead so that it's not marked as a configuration file?

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Ansgar]

On Mon, 2021-11-22 at 12:24 +, Simon McVittie wrote:
> Options to make path A an alias for path B would be a lot like --path-
> exclude in that they really only make sense in a configuration file in
> /etc/dpkg/dpkg.cfg.d, although for symmetry they would presumably also
> have to exist as command-line options just like --path-exclude does.

I would expect using --path-exclude when installing individual packages
to work: you just don't get the files from that package. (I haven't
tried.)

Merged-/usr (or --path-alias as you propose) does not work like this:
Unlike --path-exclude, you would almost always need to specify a
superset of --path-alias options for all future operations.

So I do not think there should be a command-line option for this;
unless it would behave like `--add-architecture` and register the
setting somewhere.

>     # in /etc/dpkg/dpkg.cfg.d/merged-usr
>     path-alias=/bin=usr/bin
>     path-alias=/lib32=usr/lib32
>     path-alias=/lib64=usr/lib64
>     path-alias=/lib=usr/lib
>     path-alias=/libo32=usr/libo32
>     path-alias=/libx32=usr/libx32
>     path-alias=/sbin=usr/sbin

I do not think this should be a configuration file (unless you suggest
supporting other values or removing individual lines; after all a
configuration file exists to be changed).

(Though we have various other settings that don't make sense as
configuration files in /etc, such as Debian's bugtracker and other
settings in /etc/dpkg/origins/debian, the version in
/etc/debian_version and so on.)

Ansgar

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Johannes Schauer Marin Rodrigues  writes:

> Please don't. Since 2014 it is possible to create a Debian chroot
> without taking care of the unpack order and without running any special
> setup beforehand that cannot be put into apt or dpkg itself (like
> creating /var/lib/dpkg/info which is not needed anymore since dpkg
> 1.20.0). One of my biggest motivations in writing mmdebstrap was to get
> rid of all the quirks that currently live in debootstrap, move them into
> apt and dpkg so that we end up with a declarative bootstrap process that
> is driven by the package contents, their metadata and maintainer scripts
> but does not need magic from the outside other than standardized
> interfaces. Requiring the merged-/usr setup to be done by the bootstrap
> script would be a step backwards.

I understand why you want this, but unfortunately it doesn't answer the
question of how to safely perform the upgrade and reconfigure dpkg.  Do
you have any thoughts on the drawbacks of the approach of putting the
symlinks in base-files that led me to lean away from doing that?

I'd love to get all of the machinery into packages as well, but I think
safely upgrading existing systems is even more important than that, and
I'm worried about our ability to do so if we put the machinery in
base-files.

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Johannes Schauer Marin Rodrigues]

Quoting Russ Allbery (2021-11-20 18:22:27)
> * Create a new essential package that contains these symlinks and that
>   needs to be unpacked before any binaries are executed in the target file
>   system.  This has many of the advantages and drawbacks of the approach
>   of putting the symlinks in base-files, but may make it easier to handle
>   the upgrade problem.  Ideally an upgrade would then involve installing
>   usrmerge, letting it run, and then installing this new essential package
>   so that it takes over ownership of those symlinks.
> 
>   This still feels kind of complex and messy to me, but maybe less so.
> 
> * Create the symlinks directly in the bootstrapping script.  In other
>   words, after unpacking base-files, the bootstrapping script would
>   directly create the required symlinks in the target file system, before
>   unpacking any other package.
> 
>   This has the obvious drawback of moving things outside the packaging
>   system and creating a new special case that every bootstrapping script
>   needs to be aware of (and I know we have at least four or five that
>   would need modifications).  It has the advantage that the usrmerge
>   symlinks are now not in the dpkg database and thus not subject to
>   rewriting, and therefore won't need to be special-cased.  However, that
>   comes with the obvious disadvantage that they're not in the dpkg
>   database, so for instance dpkg -S /lib wouldn't find that symlink unless
>   it was added as some sort of dpkg-query special case (which doesn't seem
>   like a great idea).
> 
>   The advantage of this approach is that it closely mimicks what's already
>   happening now with the usrmerge package, and for which we therefore have
>   a lot of existing experience.

Please don't. Since 2014 it is possible to create a Debian chroot without
taking care of the unpack order and without running any special setup
beforehand that cannot be put into apt or dpkg itself (like creating
/var/lib/dpkg/info which is not needed anymore since dpkg 1.20.0). One of my
biggest motivations in writing mmdebstrap was to get rid of all the quirks that
currently live in debootstrap, move them into apt and dpkg so that we end up
with a declarative bootstrap process that is driven by the package contents,
their metadata and maintainer scripts but does not need magic from the outside
other than standardized interfaces. Requiring the merged-/usr setup to be done
by the bootstrap script would be a step backwards. Think about the following
points:

 * you create a Debian unstable chroot using snapshot.d.o -- now you have to
   somewhere put code that identifies that this is a chroot from the past and
   that has to decide whether that chroot should be merged-/usr or not. If
   merged-/usr setup was living somewhere in the Essential:yes package set
   the chroots would be created the right way automatically
 * same for derivatives -- now every bootstrap script has to learn which
   derivative defaults to merged-/usr, which ones do not and when they switched
 * we somehow need to convert installations of users that only run testing
   or unstable but never do full stable installation to merged-/usr -- one way
   to do that is to let one package in the Essential:yes set drive the
   conversion at which point the bootstrap script doesn't need to know about
   merged-/usr because the Essential:yes set knows how to set it up
 * think about it from a software engineering point of view: Debian as a
   component based software distribution should move specific functionality
   into its packages and global functionality into declarative metadata.

Just as we are currently trying to reduce maintainer scripts and replace them
by declarative approaches, we should also try to reduce the magic in our
bootstrap scripts and move it either to our package managers or replace them by
declarative solutions. We have been moving into the right direction during the
last few years as apt and dpkg have picked up functionality I had earlier put
into mmdebstrap (thanks a lot to guillem, DonKult and juliank for picking up my
patches or writing the functionality themselves). It would mean a step
backwards if all bootstrap scripts would have to carry the setup_merged_usr()
function from debootstrap and then decide by some $magic when this function
should be executed or not.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: merged-/usr transition: debconf or not?

[Simon McVittie]

On Sat, 20 Nov 2021 at 14:54:30 -0800, Russ Allbery wrote:
> That would make it akin to dpkg
> --add-architecture, which feels like a good model (although as you mention
> I don't think we want --remove-architecture).

Instead of dpkg --add-architecture, which is a top-level "verb" like
dpkg --remove or dpkg --install [1], probably a better thing to resemble
would be the options that already exist to change dpkg's behaviour,
and in particular --path-exclude and --path-include?

Options to make path A an alias for path B would be a lot like --path-exclude
in that they really only make sense in a configuration file in
/etc/dpkg/dpkg.cfg.d, although for symmetry they would presumably also
have to exist as command-line options just like --path-exclude does.

Here's a possible design (not fully thought out, and maybe wrong or
unimplementable):

--path-alias ALIAS=TARGET
When unpacking or querying packages, paths below ALIAS will be
transformed into the corresponding path below CANONICALIZED, where
CANONICALIZED is the result of resolving TARGET relative to the
directory containing ALIAS. This transformation is done before
checking for file conflicts.

ALIAS must be an absolute path.

If the file list of any unpacked package contains a directory at
ALIAS and/or CANONICALIZED, then ALIAS will be created as a symbolic
link to TARGET, with its target directory created at CANONICALIZED.

It is an error to have multiple --path-alias directives for
the same ALIAS with different TARGET paths.

This option should usually be set via /etc/dpkg/dpkg.cfg.d rather
than as a command-line option.

For example, a series of directives like path-alias=/bin=usr/bin
can be used to implement the "/usr merge", causing /bin/bash to be
unpacked as /usr/bin/bash while creating a symbolic link
/bin -> usr/bin.

With that, I believe a full implementation of merged-usr for all known
architectures would be to install something like this, perhaps via
base-files:

# in /etc/dpkg/dpkg.cfg.d/merged-usr
path-alias=/bin=usr/bin
path-alias=/lib32=usr/lib32
path-alias=/lib64=usr/lib64
path-alias=/lib=usr/lib
path-alias=/libo32=usr/libo32
path-alias=/libx32=usr/libx32
path-alias=/sbin=usr/sbin

Rationale for TARGET being allowed to be relative: I believe the compat
symlinks in merged-/usr are preferred in the form /bin -> usr/bin, to
make sysroots (containers, chroots, btrfs root filesystem snapshots,
etc.) easier to manipulate from outside.

Rationale for ALIAS only being created if ALIAS and/or CANONICALIZED
really exist in a package: we would probably prefer the multilib
directories to be created "lazily", rather than having a /libo32 on
systems that will never install a mips* package and a /libx32 on systems
that will never install an x32 package. This is slightly nicer than the
current implementation in convert-usrmerge and debootstrap, which creates
those directories "eagerly" whenever the appropriate architectures are
configured as either primary or foreign architectures.

smcv

[1] if dpkg had been designed with more recent CLI conventions in mind,
I expect these "verbs" might have been subcommands like "dpkg remove",
etc. as seen in apt/git/flatpak, which makes it a bit more obvious
that only one is allowed

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Gabor Gombas  writes:

> If you replace "rewrite /lib64 to /usr/lib64" with "rewrite /lib64/* to
> /usr/lib64/*", then this can easily be avoided.

True, although I think you still want to ensure that dpkg never messes
with those links.  I don't think you want base-files to be able to drop
one of those links from the package and cause it to be removed by dpkg,
for instance.

Maybe I'm borrowing trouble for a scenario that won't happen (and I think
dpkg may have some special-casing of symlinks to directories anyway).

> Talking about "special casing" in dpkg is bothering me for a while. And
> there is a relatively simple way to avoid any kind of special casing:
> move the information out to a configuration file (which would _not_ be a
> conffile) - and now the code has no special casing, just
> configuration-driven logic.

All that I mean by special-case is that I think it would be a mistake to
try to shoehorn support for merged-/usr into some pre-existing dpkg
concept.  The merged-/usr links are special relative to other files that
dpkg is managing and need to be treated with special care.  We should add
a new concept to dpkg for these sorts of links.

I agree with making the exact set of links configurable.  This also allows
handling the architecture-specific links in a clean way.

> This new configuration file could be shipped by base-files itself, to
> ensure it does not get out of sync with the filesystem structure shipped
> in base-files. Then base-files' postinst would more or less need to
> include the current usrmerge package, plus whatever is needed to convert
> dpkg's database.

I don't think we want base-files to take over the logic of usrmerge.

usrmerge is both complex (in part because it relies on Perl) and not 100%
hardened against weird issues, being interrupted in the middle of running,
etc.  That's all fine for what it's doing as a one-time migration tool
that's run with care during an upgrade when the administrator already
knows Big Things are happening to the system.  Putting the logic into
base-files makes it more of a routine thing and I think it should be a lot
more hardened if we do that (and I don't think that's a good use of
anyone's time).  We ideally want usrmerge to be a tool each system runs
once, under controlled circumstances, and then never again (and isn't
needed at all for bootstrapping).

We have a migration strategy that's working.  Let's stick with that
migration strategy without messing with it and instead just add a dpkg
database conversion step to the end of that migration.

I do agree that something needs to handle installing the configuration
file and laid out some options for that in another message.

> This new configuration file would not be consumed by dpkg directly when
> installing packages, but only when the database conversion is called,
> and dpkg would keep an internal list of the rewriting rules which are
> active. Doing so would allow enforcing rewriting rules can only be added
> but never removed, which would avoid potential issues if base-files gets
> downgraded.

This is a very good idea.  I was starting to think along those lines, but
hadn't taken it quite far enough.  That would make it akin to dpkg
--add-architecture, which feels like a good model (although as you mention
I don't think we want --remove-architecture).

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Gabor Gombas]

Hi,

On Sat, Nov 20, 2021 at 09:22:27AM -0800, Russ Allbery wrote:

>   The drawback here is that dpkg is going to rewrite all paths like /lib64
>   to /usr/lib64, which would naively *also* apply to the base-files
>   package when it looks at that package, but that can't be allowed because
>   now we're back to the situation where dpkg's state database is
>   inconsistent with the file system and dpkg thinks that base-files
>   contains some nonsensical /usr/lib64 to /usr/lib64 symlink.

If you replace "rewrite /lib64 to /usr/lib64" with "rewrite /lib64/* to
/usr/lib64/*", then this can easily be avoided.

>   I think in this approach there would need to be some special-case code
>   directly in dpkg that recognizes the usrmerge symlinks [...]

Talking about "special casing" in dpkg is bothering me for a while. And
there is a relatively simple way to avoid any kind of special casing:
move the information out to a configuration file (which would _not_ be a
conffile) - and now the code has no special casing, just
configuration-driven logic.

This new configuration file could be shipped by base-files itself, to
ensure it does not get out of sync with the filesystem structure shipped
in base-files. Then base-files' postinst would more or less need to
include the current usrmerge package, plus whatever is needed to convert
dpkg's database.

This new configuration file would not be consumed by dpkg directly when
installing packages, but only when the database conversion is called,
and dpkg would keep an internal list of the rewriting rules which are
active. Doing so would allow enforcing rewriting rules can only be added
but never removed, which would avoid potential issues if base-files gets
downgraded.

Gabor

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Russ Allbery  writes:

> Well, bootstrapping a new Debian system involves running a tool that
> bootstraps a new Debian system.  I think you're constraining the problem
> too much.

> It's a nice property that everything on the system comes straight from a
> Debian package, but it's not a strict requirement, nor is it currently
> generally the case (maintainer scripts do things, for instance).  Those
> symlinks are very special from a dpkg perspective; dpkg needs to refuse
> to mess with them even when upgrading a package that provides them,
> which would mean some irritating special-casing I suspect.  It's not
> clear to me that shipping them as tar members of a package is the right
> way to go, as opposed to creating them separately as part of early
> system configuration.

Having slept on this, let me make this problem concrete.

I'm going to make the following assumptions:

* We have some mechanism to put dpkg into what I've been calling
  merged-/usr mode.  In this mode, it pre-filters all input paths from
  whatever source (including arguments to dpkg-divert,
  update-alternatives, etc.)  and canonicalizes them in a very specific
  way: the directories that become symlinks in merged /usr are replaced in
  each path with their canonical paths.  So /bin/ls becomes /usr/bin/ls,
  /lib64/ld-linux-x86-64.so.2 becomes /usr/lib64/ld-linux-x86-64.so.2, and
  so forth.

* When bootstrapping a new Debian system, we want to put dpkg into
  merged-/usr mode as early as possible.

* If dpkg is in merged-/usr mode, the first thing it does is checks the
  file system on which its operating and ensures that the expected
  symlinks already exist.  If they do not, it aborts because operating on
  that file system is unsafe.  (Leaving aside for the moment whether there
  should be some -force option, etc.)

This produces a bootstrapping problem: all dynamic binaries on, say, an
amd64 system refer to /lib64/ld-linux-x86-64.so.2 [1].  Therefore, before
the first binary is run from within the context of the newly-installed
system, either that path must exist as-is (which we don't want because we
want to create a merged-/usr system where it belongs in /usr/lib64) or the
symlink from /lib64 to /usr/lib64 must already exist.

I think it's fairly obvious that we don't want a long-term design in which
the libc6 package has to continue to ship /lib64/ld-linux-x86-64.so.2, we
originally unpack that file in that path during bootstrap, and then
something has to come along later and move it to /usr/lib64 and create a
symlink.  This has numerous drawbacks: weird special cases people have to
remember, not being able to reconcile the contents of packages with their
canonical paths in the long run, a window where we have to do file system
surgery atomically, etc.  Instead, we want to live in a world in which
libc6 can ship /usr/lib64/ld-linux-x86-64.so.2, dpkg sees it as shipping
that path, but everything works fine during bootstrap because the /lib64
to /usr/lib64 symlink is already established before we have to execute
binaries in the new environment.  (Obviously it may be some time before we
actually change the contents of the libc6 package; that's fine, the point
of adding a path filter to dpkg is that we can take our time on that.)

So, assuming we have a libc6 package that contains either
/usr/lib64/ld-linux-x86-64.so.2 or /lib64/ld-linux-x86-64.so.2, how do we
bootstrap this system?

I can see a few approaches:

* Put the required symlinks into base-files.  This preserves the nice
  property that every file on the system comes from and is registered as
  belonging to a Debian package (which we don't fully meet but which we
  always aspire to).  I haven't studied all the various tools for
  bootstrapping a Debian system, but I presume that unpacking base-files
  is the first thing that any of them do.  This will therefore lay down
  the symlinks from the start and it won't matter what path under which
  the libc6 package ships the files if it's extracted over the resulting
  file system with ar+tar.

  The drawback here is that dpkg is going to rewrite all paths like /lib64
  to /usr/lib64, which would naively *also* apply to the base-files
  package when it looks at that package, but that can't be allowed because
  now we're back to the situation where dpkg's state database is
  inconsistent with the file system and dpkg thinks that base-files
  contains some nonsensical /usr/lib64 to /usr/lib64 symlink.

  I think in this approach there would need to be some special-case code
  directly in dpkg that recognizes the usrmerge symlinks (and only
  specifically those symlinks) as special and preserves them as-is in the
  installed package database.  (I think it's probably better to
  special-case the specific symlink than to special-case base-files the
  package.)  We will then need rules that base-files must always contain
  those symlinks and they can't move between packages, etc., (or the
  system is likely to break horribly), 

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Simon Richter  writes:

> Bootstrapping a new Debian system works by unpacking Essential packages
> with ar+tar, i.e. not using dpkg. These packages will always be unpacked
> to the file names listed inside the packages.

Well, bootstrapping a new Debian system involves running a tool that
bootstraps a new Debian system.  I think you're constraining the problem
too much.

It's a nice property that everything on the system comes straight from a
Debian package, but it's not a strict requirement, nor is it currently
generally the case (maintainer scripts do things, for instance).  Those
symlinks are very special from a dpkg perspective; dpkg needs to refuse to
mess with them even when upgrading a package that provides them, which
would mean some irritating special-casing I suspect.  It's not clear to me
that shipping them as tar members of a package is the right way to go, as
opposed to creating them separately as part of early system configuration.

> Dpkg has an elaborate per-file tracking to recover from errors, and this
> change introduces several new states and transitions,

Why?  I am not convinced this is true, apart from the obviously-needed
one-time conversion of the state database.

It involves a new filter, applied whenever a package is installed, that
changes all the paths in the package to their merged-/usr paths, something
that seems like it should be done before any other package processing.
Once that's done, why would any new states or transitions be required?

You could convince me that writing the filter is complicated because there
may not be one place in dpkg where you can currently put such path
rewriting since dpkg probably wasn't designed with that operation in mind.
But it's going to be harder to convince me that there are new states or
transitions; that smells like an over-complicated design.

> That is the goal, yes, but this is a massive undertaking.

I'm still not convinced of this.

> We already have a quick hack for usrmerge support in dpkg, which is the
> check that allows moving a file from /bin to /usr/bin within the same
> package without the file being deleted in the end (because deletions are
> processed last for obvious reasons), and the obscure rule is precisely
> the limitation of this hack.

This already sounds conceptually more complicated than just solving the
problem properly, with the caveat as mentioned above that writing the
necessary filter may be difficult.

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

> > (I've used /bin -> /usr/bin as an example, but the canonicalization must
> > deal with the other merged paths too.)

/lib and /lib64 are a bit special, because they are part of the ELF ABI,
and any manipulation of these paths needs to be atomic.

Bootstrapping a new Debian system works by unpacking Essential packages
with ar+tar, i.e. not using dpkg. These packages will always be unpacked
to the file names listed inside the packages.

So, new systems will either need a package that ships the /lib ->
usr/lib symlink in the Essential set, or we need to ship the dynamic
linker in /lib, which would be a conflict during unpacking.

> > The second bit ensures that all new operations write canonicalized paths
> > (e.g. /usr/bin/foo) into the database regardless of whether the .deb 
> > shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in
> > sync with the filesystem moving forward.

This loses the information about the original path, so this would be a
trapdoor operation that can not be undone without affected packages
going into reinstallation-required state (and we probably couldn't even
identify those).

Ideally, I'd rather not hardcode the list of top-level symlinks into
dpkg, because they might be architecture-specific, or change over time,
so there should be a mechanism to add and remove transformations.

> > The one-time canonicalization can be removed once non-usrmerged systems
> > are no longer supported. The second bit needs to live (probably a lot) 
> > longer, until it's no longer reasonable to install a .deb containing
> > non-usrmerged paths (which might be old or from a third party).

My expectation would be that this code will be here to stay, because it
will be required for bootstrap.

> Are we missing something?  If not, what is blocking this solution?  Is it
> simply a matter of someone digging into dpkg's code sufficiently to put
> these changes in the correct location?  Or is there some other issue?

If it were that simple, someone would have done that already. Dpkg has
an elaborate per-file tracking to recover from errors, and this change
introduces several new states and transitions, last but not least
because we also have filters and diversions as transformations on the
database -- but the usrmerge transformation is supposed to be orthogonal
to that.

> It seems like a huge waste of resources to me to do archive-wide package
> inspection to try to find patterns that might cause problems, and to ask
> all maintainers to remember this rather obscure rule, when we could just
> fix dpkg to make the problem go away.

That is the goal, yes, but this is a massive undertaking. We already
have a quick hack for usrmerge support in dpkg, which is the check that
allows moving a file from /bin to /usr/bin within the same package
without the file being deleted in the end (because deletions are
processed last for obvious reasons), and the obscure rule is precisely
the limitation of this hack.

I've stumbled across this hack while investigating if it was possible to
"just" fix dpkg. :P

   Simon

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Simon Richter  writes:

> I think the main blocker at the moment is that the dpkg change *also*
> has the potential to break a lot of things if it isn't carefully
> designed.

I think you can simplify that problem space considerably because we know
exactly what symlinks we care about and we don't intend to support going
backwards (in other words, once you have merged /usr, it's not supported
to turn /bin, etc., back into regular directories again).

Therefore, dpkg can be configured with the list of the symlinks that
*must* exist for it to be willing to operate on the system.  Once it has
been put into merged-/usr mode (which includes updating its state database
to reflect the new canonical paths of all installed files), if those
symlinks do not exist or do not have the correct targets, the system state
is not safe and dpkg can (and probably should) abort and refuse to operate
on that system.

That modifies the problem of systems in various weird configurations from
"weird things may happen" to "dpkg will (safely) refuse to run" and we can
provide a tool to fix such a system by creating the correct symlinks.
(This tool may well just be the existing usrmerge tool.)  Getting into
that state will be hopefully rare if we structure the upgrade process so
that first one converts the system to merged-/usr if this has not already
been done, then upgrades dpkg, then tells dpkg to switch into merged-/usr
mode, and then safely upgrades the remaining packages.

> From dpkg's perspective, we now (kind of) change the policy for
> directory-vs-symlink conflicts, which dpkg currently resolves in favour
> of the directory.

I don't think we should be doing anything of the sort.  I think we should
introduce a new concept specifically for this case and have the remappings
of files for merged-/usr, once dpkg is in merged-/usr mode, take
precedence over any rules like that.  We do not want dpkg to be applying
some policy or heuristic about directory vs. symlink conflicts for
symlinks related to merged-/usr.  We know exactly what we want the rule
and behavior to be.  That sort of remapping should happen way before any
other dpkg analysis of the package or system state.

In other words, I argue that once dpkg is in merged-/usr mode, it should
treat a package with files in /bin exactly as if the files were in
/usr/bin, at an earlier stage than when it starts looking at the file
system.

We're not trying to create a generalized feature here.  Merged-/usr has a
very specific definition, with very specific and clear behavior that we
need from the package manager.  Trying to generalize this is going to make
a mess and make it much harder to reason about the behavior or test it.

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

On Fri, Nov 19, 2021 at 12:28:56PM -0700, Sam Hartman wrote:

> But we could you know fix dpkg:-)
> I'm sure that will be painful at the political level, but personally I
> think it needs doing.

I think the main blocker at the moment is that the dpkg change *also*
has the potential to break a lot of things if it isn't carefully
designed.

The problem space right now is huge:

 - for upgraded systems, the system could be pre-usrmerge, or the
   conversion could have been performed by any usrmerge version that
   ever existed (and the usrmerge package could have been deinstalled
   and reinstalled since, so the patches to the conffiles it performs
   could be inconsistently applied).
 - for upgraded systems, any version of usrmerge since the last stable
   release could be installed
 - for upgraded systems, unless the usrmerge package is discontinued,
   it could be upgraded at any point during the update.
 - for freshly installed systems, dpkg is first invoked on a
   non-usrmerged tree, and should convert the installation as soon as it
   becomes aware that conversion is desired (which might not be
   something we want to hardcode in dpkg, but possibly configure from a
   separate, Debian-specific package like base-files).
 - for freshly installed systems, initial unpack might be controlled by
   debootstrap from stable, which is usrmerge aware, so some of its
   workarounds may be active, and we need to properly capture this
   state, too.
 - freshly installed systems might be created with cdebootstrap or
   multistrap, and could be generated in --foreign mode
 - the dynamic loader is specified to be on the root filesystem, and that
   is part of the ABI standard and not under our control, so we cannot get
   a conflict-free dpkg database in bookworm.

>From dpkg's perspective, we now (kind of) change the policy for
directory-vs-symlink conflicts, which dpkg currently resolves in favour
of the directory. That was done in ancient times because it was somewhat
easy to do this in a package by accident. Not sure if that still
applies, but if it does, then we need to formulate the new policy so
that we don't catch a regression here (which is why my original idea to
just ship the symlinks in base-files is a bad idea).

>From the way dpkg and the Debian Policy are initially designed, it is
obvious to me that they started out as specification documents, and only
when these were hammered out, they were poured into code and rules, and
we've been operating from that stable base since, with two exceptions
where technical facts were created without updating Policy accordingly,
and which has both times been controversial.

My interpretation of the "political" situation is that we need this to be a
group effort, because no single person has the necessary time to do a
thorough enough job that they would feel comfortable signing off on it. I'd
be reluctant to do so even if I had three months of uninterrupted time --
that's the level of complexity I see here, and I suspect Guillem feels the
same.

So I believe the way forward will be writing a specification and giving it
enough eyeballs to identify problem cases. Finding the adversarial example
for the status quo was easy, since I had to find just one -- the goal now
is to get to a state where we don't find such an example easily anymore.

The specification should explain how the transition can be finished with
the bookworm release for all the different ways packages can be installed,
and it should describe the necessary code changes and new test cases to get
full coverage, and the document should be signed off by multiple people who
jointly take responsibility, to avoid placing all of that on a single
person -- because that's what's impeding progress right now, that no one
wants to be that person or even feels able to.

The description of the problem space I put above is likely incomplete and
overzealous at the same time (for example, I haven't checked how different
the previous usrmerge packages are, and which of them ever made it into a
stable release), so this is also just a starting point. I'll happily spend
time refining this or any other proposal, but I'm too time-constrained to
be the main driving force here -- this is not my day job, after all.

   Simon

Re: merged-/usr transition: debconf or not?

[Timo Röhling]

* Russ Allbery  [2021-11-19 11:46]:

I also don't understand why this isn't the correct solution.  It seems
obvious to me that we should simply teach dpkg about path aliases and have
it update both its internal state and its processing of new packages to
canonicalize paths so that it has an accurate representation of the world,
and then ensure dpkg is upgraded first.

+1


Are we missing something?  If not, what is blocking this solution?  Is it
simply a matter of someone digging into dpkg's code sufficiently to put
these changes in the correct location?  Or is there some other issue?

AFAIR, Guillem has not commented on the solution when it was brought
up on d-devel [1], so I'm not sure if he would accept patches, let
alone support anyone working on this. I'd be willing to assist him
and contribute code, but I'm not going to spend a good chunk of my
spare time just to see me work summarily rejected.

Cheers
Timo

[1] https://lists.debian.org/debian-devel/2021/08/msg00438.html


--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Fri, Nov 19, 2021, at 3:23 PM, Zack Weinberg wrote:
> On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
>>> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
>>> Luca Bocassi wrote:
>>> ...
 nobody has actually seen [the file disappearance bug]
 happen, to the best of my knowledge.
>>>
>>> I already explained why that doesn't prove the bug is a non-issue.
>>> To the contrary; it means there is an enormous installed base of
>>> systems where the bug is latent, waiting to cause problems under
>>> conditions which we can reasonably expect to occur shortly after
>>> the release of bookworm.
>>
>> Why would the release of bookworm make any difference?
>
> Up until the release of bookworm, all Debian packages must be
> constructed on the assumption that they _might_ be unpacked on a
> system that has not yet been converted to merged /usr.  Particularly
> for priority-required packages, this means that no one will be moving
> files from /bin, /lib, etc to /usr in the bookworm cycle.
>
> Post-bookworm, if nothing changes, that assumption will no longer be
> in force, and people who maintain packages that install files into /
> will want to simplify their packaging by installing everything in /usr
> instead.  If they also want to change the binary packages that ships
> some of those files at any point in the same release cycle -- kaboom.

After having caught up on the thread, I see that the conditions required
to trigger the bug are somewhat more complicated, but the point remains:
particularly for the packages where a lost file could render the system
unbootable, changes that would trigger the bug have been deferred until
post-bookworm, *and* we can reasonably expect the maintainers of those
packages to *want* to make changes with a high risk of triggering the bug.
I imagine the coreutils maintainers are looking forward to getting rid
of their list of programs that go in /bin, and the extra debian/rules
logic to go with it, for instance
(https://sources.debian.org/src/coreutils/8.32-4.1/debian/rules/#L16).

zw

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
>> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
>> Luca Bocassi wrote:
>> ...
>>> nobody has actually seen [the file disappearance bug]
>>> happen, to the best of my knowledge.
>>
>> I already explained why that doesn't prove the bug is a non-issue.
>> To the contrary; it means there is an enormous installed base of
>> systems where the bug is latent, waiting to cause problems under
>> conditions which we can reasonably expect to occur shortly after
>> the release of bookworm.
>
> Why would the release of bookworm make any difference?

Up until the release of bookworm, all Debian packages must be
constructed on the assumption that they _might_ be unpacked on a
system that has not yet been converted to merged /usr.  Particularly
for priority-required packages, this means that no one will be moving
files from /bin, /lib, etc to /usr in the bookworm cycle.

Post-bookworm, if nothing changes, that assumption will no longer be
in force, and people who maintain packages that install files into /
will want to simplify their packaging by installing everything in /usr
instead.  If they also want to change the binary packages that ships
some of those files at any point in the same release cycle -- kaboom.

zw

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Richard Laager  writes:

> Is this particularly hard to fix, though?

> Can't we just do something like the following pseudo-code:

[...]

> (I've used /bin -> /usr/bin as an example, but the canonicalization must
> deal with the other merged paths too.)

> The second bit ensures that all new operations write canonicalized paths
> (e.g. /usr/bin/foo) into the database regardless of whether the .deb 
> shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in
> sync with the filesystem moving forward.

> The first bit performs a one-time canonicalization of paths in the
> existing database. An on-disk flag ensures that this is run only once 
> (for performance reasons only, because it's already idempotent). This
> corrects the existing database-filesystem mismatch.

> The one-time canonicalization can be removed once non-usrmerged systems
> are no longer supported. The second bit needs to live (probably a lot) 
> longer, until it's no longer reasonable to install a .deb containing
> non-usrmerged paths (which might be old or from a third party).

> Am I missing something here?

I also don't understand why this isn't the correct solution.  It seems
obvious to me that we should simply teach dpkg about path aliases and have
it update both its internal state and its processing of new packages to
canonicalize paths so that it has an accurate representation of the world,
and then ensure dpkg is upgraded first.

Are we missing something?  If not, what is blocking this solution?  Is it
simply a matter of someone digging into dpkg's code sufficiently to put
these changes in the correct location?  Or is there some other issue?

It seems like a huge waste of resources to me to do archive-wide package
inspection to try to find patterns that might cause problems, and to ask
all maintainers to remember this rather obscure rule, when we could just
fix dpkg to make the problem go away.

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Paul Gevers]

Hi

On 18-11-2021 22:44, Marco d'Itri wrote:

On Nov 18, Zack Weinberg  wrote:


Are you seriously claiming that that phenomenon is not a severity:critical bug?

I am seriously claming that whatever you are referring to, if true, is
such a contrived example that does not actually happen in real life
(or at least, it does not happen frequently enough).


I'm thinking of bug #972936 and bug #953562. Did I remember those 
wrongly as being an example of this?


Paul



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Michael" == Michael Biebl  writes:

Michael> Am 17.11.2021 um 19:57 schrieb Sam Hartman:
Michael> AIUI simply moving files from / to /usr within the same
Michael> package is not problematic.

Sorry, I was being overly simplistic.
I think your understanding is mostly correct.
You need to make sure that in the same release you do not both
move files from / to /usr

and then later move files between packages.

It's not just that you can't do it at the same time.
It's that it cannot happen within a given release cycle.

Your packaging team may be organized enough to make sure you don't do
that.
And I guess if you've got some plan to make sure you never trip the
problematic conditions it would be okay.

I don't think the project as a whole is taht organized.
So, for example, I think it would be problematic for general tools like
debhelper to move files, because we cannot guarantee that maintainers
will not later (or earlier) move those files between packages.

But we could you know fix dpkg:-)
I'm sure that will be painful at the political level, but personally I
think it needs doing.

Anyway, thanks for calling me out on being imprecise.  There's enough
FUD going around in this discussion I do not want to add to it.

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Marco" == Marco d'Itri  writes:

Marco> On Nov 18, Zack Weinberg  wrote:
>> Are you seriously claiming that that phenomenon is not a
>> severity:critical bug?
Marco> I am seriously claming that whatever you are referring to, if
Marco> true, is such a contrived example that does not actually
Marco> happen in real life (or at least, it does not happen
Marco> frequently enough).

It was convincing enough to convince me, the TC and several other
participants in the discussion.

--Sam

Re: merged-/usr transition: debconf or not?

[Sven Joachim]

On 2021-11-19 15:37 +0100, Michael Biebl wrote:

> On 19.11.21 11:58, Philip Hands wrote:
>> Ansgar  writes:
>>
 * doing this will, in a non-negligible number of cases, trigger the
 bug to manifest on systems where that package is upgraded from a
 version where the move had not taken place to one where it has.
>>>
>>> Why do you claim that?
>>>
>>> Given packages already did such moves in the last years and you claim
>>> this happens in a non-negligible number of cases, could you please
>>> point to some examples where this already happens in practice?
>> My understanding is that in order to trigger this bug you need at
>> least
>> to both move a file from one place to the other, and also to rename the
>> package that contains that file or move ownership to another package.
>> I suspect that you might also need to be unlucky with the order that
>> apt/dpkg decides to do the installation and, depending upon how far
>> apart the move and the rename happens, also unlucky with your choice of
>> from and to versions of the packages in question.
>
> Right. I think it would be immensely useful to have an actual
> reproducer so one could study the issue more closely or at least a bug
> report, which describes the issue in more detail, like the exact
> circumstances when this can happen.
> So far this is merely theoretical, right?
> Or do we have a documented instance of this happening?

#953562 seems to be such an instance.

Cheers,
   Sven

Re: merged-/usr transition: debconf or not?

[Marc Haber]

On Fri, 19 Nov 2021 11:58:39 +0100, Philip Hands 
wrote:
>Given that these bugs are going to be utter bastards to reproduce, and
>you can be sure that we'll have enough diversity in installed systems
>that some people are going to manage to be sufficiently unlucky, it
>would be nice to know the sort of damage we might expect.
>
>It strikes me that we ought to be able to screen our own repos for
>packages that could be able to tickle this bug. That would give us the
>chance to look at what sorts of files we might realistically expect to
>be clobbered, it should give some indication of how many packages we
>should expect to be able to trigger this, and knowing this might suggest
>plausible work-arounds.

This is one of the cases where I wish that Debian would be a more
centrally organized project. Red Hat or SuSE would just fix their
package management and go on with their business.

It's a pity that we have actually THINK about alternatives to that
trivial and obvious approach because we leave our core package
maintainers too much freedom to stall.

Grüße
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: merged-/usr transition: debconf or not?

[Michael Biebl]

On 19.11.21 11:58, Philip Hands wrote:

Ansgar  writes:


* doing this will, in a non-negligible number of cases, trigger the
bug to manifest on systems where that package is upgraded from a
version where the move had not taken place to one where it has.


Why do you claim that?

Given packages already did such moves in the last years and you claim
this happens in a non-negligible number of cases, could you please
point to some examples where this already happens in practice?


My understanding is that in order to trigger this bug you need at least
to both move a file from one place to the other, and also to rename the
package that contains that file or move ownership to another package.

I suspect that you might also need to be unlucky with the order that
apt/dpkg decides to do the installation and, depending upon how far
apart the move and the rename happens, also unlucky with your choice of
from and to versions of the packages in question.


Right. I think it would be immensely useful to have an actual reproducer 
so one could study the issue more closely or at least a bug report, 
which describes the issue in more detail, like the exact circumstances 
when this can happen.

So far this is merely theoretical, right?
Or do we have a documented instance of this happening?



Given that these bugs are going to be utter bastards to reproduce, and
you can be sure that we'll have enough diversity in installed systems
that some people are going to manage to be sufficiently unlucky, it
would be nice to know the sort of damage we might expect.

It strikes me that we ought to be able to screen our own repos for
packages that could be able to tickle this bug. That would give us the
chance to look at what sorts of files we might realistically expect to
be clobbered, it should give some indication of how many packages we
should expect to be able to trigger this, and knowing this might suggest
plausible work-arounds.

Of course, that doesn't help with packages from third-party repos,
including our downstreams, but at present we seem to be discussing this
with very little hard data.

It occurs to me that one could lose quite a few files on the average
Debian install (if they were selected at random) without even noticing,


Shouldn't debsums be able to detect such missing files (at least for the 
vast majority of packages which ship a md5sums file).


I run that semi-regularly on all of my systems, most of them are 
/usr-merged and I haven't noticed any missing files yet which I could trace



whereas a very few files would render systems unbootable, so knowing a
bit more about which files are realistically at risk would be very
helpful in understanding the severity of the problem.

If anyone's got good ideas about how to gather this information, I'm
very happy to help with the effort to do so.


I'd be more then happy to help here as well.

Regards,
Michael




OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[David Kalnischkies]

On Fri, Nov 19, 2021 at 10:06:13AM +0100, Ansgar wrote:
> Given packages already did such moves in the last years and you claim
> this happens in a non-negligible number of cases, could you please
> point to some examples where this already happens in practice?

You need a / → /usr¹ and a pkg-a → pkg-b move in the same release.
Also, you need to have (the new) pkg-b be unpacked before pkg-a.

An example³ would be coreutils/util-linux/… moving everything from /bin
to /usr/bin and in the same Debian release splitting out one (or more)
of their tools into their own package (as they usually do).
As those are essential they will Pre-Depend² on the split out package
which guarantees that pkg-b will be unpacked before pkg-a.

The result is that the split out tool will be gone from /usr-merged
systems – which at that point should be all systems.


Another example would be systemd files debhelper moved for some time
already while the package does a foo and foo-data split in the same
Debian release. Just need to "solve" the unpack order now, but I will
leave that as an exercise for the reader.



The move and reorganisation is both forbidden by the CTTE for Debian 12
in "Moratorium on moving files' logical locations into /usr" which even
describes this problem as one of the reasons for it, but hopes to have
it resolved by 13 (without mentioning how).

Are you suggesting that Debian will use 13 to move each and every
file in / to its /usr counter-path while forbidding that packages
including moves are reorganised before 14 ?

Good thing 'which' isn't in /bin I guess. (SCNR)

Disclaimer: I am as usual not arguing for switching into full speed
reverse mode. I would "just" prefer if we acknowledge that problems
exist we have to deal with. Its gonna be hard enough to actually resolve
them given all bridges have been burned down years ago by pretending its
not a problem that dpkg has no idea what is done behind its back to the
files its supposed to manage.

(The problem itself isn't unique⁴ to /usr-merge, so ideally it would be
 resolved regardless, but /usr-merge undoubtedly makes heavy use of it
 so in an ideal world those interested in it would not only acknowledge
 the problems but actually work together to resolve them.
 Sadly, that isn't the world we seem to be stuck in at all.)


Best regards

David Kalnischkies

¹ You could of course also move the other way around.
² You can achieve the same with other dependency types, it is just
  harder to trigger in simple testcases as apt & dpkg try to avoid
  the auto-deconfiguration of pkg-a if there is an easy way out.
³ For fun I wrote an apt testcase with coreutils splitting out ln⁴,
  that might be a bit unrealistic, but you get the idea (attached).
⁴ as, you know, /usr-merge being the last symlink we will ever need
#!/bin/sh
set -e

TESTDIR="$(readlink -f "$(dirname "$0")")"
. "$TESTDIR/framework"

setupenvironment
configarchitecture 'native'

#mkdir -p rootdir/bin
ln -s usr/bin rootdir/bin

touch ln

mkdir -p tree/bin
cp -a ln tree/bin
buildsimplenativepackage 'coreutils' 'native' '1' 'stable' '' '' '' '' 
'tree/bin'
rm -rf tree

buildsimplenativepackage 'coreutils' 'native' '2' 'unstable' 'Pre-Depends: 
unneeded-ln'

mkdir -p tree/usr/bin
cp -a ln tree/usr/bin
buildsimplenativepackage 'unneeded-ln' 'native' '2' 'unstable' 'Breaks: 
coreutils (<< 2)
Replaces: coreutils (<< 2)' '' '' '' 'tree/usr'
rm -rf tree

setupaptarchive

testfailure test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln
testsuccess apt install coreutils=1 -y
testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln

testsuccess apt full-upgrade -y
testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Philip Hands]

Ansgar  writes:

>> * doing this will, in a non-negligible number of cases, trigger the
>> bug to manifest on systems where that package is upgraded from a
>> version where the move had not taken place to one where it has.
>
> Why do you claim that?
>
> Given packages already did such moves in the last years and you claim
> this happens in a non-negligible number of cases, could you please
> point to some examples where this already happens in practice?

My understanding is that in order to trigger this bug you need at least
to both move a file from one place to the other, and also to rename the
package that contains that file or move ownership to another package.

I suspect that you might also need to be unlucky with the order that
apt/dpkg decides to do the installation and, depending upon how far
apart the move and the rename happens, also unlucky with your choice of
from and to versions of the packages in question.

Given that these bugs are going to be utter bastards to reproduce, and
you can be sure that we'll have enough diversity in installed systems
that some people are going to manage to be sufficiently unlucky, it
would be nice to know the sort of damage we might expect.

It strikes me that we ought to be able to screen our own repos for
packages that could be able to tickle this bug. That would give us the
chance to look at what sorts of files we might realistically expect to
be clobbered, it should give some indication of how many packages we
should expect to be able to trigger this, and knowing this might suggest
plausible work-arounds.

Of course, that doesn't help with packages from third-party repos,
including our downstreams, but at present we seem to be discussing this
with very little hard data.

It occurs to me that one could lose quite a few files on the average
Debian install (if they were selected at random) without even noticing,
whereas a very few files would render systems unbootable, so knowing a
bit more about which files are realistically at risk would be very
helpful in understanding the severity of the problem.

If anyone's got good ideas about how to gather this information, I'm
very happy to help with the effort to do so.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Ansgar]

On Thu, 2021-11-18 at 23:01 -0500, The Wanderer wrote:
> * to date, package maintainers have not yet begun moving
> already-packaged files from / to /usr/ (specifically because doing so
> would break systems that have not yet been migrated to merged-/usr,
> and Debian has not yet declared that such systems are unsupported),

That claim is incorrect:

- Some packages have moved files from /(s)bin to /usr/(s)bin. (This
  sometimes requires a compat symlink.)

- More packages have moved files from /lib to /usr/lib. (This often
  doesn't require any extra care.)

> * after bookworm, package maintainers will start moving already-
> packaged files from / to /usr/, and

s/start/continue/

> * doing this will, in a non-negligible number of cases, trigger the
> bug to manifest on systems where that package is upgraded from a
> version where the move had not taken place to one where it has.

Why do you claim that?

Given packages already did such moves in the last years and you claim
this happens in a non-negligible number of cases, could you please
point to some examples where this already happens in practice?

Ansgar

Re: merged-/usr transition: debconf or not?

[Marco d'Itri]

On Nov 18, Zack Weinberg  wrote:

> Are you seriously claiming that that phenomenon is not a severity:critical 
> bug?
I am seriously claming that whatever you are referring to, if true, is 
such a contrived example that does not actually happen in real life
(or at least, it does not happen frequently enough).

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[The Wanderer]

On 2021-11-18 at 20:06, Luca Boccassi wrote:

> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
> 
>> Luca Bocassi wrote:

>>> [merged /usr] is the default. It has been the default for
>>> multiple releases of multiple distributions. All Ubuntu
>>> installations that were not using this default are now forcibly
>>> converted upon upgrade to 21.10.
>>> 
>>> And yet nobody has actually seen [the file disappearance bug]
>>> happen, to the best of my knowledge.
>> 
>> I already explained why that doesn't prove the bug is a non-issue.
>> To the contrary; it means there is an enormous installed base of
>> systems where the bug is latent, waiting to cause problems under
>> conditions which we can reasonably expect to occur shortly after
>> the release of bookworm.  Please do not make me repeat myself.
>> 
>> zw
> 
> I'm afraid you have not. Why would the release of bookworm make any
> difference? There will be nothing new that hasn't already been
> happening for years.

I interpret Zack's comment as referring to this, which he said in
https://lists.debian.org/debian-devel/2021/11/msg00205.html:

> [P]eople aren't doing the package changes that trigger the bug, yet.
> They can't, because that would break systems where /usr hasn't been
> merged. If the bug is not fixed I expect it will start causing
> problems in unstable *after* bookworm, since (as I understand the
> current transition plan) bookworm+1 development is the earliest that
> package maintainers may assume /bin is a symlink.

IOW, I interpret him as disagreeing with you that "there will be nothing
new that hasn't already been happening for years". Specifically, I parse
him as arguing that:

* to date, package maintainers have not yet begun moving
already-packaged files from / to /usr/ (specifically because doing so
would break systems that have not yet been migrated to merged-/usr, and
Debian has not yet declared that such systems are unsupported),

* after bookworm, package maintainers will start moving already-packaged
files from / to /usr/, and

* doing this will, in a non-negligible number of cases, trigger the bug
to manifest on systems where that package is upgraded from a version
where the move had not taken place to one where it has.

(Zack, if I've gotten any of those wrong, please don't hesitate to
correct me; I'll either apologize, or drop right back out of the
discussion to go hide in a metaphorical hole, if not both.)

Do you dispute any of those three points? If so, I'd be interested to
know which one(s), and why.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Luca Boccassi]

On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
Luca Bocassi wrote:
...
> [merged /usr] is the default. It has been the
> default for multiple releases of multiple distributions. All Ubuntu
> installations that were not using this default are now forcibly
> converted upon upgrade to 21.10.
> 
> And yet nobody has actually seen [the file disappearance bug]
> happen, to the best of my knowledge.

I already explained why that doesn't prove the bug is a non-issue.  To the 
contrary; it means there is an enormous installed base of systems where the bug 
is latent, waiting to cause problems under conditions which we can reasonably 
expect to occur shortly after the release of bookworm.  Please do not make me 
repeat myself.

zw

I'm afraid you have not. Why would the release of bookworm make any
difference? There will be nothing new that hasn't already been
happening for years.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

Luca Bocassi wrote:
...
> [merged /usr] is the default. It has been the
> default for multiple releases of multiple distributions. All Ubuntu
> installations that were not using this default are now forcibly
> converted upon upgrade to 21.10.
>
> And yet nobody has actually seen [the file disappearance bug]
> happen, to the best of my knowledge.

I already explained why that doesn't prove the bug is a non-issue.  To the 
contrary; it means there is an enormous installed base of systems where the bug 
is latent, waiting to cause problems under conditions which we can reasonably 
expect to occur shortly after the release of bookworm.  Please do not make me 
repeat myself.

zw

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

Marco d'Itri wrote:
> On Nov 18, Zack Weinberg  wrote:
>> as it has proven to be a genuinely critical problem
> No, it has not.

In the previous long thread on debian-devel on this subject, someone posted a 
step-by-step recipe to reproduce a phenomenon where a file that has been moved 
(in its package) from an installation path of /bin (for example) to /usr/bin, 
possibly in conjunction with a change to which package owns it, while /bin is a 
symlink to /usr/bin, will disappear from the file system when the updated set 
of packages is installed.  (I regret I do not have time today to dig up the 
exact email in question.)

Are you seriously claiming that that phenomenon is not a severity:critical bug?

zw

Re: merged-/usr transition: debconf or not?

[Luca Boccassi]

On Thu, 2021-11-18 at 13:09 +0100, Marco d'Itri wrote:
> On Nov 18, Zack Weinberg  wrote:
> 
> > And, as it has proven to be a genuinely critical problem, it is also their
> No, it has not.

Indeed - it seems to me there's a convenient tendency to forget that
this is not something new that has never been seen before, to be
introduced without any testing. This is the default. It has been the
default for multiple releases of multiple distributions. All Ubuntu
installations that were not using this default are now forcibly
converted upon upgrade to 21.10.

And yet nobody has actually seen this happen, to the best of my
knowledge. So either this combination that would allegedly not work
hasn't been used once in 4+ years across multiple distros, or the
required alignment of coincidences to make the issue happen is so rare
that it just never happens in practice.

All we have is a proof of concept. By all means, if anybody cares
enough, go and fix it, no problem with that. Propose an actual, working
alternative that gives the same result too - will happily spend my time
to test it. And put in place QA checks and whatnot to be sure - that
seems a great idea.

But talks of blocking everything and reverting things with a hacky
script that has never actually been used before, in the face of
multiple decisions by the tech committee and no evidence whatsoever of
real-world impact, and despite tens of thousands of actual, real bugs
affecting Debian that don't get even a fraction of the same treatment
(even the Replaces: feature has been affected by unrelated, actual,
reported bugs, and might very well still be, haven't checked) seems to
me a tiny bit hyperbolic and exagerated.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged-/usr transition: debconf or not?

[Marco d'Itri]

On Nov 18, Zack Weinberg  wrote:

> And, as it has proven to be a genuinely critical problem, it is also their
No, it has not.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Bjørn Mork]

Michael Biebl  writes:
> Am 17.11.2021 um 19:57 schrieb Sam Hartman:
>> The question is whether we ever get to a place where people can update
>> files in a package currently installed to /bin/foo and instead install
>> them to /usr/bin/foo.
>> We have a consensus that dpkg bugs make that a bad idea.
>
> Is that really so? If I'm not mistaken, the problematic part was
> moving files from / to /usr and at the same time moving files between
> packages. So doing that at the same time can be problematic. If that
> would affect many packages is another question.
>
> AIUI simply moving files from / to /usr within the same package is not
> problematic.

Won't you then end up with a package which cannot be split for the next
two releases?  Maybe not a big problem, but I think it will be so hard
to keep track of that it's better to not move the files.


Bjørn

Re: merged-/usr transition: debconf or not?

[Richard Laager]

On 11/17/21 8:18 PM, Zack Weinberg wrote:

However, I think it was the responsibility of the developers of usrmerge
to find out how bad that problem actually was, rather than dismissing it.
And, as it has proven to be a genuinely critical problem, it is also their
responsibility to fix it, per the 'no one can be forced to do anything' rule.

At least for this email, I'll stipulate to everything above.

Is this particularly hard to fix, though?

Can't we just do something like the following pseudo-code:



/* At dpkg start... */
usrmerged = FALSE;
if (the_flag_is_set) {
usrmerged = TRUE;
} elif (bin symlinked to /usr/bin) {
usrmerged = TRUE;
/* Update e.g. /bin/foo to /usr/bin/foo in the database. */
update_database();
set_the_flag();
}


/* As dpkg is deciding to write a file path into the database. */
/* e.g. the package is trying to install to /bin/foo, put
 * /usr/bin/foo into the database. */
if (usrmerged) {
path = canonicalize(path);
}



(I've used /bin -> /usr/bin as an example, but the canonicalization must 
deal with the other merged paths too.)


The second bit ensures that all new operations write canonicalized paths 
(e.g. /usr/bin/foo) into the database regardless of whether the .deb 
shipped with /bin/foo or /usr/bin/foo. This ensures the database stays 
in sync with the filesystem moving forward.


The first bit performs a one-time canonicalization of paths in the 
existing database. An on-disk flag ensures that this is run only once 
(for performance reasons only, because it's already idempotent). This 
corrects the existing database-filesystem mismatch.


The one-time canonicalization can be removed once non-usrmerged systems 
are no longer supported. The second bit needs to live (probably a lot) 
longer, until it's no longer reasonable to install a .deb containing 
non-usrmerged paths (which might be old or from a third party).


Am I missing something here?

--
Richard



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

>> "Sam" == Sam Hartman  wrote:
>> "Zack" == Zack Weinberg  writes:

Sam> There's a third option.  We sit around in the state where /bin is
Sam> a symlink, but where you cannot move files to /usr paths in the
Sam> package system until the bug is fixed.

Zack> I guess that is a potential outcome.  In a sense we are
Zack> already in that state, given the installed base of systems
Zack> where /bin is already a symlink.
Zack> I don't *like* that outcome; I think it's asking for lots and
Zack> lots of accidental breakage in unstable post-bookworm.

Sam> I don't like that outcome either.
Sam> I think that we have reasonable mitigations for the specific problem you
Sam> describe.
Sam> In particular, we do as I understand it have QA plans to build both with
Sam> merged /usr and without prior to bookworm to catch problems like the
Sam> ones you describe.
Sam> After bookworm releases it's totally fine if programs reference
Sam> filesystem paths like /usr/bin/bash, so long as they don't move where
Sam> things get installed.

I don't think that will mitigate the problem.  I think that post-bookworm,
packages like coreutils and libc6 are going to notice, in their upstream
configure scripts, that /bin etc are symlinks into /usr, and start installing
stuff that used to be in / into /usr.  Maintainers are going to need to take
positive action to _prevent_ the move, and there will no longer be automation
to detect that files have moved.

Sam> I.E. I don't think the transition is going to get canceled; we're
Sam> too far down that path.

Zack> *Are* we?  It seems to me that we could still, at this point,
Zack> [roll back the transition]

Sam> I don't think we would find people to do enough testing to
Sam> validate that was a reasonable thing to do.  But also, the usrmerge
Sam> proponents get most of what they wanted even if we're stuck in the
Sam> middle.

That is exactly why I think a rollback should not be taken off the table:
it is very, very bad if the usrmerge proponents get most of what they want
while the rest of the project is left to clean up their mess.  The project
cannot afford to reward such misbehavior.  I honestly think the social
fallout of allowing that to happen would be *worse* than the social and
technical fallout of forcing a rollback through.

...
Sam> I want to reiterate that the initial process surrounding usrmerge was
Sam> horrible. The dpkg maintainer had what turned out to be legitimate
Sam> concerns that were not reasonably addressed. I think that's in part
Sam> because they were presented poorly and in part because people didn't
Sam> want to hear them. That's a real process failure.
...
Sam> I think people on both sides were unwilling to listen to each other.

I'm an outside observer and I have not followed this argument closely from
the beginning, but my perception of it is much more one-sided than you are
making it out to be.  This is what _I_ saw:

 - usrmerge was deployed to unstable in late 2014.  It's not clear to me
   how much testing it got in 2015.
 - Reports of it causing problems for dpkg started to appear in early 2016
   (e.g. #810499).  These seem to have been addressed civilly, but even then
   a "well it works for _me_ so :shrug:" attitude from the merged-/usr
   proponents is evident in the bug logs.
 - The dpkg maintainer filed a severity:serious bug on usrmerge in late
   2016 (#848622), saying that it breaks dpkg -S.  This is the earliest
   instance I can find of Marco in particular overtly blowing off a bug
   report that he didn't think was significant ("Over one year of
   experience with merged-/usr systems has not exposed any failures",
   which is an invalid argument).  This bug is also significant in
   hindsight because, although not described as such, it appears to me
   to have the same root cause as the file lossage on upgrade that you
   and I, at least, agree is severity:critical.
 - Over the next, um, *five years*, Marco continued to ignore or reject
   Guillem's attempts to get him to take problems seriously that were
   caused by usrmerge rendering the dpkg database inconsistent with the
   file system.
 - Marco also reacted hostilely to concerns raised by others, e.g.
   Adrian Bunk (#863269).

 - Guillem, for his part, has displayed far more patience than I would
   have in his situation, repeatedly attempting to explain why there is
   a serious problem, offering concrete solutions - that they may not be
   practical is not the point; he's done more work toward that end than
   anyone else - and never, that I have seen, losing his temper.
   At the same time, he has said in so many words that this has caused
   him to become burnt out on Debian in general and dpkg maintenance in
   particular.

There's no "both sides were unwilling to listen" when all one side is
bringing to the table is "there's no problem, your concerns are bullshit"
(the word "bullshit" is actually used in #863269).

Maybe it was not obvious to anyone in 2016 that the package 

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Zack" == Zack Weinberg  writes:

Zack> On Wed, Nov 17, 2021, at 1:37 PM, Sam Hartman wrote:
>>> "Zack" == Zack Weinberg  writes:
Zack> Therefore: either someone fixes the bug, or the transition
Zack> will have to be canceled.  It appears to me that the tech ctte
Zack> agrees with all of this.
>> 
>> There's a third option.  We sit around in the state where /bin is
>> a symlink, but where you cannot move files to /usr paths in the
>> package system until the bug is fixed.

Zack> I guess that is a potential outcome.  In a sense we are
Zack> already in that state, given the installed base of systems
Zack> where /bin is already a symlink.

Zack> I don't *like* that outcome; I think it's asking for lots and
Zack> lots of accidental breakage in unstable post-bookworm, as
Zack> packages are rebuilt on systems that now have /bin a symlink.

I don't like that outcome either.
I think that we have reasonable migigations for the specific problem you
describe.
In particular, we do as I understand it have QA plans to build both with
merged /usr and without prior to bookworm to catch problems like the
ones you describe.
After bookworm releases it's totally fine if programs reference
filesystem paths like /usr/bin/bash, so long as they don't move where
things get installed.

So, while I agree with you getting stuck in the middle is not a good
place, 
I don't think it happens to suffer from the problems you describe.

>> I.E. I don't think the transition is going to get canceled; we're
>> too far down that path.

Zack> *Are* we?  It seems to me that we could still, at this point,
Zack> pull usrmerge from testing and stable, push point updates to
Zack> the installers for all supported releases to flip the default
Zack> back to non-merged /usr, and advise the installed base to run
Zack> dpkg-fsys-usrunmess before their next apt upgrade.

That seems crazy to me: dpkg-fsys-usrunmess has almost certainly
involved less testing than usrmerge.  Talk about not having people to do
the work: I don't think we would find people to do enough testing to
validate that was a reasonable thing to do.  But also, the usrmerge
proponents get most of what they wanted even if we're stuck in the
middle.  If you want a minimal root because you care about containers,
ostrees, and the like, you don't care that much where the package
database thinks files are being installed.

You've been working on this for years and you've gotten most of what you
hoped for.
You're going to fight really hard if someone tries to take that away.
You can easily fight hard enough to make that cost much higher than the
cost of fixing dpkg.
And you can present a reasonably looking political front: hey, we've
been working on this for years, the TC made a decision, the TC later
gave advice, we're basically just done except for this one dpkg bug.
Backing out is insane; just fix dpkg.

I want to reiterate that the initial process surrounding usrmerge was
horrible.
The dpkg maintainer had what turned out to be legitimate concerns that
were not reasonably addressed.
I think that's in part because they were presented poorly and in part
because people didn't want to hear them.
That's a real process failure.
We should learn from that mistake and do better in the future.

We don't have the political will to unwind years of work because we made
that process error, and I don't think it would be a good idea to do so
even if we did.
I do think we should be careful to be better about listening to each
other in the future.
We don't want a community where you can get your way by ignoring
legitimate objections.
I think people on both sides were unwilling to listen to each other.

And yes, if the dpkg maintainer had asked for usrmerge to block on dpkg
gaining support for aliasing, and if that had been done around the time
it was proposed to change the default for debootstrap, we as a project
should have listened.
That's not what happened though.
The dpkg maintainer refused to consider the usrmerge approach that the
usrmerge proponents wanted.
He proposed a different scheme that didn't actually meet the needs of
the usrmerge proponents.
He was unwilling to take the time to understand why his proposal didn't
meet other people's needs.

And yes, the usrmerge proponents (and especially the TC) should have
worked harder to understand the actual underlying objections.


So, there's a huge chunk of ways we could improve to go around--enough
for everyone involved.


But reliving past battles as anythying other than a way to do better in
the future will drag us down.

--Sam


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Wed, Nov 17, 2021, at 1:37 PM, Sam Hartman wrote:
>> "Zack" == Zack Weinberg  writes:
> Zack> Therefore: either someone fixes the bug,
> Zack> or the transition will have to be canceled.  It appears to me
> Zack> that the tech ctte agrees with all of this.
>
> There's a third option.
> We sit around in the state where /bin is a symlink, but where you cannot
> move files to /usr paths in the package system until the bug is fixed.

I guess that is a potential outcome.  In a sense we are already in that state, 
given the installed base of systems where /bin is already a symlink.

I don't *like* that outcome; I think it's asking for lots and lots of 
accidental breakage in unstable post-bookworm, as packages are rebuilt on 
systems that now have /bin a symlink.  But I can't personally offer to fix 
dpkg, either (way oversubscribed on other projects, and this doesn't seem like 
a job to be tackled by someone new to dpkg, tbh).

> I.E. I don't think the transition is going to get canceled; we're too
> far down that path.

*Are* we?  It seems to me that we could still, at this point, pull usrmerge 
from testing and stable, push point updates to the installers for all supported 
releases to flip the default back to non-merged /usr, and advise the installed 
base to run dpkg-fsys-usrunmess before their next apt upgrade.  It'd be ugly 
but it might well be *less* ugly than being stuck in the state you describe.  I 
understood the tech-ctte to be explicitly holding that option open.

The proponents of merged /usr would be unhappy, but that does not bother me.

zw

Re: merged-/usr transition: debconf or not?

[Michael Biebl]

Am 17.11.2021 um 19:57 schrieb Sam Hartman:

The question is whether we ever get to a place where people can update
files in a package currently installed to /bin/foo and instead install
them to /usr/bin/foo.
We have a consensus that dpkg bugs make that a bad idea.


Is that really so? If I'm not mistaken, the problematic part was moving 
files from / to /usr and at the same time moving files between packages. 
So doing that at the same time can be problematic. If that would affect 
many packages is another question.


AIUI simply moving files from / to /usr within the same package is not 
problematic.


Sam, am I missing something here?

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Marco" == Marco d'Itri  writes:

Marco> On Nov 10, Sam Hartman  wrote:
>> I'm sorry, but I think the only way in which that horse is dead
>> is that no one has proposed patches to dpkg.
Marco> Indeed, because the sides of this argument are like three
Marco> people (one of them being the dpkg maintainer) versus
Marco> everybody else.  Since some significant work on dpkg is
Marco> reasonably not forthcoming then this is clearly not a viable
Marco> transition method.

Um, no one still involved in this thread is talking about transition
methods other than what the TC proposed.
We're all talking about transition methods where /bin becomes a symlink
to /usr/bin.

The question is whether we ever get to a place where people can update
files in a package currently installed to /bin/foo and instead install
them to /usr/bin/foo.
We have a consensus that dpkg bugs make that a bad idea.
That consensus is validated by the TC.
It's not clear to me that bug is likely to get fixed in the bookworm +1
time.

I don't think that means we halt the transition: the TC has made a
decision that we're going forward.
I do think it means probably even after the release of bookworm we
cannot rearrange where packages claim files get installed.
That's the horse I think is not dead.


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Zack" == Zack Weinberg  writes:
Zack> The existence of the bug is not in question.  A step-by-step
Zack> reproduction recipe was posted in the previous thread.  Losing
Zack> files from Essential packages when they are upgraded is
Zack> severity:critical

Agreed so far.

Zack> .  Therefore: either someone fixes the bug,
Zack> or the transition will have to be canceled.  It appears to me
Zack> that the tech ctte agrees with all of this.

There's a third option.
We sit around in the state where /bin is a symlink, but where you cannot
move files to /usr paths in the package system until the bug is fixed.

I.E. I don't think the transition is going to get canceled; we're too
far down that path.
I do think that it might be a critical bug to move packages from /bin to
/usr/bin in the packaging system in bookworm+1 if this bug doesn't get
fixed.

In other words, there are people who are motivated to do the work of
creating /bin->/usr/bin symlinks.
There may not be people motivated to make it possible for the later
packaging changes to happen.
That may be the work that doesn't get done until motivated people show
up to do it.


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

Luca Bonassi wrote:
> may I also remind participants in this thread that according to our
> Constitution (2.1), no project member is obliged to do work on
> anything they don't want to

Yes, and it follows that the people who want a change to happen are
the people who will have to do the work to make that change happen,
including fixing any bugs that are exposed by that change.  If they
don't want to do that work, and nobody else does either, then maybe
the change isn't going to get done.

As I said in the previous thread about this, I personally don't care
whether merged-/usr ever actually happens.  It is not relevant to what
I use Debian for.  So I am not motivated to do any work to make it
happen.

But I do very much care that the transition isn't botched, and right
now it looks like the greatest risk of the transition being botched
comes from proponents who are trying to rush to the end state without
fixing all the bugs exposed in the process.  Since one of the exposed
bugs involves files going missing from Required and Essential packages
upon seemingly-unrelated upgrades, during some indefinite period
*after* the transition, you can, I hope, see why I feel it necessary
to make a stink.

> [The bug where files disappear from packages upon being moved from
> /bin to /usr/bin or vice versa, after / is a symlink to /usr] to the
> best of my knowledge has not been actually observed in the wild
> despite this setup being the default for 100% of Ubuntu users who
> install/upgrade to 21.10, 100% of new Ubuntu installs since ~2018
> and an unspecified number of Debian installs being the default in
> our installer too for the past two stable releases

There's a very good reason for that: people aren't doing the package
changes that trigger the bug, yet.  They can't, because that would
break systems where /usr hasn't been merged.  If the bug is not fixed
I expect it will start causing problems in unstable *after* bookworm,
since (as I understand the current transition plan) bookworm+1
development is the earliest that package maintainers may assume /bin
is a symlink.

The existence of the bug is not in question.  A step-by-step
reproduction recipe was posted in the previous thread.  Losing files
from Essential packages when they are upgraded is severity:critical.
Therefore: either someone fixes the bug, or the transition will have
to be canceled.  It appears to me that the tech ctte agrees with all
of this.

> I dare say it would help their cause a great deal more, instead of
> rekindling flame wars on a mailing list,

Marco is the person who restarted the argument.  I will cheerfully
drop the subject again as soon as Marco acknowledges that (a) the bug
exists, and (b) it is a hard blocker for the transition.

zw

p.s. apologies for breaking threading, i'm not subscribed to d-d

Re: merged-/usr transition: debconf or not?

[Luca Boccassi]

On Fri, 2021-11-12 at 08:38 +0100, Ansgar wrote:
> Hi Svante,
> 
> On Thu, 2021-11-11 at 23:22 +0100, Svante Signell wrote:
> > I'm not sure he has the skill or experience enough to submit a patch to
> > dpkg. Complaining is much easier than proposing something constructive.
> 
> I would like to remind you that Debian expects somewhat decent behavior
> of contributors. Please see https://www.debian.org/code_of_conduct
> 
> Ansgar
> 

And I want to also add that even if the same request was expressed in
an acceptable form (it was not), may I also remind participants in this
thread that according to our Constitution (2.1), no project member is
obliged to do work on anything they don't want to, and hence it follows
that it is not appropriate to berate a fellow project member for this
reason, even if it was done in a respectful tone (which again it was
not, in this occasion).

So if any project member feels particularly strongly about this issue,
which to the best of my knowledge has not been actually observed in the
wild despite this setup being the default for 100% of Ubuntu users who
install/upgrade to 21.10, 100% of new Ubuntu installs since ~2018 and
an unspecified number of Debian installs being the default in our
installer too for the past two stable releases (and despite other
unrelated Replaces: bugs actually existing and being reported over
time), the constructive way forward would seem to me to contribute the
work themselves to solve it. I dare say it would help their cause a
great deal more, instead of rekindling flame wars on a mailing list, to
come here and say "Hey, here's an alternative
tool/patchset/software/mechanism that has the exact same result but
does not suffer from the issue we are worried about. Could you help us
test it, please?". I for one would be happy to help testing such
alternatives.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Fri, 2021-11-12 at 04:57 +0100, Guillem Jover wrote:
> 
> On Sun, 2021-08-22 at 11:21:38 +0100, Luca Boccassi wrote:
> > On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> > > On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> > > > The bug is real, nobody doubts that - it has been filed on dpkg 20
> > > > years ago.
> > > 
> > > You keep repeating this, but I have no idea what bug you refer to.
> > > 
> > > There's #148258 (from 2002), which is conffile related, and not
> > > actionable and should probably just be closed.
> > > 
> > > There's #182747 (from 2003), which while apparently similar is
> > > something else completely. This is about the (IMO) misfeature of
> > > supporting a local admin to redirect (not alias) a directory using a
> > > local symlink (mainly for space management reasons). For an
> > > explanation
> > > see .
> > > 
> > > There's #406715 (from 2007) which is related to the above misfeature.
> > 
> > I am referring to #134758 since it's linked as the root cause from
> > usrmerge's #848622.
> 
> Well, that's a bogus block then, because that's obviously not the root
> cause. I see I was CCed when that block was set, I guess I missed it.
> :/ Fixed it now…
> 
> > "dpkg-query: Make -S handle unowned symlinks resolving to owned
> > pathnames" filed in February 2002 - 19 years and a half ago. I refer to
> > that because it's linked as the root cause in the BTS of the relevant
> > issue with usrmerge we are discussing.
> 
> Even if the wishlist from that report got implemented, it would still
> not fully solve all the problems, where among them «dpkg-query -S»
> is probably the lesser one, which would not work in the other direction
> anyway (querying a path under /usr/ known to dpkg as being under /).
> 
> And then I'm not convinced this should even be implemented at all,
> as it would introduce behavior differences between literal pathnames
> and patterns, and making them slower (for the first case) or potentially
> extremely slower (for the second case), in addition to making the queries
> dependent on the on-disk layout (so unreliable from the packaging PoV,
> as it would invent on the spot, pathnames not truly coming from any
> package nor otherwise known to dpkg).
> 
> This for a misfeature in dpkg (supporting redirecting symlinks) that
> allowed the current mess anyway. So I'm inclined to wontfix and close
> that one.
> 
> 
> Not looking forward to further interactions…
> Guillem

Thanks for following up! There are currently 469 open bugs against
src:dpkg, it's of course entirely up to you which ones you choose to
fix and which one you close+wontfix. As far as workarounds go, this one
is really really trivial to deal with, so I personally wouldn't mind at
all.

Thank you for your work!

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged-/usr transition: debconf or not?

[Ansgar]

Hi Svante,

On Thu, 2021-11-11 at 23:22 +0100, Svante Signell wrote:
> I'm not sure he has the skill or experience enough to submit a patch to
> dpkg. Complaining is much easier than proposing something constructive.

I would like to remind you that Debian expects somewhat decent behavior
of contributors. Please see https://www.debian.org/code_of_conduct

Ansgar

Re: merged /usr vs. symlink farms

[Guillem Jover]

unblock 848622 by 134758
thanks

On Sun, 2021-08-22 at 11:21:38 +0100, Luca Boccassi wrote:
> On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> > On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> > > The bug is real, nobody doubts that - it has been filed on dpkg 20
> > > years ago.
> > 
> > You keep repeating this, but I have no idea what bug you refer to.
> > 
> > There's #148258 (from 2002), which is conffile related, and not
> > actionable and should probably just be closed.
> > 
> > There's #182747 (from 2003), which while apparently similar is
> > something else completely. This is about the (IMO) misfeature of
> > supporting a local admin to redirect (not alias) a directory using a
> > local symlink (mainly for space management reasons). For an
> > explanation
> > see .
> > 
> > There's #406715 (from 2007) which is related to the above misfeature.
> 
> I am referring to #134758 since it's linked as the root cause from
> usrmerge's #848622.

Well, that's a bogus block then, because that's obviously not the root
cause. I see I was CCed when that block was set, I guess I missed it.
:/ Fixed it now…

> "dpkg-query: Make -S handle unowned symlinks resolving to owned
> pathnames" filed in February 2002 - 19 years and a half ago. I refer to
> that because it's linked as the root cause in the BTS of the relevant
> issue with usrmerge we are discussing.

Even if the wishlist from that report got implemented, it would still
not fully solve all the problems, where among them «dpkg-query -S»
is probably the lesser one, which would not work in the other direction
anyway (querying a path under /usr/ known to dpkg as being under /).

And then I'm not convinced this should even be implemented at all,
as it would introduce behavior differences between literal pathnames
and patterns, and making them slower (for the first case) or potentially
extremely slower (for the second case), in addition to making the queries
dependent on the on-disk layout (so unreliable from the packaging PoV,
as it would invent on the spot, pathnames not truly coming from any
package nor otherwise known to dpkg).

This for a misfeature in dpkg (supporting redirecting symlinks) that
allowed the current mess anyway. So I'm inclined to wontfix and close
that one.


Not looking forward to further interactions…
Guillem

Re: merged-/usr transition: debconf or not?

[Svante Signell]

On Thu, 2021-11-11 at 16:07 -0500, Zack Weinberg wrote:
> Marco d'Itri  wrote:
> > Since some significant work on dpkg is reasonably not forthcoming
> 
> Yeah, because _you,_ Marco, prefer to spend your time trying to
> gaslight the project into thinking there isn't a critical-severity
> bug in usrmerge.  This could have all been over by now if you had
> rolled up your sleeves and written the necessary patches for dpkg
> when Guillem originally notified you of the problem (in 2016;
> #848622; the bug log does not reflect the actual severity, but again
> that appears to be all on you).

I'm not sure he has the skill or experience enough to submit a patch to
dpkg. Complaining is much easier than proposing something constructive.

Just my $5c.

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

Marco d'Itri  wrote:
> On Nov 10, Sam Hartman  wrote:
> > I'm sorry, but I think the only way in which that horse is dead is that
> > no one has proposed patches to dpkg.
> Indeed, because the sides of this argument are like three people (one of
> them being the dpkg maintainer) versus everybody else.

It's not a subject of debate.  The dpkg maintainer says that dpkg
does not support what usrmerge does, and that it can lead to package
corruption.  In the previous debian-devel thread on this, it was proven
empirically that he is correct.

> Since some significant work on dpkg is reasonably not forthcoming

Yeah, because _you,_ Marco, prefer to spend your time trying to
gaslight the project into thinking there isn't a critical-severity bug
in usrmerge.  This could have all been over by now if you had rolled
up your sleeves and written the necessary patches for dpkg when
Guillem originally notified you of the problem (in 2016; #848622; the
bug log does not reflect the actual severity, but again that appears
to be all on you).

zw

Re: merged-/usr transition: debconf or not?

[David Kalnischkies]

On Wed, Nov 10, 2021 at 01:48:07AM +0200, Uoti Urpala wrote:
> David Kalnischkies wrote:
> > As the transition hasn't started everyone not already merged is currently
> > deferring it. That is true for those who upgrade daily as well as for
> > those people who seemingly only upgrade their sid systems once in a blue
> > moon. So, at which point have all those systems stopped deferring?
> 
> I think the logical answer is that you're "deferring" in this sense if
> you are using the suggested flag file or whatever other mechanism to
> prevent the merge. Until you do an upgrade which would perform the
> merge without use of such a mechanism, your system is just out of date,
> not deferring.

A distribution upgrade is not atomic. Between an unpack of package foo
and the configure of foo a million other packages can pass through
various stages. Ideally, that window will be pretty small for usrmerge
the package (or whatever the transition mechanism will be in the end),
but that depends on various factors and easily balloons out of hand.
In a previous thread I mentioned how not too long ago the entire KDE
desktop environment had to be at least unpacked before dpkg could be
upgraded due to one tiny Conflicts (which was correct). If you hadn't
KDE installed dpkg was one of the first things upgraded even without
users going out of their way to explicitly request it (as it should
be, as its essential and apt does special dances for those).

So the easiest way to check if an upgrade on a "quantum state merge"
system is going to work is to keep it at unmerged for the entire time
and manually trigger the merge at the end as that is what could
theoretically happen, but is likely not for most testers.
If it works with merged is already checked by already merged systems.


> So presumably it is valid for packages to gain dependencies which force
> merge or "deferring" state on installation.

Valid perhaps, but I would hope that it isn't lightheartedly plastered
all around just in case as the guarantees it provides for the package
depending on the transition mechanism package are slim (as in, the
system might or might not be merged, regardless of deferred or not¹,
while depending package itself passes through various stages) to
non-existent² depending on the specific implementation of the transition
while it puts potentially enormous problems on the shoulders of dpkg and
apt to produce an acceptable ordering:

The package usrmerge is e.g. currently implemented in perl (the big one,
not -base) and so any other package implemented in perl is effectively
forbidden from forming dependencies on usrmerge as we otherwise run into
loops of the form app -> usrmerge -> perl -> app which might or might
not be breakable based on the dependency type (and version) of each ->.
Oh, and if you happen to have a dependency on something written in perl,
congrats, you are part of this elusive group as well as everything else
depending on you…

It will be hard enough to have one essential package trigger the
mechanism without running into issues, the last we need is a couple
other packages inserting themselves needlessly into the loop just
because "it is valid".


Best regards

David Kalnischkies

¹ Spoiler alert: Even a Pre-Depends technically only makes guarantees
  for the preinst scripts, not for the unpack itself, but that is fine
  print usually only encountered in the deeper horrors of loops… you
  need explicit approval for Pre-Depends anyhow.

² Spoiler alert: You can e.g. Pre-Depend all you want on dpkg, but that
  doesn't mean that the version you are pre-depending on is actually
  used to work on your package instead of just lying around on disk.
  That is true for a few other packages, the most obvious perhaps apt
  and the kernel.


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Adam Borowski]

On Wed, Nov 10, 2021 at 07:01:15PM +0100, Marco d'Itri wrote:
> On Nov 10, Sam Hartman  wrote:
> 
> > I'm sorry, but I think the only way in which that horse is dead is that
> > no one has proposed patches to dpkg.
> Indeed, because the sides of this argument are like three people (one of 
> them being the dpkg maintainer) versus everybody else.
> Since some significant work on dpkg is reasonably not forthcoming then 
> this is clearly not a viable transition method. 

Until we have that in place, going any further with usrmerge is not viable.

All the TC brought us so far is a wishlist, not a working implementation.
Thus, I'd say it's time to consider whether the trouble we have
(substantial, as evidenced by discussions still going and going) is worth
the gains (are there even any?).

So either you do the work to revive the horse or it stays put.


Meow.
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Polexit is brewing?  Let's skip that smelly Polsha and reactivate
⢿⡄⠘⠷⠚⠋⠀ the Free City of Danzig and/or reapply to the Hansa.
⠈⠳⣄

Re: merged-/usr transition: debconf or not?

[Marco d'Itri]

On Nov 10, Sam Hartman  wrote:

> I'm sorry, but I think the only way in which that horse is dead is that
> no one has proposed patches to dpkg.
Indeed, because the sides of this argument are like three people (one of 
them being the dpkg maintainer) versus everybody else.
Since some significant work on dpkg is reasonably not forthcoming then 
this is clearly not a viable transition method. 

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Marco" == Marco d'Itri  writes:

Marco> On Nov 08, Simon Richter  wrote:
>> Yes, but the conversion needs to be performed by dpkg, because
>> the usrmerge
Marco> Please kindly stop beating this long-dead horse.

I'm sorry, but I think the only way in which that horse is dead is that
no one has proposed patches to dpkg.

Simon has raised a valid issue with doing the transition in something
other than dpkg.
I don't quite agree with everything he says--for example I think it
might be possible for a package to cooperate with dpkg--but I do agree
with the base issue that eventually dpkg needs to learn about aliasing,
and that likely means that dpkg needs to be involved with usrmerge
transition either during the transition or after the fact.
(And if it is after, it's going to be more clunky).


Moreover, the TC  has effectively agreed with the issue by talking about
how  we don't want to move files in packages.

I don't think you need to block waiting for dpkg work, and I think it's
fine for usrmerge to become transitively essential and to go transition
systems.
But I don't think it is fair to say Simon's horse is dead or to try and
shut down discussion about how we're going to  eventually solve the dpkg
aliasing problem.
--Sam


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[David Kalnischkies]

On Tue, Nov 09, 2021 at 08:44:52PM +, Simon McVittie wrote:
> On Tue, 09 Nov 2021 at 19:01:18 +0100, David Kalnischkies wrote:
> > On Tue, Nov 09, 2021 at 03:21:25PM +, Simon McVittie wrote:
> > (Minus that for 12 it is technically still supported as long as it
> >  remains 12
> 
> No, it doesn't have to be supported, and the TC resolution explicitly
> said that it doesn't have to be supported.
> 
> What *does* need to be supported is the upgrade path from 11 to 12,
> or from current testing (11-and-a-bit) to 12, with any ordering of apt
> transactions that doesn't violate the packages' dependency conditions -
> and the TC's reasoning was that the simplest, most conservative, most
> robust way to make sure that continues to work was to mandate that all
> Debian 12 packages, individually, are installable onto unmerged-/usr
> Debian 11 (assuming that "installing a package" implies installing its
> dependencies, in any order that apt/dpkg consider to be valid and not
> breaking any dependency relationships).

Yes, any Debian 12.x package (even the very last security fix build for
it) needs to be installable on a Debian 11.y system as it could be part
of the upgrade from 11 to 12 and as it has no idea if its the first or
last package in that upgrade (within reason) it has to work on 11 as
well as on very-very-close-to-12.

As such, if you promise 11.x to 12.y upgrades I would expect 12.x to
12.y to work just as well as 12.x is very-very-close-to-12(.y).

If you say 12.x to 12.y isn't supported on unmerged it means effectively
that all "cattle" have to be constantly recreated as you can't have
a single package be considered an 'upgrade', they all need to be 'new
install' while e.g. installing build dependencies (as ironically a fully
upgraded 12 system is indistinguishable from an upgrade-in-progress-from-
11 system which just happens to install a bunch of new packages in the
end).

Its also quite a disaster for all systems already technically bookworm
like testing and sid as any upgrade, including to the release 12.0, will
be unsupported in your logic. Unsupported machines (aka our buildds)
building supported packages seems sad and I thought we had talked about
this before.


> > (Perhaps it comes with the job as apt maintainer, but I don't "discard
> >  and redo" systems or even chroots… upgrades until hardware failure…
> >  my current build chroots are from 2013. So I can totally see me opt-out
> >  first and later in… although probably more with apt_preferences)
> 
> For full systems that are managed as full systems ("pets" in the cattle
> vs. pets terminology), sure, do that; the Debian installation I'm typing
> this into has been copied from several older machines. However, deferring
> or avoiding the merged-/usr transition on these systems is not intended
> to be something that is considered valid for bookworm.

As the transition hasn't started everyone not already merged is currently
deferring it. That is true for those who upgrade daily as well as for
those people who seemingly only upgrade their sid systems once in a blue
moon. So, at which point have all those systems stopped deferring?

I would say that the first time you can say with absolute certainty that
a given system is no longer deferring the transition is the moment an
unpack of a trixie pkg is attempted as skipping releases is not
supported. All unpacks before that could happened on an unmerged
system as that system might very well be in the process of upgrading
from 11 to 12 at the moment.

(and btw, what I meant with me opting out for a while was delaying the
 upgrade of my sid "beasts" to a more exciting problem space than the
 first possible moment as that wouldn't be much of a test for apt, /usr
 merge and all those other packages installed around here. If I am
 asking for an upgrade path its only fair to not take the easiest road
 of transitioning to merged before anything could implicitly require
 it and hence fail for less lucky people not equipped to deal with it.
 My "eldritch horrors" are fine and behave, thanks for asking. )


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Simon McVittie]

On Tue, 09 Nov 2021 at 19:01:18 +0100, David Kalnischkies wrote:
> On Tue, Nov 09, 2021 at 03:21:25PM +, Simon McVittie wrote:
> > > As I see it the CTTE decision effectively allows the transition to be
> > > deferred until the moment you want to upgrade to 13.
> > 
> > I think you mean: until the moment you want to upgrade to testing after
> > Debian 12 release day. That's not Debian 13 *yet*
> 
> Yes, I meant that indeed… should have used codenames after all.

To be clear about this: when drafting the TC resolution, I did not intend
for this to "allow the transition to be deferred" and still leave you with
a supported system.

All I was aiming to do there was to make it possible to create an
unsupported, not-quite-Debian system that our QA infra can compare with
the supported Debian system that closely resembles it, so that for example
if a package builds differently on unmerged-/usr and merged-/usr systems,
we can flag it as "this is likely to break upgrades" and look more closely.

> > Speaking only for myself and not for the TC, I think a debconf question
> > would be OK as an implementation of this, but the debconf question should
> > indicate that the result of opting out is an unsupported system.
> 
> Sure.
> 
> (Minus that for 12 it is technically still supported as long as it
>  remains 12

No, it doesn't have to be supported, and the TC resolution explicitly
said that it doesn't have to be supported.

What *does* need to be supported is the upgrade path from 11 to 12,
or from current testing (11-and-a-bit) to 12, with any ordering of apt
transactions that doesn't violate the packages' dependency conditions -
and the TC's reasoning was that the simplest, most conservative, most
robust way to make sure that continues to work was to mandate that all
Debian 12 packages, individually, are installable onto unmerged-/usr
Debian 11 (assuming that "installing a package" implies installing its
dependencies, in any order that apt/dpkg consider to be valid and not
breaking any dependency relationships).

> (Perhaps it comes with the job as apt maintainer, but I don't "discard
>  and redo" systems or even chroots… upgrades until hardware failure…
>  my current build chroots are from 2013. So I can totally see me opt-out
>  first and later in… although probably more with apt_preferences)

For full systems that are managed as full systems ("pets" in the cattle
vs. pets terminology), sure, do that; the Debian installation I'm typing
this into has been copied from several older machines. However, deferring
or avoiding the merged-/usr transition on these systems is not intended
to be something that is considered valid for bookworm.

For installations that are meant to be predictable, like build chroots
and the chroots/containers used for automated QA systems, keeping a 2013
installation and successively upgrading it comes with a significant risk
of ending up with an installation that nobody else (including you, if
the underlying hardware fails!) can replicate without starting from a 1:1
backup of that installation. I would not recommend this approach:
I think installations that are meant to be predictable should be "cattle".

I had intended that preventing the /usr merge (with debconf preseeding or
Marco's flag file or whatever) would be something that people should only
be doing in those "cattle": the steps to reproduce those systems would
include applying the debconf preseeding or creating the flag file.

smcv

Re: merged-/usr transition: debconf or not?

[Simon McVittie]

On Tue, 09 Nov 2021 at 20:27:24 +0100, Bastian Blank wrote:
> On Tue, Nov 09, 2021 at 08:19:40PM +0100, Michael Biebl wrote:
> > I'm worried that by saying that unmerged is still supported in 12, we open a
> > can of worms and just punt this down to yet another release cycle.
> 
> No, unmerged will not be supported in 12.  Having the ability to create
> something does not make it supported.

That was my intention, yes. My intention when drafting the TC resolution
was that unmerged-/usr Debian 12 'bookworm' systems should be technically
possible but unsupported - similar to how downgrading packages is possible
but unsupported, and configuring dpkg --path-exclude is possible but
unsupported. If you contrive to create an unmerged bookworm system
and then try to upgrade it, that is likely to fail, and that's on you
to resolve.

> > E.g., what exactly does this mean for backports?
> 
> Stuff from backports is post-release, so requires a merged system.

Yes, I think this is right. IMO you can't validly enable bookworm-backports
without first upgrading to bookworm, by which time the automatic transition
mechanism called for by the TC resolution should have taken effect.

However, I think bookworm-updates and bookworm-security still need
to cope with unmerged-/usr systems to the same extent that bookworm
itself does, because it's valid to upgrade from bullseye to bookworm +
bookworm-updates + bookworm-security in a single step.

smcv

Re: merged-/usr transition: debconf or not?

[Bastian Blank]

On Tue, Nov 09, 2021 at 08:19:40PM +0100, Michael Biebl wrote:
> I'm worried that by saying that unmerged is still supported in 12, we open a
> can of worms and just punt this down to yet another release cycle.

No, unmerged will not be supported in 12.  Having the ability to create
something does not make it supported.

> E.g., what exactly does this mean for backports?

Stuff from backports is post-release, so requires a merged system.

Bastian

-- 
Star Trek Lives!

Re: merged-/usr transition: debconf or not?

[Michael Biebl]

On 09.11.21 19:01, David Kalnischkies wrote:

(Minus that for 12 it is technically still supported as long as it
  remains 12, but those who have to know will know that and everyone else
  is better of following the default anyhow)


I'm worried that by saying that unmerged is still supported in 12, we 
open a can of worms and just punt this down to yet another release cycle.

E.g., what exactly does this mean for backports?
If I provide a backport from bookworm+1, I don't think it's feasible 
(and intended) to roll back any changes related to usrmerge, so I think 
backports should be able to assume a bookworm system is usrmerged?


Simon, could you clarify your thoughts regarding this aspect?



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[David Kalnischkies]

On Tue, Nov 09, 2021 at 03:21:25PM +, Simon McVittie wrote:
> > As I see it the CTTE decision effectively allows the transition to be
> > deferred until the moment you want to upgrade to 13.
> 
> I think you mean: until the moment you want to upgrade to testing after
> Debian 12 release day. That's not Debian 13 *yet*, although you could

Yes, I meant that indeed… should have used codenames after all.


> > So, wouldn't it make sense to go with an (extreme) low priority debconf
> > question defaulting to 'yes, convert now' which [I think] non-experts
> > aren't bothered with and users/systems wanting to opt-out for the moment
> > (like buildds) have a standard way of preseeding rather than inventing a
> > homegrown flag-file and associated machinery?
> 
> Speaking only for myself and not for the TC, I think a debconf question
> would be OK as an implementation of this, but the debconf question should
> indicate that the result of opting out is an unsupported system.

Sure.

(Minus that for 12 it is technically still supported as long as it
 remains 12, but those who have to know will know that and everyone else
 is better of following the default anyhow)


> I had intended this to be for the class of systems that you would expect
> to discard and re-bootstrap rather than upgrading (chroots, lxc/Docker
> containers, virtual machines, etc. used for autopkgtest, piuparts,
> reproducible-builds, etc.), where a way to undo the opt-out isn't really
> necessary because the system is treated as disposable.

That is likely what happens to most of them, but as we support running
the merge somewhere between a few years ago and first unpack of a trixie
package anyhow I don't see the harm of having an official opt-out of the
opt-out as long as it happens in time.

(Perhaps it comes with the job as apt maintainer, but I don't "discard
 and redo" systems or even chroots… upgrades until hardware failure…
 my current build chroots are from 2013. So I can totally see me opt-out
 first and later in… although probably more with apt_preferences)


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Simon McVittie]

(Speaking only for myself, not the TC as a whole - but I wrote the first
draft of what became the TC resolution we're discussing.)

On Tue, 09 Nov 2021 at 15:19:53 +0100, David Kalnischkies wrote:
> On Mon, Nov 08, 2021 at 12:56:49PM +0100, Marco d'Itri wrote:
> > On Nov 08, Simon Richter  wrote:
> > > Right now, it is sufficient to preseed debconf to disallow the usrmerge
> > > package messing with the filesystem tree outside dpkg. Managed 
> > > installations
> > > usually have a ready-made method to do so.
> > This is not really relevant, since the conversion is mandatory.
> > The CTTE stated that the only exception is "Testing and QA systems 
> > should be able to avoid this transition, but if they do, they cannot be 
> > upgraded beyond Debian 12", and my plan is to arrange for this with 
> > a flag file.

For reference, the wording in the TC resolution is:

We anticipate that during the development cycle that will lead to
Debian 12, it is likely to be useful for testing and QA infrastructure
(such as autopkgtest, piuparts and/or reproducible-builds) to be able
to produce an installation of Debian testing/unstable that is not
merged-/usr, in order to be able to verify that packages targeted
for inclusion in Debian 12 can still be installed and/or built
successfully in a non-merged-/usr environment during partial upgrades.

As a result, we recommend that if there is an automatic migration from
non-merged-/usr to merged-/usr, it should be possible to prevent
that migration. **However, systems where that migration has been
prevented are not required to be fully-supported**, and in particular,
upgrading them to Debian 13 or to the state of testing/unstable
during development of Debian 13 should be considered unsupported.

(emphasis added in this mail, not present in the TC resolution)

As Marco implies, it was not my intention to have this as a
general-purpose way for change-averse sysadmins to avoid or defer the
automatic transition. I had intended this to be specifically there as a
way to do the testing and QA that will ensure that the transition can go
as smoothly as possible (for example, reproducible-builds does one build
with merged-/usr and one with unmerged /usr, and this should continue,
otherwise we'll lose the ability to detect packages that build differently
in those two scenarios, which in practice is strongly correlated with
packages that won't work on non-merged-/usr systems if they have been
built on merged-/usr).

I can see that if people are developing dpkg enhancements that allow it to
reconcile its view of the filesystem with merged-/usr by having a record
of the directories that "should be" merged (perhaps a --paths-aliased
analogous to --path-include?), then they would also benefit from the
ability to construct up-to-date systems that have and haven't undergone
this transition, so they can compare the two.

> As I see it the CTTE decision effectively allows the transition to be
> deferred until the moment you want to upgrade to 13.

I think you mean: until the moment you want to upgrade to testing after
Debian 12 release day. That's not Debian 13 *yet*, although you could
think of it as some sort of Debian 13~alpha, and the TC resolution allows
packages in testing/unstable to start requiring merged-/usr immediately
after Debian 12 is released.

I tried to make sure the TC resolution distinguished carefully between
Debian stable releases, and the states of testing/unstable that will
exist at particular points in the stable release cycle.

> So, wouldn't it make sense to go with an (extreme) low priority debconf
> question defaulting to 'yes, convert now' which [I think] non-experts
> aren't bothered with and users/systems wanting to opt-out for the moment
> (like buildds) have a standard way of preseeding rather than inventing a
> homegrown flag-file and associated machinery?

Speaking only for myself and not for the TC, I think a debconf question
would be OK as an implementation of this, but the debconf question should
indicate that the result of opting out is an unsupported system.

I would personally see this as a bit like using dpkg --path-exclude, or
downgrading packages: it's allowed, but it might break things immediately,
or it might have weird side-effects later on (even after the configuration
change has been reverted).

> As a bonus, if I had previously decided to forgo the automatic
> transition for whatever reason (lets say to test build packages on that
> box) I also have a standard way of triggering the conversion by calling
> dpkg-reconfigure on usrmerge at leisure before the 13 upgrade.

I had intended this to be for the class of systems that you would expect
to discard and re-bootstrap rather than upgrading (chroots, lxc/Docker
containers, virtual machines, etc. used for autopkgtest, piuparts,
reproducible-builds, etc.), where a way to undo the opt-out isn't really
necessary because the system is treated as 

Re: merged-/usr transition: debconf or not?

[David Kalnischkies]

On Mon, Nov 08, 2021 at 12:56:49PM +0100, Marco d'Itri wrote:
> On Nov 08, Simon Richter  wrote:
> > Right now, it is sufficient to preseed debconf to disallow the usrmerge
> > package messing with the filesystem tree outside dpkg. Managed installations
> > usually have a ready-made method to do so.
> This is not really relevant, since the conversion is mandatory.
> The CTTE stated that the only exception is "Testing and QA systems 
> should be able to avoid this transition, but if they do, they cannot be 
> upgraded beyond Debian 12", and my plan is to arrange for this with 
> a flag file.

As I see it the CTTE decision effectively allows the transition to be
deferred until the moment you want to upgrade to 13. Ideally the
transition is performed already in the 11→12 upgrade automatically for
you, but you could prevent that automatism and do it manually someday
while you have 12 installed (as no 12 package can depend on merged /usr
as it would not be installable on upgrade from 11 and/or executable on
buildds/testing/qa systems at the least).


So, wouldn't it make sense to go with an (extreme) low priority debconf
question defaulting to 'yes, convert now' which [I think] non-experts
aren't bothered with and users/systems wanting to opt-out for the moment
(like buildds) have a standard way of preseeding rather than inventing a
homegrown flag-file and associated machinery?

As a bonus, if I had previously decided to forgo the automatic
transition for whatever reason (lets say to test build packages on that
box) I also have a standard way of triggering the conversion by calling
dpkg-reconfigure on usrmerge at leisure before the 13 upgrade.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Marco d'Itri]

On Nov 08, Simon Richter  wrote:

> Yes, but the conversion needs to be performed by dpkg, because the usrmerge
Please kindly stop beating this long-dead horse.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

On 11/8/21 12:56 PM, Marco d'Itri wrote:


Right now, it is sufficient to preseed debconf to disallow the usrmerge
package messing with the filesystem tree outside dpkg. Managed installations
usually have a ready-made method to do so.



This is not really relevant, since the conversion is mandatory.


Yes, but the conversion needs to be performed by dpkg, because the 
usrmerge package is not in a position to complete the transition, it 
will always leave a filesystem that is inconsistent with the dpkg 
database, and it can not fix up the database because there is neither a 
time during package installation where the dpkg database is unlocked, 
nor is it desirable that another package besides dpkg changes the dpkg 
database, nor is it safe to do so (in fact, that would almost certainly 
cause files to be lost).


Ergo, the usrmerge package needs to be phased out before the bookworm 
release, it only creates complications for finishing the transition, and 
while I find that useful with my Devuan user hat on (as it means that 
Debian packages will continue to ship essential files in /bin until at 
least bookworm+1, and we don't need to fork these), my Debian hat says 
that we should complete this with bookworm.


Last but not least, it is disingenuous to remove control from users 
(especially those who have previously objected) and pretend they have 
consented. I would, at the very least, expect the package to honor 
previous debconf answers (and these would remain fixed if the question 
were removed).


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Marco d'Itri]

On Nov 08, Simon Richter  wrote:

> Right now, it is sufficient to preseed debconf to disallow the usrmerge
> package messing with the filesystem tree outside dpkg. Managed installations
> usually have a ready-made method to do so.
This is not really relevant, since the conversion is mandatory.
The CTTE stated that the only exception is "Testing and QA systems 
should be able to avoid this transition, but if they do, they cannot be 
upgraded beyond Debian 12", and my plan is to arrange for this with 
a flag file.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

On 11/8/21 12:03 AM, Marco d'Itri wrote:


I have been asked to remove from the usrmerge package the debconf
question which asks to confirm the conversion.
Does anybody REALLY believe that it should not be removed?


While it may be more convenient for home users to not be bothered with 
questions they don't understand, I believe predictable behaviour would 
be preferable in a professional environment.


Right now, it is sufficient to preseed debconf to disallow the usrmerge 
package messing with the filesystem tree outside dpkg. Managed 
installations usually have a ready-made method to do so.


With that change, the only way to do so would be to blacklist usrmerge 
via APT preferences, and while that is certainly possible for management 
frameworks to handle, the resulting ripple effects from packages 
becoming uninstallable through this might cause additional problems.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Andreas Metzler]

On 2021-08-26 Timo Röhling  wrote:
[...]
> However, Guillem also seems to think that dpkg can manage file
> symlinks in a real directory better than an directory symlinks in /,
> which is why he proposed symlink farms in the first place.

Hello,

Afaiui, the symlink farm would just work from dpkg's point of view, even
with dpkg from oldoldoldoldstable. It can handle symlinks to files just
fine.

usrmerge is conceptually a lot harder because dpkg needs to get to handle
the fact that /bin/foo and /usr/bin/foo conflict.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: merged /usr vs. symlink farms

[Timo Röhling]

Hi,

* Sam Hartman  [2021-08-26 08:56]:

That may not matter so much for Debian (at least in some people's minds)
but being compatible with the rest of the world has enough value in this
instance that I consider the issue moot.

I agree that this is a very convincing argument. A considerably
weaker supportive argument would also be that these symlinks are
here to stay for years, possibly even decades, which makes four or
five static symlinks much simpler to maintain long-term.

Still, I would prefer to know in advance if there are additional
technical concerns which need to be fixed or worked around.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Sam Hartman]

> "Timo" == Timo Röhling  writes:
Timo> However, Guillem also seems to think that dpkg can manage file
Timo> symlinks in a real directory better than an directory symlinks
Timo> in /, which is why he proposed symlink farms in the first
Timo> place.  Assuming dpkg implements the proposed
Timo> canonicalization, is there a scenario where the aliased dirs
Timo> break, but would not break with a symlink farm?

Quite possibly.
However, I haven't been examining that part of the design space because
the symlink farms approach misses key features that are important to
some usrmerge use cases.

In particular, if you're sharing a /usr between containers or similar,
and moving toward the empty /etc at boot ostree style approach, symlink
farms don't work and base level symlinks do work.

That may not matter so much for Debian (at least in some people's minds)
but being compatible with the rest of the world has enough value in this
instance that I consider the issue moot.

Symlink farms are a different thing, and that's not what we're trying to
get here.

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Thu, 2021-08-26 at 14:51 +0200, Simon Richter wrote:
> Hi,
> 
> On 8/26/21 1:17 PM, Luca Boccassi wrote:
> 
> > > Ideally the question whether a system works correctly would be a
> > > technical, not a political one that is based on a majority vote of
> > > people who do not look behind the facade.
> 
> > Precisely - and the correct technical question is, how many bug reports
> > were opened by users?
> 
> That is the majority vote by people who do not look behind the facade.

A vote is a conscious decision and arbitrary act. Your system breaking
is not.

> > Strawmen can always be constructed, for example
> > you can completely brick a system by running 'apt install bash:armhf'
> > but that doesn't mean multiarch should be scrapped and reverted, it
> > would be silly to suggest that - and yet here we are.
> 
> This is precisely the debate we would be having if there was a package 
> that enables armhf as a foreign architecture whenever it is installed: 
> it would not break systems on its own, but we'd have to be a lot more 
> careful in other packages about expressing dependencies.
> 
> I have armhf enabled on my laptop, and the resolver has suggested 
> replacing bash:amd64 with bash:armhf a few times -- but I know that my 
> setup is unusual.

It's exactly the same. This dpkg bug manifests itself via a carefully
crafted sequence of package upgrades/downgrades with a specific set of
metadata between them. If I uploaded a new version of iproute2 which
enabled armhf and installed bash:armhf the suggestion wouldn't be to
revert the debian multiarch implementation.

> > Conversely this
> > doesn't mean the dpkg database bug shouldn't be fixed,
> 
> Keep in mind that this isn't a dpkg bug, because dpkg never created a 
> filesystem layout that was inconsistent with its database; usrmerge did.
> 
> What we are talking about is adding a workaround for the usrmerge 
> package to dpkg, and how to make sure that it works, regardless of 
> whether the system has already been converted or not, if it has, which 
> version of usrmerge was used, whether the package is currently installed 
> or not, whether it will be installed as part of an upgrade and if so, 
> which state the dpkg package is in at the time it is installed.
> 
> For example, a possible scenario could be that the system was converted 
> with usrmerge version 9 (which modified dpkg.cfg), then the usrmerge 
> package was deinstalled as a later version declared a conflict that the 
> user resolved in favour of the other package, usrmerge hasn't been 
> installed since, then during the upgrade to bookworm, a new version of 
> dpkg is unpacked (but not configured yet), then a new version of 
> usrmerge is unpacked, and then dpkg --configure -a is run.
> 
> So we need a list of things that need to happen to get back into a 
> consistent state and a way of sequencing that inside the package 
> dependency DAG.

This is 100% a dpkg bug. rpm doesn't have it. Other package managers
don't have it. And it's fine - all software has bugs.

> > so that the
> > replace+move bug can't happen in the future, but it simply makes
> > overblown and hyperbolic ideas such as "all systems are broken, revert
> > everything" and "let's cancel the TC decision" appear counter-
> > productive and deeply unhelpful toward finding an actual, realistic
> > solution.
> 
> The TC decision does not specify technical details, which was a grave 
> mistake because the implementation was so shoddy that I doubt the TC 
> would have signed off on that, had they been consulted on the details.

You are missing a gigantic 'IMHO'. For me the implementation was great
- simple, to the point, matching other distros so that we don't get
yet-another-incompatibility for the sake of being 'special', and
working just fine across millions of installations for 2+ years and
counting with hardly a hiccup. Of course it has bugs, because it's
software, or more precisely yet it makes old bugs in other packages
(dpkg) come to light, which happens and it's not the end of the world -
we'll fix them or work around them, like the tens of thousands of other
bugs affecting debian. Also, from what we have heard "unofficially"
here and elsewhere, it appears that the TC has always indeed preferred
the current approach of matching other distros and symlinking the top
dirs, rather than the objectively failed symlinks farm approach.

> Pretty much no one is interested in rolling this back, but a lot of 
> people, me included, are indifferent and cannot see the benefits, 
> especially as they consist mainly of things that used to work in the 
> past, were broken during the systemd rollout and subsequently declared 
> out of scope when people complained, so all users of these features have 
> already left anyway.
> 
> An actual, realistic solution will work for all users, including those 
> who have now installed or upgraded to buster from DVD, run offline and 
> will upgrade to bookworm from DVD when it comes out.

You say 

Re: merged /usr vs. symlink farms

[Timo Röhling]

* Simon Richter  [2021-08-26 14:51]:
It makes a lot more sense to have a dpkg-internal tool that can 
investigate the differences between the file system and the dpkg 
database, resolve conflicts (possibly in an interactive process when 
required by a corner case like the one I mentioned earlier -- luckily, 
these should be really rare) and then leave a consistent state again 
that allows package maintainers to use all dpkg features again after 
the bookworm release.

Agreed. Having yet another tool messing with dpkg "behind its back"
will only exacerbate the problems we're trying to solve.

Also, I wouldn't say it is the *only* possible way to move forward,
but it is *a* possible way. I am also still trying to understand
Guillem position and his objections better.

This proposal addresses one half on Guillem's objections by making
dpkg's point of view consistent with the actual filesystem reality. 


However, Guillem also seems to think that dpkg can manage file
symlinks in a real directory better than an directory symlinks in /,
which is why he proposed symlink farms in the first place.  Assuming
dpkg implements the proposed canonicalization, is there a scenario
where the aliased dirs break, but would not break with a symlink
farm?

My apologies if this has been answered already. Just point me to the
relevant emails/wiki pages then.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 8/26/21 1:17 PM, Luca Boccassi wrote:


Ideally the question whether a system works correctly would be a
technical, not a political one that is based on a majority vote of
people who do not look behind the facade.



Precisely - and the correct technical question is, how many bug reports
were opened by users?


That is the majority vote by people who do not look behind the facade.


Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are.


This is precisely the debate we would be having if there was a package 
that enables armhf as a foreign architecture whenever it is installed: 
it would not break systems on its own, but we'd have to be a lot more 
careful in other packages about expressing dependencies.


I have armhf enabled on my laptop, and the resolver has suggested 
replacing bash:amd64 with bash:armhf a few times -- but I know that my 
setup is unusual.



Conversely this
doesn't mean the dpkg database bug shouldn't be fixed,


Keep in mind that this isn't a dpkg bug, because dpkg never created a 
filesystem layout that was inconsistent with its database; usrmerge did.


What we are talking about is adding a workaround for the usrmerge 
package to dpkg, and how to make sure that it works, regardless of 
whether the system has already been converted or not, if it has, which 
version of usrmerge was used, whether the package is currently installed 
or not, whether it will be installed as part of an upgrade and if so, 
which state the dpkg package is in at the time it is installed.


For example, a possible scenario could be that the system was converted 
with usrmerge version 9 (which modified dpkg.cfg), then the usrmerge 
package was deinstalled as a later version declared a conflict that the 
user resolved in favour of the other package, usrmerge hasn't been 
installed since, then during the upgrade to bookworm, a new version of 
dpkg is unpacked (but not configured yet), then a new version of 
usrmerge is unpacked, and then dpkg --configure -a is run.


So we need a list of things that need to happen to get back into a 
consistent state and a way of sequencing that inside the package 
dependency DAG.



so that the
replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken, revert
everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic
solution.


The TC decision does not specify technical details, which was a grave 
mistake because the implementation was so shoddy that I doubt the TC 
would have signed off on that, had they been consulted on the details.


Pretty much no one is interested in rolling this back, but a lot of 
people, me included, are indifferent and cannot see the benefits, 
especially as they consist mainly of things that used to work in the 
past, were broken during the systemd rollout and subsequently declared 
out of scope when people complained, so all users of these features have 
already left anyway.


An actual, realistic solution will work for all users, including those 
who have now installed or upgraded to buster from DVD, run offline and 
will upgrade to bookworm from DVD when it comes out.



Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other day,
namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?


No. We couldn't sequence that correctly, we'd have to somehow make 
installation of the usrmerge package mandatory, and that tool would need 
to touch the dpkg database at a point where dpkg is active in the 
background.


It makes a lot more sense to have a dpkg-internal tool that can 
investigate the differences between the file system and the dpkg 
database, resolve conflicts (possibly in an interactive process when 
required by a corner case like the one I mentioned earlier -- luckily, 
these should be really rare) and then leave a consistent state again 
that allows package maintainers to use all dpkg features again after the 
bookworm release.


That will be a lot of work, and we cannot speed it up anyway because 
we're tied to the release cycle here, so I'd rather do this properly 
than deploy another paper mâché thing that then turns out to have 
another weird bug that will delay the transition to bookworm+2.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Thu, 2021-08-26 at 13:50 +0200, Philip Hands wrote:
> Luca Boccassi  writes:
> 
> > On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:
> > > Hi,
> > > 
> > > On 8/26/21 8:38 AM, Marco d'Itri wrote:
> > > 
> > > > > By my definition, these have never been working correctly, but
> > > > > semantics I guess.
> > > 
> > > > It is not semantics. You keep saying that countless Debian and Ubuntu
> > > > systems are not working correctly, but since this obviously does not
> > > > reflect the experience of the owners of these systems then just about
> > > > everybody will believe that you are wrong about merged-/usr.
> > > 
> > > Ideally the question whether a system works correctly would be a 
> > > technical, not a political one that is based on a majority vote of 
> > > people who do not look behind the facade.
> > > 
> > > Simon
> > 
> > Precisely - and the correct technical question is, how many bug reports
> > were opened by users?
> 
> While I'm sympathetic with the argument that a lack of bug reports
> suggests that the theoretical problems (that I hope we all agree, do
> exist) seem not to be exhibiting themselves in the wild, it is very far
> from proof.
> 
> If some random file disappears on your system one day, what happens?
> 
> Most likely, nothing, and you never notice.
> 
> Possibly, your system starts to misbehave.  What do you do then?
> 
>   If you're a naive user, such as a recent arrival from Windows, then
>   misbehaviour is something that you've been trained to expect and you
>   install from scratch.  The idea of reporting a bug never enters your
>   head.
> 
>   If you're aware that this sort of thing really should not happen, then
>   what happens next is probably down to how busy you are.  If you're
>   busy, you probably check your SMART stats, and having convinced
>   yourself that the disks are OK, either restore from backup and check
>   for what ealse is missing, or use debsums to see the extent of the
>   damage and reinstall a package or two.  Again, since you're only
>   trying to get back to work, and didn't track down the cause, no bug
>   report.
> 
> The only bug reports you're going to get about this are either going to
> be the useless "Something didn't work" bugs, that I doubt would ever get
> tied to this cause, or the ones submitted by experienced, diligent, and
> time-rich bug reporters (which is a rather rare combination).
> 
> The fact that we don't see bug reports should surprise no one.
> 
> Cheers, Phil.

Actually in unstable (not in a release) this bug was observed, caught,
properly reported and fixed: https://bugs.debian.org/953562 which would
suggest these instances can and have been indeed detected when they
happened - but anyway, that's why the full quote without the cut is:

> Precisely - and the correct technical question is, how many bug
> reports
> were opened by users? Strawmen can always be constructed, for example
> you can completely brick a system by running 'apt install bash:armhf'
> but that doesn't mean multiarch should be scrapped and reverted, it
> would be silly to suggest that - and yet here we are. Conversely this
> doesn't mean the dpkg database bug shouldn't be fixed, so that the
> replace+move bug can't happen in the future, but it simply makes
> overblown and hyperbolic ideas such as "all systems are broken,
> revert
> everything" and "let's cancel the TC decision" appear counter-
> productive and deeply unhelpful toward finding an actual, realistic
> solution.

IE, it's fine to think about all the possible ways things can go wrong,
but the solutions need to be proportionate. So proposing to trash
everything and going against the TC decision in absence of any user
report and any knowledge from anybody of an actual breakage in
buster/bullseye/19.10/20.04/20.10/21.04 is as far from proportionate as
one can possibly be, just as suggesting to revert multiarch because
'apt install bash:ppc64el' breaks an amd64 machine beyond repair would
be completely unreasonable. Am I suggesting the replaces+move issue
should be ignored? No, hence the second part of the quote that was cut:

> Given this and the last few mails, it seems to me at this point the
> only possible way forward is what Timo and Ted suggested the other
> day,
> namely to have an external tool, possibly part of src:usrmerge, scan
> the dpkg database and fix it? Possibly on a trigger?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Philip Hands]

Luca Boccassi  writes:

> On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:
>> Hi,
>> 
>> On 8/26/21 8:38 AM, Marco d'Itri wrote:
>> 
>> > > By my definition, these have never been working correctly, but
>> > > semantics I guess.
>> 
>> > It is not semantics. You keep saying that countless Debian and Ubuntu
>> > systems are not working correctly, but since this obviously does not
>> > reflect the experience of the owners of these systems then just about
>> > everybody will believe that you are wrong about merged-/usr.
>> 
>> Ideally the question whether a system works correctly would be a 
>> technical, not a political one that is based on a majority vote of 
>> people who do not look behind the facade.
>> 
>> Simon
>
> Precisely - and the correct technical question is, how many bug reports
> were opened by users?

While I'm sympathetic with the argument that a lack of bug reports
suggests that the theoretical problems (that I hope we all agree, do
exist) seem not to be exhibiting themselves in the wild, it is very far
from proof.

If some random file disappears on your system one day, what happens?

Most likely, nothing, and you never notice.

Possibly, your system starts to misbehave.  What do you do then?

  If you're a naive user, such as a recent arrival from Windows, then
  misbehaviour is something that you've been trained to expect and you
  install from scratch.  The idea of reporting a bug never enters your
  head.

  If you're aware that this sort of thing really should not happen, then
  what happens next is probably down to how busy you are.  If you're
  busy, you probably check your SMART stats, and having convinced
  yourself that the disks are OK, either restore from backup and check
  for what ealse is missing, or use debsums to see the extent of the
  damage and reinstall a package or two.  Again, since you're only
  trying to get back to work, and didn't track down the cause, no bug
  report.

The only bug reports you're going to get about this are either going to
be the useless "Something didn't work" bugs, that I doubt would ever get
tied to this cause, or the ones submitted by experienced, diligent, and
time-rich bug reporters (which is a rather rare combination).

The fact that we don't see bug reports should surprise no one.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:
> Hi,
> 
> On 8/26/21 8:38 AM, Marco d'Itri wrote:
> 
> > > By my definition, these have never been working correctly, but
> > > semantics I guess.
> 
> > It is not semantics. You keep saying that countless Debian and Ubuntu
> > systems are not working correctly, but since this obviously does not
> > reflect the experience of the owners of these systems then just about
> > everybody will believe that you are wrong about merged-/usr.
> 
> Ideally the question whether a system works correctly would be a 
> technical, not a political one that is based on a majority vote of 
> people who do not look behind the facade.
> 
> Simon

Precisely - and the correct technical question is, how many bug reports
were opened by users? Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are. Conversely this
doesn't mean the dpkg database bug shouldn't be fixed, so that the
replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken, revert
everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic
solution.

Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other day,
namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 8/26/21 8:38 AM, Marco d'Itri wrote:


By my definition, these have never been working correctly, but
semantics I guess.



It is not semantics. You keep saying that countless Debian and Ubuntu
systems are not working correctly, but since this obviously does not
reflect the experience of the owners of these systems then just about
everybody will believe that you are wrong about merged-/usr.


Ideally the question whether a system works correctly would be a 
technical, not a political one that is based on a majority vote of 
people who do not look behind the facade.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Marco d'Itri]

On Aug 26, Guillem Jover  wrote:

> By my definition, these have never been working correctly, but
> semantics I guess.
It is not semantics. You keep saying that countless Debian and Ubuntu 
systems are not working correctly, but since this obviously does not 
reflect the experience of the owners of these systems then just about 
everybody will believe that you are wrong about merged-/usr.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Danilo Santos]

Today's update, Debian test can't read my windows partition. I fix it
inside the bios configuration.

El mié, 25 de ago. de 2021 a la(s) 15:27, Aurelien Jarno (
aurel...@aurel32.net) escribió:

> On 2021-08-20 23:15, Simon Richter wrote:
> > I think that one of the release goals should be that any freshly
> installed
> > or upgraded system should have a dpkg database that is consistent with
> > reality, and I'd prioritize that higher than actually finishing the
> > transition, because as long as we can have files vanish from under us, we
> > can't expect that the transition will be reliable for bullseye->bookworm
> > updates.
> >
> > Basically, we've been lucky so far.
>
> I am not sure we have been so lucky. Problems have been found during the
> release cycles, some of them got fixed (#953562), some other are still
> there (#926699).
>
> --
> Aurelien Jarno  GPG: 4096R/1DDD8C9B
> aurel...@aurel32.net http://www.aurel32.net
>


-- 
*Grupo Santos Frias S.R.L*
*Danilo Alejandro Santos Frias*
*Cel: (809) 708-7460*

Re: merged /usr vs. symlink farms

[Guillem Jover]

Hi!

On Sun, 2021-08-22 at 09:18:25 +0200, Andreas Metzler wrote:
> Afaict we have still no idea on how to move on.
> 
> 1 I think you agree that there is a significant number of usrmerged Debian
>   installations out there. It does not really matter whether there are 7% or
>   40%. They exist and should (keep) working.

By my definition, these have never been working correctly, but
semantics I guess. My wish would be to indeed salvage those systems,
that's why I implemented dpkg-fsys-usrunmess. But if people refuse,
then at some point I'd personally consider them lost, TBH. :/

> 2 As you have stated there are known issues with dpkg and usrmerged
>   systems. Some of them are are triggered by moving files from / to /usr.
> 
> Starting from here it is hard to see a way forward.

Well, in my mind the first and most immediate action that would be
done, is to stop the bleeding, by:

  - reverting the changes in deboostrap in sid, bullseye (and ideally
in buster too),
  - reverting the notion that split-/usr is unsupported (which includes
the extremely confusing interpretation about this applying to
sid/testing too), and update documentation such as release-notes,
etc.,
  - adding a Conflicts against usrmerge somewhere (dpkg, base-files or
similar) for extra caution,
  - (optionally) removing usrmerge from buster, bullseye and sid,

I'd be ready to prepare patches or file reports for any of these.

But then, my expectations regarding Debian have plummeted so…

> Is it a realistic option to require something like
> -
> dpkg --purge usrmerge
> dpkg-fsys-usrunmess
> -
> pre dist-upgrade to bookworm to get back all systems to a sane (from
> dpkg POV) state and start fresh?

If by requiring you mean documenting the requirement, then I don't
think that would be enough.

But I could see introducing a preinst somewhere (perhaps base-files,
to permit upgrading dpkg itself) which would check for those unsupported
systems and emit a big fat message explaining the situation and
instructions to follow to be able to proceed, then abort. Similar in vein
with how other preventions (AFAIR) have been implemented for example on
glibc or udev for environment, multi-arch or kernel requirements.

Sadly, even introducing that into bullseye would not guarantee that the
bookworm upgrade would already be in a sane state, as there's no
guarantee people do not skip point releases. But I assume it would cover
most of the installations. So the problem would be greatly minimized.
Only after the bookworm upgrade has been completed this would be
guaranteed.

> Is this robust (except for crash as stated in the manpage), or would
> you consider it beta-quality?

There's a known problematic interaction with a systemd's emergency mode
misbehavior/bug, which I've worked around, plus some other robustness
improvements I've prepared that will be included in 1.21.0, and targeted
for 1.20.10. I should probably also add a temporary policy-rc.d to
batch and defer all service actions for after the mass reconfiguration,
which I'll try to code up soonish to further improve robustness.

Otherwise I've taken great care and consideration on trying to make it
as robust as possible, but I guess out of cautiousness and given the
nature of what the script is doing (which is still a hack, although a
self-contained one at least), I'd be hard pressed to probably ever
declare it 100% safe. :/ Of course more testing is always helpful!

Thanks,
Guillem

Re: merged /usr vs. symlink farms

[Russ Allbery]

Guillem Jover  writes:

> The fact that the supporters of a *filesystem layout* have been happy to
> dismiss and ignore this and have been pushing for what I think can be
> easily described as the worst ever "transition" done in Debian, very
> sadly, for me this whole topic marks a before and after in Debian, and
> has put my trust in the technical side of the project into question.

I agree that you've been pointing out potential problems for years and
that your warnings have been at least partly vindicated by the problems we
are indeed seeing.

My understanding (which may be entirely incorrect, in which case please
let me know what your preferred solution actually is!) is that your
preferred solution was to migrate individual packages.  What I'm hearing
from the frequent threads in debian-devel is that a sufficient number of
Debian contributors have rejected that approach (for various reasons) as
to make that solution not viable.  In other words, my reading of the
consensus is that this option has effectively been vetoed by the rest of
the project (noting that this veto was certainly not unanimous).

Please note that I'm not taking a position on the merits of that decision,
simply noting that, based on my reading of debian-devel, this is has what
has happened, and I do not believe it will be possible at this point to
convince the project as a whole to unwind usrmerge and go back to doing
individual package migrations.

Given that as a design constraint (we will not be doing this transition
via one-by-one changes to each package), what would you support as a good
architectural solution to this transition?  Even ruling out that approach,
the design space seems large and flexible; surely there must be some
coherent way of doing this transition that does not require individual
action for each affected package and preserves some of the "be done with
it" design goals of the current usrmerge approach while avoiding the
problems you have pointed out.

I think it's obvious that any such design will require support from dpkg.

You're very understandably upset that people are insisting on a solution
that you feel is clearly incorrect, but I think that's also how the people
who are disagreeing with you are feeling, and as long as that's true on
both sides it's hard to see how we're going to arrive at a solution that
isn't going to make you even more unhappy.  In order to break this
deadlock, I think we have to have a design discussion in the shared space
of mutually agreeable solutions and not (on all sides) retreat back to a
single preferred architectural decision and only point out the problems
with any other approach.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Russ Allbery]

Simon Richter  writes:

> I'd expand the definition of Conflicts/Replaces though: packages that
> use names that conflict because of usrmerge would need to declare it,
> because as soon as we teach dpkg to recognize these conflicts, the
> packages would fail to install on stable.

Yes, that's probably the most appropriate place to make a change to Policy
to clarify this.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 25.08.21 18:57, Russ Allbery wrote:


The problem here is also that if there are two packages like that, on an
usrmerge system, we would not know this is happening.



I agree, of course, but I don't see a way in which Policy can help with
that problem unless this packaging decision was intentional and the person
who made that decision would have chosen otherwise if Policy had said to
not do it.


I'd expand the definition of Conflicts/Replaces though: packages that 
use names that conflict because of usrmerge would need to declare it, 
because as soon as we teach dpkg to recognize these conflicts, the 
packages would fail to install on stable.



This seems more like an appropriate check for an archive-wide QA tool
looking for cross-package problems.


We have a tool that looks for missing Conflicts/Replaces, and that tool 
needs to be extended as well.


   Simon

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Wed, Aug 25, 2021 at 09:57:09AM -0700, Russ Allbery wrote:
> Wouter Verhelst  writes:
> > On Mon, Aug 23, 2021 at 08:23:50AM -0700, Russ Allbery wrote:
> 
> >> If we tried to document every random bit of buggy packaging behavior
> >> anyone thought of in Policy, Policy would become unwieldy, so I want to
> >> verify here that someone really thought having one package containing a
> >> file in /bin and another package containing the same file in /usr/bin
> >> was was a reasonable thing to do (as opposed to accidental).  Are there
> >> packages in the archive like this?  Or could you point me at the
> >> message in the thread that said this was non-buggy?  I think I missed
> >> it.
> 
> > The problem here is also that if there are two packages like that, on an
> > usrmerge system, we would not know this is happening.
> 
> I agree, of course, but I don't see a way in which Policy can help with
> that problem unless this packaging decision was intentional and the person
> who made that decision would have chosen otherwise if Policy had said to
> not do it.
>
> This seems more like an appropriate check for an archive-wide QA tool
> looking for cross-package problems.

Indeed, and that was the point I was trying to make: it's not something
Policy can help with, unless it is intentional (and it is my belief that
this is not likely to be the case).

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: merged /usr vs. symlink farms

[Aurelien Jarno]

On 2021-08-20 23:15, Simon Richter wrote:
> I think that one of the release goals should be that any freshly installed
> or upgraded system should have a dpkg database that is consistent with
> reality, and I'd prioritize that higher than actually finishing the
> transition, because as long as we can have files vanish from under us, we
> can't expect that the transition will be reliable for bullseye->bookworm
> updates.
> 
> Basically, we've been lucky so far.

I am not sure we have been so lucky. Problems have been found during the
release cycles, some of them got fixed (#953562), some other are still
there (#926699).

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Wed, 2021-08-25 at 09:57:09 -0700, Russ Allbery wrote:
> Wouter Verhelst  writes:
> > The problem here is also that if there are two packages like that, on an
> > usrmerge system, we would not know this is happening.

Also this does not need to come from "buggy" packaging practices.

> I agree, of course, but I don't see a way in which Policy can help with
> that problem unless this packaging decision was intentional and the person
> who made that decision would have chosen otherwise if Policy had said to
> not do it.

> This seems more like an appropriate check for an archive-wide QA tool
> looking for cross-package problems.

I've said this many times over the months (years!?), that while this
can easily affect stuff from the archive, which we do control, where
policy applies and where we could try to poorly reimplement the checks
that dpkg does to detect them in some QA checker, even though the
following problems would still apply:

  - the checks cannot be performed (only) over static snapshots of
the archive, as this would affect partial upgrades too,
  - the checks would need to take into account packages we have
stopped shipping way in the past as those can linger around
installed,
  - the checks would have a hard time with the non-declarative side
of the packaging stack,

is that something else I expect to have a non-zero chance to bite users
are all those common practices that people give for granted and that we
have supposedly supported in the past, as guaranteed by our packaging
system, such as:

  - keeping installed packages that have stopped shipping in Debian,
  - installation of packages from third-parties, or from local
overlays or similar, or even rebuilt forks of packages from Debian,
  - installation or holding of Debian packages from older releases,
  - local diversions or alternatives,

which are out of our QA reach.


The fact that the supporters of a *filesystem layout* have been happy
to dismiss and ignore this and have been pushing for what I think can
be easily described as the worst ever "transition" done in Debian, very
sadly, for me this whole topic marks a before and after in Debian, and
has put my trust in the technical side of the project into question.

Of course some of those supporters are now agreeing these problems can
be insidious, progress I guess. And at the same time others are claiming
that acknowledging those problems would hold back the entire distribution,
while others are now saying we should stop doing packaging changes because,
well, these problems perhaps are potentially problematic, the irony.

And then we get people raving over what's the worst and most atrocious
hacks to pile on into dpkg to try to workaround the actual root cause
(turtles^Wnasty hacks and kludges all the way down I guess). Which I
obviously expect to eventually be coerced into merging into dpkg at
some point or another via our esteemed Authority.

From where I'm sitting Debian is the project that lost its way…

Sadly,
Guillem

Re: merged /usr vs. symlink farms

[Russ Allbery]

Wouter Verhelst  writes:
> On Mon, Aug 23, 2021 at 08:23:50AM -0700, Russ Allbery wrote:

>> If we tried to document every random bit of buggy packaging behavior
>> anyone thought of in Policy, Policy would become unwieldy, so I want to
>> verify here that someone really thought having one package containing a
>> file in /bin and another package containing the same file in /usr/bin
>> was was a reasonable thing to do (as opposed to accidental).  Are there
>> packages in the archive like this?  Or could you point me at the
>> message in the thread that said this was non-buggy?  I think I missed
>> it.

> The problem here is also that if there are two packages like that, on an
> usrmerge system, we would not know this is happening.

I agree, of course, but I don't see a way in which Policy can help with
that problem unless this packaging decision was intentional and the person
who made that decision would have chosen otherwise if Policy had said to
not do it.

This seems more like an appropriate check for an archive-wide QA tool
looking for cross-package problems.

(That said, the molly-guard example does seem to indicate that at least
one packager in the past did not realize this would be a problem.)

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Mon, Aug 23, 2021 at 08:23:50AM -0700, Russ Allbery wrote:
> Luca Boccassi  writes:
> 
> > Thank you - it has been brought up in this thread as an example of a
> > valid setup, so if it is not, I think it could be good to be extra clear
> > in the policy? How about the following:
> 
> If we tried to document every random bit of buggy packaging behavior
> anyone thought of in Policy, Policy would become unwieldy, so I want to
> verify here that someone really thought having one package containing a
> file in /bin and another package containing the same file in /usr/bin was
> was a reasonable thing to do (as opposed to accidental).  Are there
> packages in the archive like this?  Or could you point me at the message
> in the thread that said this was non-buggy?  I think I missed it.

The problem here is also that if there are two packages like that, on an
usrmerge system, we would not know this is happening.

Let's say there's a package "foo" which installs /usr/bin/foo, and an
package "binfoo" which installs "/bin/foo". In the current situation,
dpkg would not know that the two files are equivalent, and would happily
overwrite /usr/bin/foo with /bin/foo if "binfoo" was installed after
"foo".

Then when the user notices the "foo" program is not doing what they
believe it should be doing and runs "reportbug /usr/bin/foo", reportbug
will file the bug against the package "foo" rather than the package
"binfoo" which is the actual package whose binary they are trying to
use.

In contrast, if foo and binfoo both install "/bin/foo" (or both install
"/usr/bin/foo", either way works), then dpkg will complain at
installation time that one of the two packages tries to overwrite a file
from the other and refuse to continue.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Simon McVittie]

On Tue, 24 Aug 2021 at 10:46:45 -0400, Theodore Ts'o wrote:
> the definition of usrmerged is
> relatively well understood (symlinks for /{bin,lib,sbin} to
> /usr/{bin,lib,sbin})

For completeness: also /libQUAL to usr/libQUAL, for each libQUAL
that either participates in multilib or contains ld.so(8). See
setup_merged_usr() in debootstrap/functions or directories_to_merge()
in usrmerge/convert-usrmerge.

(For x86_64 users this means lib32, lib64 and libx32.)

smcv

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Theodore Ts'o]

On Tue, Aug 24, 2021 at 11:57:27AM +0200, Simon Richter wrote:
> Hi,
> 
> On 8/24/21 2:48 AM, Theodore Ts'o wrote:
> 
> > So in theory, if we had a program which looked for the top-level
> > symlinks /{bin,lib,sbin} -> /usr/{bin,lib,sbin}, and if they exist,
> > scans dpkg database is scanned looking for of the form
> > /{bin,lib,sbin}/$1, and updates them with /usr/{bin,lib,sbin}/$1, and
> > then in the future, if dpkg sees the top-level symlink, canonicalizes
> > any files referenced in the packages to /usr/{bin,lib,sbin}/$1, with a
> > fallback searching for /{bin,lib,sbin}/$1 in the file system, this
> > would solve the problem.
> 
> Yes. To apply the transformation, this would likely have to happen in the
> dpkg package itself, so the one-time transformation is applied only when
> dpkg can maintain the workaround from that point on.

Sure, since we're talking about dpkg database surgery, it's better
that the sources of such a program be located in the dpkg sources, and
regardless of who does the work, the dpkg maintainers should be
involved in the code review of any such change.

> That is the half that is missing from my proposal, as I was focusing on how
> to transition non-usrmerged systems from within dpkg.

I've been more focused on the usrmerged systems since nearly all of my
personal systems are already usrmerged, since I tend to reinstall my
systems whenver I upgrade my hardware, and deboostrap has installing
systems with the usrmerge top-level symlinks since Buster.
Furthermore, people have been arguing strenuously that the possibility
of file loss is real for these usrmerged systems, so fixing this
seemed to be high priority, regardless of how many systems use the
usrmerge setup.

So protecting against lost files was higher priority in my mind,
whether the people arguing that it's rare because Ubuntu users having
been losing files, or not, I accept the argument that if it can happen
(and you've demonstrated via adversarial test cases that it *can*), we
should fix that bug, whether it's considered an RC bug or not.

(Although my personal belief is that we should be a lot more open to
fixing less critical bugs in stable releases, so if we have a fix, I'd
be all for rolling it out early to Bullseye regardless of whether it's
considered "RC" or not.)

> > Let's ignore how we would deploy this helper program and the update
> > dpkg from a stable upgrade perspective, but in terms of preventing
> > potential problems during the testing window, getting an update to
> > dpkg which included the database fixup program and which ran from the
> > maintainer script would be a potential solution path.
> 
> Yes-ish. We'd also need to make sure that installing the usrmerge package
> after that dpkg upgrade does not make things inconsistent again, so it would
> make sense for the dpkg source package to provide a "usrmerge" package that
> is well integrated, and for dpkg to conflict with older versions of
> usrmerge.

I'm less worried about whether usrmerge is part of dpkg or not, since
most of my usrmerged systems are due to them being reinstalled since
Debian Buster has been around, and the definition of usrmerged is
relatively well understood (symlinks for /{bin,lib,sbin} to
/usr/{bin,lib,sbin}).  But it wouldn't hurt for dpkg to provide its
own "usrmerge" functionality, and I certainly wouldn't argue against it.

> > Furthermore, if dpkg knew to always canonicalize filenames from
> > /{bin,lib,sbin}/XXX to /usr/{bin,lib,sbin}/XXX when adding to the
> > database, and when it looked for files in the file system looked first
> > in /usr/{bin,lib,sbin}/XXX, with a fallback to /usr/{bin,lib,sbin}/XXX
> > the file names in the package would not need to change all, since all
> > of the magic fixup work would be happening inside dpkg.
> 
> Yes. In an ideal world, we wouldn't hardcode the list of symlinks in dpkg
> though, that's why I proposed shipping symlinks in base-files and having
> dpkg recognize the symlink-vs-directory conflict as intent to move a
> filesystem tree around.

That's certainly the more general solution, although again, I think
the definition of usrmerge is well understood, especially since Debian
has been on the trailing edge of the usrmerge transition across the
Linux ecosystem, and so the high priority should be for fixing the
specific case of moving the contents to /{bin,lib,sbin} to
/usr/{bin,lib,sbin} and leaving symlinks behind for the directories.

If we want to support people who want to say, move /usr/bin to
/u1/bin, and /usr/lib to /u2/lib, or even /usr/lib/X11 to
/funky/dir/X11, great!  Personally, I wouldn't consider that a high
priority item on the requirements list, though, unless it comes
essentially for free.

Cheers,

- Ted

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Simon Richter]

Hi,

On 8/24/21 2:48 AM, Theodore Ts'o wrote:


So in theory, if we had a program which looked for the top-level
symlinks /{bin,lib,sbin} -> /usr/{bin,lib,sbin}, and if they exist,
scans dpkg database is scanned looking for of the form
/{bin,lib,sbin}/$1, and updates them with /usr/{bin,lib,sbin}/$1, and
then in the future, if dpkg sees the top-level symlink, canonicalizes
any files referenced in the packages to /usr/{bin,lib,sbin}/$1, with a
fallback searching for /{bin,lib,sbin}/$1 in the file system, this
would solve the problem.


Yes. To apply the transformation, this would likely have to happen in 
the dpkg package itself, so the one-time transformation is applied only 
when dpkg can maintain the workaround from that point on.


That is the half that is missing from my proposal, as I was focusing on 
how to transition non-usrmerged systems from within dpkg.



Let's ignore how we would deploy this helper program and the update
dpkg from a stable upgrade perspective, but in terms of preventing
potential problems during the testing window, getting an update to
dpkg which included the database fixup program and which ran from the
maintainer script would be a potential solution path.


Yes-ish. We'd also need to make sure that installing the usrmerge 
package after that dpkg upgrade does not make things inconsistent again, 
so it would make sense for the dpkg source package to provide a 
"usrmerge" package that is well integrated, and for dpkg to conflict 
with older versions of usrmerge.



Furthermore, if dpkg knew to always canonicalize filenames from
/{bin,lib,sbin}/XXX to /usr/{bin,lib,sbin}XXX when adding to the
database, and when it looked for files in the file system looked first
in /usr/{bin,lib,sbin}/XXX, with a fallback to /usr/{bin,lib,sbin}/XXX
the file names in the package would not need to change all, since all
of the magic fixup work would be happening inside dpkg.


Yes. In an ideal world, we wouldn't hardcode the list of symlinks in 
dpkg though, that's why I proposed shipping symlinks in base-files and 
having dpkg recognize the symlink-vs-directory conflict as intent to 
move a filesystem tree around.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Timo Röhling]

Hi,

* Theodore Ts'o  [2021-08-23 20:48]:

I want to ask a potentially stupid question.
[...]

This is pretty much what I was wondering about in
https://lists.debian.org/debian-devel/2021/08/msg00372.html
You, however, phrased it much more eloquently than I could.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Theodore Ts'o]

I want to ask a potentially stupid question.

As I understand things, the problem is that in a usrmerge'd file
system where we have the top-level symlinks /{bin,lib,sbin} which
point at /usr/{bin,lib,sbin}, the problem is if we have a package
which contains the file in /sbin/blart, it gets installed in
/usr/sbin/blart thanks to the symlink, but the dpkg database has an
entry for /sbin/blart, and that is the conflict which is the problem.

So in theory, if we had a program which looked for the top-level
symlinks /{bin,lib,sbin} -> /usr/{bin,lib,sbin}, and if they exist,
scans dpkg database is scanned looking for of the form
/{bin,lib,sbin}/$1, and updates them with /usr/{bin,lib,sbin}/$1, and
then in the future, if dpkg sees the top-level symlink, canonicalizes
any files referenced in the packages to /usr/{bin,lib,sbin}/$1, with a
fallback searching for /{bin,lib,sbin}/$1 in the file system, this
would solve the problem.

Let's ignore how we would deploy this helper program and the update
dpkg from a stable upgrade perspective, but in terms of preventing
potential problems during the testing window, getting an update to
dpkg which included the database fixup program and which ran from the
maintainer script would be a potential solution path.

Furthermore, if dpkg knew to always canonicalize filenames from
/{bin,lib,sbin}/XXX to /usr/{bin,lib,sbin}XXX when adding to the
database, and when it looked for files in the file system looked first
in /usr/{bin,lib,sbin}/XXX, with a fallback to /usr/{bin,lib,sbin}/XXX
the file names in the package would not need to change all, since all
of the magic fixup work would be happening inside dpkg.

Is this a viable path forward, or am I missing something?

Thanks,

- Ted

Re: merged /usr vs. symlink farms

[Russ Allbery]

Ansgar  writes:

> Different, non-conflicting packages shipping binaries with the same name
> in /bin and /usr/bin (or similar) should be resolved for a while
> now. That as looked at when usrmerge was first introduced. I'm aware of
> one instance where this was intentional to prefer one program over the
> other (molly-guard), but it since uses diversions.

> (Actually molly-guard ships several /sbin/pm-* binaries, while pm-utils
> ships them as /usr/sbin/pm-*, but I think this is fine as the ones in
> pm-utils are diverted by molly-guard's preinst script.)

> For what it is worth I could find the following instances in Debian
> bookworm (amd64 + all), not taking into account package conflicts,
> covering /bin, /sbin and /lib*:

Thanks for the data!

> sbin/pm-hibernate: root=admin/molly-guard 
> usr=admin/pm-utils,metapackages/progress-linux-container
> sbin/pm-suspend: root=admin/molly-guard 
> usr=admin/pm-utils,metapackages/progress-linux-container
> sbin/pm-suspend-hybrid: root=admin/molly-guard 
> usr=admin/pm-utils,metapackages/progress-linux-container

I will assume this uses diversions, but probably the strongest argument
for saying something in Policy was this confusion.  At least one Debian
maintainer didn't realize that diversions were the best way of doing this
rather than shipping binaries in a different PATH.

> sbin/syslogd: root=utils/busybox-syslogd usr=net/inetutils-syslogd
> bin/systemctl: root=admin/systemd usr=admin/systemctl
> sbin/update-service: root=admin/runit usr=admin/daemontools-run
> lib/x86_64-linux-gnu/libsystemd.so.0: root=libs/libelogind0 
> usr=libs/libsystemd0
> sbin/exfatlabel: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
> sbin/fsck.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
> sbin/mkfs.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs

> And one additional instance in unstable:

> lib/systemd/system/ifup@.service: root=admin/ifupdown2 usr=admin/ifupdown
> lib/systemd/system/networking.service: root=admin/ifupdown2 usr=admin/ifupdown

I've chcked these and they all declare explicit Conflicts.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Ansgar]

Hi Russ,

On Mon, 2021-08-23 at 13:41 -0700, Russ Allbery wrote:
> Right now, in the absence of such a plan, it's obvious that having
> two
> unrelated packages (that do not Conflict) ship a binary with the same
> name
> in /bin and /usr/bin is not sensible, yes?  (I believe that's the
> topic
> under discussion in this thread.)  I'm trying to understand if enough
> people thought this was a sensible, non-buggy thing to do today that
> it's
> worthwhile adding something to Policy explicitly saying that it's
> not.

Different, non-conflicting packages shipping binaries with the same
name in /bin and /usr/bin (or similar) should be resolved for a while
now. That as looked at when usrmerge was first introduced. I'm aware of
one instance where this was intentional to prefer one program over the
other (molly-guard), but it since uses diversions.

(Actually molly-guard ships several /sbin/pm-* binaries, while pm-utils
ships them as /usr/sbin/pm-*, but I think this is fine as the ones in
pm-utils are diverted by molly-guard's preinst script.)

For what it is worth I could find the following instances in Debian
bookworm (amd64 + all), not taking into account package conflicts,
covering /bin, /sbin and /lib*:

sbin/pm-hibernate: root=admin/molly-guard 
usr=admin/pm-utils,metapackages/progress-linux-container
sbin/pm-suspend: root=admin/molly-guard 
usr=admin/pm-utils,metapackages/progress-linux-container
sbin/pm-suspend-hybrid: root=admin/molly-guard 
usr=admin/pm-utils,metapackages/progress-linux-container
sbin/syslogd: root=utils/busybox-syslogd usr=net/inetutils-syslogd
bin/systemctl: root=admin/systemd usr=admin/systemctl
sbin/update-service: root=admin/runit usr=admin/daemontools-run
lib/x86_64-linux-gnu/libsystemd.so.0: root=libs/libelogind0 usr=libs/libsystemd0
sbin/exfatlabel: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
sbin/fsck.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
sbin/mkfs.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs

And one additional instance in unstable:

lib/systemd/system/ifup@.service: root=admin/ifupdown2 usr=admin/ifupdown
lib/systemd/system/networking.service: root=admin/ifupdown2 usr=admin/ifupdown

Ansgar

Re: merged /usr vs. symlink farms

[Russ Allbery]

Simon Richter  writes:

> It is less nonsensical because usrmerge exists, since we presumably
> don't want to keep the /bin paths in the packages, so at some point we
> need to move /bin/foo to /usr/bin/foo inside a package.  That is safe
> with current dpkg, as dpkg will not delete /bin/foo if it has the same
> inode as a just-unpacked file.

[...]

I think this implies that writing something in Policy about this would be
premature.  The issues you raise are related to something new that is
being discussed and would be part of a migration plan to a merged /usr
world.  The appropriate time to document those details in Policy would be
after we agreed on a plan, not now when they're just tentative ideas.

Right now, in the absence of such a plan, it's obvious that having two
unrelated packages (that do not Conflict) ship a binary with the same name
in /bin and /usr/bin is not sensible, yes?  (I believe that's the topic
under discussion in this thread.)  I'm trying to understand if enough
people thought this was a sensible, non-buggy thing to do today that it's
worthwhile adding something to Policy explicitly saying that it's not.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 23.08.21 17:23, Russ Allbery wrote:

[one package with /bin/foo, another with /usr/bin/foo]


This seems clearly nonsensical to me even if usrmerge was never on the
horizon, since which binary you got would randomly depend on the PATH
ordering and the order of /bin vs. /usr/bin in user-set PATHs is not fixed
and has never mattered.


It is less nonsensical because usrmerge exists, since we presumably 
don't want to keep the /bin paths in the packages, so at some point we 
need to move /bin/foo to /usr/bin/foo inside a package. That is safe 
with current dpkg, as dpkg will not delete /bin/foo if it has the same 
inode as a just-unpacked file.


We have another kind of common transition: moving files between packages 
with a Replaces: relation.


If a package undergoes both transitions in the same release cycle, then 
dpkg would indeed see package A containing /bin/foo, and package B with 
Replaces: A containing /usr/bin/foo.


And in this case, /bin/foo can be removed after /usr/bin/foo is 
unpacked, and then the file vanishes because dpkg did not register the 
ownership transfer.


   Simon

Re: merged /usr vs. symlink farms

[Zack Weinberg]

Tomas Pospisek  wrote:
> On 22.08.21 00:11, Guillem Jover wrote:
>> I'm personally just not seeing such consensus, despite the attempts of
>> some to make it pass as so. My perception is that this topic has become
>> such a black hole of despair, that people that take issue with it, are
>>  simply stepping away.
...
> I do not really care which solution will be chosen. I hope it will
> be one that doesn't break my system(s) too hard so I'll be able to
> ask a search engine and follow the hints and instructions.

I'm just a user, but this seems like the right moment for me to speak up:

Whether or not / and /usr ever get merged *doesn't affect me as a
user.*   All the stuff I use will be in my $PATH either way.  The
benefits, AIUI, are all for people developing or packaging software
that has to be compatible with many different Linux distributions, but
does not care about portability to non-Linux environments.  That
simply isn't me.  So I don't care if the transition happens or not,
nor about the timeframe, and I'd *like* to not have to care about how
it gets done.

What I *do* very much care about is whether I can trust package
installation (of official, stable packages) to not break my systems,
and the way this discussion is going, it seems like I might not be
able to, and yes, that is disheartening.

The chief dpkg maintainer has given clear technical reasons why the
approach taken by usrmerge risks breaking people's systems.  There is
a proof-of-concept in this very thread, demonstrating that the bugs
are real.  From where I'm sitting, that should have been the end of
the argument.  Hard stop on further merge-related changes in unstable
until a transition plan has been worked out that *won't* tickle dpkg
bugs; if that means waiting another release cycle in order to ship
dpkg bugfixes, *so be it*.  (I reiterate that as a user I don't care
whether this transition ever actually happens.)  Maybe even revert to
non-merged mode in the installer and drop usrmerge from testing and
stable, as a precaution.

But somehow what I see happening instead, is that Guillem's concerns
are being brushed aside, demoralizing him to the point where, if it
were me in his shoes, I would have resigned from the entire project;
and half the posters in this thread are raring to push ahead with a
transition that has a nonzero chance of wrecking one of my servers the
next time unattended-upgrades does its thing.  (If I understand the
issue correctly, it *could* bite on a point upgrade or even a security
patch of any system that has already been transitioned the way
usrmerge/d-i do it, including all fresh bullseye and now also bookworm
installations.)  That's a horrifying failure of both technical and
social project management.

I used the word "nonzero" in the last paragraph intentionally.
I don't want to hear any probabilistic arguments why everything should
be fine.  I want a transition plan that *guarantees* no breakage.
That's what Debian has given us end-users for 20+ years and I hate
that I might have to worry about not getting it again.

And I think several people here owe Guillem an apology.

zw

Re: merged-/usr vs. partially-symlink-farmed-root

[Marvin Renich]

* Ansgar  [210823 11:16]:
> Hi Marvin,
> 
> On Mon, 2021-08-23 at 10:29 -0400, Marvin Renich wrote:
> > Yet they cannot be counted on to work on Debian now, nor will they on
> > non- or partially-merged systems.  You are saying "the end result is
> > thus, so the partially merged system must have this property."
> 
> No. I am comparing end results from two different proposals. I am not
> talking about any intermediate state.
> 
> There is no replacing /bin with a /bin -> /usr/bin symlink ever in the
> partially-symlink-farmed-root proposal. So you only get the symlinks
> provided explicitly in /bin by packages and, e.g., dash would ship
> forever a /bin/sh -> /usr/bin/sh symlink and this would be present on
> all systems.
> 
> Only non-required symlinks like for, say, run-parts could be dropped.
> 
> See [1] about "finishing the transition".
> 
>   [1]: https://lists.debian.org/debian-devel/2019/02/msg00321.html

I did, indeed, miss this.  I stand corrected; please accept my
apologies.

While I understand Guillem's rational, I believe that the effort to
clean up once there are only symlinks in /{,s}bin could be easily
automated and would technically be worth the effort (including code in
dpkg to recognize symlinks such as /bin/dash → /usr/bin/dash and mark
them in its database as "intentionally not placed on the filesystem").

However, at this point, I don't believe it is socially feasible to
continue down the symlink-farm path.

Again, I apologize for the misinformed argument.

...Marvin

Re: merged /usr vs. symlink farms

[Russ Allbery]

Luca Boccassi  writes:

> Thank you - it has been brought up in this thread as an example of a
> valid setup, so if it is not, I think it could be good to be extra clear
> in the policy? How about the following:

If we tried to document every random bit of buggy packaging behavior
anyone thought of in Policy, Policy would become unwieldy, so I want to
verify here that someone really thought having one package containing a
file in /bin and another package containing the same file in /usr/bin was
was a reasonable thing to do (as opposed to accidental).  Are there
packages in the archive like this?  Or could you point me at the message
in the thread that said this was non-buggy?  I think I missed it.

This seems clearly nonsensical to me even if usrmerge was never on the
horizon, since which binary you got would randomly depend on the PATH
ordering and the order of /bin vs. /usr/bin in user-set PATHs is not fixed
and has never mattered.

(It may be that someone has done this *accidentally* and thus created an
edge case that the package management system has to cope with, but that's
a question of finding buggy packages, which is not something Policy can
really help with.)

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr vs. partially-symlink-farmed-root

[Ansgar]

Hi Marvin,

On Mon, 2021-08-23 at 10:29 -0400, Marvin Renich wrote:
> Yet they cannot be counted on to work on Debian now, nor will they on
> non- or partially-merged systems.  You are saying "the end result is
> thus, so the partially merged system must have this property."

No. I am comparing end results from two different proposals. I am not
talking about any intermediate state.

There is no replacing /bin with a /bin -> /usr/bin symlink ever in the
partially-symlink-farmed-root proposal. So you only get the symlinks
provided explicitly in /bin by packages and, e.g., dash would ship
forever a /bin/sh -> /usr/bin/sh symlink and this would be present on
all systems.

Only non-required symlinks like for, say, run-parts could be dropped.

See [1] about "finishing the transition".

  [1]: https://lists.debian.org/debian-devel/2019/02/msg00321.html

> Anyone who has a '#!/bin/python3' script now, must ensure the link is
> there themselves, and that would not change in the middle of a
> symlink-farm transition, nor would it hinder such transition.

Yes, but where it would work once the transition to the new proposed
layout differs between merged-/usr and partially-symlink-farmed-root
(unless one does fully-symlink-farmed-root including symlinks for all
binaries in /usr/bin and /usr/sbin, but that would be yet another
different proposal).

> However, do not use bogus arguments such as the one above to try to
> maken your point.  It clutters the discussion with needless
> debunking.

I think you misunderstand how the partially-symlink-farmed-root
proposal is different from the merged-/usr proposal.  Exactly to avoid
such misunderstandings the partially-symlink-farmed-root proposal
should not be named merged-/usr.

Ansgar

Re: merged /usr vs. symlink farms

[Sam Hartman]

> "Simon" == Simon Richter  writes:

>> I can see two arguments why we might need a dpkg update:
>> 
>> 1) To fix bugs related to directory aliasing.
>> 
>> I don't think that there is a consensus those bugs need to be
>> fixed to move forward.  (Put another way it's not clear the
>> community agrees they are RC).

Simon> dpkg as it is now will detect only the case when a file is
Simon> moved to the usrmerge path inside a package, or when a file
Simon> is moved from one package to another (with Replaces), but not
Simon> both at the same time.

I confirm I do not consider this RC nor do I believe it should block the
transition.
[explanation removed from quoted material]
Simon> We can work around that by introducing a policy that no files
Simon> that are currently registered as living outside /usr may be
Simon> moved between packages, but... just no.

When combined with a policy that you should not move files from
/bin|/lib to /usr/bin|/usr/lib in the bookworm cycle, that seems
workable to me.
In other words, individual packages should not transition to usrmerged
paths on their own during this cycle.
There may be some corner case where we need this.
If so, we'll have to consider the replaces issues.
Presumably we could replace and explicitly conflict rather than
breaking.
And yes that's not ideal but it will be sufficient to make sure that
both packages are not unpacked at the same time.


>> Yes, there are bugs.  Yes, it would be good to get them fixed in
>> the bookworm cycle.  But despite the issue being brought up,
>> there is not strong support for the idea that we must block on a
>> solution to the dpkg directory aliasing bugs.

Simon> I think that one of the release goals should be that any
Simon> freshly installed or upgraded system should have a dpkg
Simon> database that is consistent with reality, and I'd prioritize
Simon> that higher than actually finishing the transition, because
Simon> as long as we can have files vanish from under us, we can't
Simon> expect that the transition will be reliable for
bullseye-> bookworm updates.

I think you are in the rough (not part of the consensus) on this desire.



signature.asc
Description: PGP signature

Re: merged-/usr vs. partially-symlink-farmed-root

[Marvin Renich]

* Ansgar  [210822 17:29]:
> Hi,
> 
> On Sun, 2021-08-22 at 12:29 -0400, Marvin Renich wrote:
> > * Ansgar  [210822 05:08]:
> > > To get a filesystem layout equivalent to merged-/usr via symlinks
> > > farming *every* package shipping files in at least /usr/bin,
> > > /usr/sbin and possibly some of /usr/lib would need to include
> > > symlinks in /bin, /sbin, /lib.  This would affect far more
> > > packages than updating the packages currently shipping files in
> > > /bin, /sbin and /lib* to ship these under /usr instead.
> > 
> > It is true that for a symlink-farm-usr-merge system to be strictly
> > equivalent to a symlink-dir-usr-merge system, many packages that
> > never had /bin/foo but had /usr/bin/foo would have to add a symlink
> > /bin/foo, however this is clearly unnecessary.
> 
> No, it is required if we want users to be able to run scripts that use
> `#!/bin/python3` (or similar for other interpreters).  Such scripts
> exist in the wild, they were not initially written on Debian.
> 
> These would need Debian-specific patches to work on Debian.  It is just
> the opposite of now having to patch software packaged in Debian to use
> /bin/rm instead of /usr/bin/rm (that also happens in the wild for
> software not developed on Debian).  We also happen to miss such
> instances and needlessly make software buggy on Debian.
> 
> Not having to worry about yet another small incompatibility between
> distributions is a nice feature.

Yet they cannot be counted on to work on Debian now, nor will they on
non- or partially-merged systems.  You are saying "the end result is
thus, so the partially merged system must have this property."  That
does not follow.  It is part of the impatience for a finished product
that usually ends up with something less ideal than would be possible
with a more patient approach.  (And the symlink-dir approach appears to
me to be driven primarily by impatience.)

Anyone who has a '#!/bin/python3' script now, must ensure the link is
there themselves, and that would not change in the middle of a
symlink-farm transition, nor would it hinder such transition.

I was initially in favor of the symlink-farm approach, because it is the
correct technical solution.  I have become convinced that the
symlink-dir approach is the correct social solution (which is often, and
I believe in this case it is, worse than the correct technical
solution).  I also believe that fixing dpkg to handle the general
"symlink dir on file system where .deb has a real dir" case is useful
for other cases.

However, do not use bogus arguments such as the one above to try to make
your point.  It clutters the discussion with needless debunking.

...Marvin