Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Askar Safin]

Hi, Helmut. I'm very sorry for responding to an 8-months old letter,
but I think my message is important.

Helmut Grohne:
> * mmdebstrap operates in two phases. It first unpacks and configures a
>   rather minimal set of packages and then proceeds to adding packages
>   passed to --include in a second phase once essential is fully
>   configured while debootstrap immediately unpacks everything.
>
>   I think the debootstrap approach is slightly worse here, because it
>   means that preinst scripts of non-essential packages cannot rely on
>   essential packages having been configured.

This is not true. Here is output of debootstrap:
https://paste.debian.net/1304816/ .

We created current sid from sid. As you can see, mc unpacked in very last stage.

The same can be seen in debootstrap.log: https://paste.debian.net/1304817/

Moreover, I did another experiment: I did run debootstrap and aborted
it immediately after output "Unpacking the base system..."
The resulting system did not contain "mc" binary at all!

(This is answer to https://lists.debian.org/debian-dpkg/2023/05/msg00080.html )
-- 
Askar Safin

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Sun, 11 Jun 2023, 14:32 Jeroen Dekkers,  wrote:

> On Fri, 09 Jun 2023 22:14:16 +0200,
> Helmut Grohne wrote:
> > On Fri, Jun 09, 2023 at 02:42:25PM -0500, Richard Laager wrote:
> > > Later, whatever replaces /lib64 with a symlink needs to deal with
> this, but
> > > that's not significantly different than whatever it was going to do
> anyway,
> > > right? Just do this:
> > >
> > > 1. Whatever safety checks are appropriate.
> > > 2. Unless already verified to be identical by #1, hardlink
> > > /lib64/ld-linux-x86-64.so.2 to /usr/lib64/ld-linux-x86-64.so.2. This
> might
> > > be just a particular instance of the more general case of hardlink
> > > everything from /lib64 into /usr/lib64.
> > > 3. Unlink everything from /lib64.
> > > 4. Unlink /lib64.
> > > 5. Symlink /lib64 to /usr/lib64
> >
> > I think we start from the premise that /lib64 already is a symlink and
> > as long as libc6 actually ships /lib64 (even if empty), dpkg won't
> > delete it. What we will not get here is getting rid of the aliasing and
> > we will also be unable to ship /lib64 as a symlink in any data.tar
> > (since that would be a directory vs symlink conflict, which has
> > unpack-order-dependent behaviour, which is bad).
>
> But if all packages in trixie are changed to not ship /lib64 anymore, there
> wouldn't be a conflict in trixie anymore? If we have the following
> situation:
>
> - We change dpkg to never delete /bin, /lib, /lib32, /lib64, /libo32,
> /libx32,
>   and /sbin. We can also change dpkg in bookworm to not do this so that we
> are
>   sure that when upgrading to trixie we have a dpkg that won't delete any
> of
>   these symlinks.


Changing dpkg is a non starter. Even assuming it's doable (it's not) it
would take years of fighting before a single line of code was merged, and
would require finding a new maintainer at a minimum. Are you volunteering
for the job? The file move moratorium would have to remain in place for
2/3/4 releases on top of that while all of this is sorted.

Helmut's idea is good because it's doable in the real world. Are there
theoretical alternatives? Most likely. But let's focus on what's possible
and really doable in the next month or so and get this over with, please.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Jeroen Dekkers]

On Fri, 09 Jun 2023 22:14:16 +0200,
Helmut Grohne wrote:
> On Fri, Jun 09, 2023 at 02:42:25PM -0500, Richard Laager wrote:
> > Because you want to support non-usr-merged systems, e.g. for derivatives?
>
> dpkg is used in any different contexts. A very simple example of a
> non-merged system would be Debian stretch. For another dpkg really is
> being used for things that are not based on Debian. While it is the
> Debian package manager, it has uses beyond dpkg has (thus far) stayed
> away from imposing policy on the filesystem layout.

Refusing to delete /bin etc. doesn't mean that dpkg imposes any policy on the
filesystem layout. The change would also be small enough that it would be easy
to disable it, but I can't think of any usage of dpkg for which that would be
needed.

> > They aren't going to want to delete /bin either, so I don't see how a
> > special-case preventing deletion of /bin would be problematic.
>
> Indeed. However, if you actually manage to trigger this, it can be very
> surprising. Your hard coded list would also contain /lib32, /libx32 and
> /libo32. Then you install some mipsen packages, remove them and wonder
> why /libo32 does not go away. piuparts is definitely unhappy at this
> point. I am quite sure that this behaviour would break something. What I
> am not sure about is whether accepting such breakage is a reasonable
> trade-off.

I can't really think of anything that would break with having an extra directory
or symlink around. And if base-files ships the symlinks they would always be
there and piuparts would be happy.

> > Later, whatever replaces /lib64 with a symlink needs to deal with this, but
> > that's not significantly different than whatever it was going to do anyway,
> > right? Just do this:
> >
> > 1. Whatever safety checks are appropriate.
> > 2. Unless already verified to be identical by #1, hardlink
> > /lib64/ld-linux-x86-64.so.2 to /usr/lib64/ld-linux-x86-64.so.2. This might
> > be just a particular instance of the more general case of hardlink
> > everything from /lib64 into /usr/lib64.
> > 3. Unlink everything from /lib64.
> > 4. Unlink /lib64.
> > 5. Symlink /lib64 to /usr/lib64
>
> I think we start from the premise that /lib64 already is a symlink and
> as long as libc6 actually ships /lib64 (even if empty), dpkg won't
> delete it. What we will not get here is getting rid of the aliasing and
> we will also be unable to ship /lib64 as a symlink in any data.tar
> (since that would be a directory vs symlink conflict, which has
> unpack-order-dependent behaviour, which is bad).

But if all packages in trixie are changed to not ship /lib64 anymore, there
wouldn't be a conflict in trixie anymore? If we have the following situation:

- We change dpkg to never delete /bin, /lib, /lib32, /lib64, /libo32, /libx32,
  and /sbin. We can also change dpkg in bookworm to not do this so that we are
  sure that when upgrading to trixie we have a dpkg that won't delete any of
  these symlinks.
- All packages in trixie are changed to have their files moved to /usr.
- Base-files will include the symlinks

In the case of bootstrapping no trixie package will have any files in /bin etc.
so there wouldn't be a directory vs symlink conflict. From what I understood we
need to change the bootstrap protocol to make sure that base-files is unpacked
before anything is run so that /bin/sh and ld-linux.so are available. To me that
seems to be a simple change to make and it should also be possible to make this
small change in the bootstrapping tools in bookworm.

In the case of upgrading we already have all the symlinks in the filesystem.
Installing base-files when there are still packages installed with files in /bin
etc. shouldn't be a problem as far as I understand it. And we changed dpkg to
never delete the symlinks when no package ships the /bin directory anymore so it
also shouldn't be a problem if all packages that used to ship /bin etc/ are
upgraded before base-files is upgraded.

Kind regards,

Jeroen Dekkers

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Helmut Grohne]

Hi,

On Fri, Jun 09, 2023 at 09:57:21PM +0200, HW42 wrote:
> Did you consider just having one package keep one dummy file in /bin?
> While this isn't elegant it sounds much less complex than diversions and
> tricky pre-depend loops, etc.

The dummy file is not necessary. Debian packages can ship empty
directories. Having any package ship /bin (empty or not) is fully
sufficient to prevent dpkg from removing it.

> I might be very well missing something here (for example maybe it's
> really essential that no files remain in /bin, even not a dummy file).
> But in the other branch of this thread you welcomed "dumb" questions, so
> here you go ;]

Yeah, I consider the property that nothing ships anything in aliased
locations an important one. So let us go down for the consequences of
not doing that.

So some package will keep shipping /bin. It does't really matter which,
but clearly this package must be part of the essential set (otherwise
you could remove it and with it /bin would be deleted). This is cool for
upgrades, but less so for bootstrapping tools.

One of the approaches to making bootstrapping work was adding the
symlinks to some data.tar. That has been category 2 from my earlier
mail. We definitely cannot add /bin as a directory to one package and
/bin as a symlink to another (unless using diversions), because the
resulting behaviour is dependent on the unpack order when used with
dpkg. Also any bootstrap tool that unpacks with tar -k (such as
debootstrap) requires changes to support this. So this pretty much
precludes completing the transition in a way that just unpacking all
data.tar of essential packages gives you a working chroot. In effect,
this requires a proposal to change the bootstrap protocol (category 4)
in order to make sense.

There is a loop hole that I ignored here. While /bin cannot be both a
directory and a symlink at the same time, we can upgrade it. So if we
somehow managed to get one and only one package to contain /bin as a
directory, we could upgrade that to a symlink. Unfortunately, any
external package that still ships stuff in /bin breaks this. In effect,
any addon repository or old package can break your system.

The other way of seeing us keep /bin as a directory is to not
canonicalize (i.e. category 1). Then we'd simply keep (wlog) /bin/sh in
/bin and not move it to /usr.

Can you elaborate in what way you see protective diversions as adding
complexity? It can be as simple as:

dpkg-divert --add /bin --divert someplacewedontcare --package base-files 
--no-rename

We'd add this to one package and everyone else can issue Pre-Depends.

The benefit we gain from keeping /bin is not clear to me (beyond
avoiding a diversion). At this time, it seems to me that doing that
either requires changing all bootstrapping tools (in yet unspecified
ways) or never canonicalizing all paths (according to the dpkg
database).

Now I'm wondering what I am missing here.

Helmut

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Simon Richter]

Hi,

On 10.06.23 00:41, Steve McIntyre wrote:


What exactly do you mean here? You know that even a statically linked
executable needs an interpreter defined in the ELF header?


/sbin/ldconfig has no PT_INTERP segment.

If you use libdl, you need to be loaded through ld.so, and since PAM 
uses libdl to load plugins, accessing databases doesn't work when 
statically linked without an interpreter, so a lot of people use an 
interpreter even in a static link, but it's not required (and would be 
counterproductive, the interpreter chain needs to terminate somewhere).


   Simon


OpenPGP_signature
Description: OpenPGP digital signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[HW42]

Helmut Grohne:
> Hi Johannes,
> 
> On Fri, Jun 09, 2023 at 05:47:56PM +0200, Johannes Schauer Marin Rodrigues 
> wrote:
>> if I understand that plan correctly, the usrmerge-support package
>> setting up diversions is only necessary because you want to avoid
>> having to do the move to /usr of *all* affected packages in the
>> essential set in a single dinstall? Is that correct?
> 
> This is not correct. In bookworm -> trixie upgrade scenario, we intend
> to move all the files from / to /usr. Now we look into how this
> happens focus on one particular symlink, without loss of generality
> choose /bin. Since /bin is no longer in the dpkg database at the end
> of the upgrade, some package must be the last one to contain /bin.
> When upgrading (or removing that package), dpkg will attempt to remove
> /bin (which in its opinion is an empty directory and the last consumer
> is releasing it). However, since dpkg has no clue about file types, it
> doesn't actually know that this is a directory and takes care of the
> /bin -> /usr/bin symlink using unlink(). And this is where /bin
> vanishes. Oops.
> 
> So the idea here is to add a protective diversion for /bin such that
> removing /bin instead removes some path we don't care about. [...]

Did you consider just having one package keep one dummy file in /bin?
While this isn't elegant it sounds much less complex than diversions and
tricky pre-depend loops, etc.

I might be very well missing something here (for example maybe it's
really essential that no files remain in /bin, even not a dummy file).
But in the other branch of this thread you welcomed "dumb" questions, so
here you go ;]


OpenPGP_signature
Description: OpenPGP digital signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Helmut Grohne]

Hi Richard,

On Fri, Jun 09, 2023 at 02:42:25PM -0500, Richard Laager wrote:
> Is the broader context here that this is an alternative to teaching dpkg
> about aliasing? That is, we just arrange the transition correctly such that
> we get out of the aliased situation as part of upgrading to trixie?

Yes, I think the idea that we are mostly exploring now is not teaching
dpkg about aliasing and rather move all the files to their canonical
location such that there no longer is any aliasing that dpkg would have
to deal with.

I don't think this is complete consensus at this time, but the majority
of discussion participants appear to favour this approach.

> Because you want to support non-usr-merged systems, e.g. for derivatives?

dpkg is used in any different contexts. A very simple example of a
non-merged system would be Debian stretch. For another dpkg really is
being used for things that are not based on Debian. While it is the
Debian package manager, it has uses beyond dpkg has (thus far) stayed
away from imposing policy on the filesystem layout.

> They aren't going to want to delete /bin either, so I don't see how a
> special-case preventing deletion of /bin would be problematic.

Indeed. However, if you actually manage to trigger this, it can be very
surprising. Your hard coded list would also contain /lib32, /libx32 and
/libo32. Then you install some mipsen packages, remove them and wonder
why /libo32 does not go away. piuparts is definitely unhappy at this
point. I am quite sure that this behaviour would break something. What I
am not sure about is whether accepting such breakage is a reasonable
trade-off.

> Am I understanding the problem correctly?

I confirm.

> What would happen if, for trixie only, bin:libc6 shipped two identical
> copies of ld-linux-x86-64.so.2, one in each of /lib64 and /usr/lib64?

That's an interesting idea. Do note that we don't actually have to ship
ld-linux in both locations. We can actually move it in a safe way
(unless we also move it between packages, which we don't). So let me
change that to: We keep /lib64 (the directory) in addition to
/usr/lib64. Keeping the directory prevents dpkg from deleting the
symlink (as it doesn't know about the filetype).

> Then at step 2, /lib64 does not get deleted and nothing breaks.

Confirmed (with the simplified variant).

> Later, whatever replaces /lib64 with a symlink needs to deal with this, but
> that's not significantly different than whatever it was going to do anyway,
> right? Just do this:
> 
> 1. Whatever safety checks are appropriate.
> 2. Unless already verified to be identical by #1, hardlink
> /lib64/ld-linux-x86-64.so.2 to /usr/lib64/ld-linux-x86-64.so.2. This might
> be just a particular instance of the more general case of hardlink
> everything from /lib64 into /usr/lib64.
> 3. Unlink everything from /lib64.
> 4. Unlink /lib64.
> 5. Symlink /lib64 to /usr/lib64

I think we start from the premise that /lib64 already is a symlink and
as long as libc6 actually ships /lib64 (even if empty), dpkg won't
delete it. What we will not get here is getting rid of the aliasing and
we will also be unable to ship /lib64 as a symlink in any data.tar
(since that would be a directory vs symlink conflict, which has
unpack-order-dependent behaviour, which is bad).

> However, note that this cannot be a shell script, as then step 3 would
> delete /lib64/ld-linux-x86-64.so.2 and everything after that would fail.

Non-issue since we assume that bookworm is merged already.

> At that point, everything is fine, EXCEPT that dpkg now thinks it has a
> /lib64/ld-linux-x86-64.so.2 file installed, but really that is aliasing
> /usr/lib64/ld-linux-x86-64.so.2. When bin:lib6:amd64 is later upgraded (e.g.
> in forky) to a version that stops shipping /lib64/ld-linux-x86-64.so.2, dpkg
> will unlink /lib64/ld-linux-x86-64.so.2 and then everything breaks.

Simplified: dpkg thinks that it has /lib64 while it should not. When we
drop that in forky, stuff breaks.

> The fix to that is either whatever separate general case fix is being done
> for aliasing, or if the whole point is we are trying to avoid having that
> sort of thing at all, then just put in a special case that dpkg will not
> unlink /lib64/ld-linux-x86-64.so.2.

Yes, if we add that special casing to dpkg, we can remove /lib64 in forky.

> So we end up with something roughly like this in dpkg (please excuse
> syntax/pointer errors):
> 
> Wherever file deletions are handled, make this change:
> - unlink(pathname);
> + special_unlink(pathname);
> 
> to use this:
> 
> char *SPECIAL_PATHS[] = {
> "/bin",
> "/lib",
> "/lib64",
> "/lib64/ld-linux-x86-64.so.2",
> "/sbin",
> NULL,
> }
> 
> void special_unlink(const char *pathname) {
> const char **special;
> for (special = SPECIAL_PATHS ; *special ; special++) {
> if (strcmp(pathname, special) == 0) {
> return;
> }
> }
> unlink(pathname);
> }

Might work, but the list of SPECIAL_PATH

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Richard Laager]

I know I haven't thought about this as much as others, so I might be 
naively missing something here.


Is the broader context here that this is an alternative to teaching dpkg 
about aliasing? That is, we just arrange the transition correctly such 
that we get out of the aliased situation as part of upgrading to trixie?


On 2023-06-09 13:47, Helmut Grohne wrote:

I caution that this protection mechanism of symlinks is a property of
the installation and not of dpkg. Depending on what dpkg is operating
on, we expect it to handle this or not. So we'd need a way to tell
whether an installation needs this kind of special handling.


Because you want to support non-usr-merged systems, e.g. for 
derivatives? They aren't going to want to delete /bin either, so I don't 
see how a special-case preventing deletion of /bin would be problematic.



On amd64, we'd upgrade libc6 before dpkg and then /lib64 would go
missing, because libc6 already is the last package that ships files in
/lib64.


Am I understanding the problem correctly?

1. src:libc "moves" /lib64/ld-linux-x86-64.so.2 to 
/usr/lib64/ld-linux-x86-64.so.2, i.e. it builds bin:libc6:amd64 such 
that it ships the latter path.
2. bin:libc6:amd64 gets upgraded before dpkg. The normal unpacking 
behavior of dpkg results in /lib64/ld-linux-x86-64.so.2 being deleted 
and since /lib64 is now empty, dpkg deletes /lib64.
3. Everything breaks, because /lib64/ld-linux-x86-64.so.2 is special in 
the ELF ABI.



What would happen if, for trixie only, bin:libc6 shipped two identical 
copies of ld-linux-x86-64.so.2, one in each of /lib64 and /usr/lib64?


Then at step 2, /lib64 does not get deleted and nothing breaks.

Later, whatever replaces /lib64 with a symlink needs to deal with this, 
but that's not significantly different than whatever it was going to do 
anyway, right? Just do this:


1. Whatever safety checks are appropriate.
2. Unless already verified to be identical by #1, hardlink 
/lib64/ld-linux-x86-64.so.2 to /usr/lib64/ld-linux-x86-64.so.2. This 
might be just a particular instance of the more general case of hardlink 
everything from /lib64 into /usr/lib64.

3. Unlink everything from /lib64.
4. Unlink /lib64.
5. Symlink /lib64 to /usr/lib64

However, note that this cannot be a shell script, as then step 3 would 
delete /lib64/ld-linux-x86-64.so.2 and everything after that would fail.


At that point, everything is fine, EXCEPT that dpkg now thinks it has a 
/lib64/ld-linux-x86-64.so.2 file installed, but really that is aliasing 
/usr/lib64/ld-linux-x86-64.so.2. When bin:lib6:amd64 is later upgraded 
(e.g. in forky) to a version that stops shipping 
/lib64/ld-linux-x86-64.so.2, dpkg will unlink 
/lib64/ld-linux-x86-64.so.2 and then everything breaks.


The fix to that is either whatever separate general case fix is being 
done for aliasing, or if the whole point is we are trying to avoid 
having that sort of thing at all, then just put in a special case that 
dpkg will not unlink /lib64/ld-linux-x86-64.so.2.


So we end up with something roughly like this in dpkg (please excuse 
syntax/pointer errors):


Wherever file deletions are handled, make this change:
- unlink(pathname);
+ special_unlink(pathname);

to use this:

char *SPECIAL_PATHS[] = {
"/bin",
"/lib",
"/lib64",
"/lib64/ld-linux-x86-64.so.2",
"/sbin",
NULL,
}

void special_unlink(const char *pathname) {
const char **special;
for (special = SPECIAL_PATHS ; *special ; special++) {
if (strcmp(pathname, special) == 0) {
return;
}
}
unlink(pathname);
}

--
Richard


OpenPGP_signature
Description: OpenPGP digital signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Helmut Grohne]

Hi Richard,

On Fri, Jun 09, 2023 at 01:07:13PM -0500, Richard Laager wrote:
> On 2023-06-09 11:26, Helmut Grohne wrote:
> > When upgrading (or
> > removing that package), dpkg will attempt to remove /bin (which in its
> > opinion is an empty directory and the last consumer is releasing it).
> > However, since dpkg has no clue about file types, it doesn't actually
> > know that this is a directory and takes care of the /bin -> /usr/bin
> > symlink using unlink(). And this is where /bin vanishes. Oops.
> 
> This might be a dumb question, but could we just special-case this? That is,
> dpkg would simply not remove /bin specifically? If the list of directories
> is small, known, and relatively fixed (e.g. /bin, /usr/bin, /lib), that
> might be workable.

Even if this was a dumb question, it's these kind of questions that -
surprisingly often - lead to new insights. So thanks for asking.

I caution that this protection mechanism of symlinks is a property of
the installation and not of dpkg. Depending on what dpkg is operating
on, we expect it to handle this or not. So we'd need a way to tell
whether an installation needs this kind of special handling. Anyway,
let's for now just assume that magically dpkg would magically save those
symlinks when we want to save them.

Now any package that moves files from / to /usr, needs to ensure that
the dpkg doing that move is recent. That's a dependency we cannot
express in theory. David Kalnischkies spent some time going into
detail[1] about this aspect. I think the bottom line is that for all
practical purposes we're probably fine if everything that moves also
gains a Pre-Depends on dpkg. Except that dpkg Pre-Depends on libc6,
which would now Pre-Depends: dpkg and we're back to our Pre-Depends
loop. So now we say "screw it" and let libc6 get a pass without this
Pre-Depends, because so many packages already have a Pre-Dependency on
dpkg, it'll probably get upgraded early and what could possibly go
wrong? On amd64, we'd upgrade libc6 before dpkg and then /lib64 would go
missing, because libc6 already is the last package that ships files in
/lib64. So really, libc6 is one of the few packages that really must
depend on that fixed dpkg. It also is one of the few packages that
really cannot.

As we cannot get out of this loop, we consider stretching the transition
over two releases. For trixie, we just update dpkg without moving files
and then for the trixie -> forky upgrade, we know (since we forbid skip
upgrades) that dpkg is fixed and then it actually works out without this
mess of Pre-Depends on dpkg.

My impression is that we'd like to have this done sooner rather than
later. At this time, the protective diversion seems like a fairly easy
and reliable mitigation with little downsides (except for having a new
transitively essential package) that helps us move forward faster.

Please don't stop asking. The chances that something about this is wrong
or missing something is significantly non-zero. I hope that this kind of
peer-review will get us to a solution that actually works in practice.

Helmut

[1] https://lists.debian.org/20230503130026.ixu4zlymo4fykdru@crossbow

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Richard Laager]

On 2023-06-09 11:26, Helmut Grohne wrote:

When upgrading (or
removing that package), dpkg will attempt to remove /bin (which in its
opinion is an empty directory and the last consumer is releasing it).
However, since dpkg has no clue about file types, it doesn't actually
know that this is a directory and takes care of the /bin -> /usr/bin
symlink using unlink(). And this is where /bin vanishes. Oops.


This might be a dumb question, but could we just special-case this? That 
is, dpkg would simply not remove /bin specifically? If the list of 
directories is small, known, and relatively fixed (e.g. /bin, /usr/bin, 
/lib), that might be workable.


--
Richard



OpenPGP_signature
Description: OpenPGP digital signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Helmut Grohne]

Hi Johannes,

On Fri, Jun 09, 2023 at 05:47:56PM +0200, Johannes Schauer Marin Rodrigues 
wrote:
> if I understand that plan correctly, the usrmerge-support package setting up
> diversions is only necessary because you want to avoid having to do the move 
> to
> /usr of *all* affected packages in the essential set in a single dinstall? Is
> that correct?

This is not correct. In bookworm -> trixie upgrade scenario, we intend
to move all the files from / to /usr. Now we look into how this happens
focus on one particular symlink, without loss of generality choose /bin.
Since /bin is no longer in the dpkg database at the end of the upgrade,
some package must be the last one to contain /bin. When upgrading (or
removing that package), dpkg will attempt to remove /bin (which in its
opinion is an empty directory and the last consumer is releasing it).
However, since dpkg has no clue about file types, it doesn't actually
know that this is a directory and takes care of the /bin -> /usr/bin
symlink using unlink(). And this is where /bin vanishes. Oops.

So the idea here is to add a protective diversion for /bin such that
removing /bin instead removes some path we don't care about. The
important thing now is that every package that moves stuff from /bin to
/usr/bin needs to ensure that this diversion exists. We can achieve that
in one of two ways. Either that some package (and with that I mean every
package that ships stuff in /bin, because we cannot predict which
package will be last) gains a preinst that sets up this diversion (on
behalf of base-files) or it Pre-Depends on some package that handles
setting up this diversion. It seems rather obvious that we might just
have a versioned "Pre-Depends: base-files (>= version that introduces
the diversion)", but then we get a pre-dependency loop from base-files
via an awk implementation to libc6 and then (via this new Pre-Depends)
back to base-files. So base-files cannot be the package that we list in
Pre-Depends here. And this is where usrmerge-support comes into the
picture. Any package that moves stuff out of one of the aliased
directories gains a Pre-Depends: usrmerge-support to protect the
aliasing symlinks from deletion.

Please note that this hasn't been obvious to me at all. I totally didn't
see this pre-dependency loop coming until dpkg told me when I actually
tried this.

So this usrmerge-support package very much is not for reducing that set
that we have to upload in one dinstall, but for making smooth upgrades
work at all.

This really is an important detail and I'm sorry for having missed it in
my previous mail. Thanks for asking.

> If yes, how many source packages are we have to be modified part from
> base-files, dash, bash, libc6, and util-linux?

Given the above, I think this no longer is relevant.

> Would it be too much to prepare patches for all of these, test that everything
> works with some QA setup and then NMU all 22 source packages with pre-approved
> patches in a single dinstall? Would that avoid having to temporarily go via a
> usrmerge-support package?

I have considered this approach and if it gains us something I
definitely see it as something to consider, but given the above, it
doesn't save us from usrmerge-support.

The other thing that we need usrmerge-support for is the dpkg-divert
wrapper. Any package that contains an aliased diversion or moves a
diverted file from an aliased location will likewise have to gain a
pre-dependency on usrmerge-support. Again, we cannot do this in
usr-is-merged, because that would kick usrmerge out of the default
essential set and thus break mmdebstrap.

Helmut

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2023-06-09 15:22:39)
> Add a new package usrmerge-support (or whatever). It is a bit similar to
> multiarch-support: It must not have any dependencies or pre-dependencies. It
> will not have files, but maintainer scripts. Those scripts set up protective
> diversions on behalf of base-files for the symbolic links that cause
> aliasing. Then base-files will issue a Pre-Depends on usrmerge-support (but
> not yet ship symlinks). I initially thought, this could be part of
> usr-is-merged, but then base-files would pull that and standard mmdebstrap
> would no longer pull usrmerge and break. So it really needs to be a separate
> package. Anyway, once we have protective diversions, we can move files
> without risking that dpkg deletes the symbolic links.
> 
> Then we can actually perform that move of files to their canonical
> locations except for a small set of locations including dash, bash,
> libc6, and util-linux (maybe not exhaustive). [There is a lot of missing
> detail about non-bootstrap aspects here.]
> 
> Once all essential packages (but the exceptions) have no files left in
> aliased locations, we can upload base-files adding the symlinks together
> with the packages previously kept unmodified in one dinstall. Before
> that dinstall, things will continue to work normally. The protective
> diversions will not affect unpacking, because dpkg only performs exact
> matches on diversions. After that dinstall, base-files will create the
> symlinks and things will hopefully work (because the patched debootstrap only
> creates them after the initial unpack).

if I understand that plan correctly, the usrmerge-support package setting up
diversions is only necessary because you want to avoid having to do the move to
/usr of *all* affected packages in the essential set in a single dinstall? Is
that correct?

If yes, how many source packages are we have to be modified part from
base-files, dash, bash, libc6, and util-linux?

Is it just these? audit bzip2 coreutils debianutils dpkg gcc-13 grep gzip
hostname libcap2 libcap-ng libgpg-error libselinux libxcrypt ncurses pam sed
shadow sysvinit tar xz-utils zlib

Would it be too much to prepare patches for all of these, test that everything
works with some QA setup and then NMU all 22 source packages with pre-approved
patches in a single dinstall? Would that avoid having to temporarily go via a
usrmerge-support package?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Steve McIntyre]

Raphaël Hertzog wrote:
>
>In the same spirit, I'd like to throw an idea... could we decide that
>base-files is the first package to be configured as part of the bootstrap
>protocol and change base-files maintainer's scripts into statically linked
>executables so that they can work even if we don't have the library loader
>on the ABI-compliant path?

What exactly do you mean here? You know that even a statically linked
executable needs an interpreter defined in the ELF header?

--
Steve McIntyre, Cambridge, UK.st...@einval.com
< sladen> I actually stayed in a hotel and arrived to find a post-it
  note stuck to the mini-bar saying "Paul: This fridge and
  fittings are the correct way around and do not need altering"

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Johannes Schauer Marin Rodrigues]

Quoting Marco d'Itri (2023-06-09 09:41:43)
> On Jun 08, Raphael Hertzog  wrote:
> > And creating the required symlinks would be done by those (standalone)
> > maintainer scripts...
> > 
> > I don't know if we already have some rule/invariant in the configuration
> > order of the unpacked packages, but I doubt so.
> Indeed, this would be very simple and it has already been proposed.
> But somebody then complained that special-casing a package would violate 
> the design contraints he self-imposed to his own image building tool, 
> and as we all know every Debian maintainer can veto any systemic changes that
> they do not like.

I definitely complained about special-casing a package because it would violate
the design contract for my own image building tool. You would not by any chance
be talking about me in your last message, would you?

Anyway, great communication style. This will totally help bringing us all
together to create a great operating system. Very productive. I already feel a
lot more motivated to discuss technical matters with you.

Now who do I have to talk to in case I'd really like to veto something?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Helmut Grohne]

Hi Raphaël,

On Thu, Jun 08, 2023 at 10:46:24AM +0200, Raphael Hertzog wrote:
> In the same spirit, I'd like to throw an idea... could we decide that
> base-files is the first package to be configured as part of the bootstrap
> protocol and change base-files maintainer's scripts into statically linked
> executables so that they can work even if we don't have the library loader
> on the ABI-compliant path?

Thanks for putting effort into this questions. You already figured that
this poses a problem to dpkg calling the maintainer script. Let me add
two further observations to further understand the solution space here.

dpkg has a --root flag and can be called externally. This is something
mmdebstrap already uses in some modes. That way, we avoid the issue you
presented for dpkg itself. Unfortunately, we cannot assume presence of
dpkg outside the chroot as debootstrap supports running on non-Debian
systems, so the --root flag doesn't actually help us here.

The other aspect is that maintainer scripts that are not interpreted
break chrootless foreign architecture bootstrap as the
base-files.preinst would be an executable that the processor cannot
execute.

In a vague reply to the other messages as well: I repeatedly got the
feedback that I have not sufficiently exploited the solution space. I
hear you. My lack of replies here shall indicate that I'm not done.

I have a vague sketch that seems to kinda work out for everything, but
maybe it still has some problems that I don't see yet. Let me summarize
it even though this very much is unfinished. Given my earlier
categorization of the solution space, this is a category 2 solution
addressing many of the problems mentioned there.

Update debootstrap (in bookworm and unstable) to create the symbolic
links after unpacking rather than before while still doing it before
running any maintainer scripts. This enables us to ship the symbolic
links in some data.tar while keeping bootstraps of bookworm and earlier
working as before.

Add a new package usrmerge-support (or whatever). It is a bit similar to
multiarch-support: It must not have any dependencies or
pre-dependencies. It will not have files, but maintainer scripts. Those
scripts set up protective diversions on behalf of base-files for the
symbolic links that cause aliasing. Then base-files will issue a
Pre-Depends on usrmerge-support (but not yet ship symlinks). I initially
thought, this could be part of usr-is-merged, but then base-files would
pull that and standard mmdebstrap would no longer pull usrmerge and
break. So it really needs to be a separate package. Anyway, once we have
protective diversions, we can move files without risking that dpkg
deletes the symbolic links.

Then we can actually perform that move of files to their canonical
locations except for a small set of locations including dash, bash,
libc6, and util-linux (maybe not exhaustive). [There is a lot of missing
detail about non-bootstrap aspects here.]

Once all essential packages (but the exceptions) have no files left in
aliased locations, we can upload base-files adding the symlinks together
with the packages previously kept unmodified in one dinstall. Before
that dinstall, things will continue to work normally. The protective
diversions will not affect unpacking, because dpkg only performs exact
matches on diversions. After that dinstall, base-files will create the
symlinks and things will hopefully work (because the patched debootstrap
only creates them after the initial unpack).

This still is a lot of wishful thinking. I've prototyped parts of this,
but not the entire story. I'm pretty sure it'll not work out as written
here, but maybe some adaption of it will unless insurmountable issues
pop up. For instance, debootstrap --variant=buildd (which currently
implies --no-merged-usr) will need a second thought.

You may now tell me why this is utter nonsense and why it cannot work at
all. Thanks.

Helmut

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Fri, 9 Jun 2023 at 10:53, Raphael Hertzog  wrote:
>
> On Fri, 09 Jun 2023, Marco d'Itri wrote:
> > On Jun 08, Raphael Hertzog  wrote:
> >
> > > In the same spirit, I'd like to throw an idea... could we decide that
> > > base-files is the first package to be configured as part of the bootstrap
> > > protocol and change base-files maintainer's scripts into statically linked
> > > executables so that they can work even if we don't have the library loader
> > > on the ABI-compliant path?
> > It could be even easier: base-files could be unpacked once without
> > running the maintainer scripts and then "reinstalled" again later as
> > usual.
>
> I think you are missing the point here, that only works if the package is
> shipping the symlinks. And the idea is to not do this immediately because
> it breaks debootstrap: if I understood correctly unpacking base-files
> with the symlinks would fail if debootstrap had already pre-created those
> symlinks (due to a -k option that we should get rid of in
> /usr/share/debootstrap/scripts/debian-common).
>
> Hence the special maintainer script to create the required symlinks
> without relying on /bin/sh or any dynamically linked executable.

Yes I think this will necessarily require another round of debootstrap
changes once we've locked in on what we want to do, and go via the
various -p-u queues. I'm pretty sure some buildds will still be stuck
on Buster for example. I've done this last year and I'm happy to do it
again once we have a plan.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Raphael Hertzog]

On Fri, 09 Jun 2023, Marco d'Itri wrote:
> On Jun 08, Raphael Hertzog  wrote:
> 
> > In the same spirit, I'd like to throw an idea... could we decide that
> > base-files is the first package to be configured as part of the bootstrap
> > protocol and change base-files maintainer's scripts into statically linked
> > executables so that they can work even if we don't have the library loader
> > on the ABI-compliant path?
> It could be even easier: base-files could be unpacked once without 
> running the maintainer scripts and then "reinstalled" again later as 
> usual.

I think you are missing the point here, that only works if the package is
shipping the symlinks. And the idea is to not do this immediately because
it breaks debootstrap: if I understood correctly unpacking base-files
with the symlinks would fail if debootstrap had already pre-created those
symlinks (due to a -k option that we should get rid of in
/usr/share/debootstrap/scripts/debian-common).

Hence the special maintainer script to create the required symlinks
without relying on /bin/sh or any dynamically linked executable.

> > And creating the required symlinks would be done by those (standalone)
> > maintainer scripts...
> > 
> > I don't know if we already have some rule/invariant in the configuration
> > order of the unpacked packages, but I doubt so.
>
> Indeed, this would be very simple and it has already been proposed.
> But somebody then complained that special-casing a package would violate 
> the design contraints he self-imposed to his own image building tool, 
> and as we all know every Debian maintainer can veto any systemic changes 
> that they do not like.

That's not very helpful. Nobody has vetoed anything here. But I agree that
it would be cleaner if we could reach a situation where we can just unpack
all packages and have a working system where we can just "dpkg --configure
-a" and be done.

You don't care about this goal, it's fine, but it's not a reason to paint
this as a black/white picture. We can have both, we just need an
intermediate step.

I understand some would rather just be done with this transition (so am
I...), but going the extra mile here doesn't seem unreasonable. 

---

Coming back to my initial suggestion, I realize however that while the
maintainer script can run, dpkg itself will not run in the chroot so if
debootstrap is relying on dpkg to do the initial base-files configuration,
this will not work.

So this looks like that we will have to continue to rely on debootstrap
to create the symlinks and we will have to fix it so that it can properly
unpack a base-files containing /bin and /lib as symlinks on top of
existing symlinks.

And the actual switch to include /bin and /lib symlinks in base-files can
be done once we have fixed debootstrap in all relevant releases. And after
we can stop pre-creating those symlinks in debootstrap for all future
releases.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS


signature.asc
Description: PGP signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Marco d'Itri]

On Jun 08, Raphael Hertzog  wrote:

> In the same spirit, I'd like to throw an idea... could we decide that
> base-files is the first package to be configured as part of the bootstrap
> protocol and change base-files maintainer's scripts into statically linked
> executables so that they can work even if we don't have the library loader
> on the ABI-compliant path?
It could be even easier: base-files could be unpacked once without 
running the maintainer scripts and then "reinstalled" again later as 
usual.

> And creating the required symlinks would be done by those (standalone)
> maintainer scripts...
> 
> I don't know if we already have some rule/invariant in the configuration
> order of the unpacked packages, but I doubt so.
Indeed, this would be very simple and it has already been proposed.
But somebody then complained that special-casing a package would violate 
the design contraints he self-imposed to his own image building tool, 
and as we all know every Debian maintainer can veto any systemic changes 
that they do not like.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Thu, 8 Jun 2023 at 09:46, Raphael Hertzog  wrote:
>
> Hi,
>
> On Wed, 17 May 2023, Helmut Grohne wrote:
> > For completeness sake, there is one more entry in category 3: We can run
> > the dynamic loader from its canonical location explicitly, so we'd
> > modify maintainer scripts to start with:
> >
> > #!/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/sh
> >
> > This would only affect preinst scripts participating in bootstrap and
> > we'd not bother with changing PT_INTERP in the toolchain nor in
> > packages. Unfortunately, this completely breaks the DPKG_ROOT work and
> > with that, I see no viable entries in category 3 for moving forward.
> >
> > Moving on to category 4 feels rather obvious, especially because work
> > has been done there in debootstrap. The approach in debootstrap however
> > is one that I see as a dead end, because it causes us to maintain this
> > code multiple times. It's the number of derivatives times the number of
> > bootstrap tools and that doesn't scale.
> >
> > Category 4 is wider though and we also have other prior art at
> > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap. In particular,
> > making architecture bootstrap become chrootless would likely be a
> > generic solution to this and other problems.  However, any change needs
> > to propagate to a stable release in all bootstrapping tools.  Therefore
> > we cannot reasonably finish the transition before forky.  This makes
> > category 4 rather unattractive in a short term, but still worth pursuing
> > in a long term.
>
> In the same spirit, I'd like to throw an idea... could we decide that
> base-files is the first package to be configured as part of the bootstrap
> protocol and change base-files maintainer's scripts into statically linked
> executables so that they can work even if we don't have the library loader
> on the ABI-compliant path?
>
> And creating the required symlinks would be done by those (standalone)
> maintainer scripts...
>
> I don't know if we already have some rule/invariant in the configuration
> order of the unpacked packages, but I doubt so.
>
> > Having ruled out categories 3 and 4 maybe category 2 would be good?  We
> > could just ship those symlinks in base-files and be done, right?
> > Unfortunately, we pass -k to tar in debootstrap, so when it extracts
> > base-files and tries to unpack the bin -> usr/bin symlink, it sees that
> > oh no, there already is a symlink at bin (as debootstrap placed it
> > there) and thus fails. So in order to make this work, we also have to
> > modify debootstrap (and thus are in a combination of category 3 and 4).
>
> So when we will have fixed this, and waited for a release cycle, we can
> get rid of the statically compiled maintainer scripts and simply ship the
> symlinks in base-files.

Just a note that we can always change debootstrap via bookworm-p-u,
we've already done so in the past for this, so there's no requirement
to wait an extra release.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Raphael Hertzog]

Hi,

On Wed, 17 May 2023, Helmut Grohne wrote:
> For completeness sake, there is one more entry in category 3: We can run
> the dynamic loader from its canonical location explicitly, so we'd
> modify maintainer scripts to start with:
> 
> #!/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/sh
> 
> This would only affect preinst scripts participating in bootstrap and
> we'd not bother with changing PT_INTERP in the toolchain nor in
> packages. Unfortunately, this completely breaks the DPKG_ROOT work and
> with that, I see no viable entries in category 3 for moving forward.
> 
> Moving on to category 4 feels rather obvious, especially because work
> has been done there in debootstrap. The approach in debootstrap however
> is one that I see as a dead end, because it causes us to maintain this
> code multiple times. It's the number of derivatives times the number of
> bootstrap tools and that doesn't scale.
> 
> Category 4 is wider though and we also have other prior art at
> https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap. In particular,
> making architecture bootstrap become chrootless would likely be a
> generic solution to this and other problems.  However, any change needs
> to propagate to a stable release in all bootstrapping tools.  Therefore
> we cannot reasonably finish the transition before forky.  This makes
> category 4 rather unattractive in a short term, but still worth pursuing
> in a long term.

In the same spirit, I'd like to throw an idea... could we decide that
base-files is the first package to be configured as part of the bootstrap
protocol and change base-files maintainer's scripts into statically linked
executables so that they can work even if we don't have the library loader
on the ABI-compliant path?

And creating the required symlinks would be done by those (standalone)
maintainer scripts...

I don't know if we already have some rule/invariant in the configuration
order of the unpacked packages, but I doubt so.

> Having ruled out categories 3 and 4 maybe category 2 would be good?  We
> could just ship those symlinks in base-files and be done, right?
> Unfortunately, we pass -k to tar in debootstrap, so when it extracts
> base-files and tries to unpack the bin -> usr/bin symlink, it sees that
> oh no, there already is a symlink at bin (as debootstrap placed it
> there) and thus fails. So in order to make this work, we also have to
> modify debootstrap (and thus are in a combination of category 3 and 4).

So when we will have fixed this, and waited for a release cycle, we can
get rid of the statically compiled maintainer scripts and simply ship the
symlinks in base-files.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: DEP 17: Improve support for directory aliasing in dpkg

[Matthew Vernon]

On 26/05/2023 09:24, Luca Boccassi wrote:

On Fri, 26 May 2023 at 08:39, Matthew Vernon  wrote:



Consider: it is consistent to believe that it would have been better for
dpkg not to have had that warning added (quite some time ago now), but
that by now most derivatives that care will likely have patched it out
again (mitigating the harm); and if the current work on dpkg is allowed
to run its course then the warning will probably go away anyway.


That assumes all derivatives track unstable/testing and have taken
action, but it is possible for derivatives to track stable only, and
those would be broken.


I agree such distributions would be left with a confusing disagreement 
between the release notes "only /usg-merged systems are supported" and 
dpkg's warning. I agree this isn't ideal; but the release notes will 
mitigate the risk to such derivatives.


And as I said up-thread (and I'm trying not to repeat myself too much), 
I'm not sure why this is suddenly urgent so late in the release cycle, 
nor that we wouldn't be better off working on fixing the issues around 
dpkg and /usr-merge (which some people are currently doing).


Regards,

Matthew

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Fri, 26 May 2023 at 08:39, Matthew Vernon  wrote:
>
> Hi,
>
> On 26/05/2023 07:03, Ansgar wrote:
> > On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
> >> Ansgar  writes:
> >>> Debian going out of its way to tell derivative users to switch back from
> >>> merged-/usr to split-/usr is the *opposite* of trying to make things as
> >>> smooth for them as possible.
> >>
> >> Yes, I agree with that part and I think I objected to that at the time.
> >> Nonetheless, one bad decision doesn't mean that it is Debian policy that
> >> we don't care about derivatives or their users.  I think we made a mistake
> >> there which is not in alignment with our ideals or our goals.  We should
> >> try to reverse that mistake, not double down on it.
> >
> > My impression is that the tech-ctte disagrees on this point and would
> > not want to reverse the mistake, but double down on it (in your words).
>
> Your impression is incorrect. And assigning motivations to other parties
> during contentious discussions should be done with care if at all.
>
> Consider: it is consistent to believe that it would have been better for
> dpkg not to have had that warning added (quite some time ago now), but
> that by now most derivatives that care will likely have patched it out
> again (mitigating the harm); and if the current work on dpkg is allowed
> to run its course then the warning will probably go away anyway.

That assumes all derivatives track unstable/testing and have taken
action, but it is possible for derivatives to track stable only, and
those would be broken.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Ansgar]

On Fri, 2023-05-26 at 08:39 +0100, Matthew Vernon wrote:
> > So let me summarize Debian's "official" position as I understand it: we
> > do *NOT* care how dpkg's recommendations will break derivative
> > installations at all; if systems become unbootable, cause data loss,
> > ... now or in the future that is explicitly fine.
> 
> This is also unhelpful (and incorrect).

No, it is correct.

We allow boot-critical parts to refer to files using either the path in
/ or /usr; on systems following the recommendations from dpkg's warning
this might result in non-booting systems.

That is what we sign up to accept by having the warning in dpkg.

Ansgar

Re: DEP 17: Improve support for directory aliasing in dpkg

[Matthew Vernon]

Hi,

On 26/05/2023 07:03, Ansgar wrote:

On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:

Ansgar  writes:

Debian going out of its way to tell derivative users to switch back from
merged-/usr to split-/usr is the *opposite* of trying to make things as
smooth for them as possible.


Yes, I agree with that part and I think I objected to that at the time.
Nonetheless, one bad decision doesn't mean that it is Debian policy that
we don't care about derivatives or their users.  I think we made a mistake
there which is not in alignment with our ideals or our goals.  We should
try to reverse that mistake, not double down on it.


My impression is that the tech-ctte disagrees on this point and would
not want to reverse the mistake, but double down on it (in your words).


Your impression is incorrect. And assigning motivations to other parties 
during contentious discussions should be done with care if at all.


Consider: it is consistent to believe that it would have been better for 
dpkg not to have had that warning added (quite some time ago now), but 
that by now most derivatives that care will likely have patched it out 
again (mitigating the harm); and if the current work on dpkg is allowed 
to run its course then the warning will probably go away anyway.



Or rather my impression is that they would like to avoid any decision
on the dpkg mess situation. (Though not making a decision when asked is
of course also an explicit decision.)


There is currently a pile of ongoing work and discussion about 
/usr-merge and dpkg (in -devel at least). It seems to me that the right 
thing to do is to see how that work pans out, and let the people doing 
that work do so in peace.



So let me summarize Debian's "official" position as I understand it: we
do *NOT* care how dpkg's recommendations will break derivative
installations at all; if systems become unbootable, cause data loss,
... now or in the future that is explicitly fine.


This is also unhelpful (and incorrect). I do not think the case has been 
made that it is urgent that we remove (or revise) the warning from dpkg 
Right Now; if you want to attempt to do so, please do so without 
impugning those who disagree with you.


Regards,

Matthew

Re: DEP 17: Improve support for directory aliasing in dpkg

[Ansgar]

Hi Russ,

On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
> Ansgar  writes:
> > Debian going out of its way to tell derivative users to switch back from
> > merged-/usr to split-/usr is the *opposite* of trying to make things as
> > smooth for them as possible.
> 
> Yes, I agree with that part and I think I objected to that at the time.
> Nonetheless, one bad decision doesn't mean that it is Debian policy that
> we don't care about derivatives or their users.  I think we made a mistake
> there which is not in alignment with our ideals or our goals.  We should
> try to reverse that mistake, not double down on it.

My impression is that the tech-ctte disagrees on this point and would
not want to reverse the mistake, but double down on it (in your words).

Or rather my impression is that they would like to avoid any decision
on the dpkg mess situation. (Though not making a decision when asked is
of course also an explicit decision.)

So let me summarize Debian's "official" position as I understand it: we
do *NOT* care how dpkg's recommendations will break derivative
installations at all; if systems become unbootable, cause data loss,
... now or in the future that is explicitly fine.

Ansgar

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Fri, 19 May 2023 at 01:30, Simon Richter  wrote:
>
> Hi,
>
> On 5/18/23 18:08, Luca Boccassi wrote:
>
> >> Without it, leaving them in place makes no difference for usrmerged
> >> systems, and allows derived distributions that don't need usrmerge to
> >> continue using our packages.
>
> > Not quite. Having packages only ship files under /usr (and possibly
> > /etc) is very much a goal in itself for a lot of us.
>
> My point is: that is not an end goal, because it provides no real
> tangible benefit on its own.

For yourself. Again, for others, it is and it does.

> It does make sense in the context of building immutable images, which is
> *also* a bootstrapping problem, and probably worth being supported with
> proper tooling.
>
> >>- there is no guarantee that usrmerge will be permanent or the last
> >> transition of this kind
>
> > It is permanent, there are several upstream projects that will drop
> > support for legacy layouts very soon, and it will not be re-added
> > back.
>
> You are currently building a "legacy" system, it will just take a bit of
> time to reach that status. The less you anticipate future needs, the
> faster this will be.
>
> I understand that you are also a member of one of these upstream
> projects, and that you are taking the interests of this project to
> "pretty much the last relevant holdout" here. Has it occurred to you
> that you are also wearing a Debian hat, and you could be taking the
> interests of the Debian project to said upstream project?

Has it occurred to you that there might be a specific reason why said
upstream project has kept compatibility with legacy cruft that only
benefits Debian for so many years, at non-zero cost for developers,
despite everything else that's relevant having long since moved on? It
is really not difficult to find out what that reason might be, and
would have been better to do so before throwing around such
accusations. And if you can't find it, you can always go to one of the
numerous available channels and ask other maintainers. Why don't you
do that, ask how come support for Debian stable for this and many
other aspects is kept intact and who's behind that effort, and report
back the answer?

> >>- it also solves the bootstrap problem
>
> > It also is the least likely to succeed, and the most likely to cause
> > significant "social" upheavals.
>
> The only social problem I see is that you are trying to create a
> situation in which other people are compelled to do your work for you if
> they want it to be done properly.

No, the social problem is that there is one maintainer who is allowed
to ignore the TC and roadblock the entire distribution, so that
something that's been trivially and quickly done pretty much literally
everywhere else is instead made incredibly difficult and long-winded.
But despite that, we are getting there, slow and steady, and
remarkably well given the difficult circumstances outwith our control.

> So far, you have been throwing out "solutions", and left the analysis of
> the feasibility of those to other people, then, after three iterations,
> you demanded a full write-up of all existing use cases and blanket
> permission to ignore anything not brought up in this list.

I have literally no idea what you are talking about. I have
contributed what I can in my spare time, unblocking the transition
last year and helping out Helmut and others this year. This is not my
day job, by the way. What have you done to help, precisely?

> The thing is: I see more enthusiasm and self-directed problem solving
> skills from the interns at the company where I work, and at the same
> time you are one of the top contributors of the upstream project whose
> ideas of a "supported" configuration we are supposed to follow.

I seriously do not appreciate your tone, you are out of line. Please stop.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Simon Richter]

Hi,

On 5/18/23 18:08, Luca Boccassi wrote:


Without it, leaving them in place makes no difference for usrmerged
systems, and allows derived distributions that don't need usrmerge to
continue using our packages.



Not quite. Having packages only ship files under /usr (and possibly
/etc) is very much a goal in itself for a lot of us.


My point is: that is not an end goal, because it provides no real 
tangible benefit on its own.


It does make sense in the context of building immutable images, which is 
*also* a bootstrapping problem, and probably worth being supported with 
proper tooling.



   - there is no guarantee that usrmerge will be permanent or the last
transition of this kind



It is permanent, there are several upstream projects that will drop
support for legacy layouts very soon, and it will not be re-added
back.


You are currently building a "legacy" system, it will just take a bit of 
time to reach that status. The less you anticipate future needs, the 
faster this will be.


I understand that you are also a member of one of these upstream 
projects, and that you are taking the interests of this project to 
"pretty much the last relevant holdout" here. Has it occurred to you 
that you are also wearing a Debian hat, and you could be taking the 
interests of the Debian project to said upstream project?



   - it also solves the bootstrap problem



It also is the least likely to succeed, and the most likely to cause
significant "social" upheavals.


The only social problem I see is that you are trying to create a 
situation in which other people are compelled to do your work for you if 
they want it to be done properly.


So far, you have been throwing out "solutions", and left the analysis of 
the feasibility of those to other people, then, after three iterations, 
you demanded a full write-up of all existing use cases and blanket 
permission to ignore anything not brought up in this list.


The thing is: I see more enthusiasm and self-directed problem solving 
skills from the interns at the company where I work, and at the same 
time you are one of the top contributors of the upstream project whose 
ideas of a "supported" configuration we are supposed to follow.


   Simon

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Thu, 18 May 2023 at 07:39, Simon Richter  wrote:
>
> Hi,
>
> On 5/18/23 02:15, Sam Hartman wrote:
>
> >  Helmut> I think at this point, we have quite universal consensus
> >  Helmut> about the goal of moving files to their canonical location
> >  Helmut> (i.e. from / to /usr) as a solution to the aliasing problems
> >  Helmut> while we do not have consensus on precisely how to do this
> >  Helmut> (i.e. with changing dpkg or without). If you believe that
> >  Helmut> this is not consensus, please speak up.
>
> > I agree we have strong consensus that we want to move files to their
> > canonical locations.
>
> > I'm not entirely sure I'd agree that we have consensus that's our
> > solution to the aliasing problem.
>
> It's the other way around: moving the files as a solution to the
> aliasing problem is the strongest argument in favour of moving the files
> inside the packages.
>
> Without it, leaving them in place makes no difference for usrmerged
> systems, and allows derived distributions that don't need usrmerge to
> continue using our packages.

Not quite. Having packages only ship files under /usr (and possibly
/etc) is very much a goal in itself for a lot of us.

> > If for example we accomplish the move to canonical locations by changing
> > dpkg, we might well get some form of aliasing support in dpkg.
>
> IMO, that is still the preferred solution:
>
>   - it is actually safe, because dpkg knows what is going on and can
> reject conflicting changes
>   - there is no guarantee that usrmerge will be permanent or the last
> transition of this kind

It is permanent, there are several upstream projects that will drop
support for legacy layouts very soon, and it will not be re-added
back. This will become more and more common, as simply most will stop
caring and paying any attention to this detail. Debian is pretty much
the last relevant holdout here, and that's going to end in a couple of
weeks.

>   - it also solves the bootstrap problem

It also is the least likely to succeed, and the most likely to cause
significant "social" upheavals.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Simon Richter]

Hi,

On 5/18/23 02:15, Sam Hartman wrote:


 Helmut> I think at this point, we have quite universal consensus
 Helmut> about the goal of moving files to their canonical location
 Helmut> (i.e. from / to /usr) as a solution to the aliasing problems
 Helmut> while we do not have consensus on precisely how to do this
 Helmut> (i.e. with changing dpkg or without). If you believe that
 Helmut> this is not consensus, please speak up.



I agree we have strong consensus that we want to move files to their
canonical locations.



I'm not entirely sure I'd agree that we have consensus that's our
solution to the aliasing problem.


It's the other way around: moving the files as a solution to the 
aliasing problem is the strongest argument in favour of moving the files 
inside the packages.


Without it, leaving them in place makes no difference for usrmerged 
systems, and allows derived distributions that don't need usrmerge to 
continue using our packages.



If for example we accomplish the move to canonical locations by changing
dpkg, we might well get some form of aliasing support in dpkg.


IMO, that is still the preferred solution:

 - it is actually safe, because dpkg knows what is going on and can 
reject conflicting changes
 - there is no guarantee that usrmerge will be permanent or the last 
transition of this kind

 - it also solves the bootstrap problem

   Simon

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

Helmut> Moving on to category 4 feels rather obvious, especially
Helmut> because work has been done there in debootstrap. The
Helmut> approach in debootstrap however is one that I see as a dead
Helmut> end, because it causes us to maintain this code multiple
Helmut> times. It's the number of derivatives times the number of
Helmut> bootstrap tools and that doesn't scale.

Like others, I don't find this analysis compelling.
I'd like to better understand why the number of derivatives is in the
cross product.
I suspect most derivatives are going to move to merged /usr, and so I
suspect most if not all derivatives can be treated the same.

What am I missing?


signature.asc
Description: PGP signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

Helmut> I think at this point, we have quite universal consensus
Helmut> about the goal of moving files to their canonical location
Helmut> (i.e. from / to /usr) as a solution to the aliasing problems
Helmut> while we do not have consensus on precisely how to do this
Helmut> (i.e. with changing dpkg or without). If you believe that
Helmut> this is not consensus, please speak up.

I agree we have strong consensus that we want to move files to their
canonical locations.

I'm not entirely sure I'd agree that we have consensus that's our
solution to the aliasing problem.
there are other competing reasons why we might want to move files to
their canonical locations.

If for example we accomplish the move to canonical locations by changing
dpkg, we might well get some form of aliasing support in dpkg.

This mostly doesn't matter.
If we're going to move files to their canonical location, it doesn't
matter much why.
There is one potential area where it might.

If the reason for preferring one bootstrap protocol over another depends
on why we're moving files to their canonical locations, I think we'd
need to dig into that very carefully and examine that part of your
consensus call a bit more.


signature.asc
Description: PGP signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Wed, 17 May 2023 at 10:31, Helmut Grohne  wrote:
>
> Hi,
>
> This bootstrap aspect got me and I discussed this with a number of
> people and did some research.
>
> On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> > I don't think this is true? At least not in the broader sense: if you
> > compile something on Debian, it will obviously get linked against
> > libraries and dependencies as they are in Debian.
> > Perhaps what you mean is that, given an entire separate sysroot-like
> > tree, passing the appropriate compiler and linker flags and
> > environment variables, you can use the local compiler we ship to build
> > 'foreign' programs. That is true, but again it requires to set up the
> > environment appropriately, including linker flags. And the caller
> > needs to ensure the environment, including linker flags, is
> > appropriate for the target environment (I guess 'host' environment, in
> > GNU parlance). Therefore, I don't think it would be unreasonable to
> > require that if the target environment is split-usr, then the caller
> > also needs to specify an appropriate
> > '-Wl,--dynamic-linker=/lib/ld-whatever' option.
>
> Given the feedback, I am convinced that changing PT_INTERP is a stupid
> idea regardless of whether it is technically feasible. There must be a
> better way. Let's step back a bit.
>
> The underlying problem here is performing the initial filesystem
> bootstrap. The semantics of this are a bit vague as they are not spelled
> out in policy, so we will have to derive them from implementations.
>
> I think the major players are (in descending popularity):
>  * debootstrap
>  * mmdebstrap
>  * cdebootstrap
>  * multistrap
>
> multistrap predates mmdebstrap and when there was no mmdebstrap, I used
> it a lot. When attempting to test it, I totally couldn't convince it to
> bootstrap from an unsigned or locally signed repository.  The patch in
> #908451 didn't cut it. I also note that it creates a /lib64 -> /lib
> symbolic link which feels quite incompatible with merged-/usr.  For
> these reasons, I am dropping multistrap from the tools under
> consideration and recommend removing it from the archive. If you happen
> to use multistrap, now would be a good moment to tell me. Personally,
> all of my use cases of multistrap have been converted to mmdebstrap and
> that made a lot of things simpler.
>
> cdebootstrap vaguely works though unsigned operation seems dysfunctional
> as it runs apt-get update during cdebootstrap-helper-apt.postinst and
> that fails. I happen to not have figured out why and treat this failure
> as a success.
>
> So the most popular implementations quite evidently are debootstrap and
> mmdebstrap and both "just work". I note though that they work quite
> differently:
>  * debootstrap (depending on flags including --variant) pre-merges its
>chroot while mmdebstrap relies on packages doing it.
>
>I think that the question whether a distribution is merged is a
>property of the distribution and not the bootstrap tool, so I
>strongly recommend following mmdebstrap's view on this. The
>debootstrap way means that we have to include patches for every
>derivative, which is a process that does not scale well.
>
>  * mmdebstrap operates in two phases. It first unpacks and configures a
>rather minimal set of packages and then proceeds to adding packages
>passed to --include in a second phase once essential is fully
>configured while debootstrap immediately unpacks everything.
>
>I think the debootstrap approach is slightly worse here, because it
>means that preinst scripts of non-essential packages cannot rely on
>essential packages having been configured.
>
> In any case, we have to deal with both behaviours.
>
> After this little excursion into bootstrap technology, let's go back to
> the /usr-merge and its effects.
>
> I think at this point, we have quite universal consensus about the goal
> of moving files to their canonical location (i.e. from / to /usr) as a
> solution to the aliasing problems while we do not have consensus on
> precisely how to do this (i.e. with changing dpkg or without). If you
> believe that this is not consensus, please speak up.
>
> So in a distant future our packages will not contain any files in /bin
> or /lib. In particular, this affects /bin/sh and the dynamic loader,
> both of which are required to run maintainer scripts, which are
> currently required for creating the symbolic links. Boom.
>
> Solutions have been proposed to this and I think they all fall into one
> of the following four categories.
>
>  1. Don't move. We just keep those files that require a particular
> location (such as /bin/sh or the dynamic loader) in their
> non-canonical location. As such, maintainer scripts will be able to
> run and perform the conversion to symbolic links afterwards.
>
>  2. Move and ship links. Since we unpack all essential data.tar before
> running the first maintainer scr

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Marco d'Itri]

On May 17, Helmut Grohne  wrote:

> Given the feedback, I am convinced that changing PT_INTERP is a stupid
> idea regardless of whether it is technically feasible. There must be a
> better way. Let's step back a bit.
Me too, I was never persuaded.

>  4. Change the bootstrap protocol. In essence, this has been attempted
> in debootstrap by creating these symlinks prior to unpack, but no
> consensus has evolved around this approach yet. The category is
> wider though and generally requires changes to all bootstrapping
> tools.
I think that this is being dismissed too easily, mostly because the
mmdebstrap maintainer has been fighting it due to a philosophical 
preference.

> Moving on to category 4 feels rather obvious, especially because work
> has been done there in debootstrap. The approach in debootstrap however
> is one that I see as a dead end, because it causes us to maintain this
> code multiple times. It's the number of derivatives times the number of
> bootstrap tools and that doesn't scale.
But the code is trivial. It is currently more complex than it is needed 
in debootstrap only because initially it needed to support the biarch 
libc packages, but since nowadays they create the top level symlinks 
themselves then it can be made as simple as:

for dir in bin sbin lib; do
ln -s usr/"$dir" "$TARGET/$dir"
mkdir -p "$TARGET/usr/$dir"
done

Indeed, usrmerge does not have any architecture-specific knowledge 
anymore.

> https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap. In particular,
> making architecture bootstrap become chrootless would likely be a
> generic solution to this and other problems.  However, any change needs
> to propagate to a stable release in all bootstrapping tools.  Therefore
> we cannot reasonably finish the transition before forky.  This makes
Why not?

> So what's left is category 1. I looked into what the minimum set of
> files to be retained could be. To do that end, I moved everything and
> then reverted as much as was needed to make bootstrapping work.
>  * /lib64/ld-linux-x86-64.so.2 (hopefully obvious)
>  * /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (link target)
>  * /bin/sh (hopefully obvious)
>  * /bin/dash (link target)
Interesting approach. If it is so much simple, why are you not persuaded 
that it could be a good solution until the bootstrapping tools are 
updated?

>  * /bin/bash (usrmerge runs ldd, which is a #!/bin/bash script)
>  * /bin/more (update-alternatives doesn't like its absence)
>  * /bin/cp (unless usrmerge stops hard coding its path)
These can be easily fixed, maybe the ldd issue too.

> The other major takeaway is that a significant
> chunk of the problems mentioned in this mail cannot be fixed by
> modifying dpkg only.
Agreed.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[G. Branden Robinson]

At 2023-05-17T11:30:36+0200, Helmut Grohne wrote:
> This bootstrap aspect got me and I discussed this with a number of
> people and did some research.

I'd like to nominate you for a Russ Allbery Award for the most useful
post to the thread.  Your attention to concrete, empirical details,
arising necessarily from practical experimentation rather than pure
cogitation, made the nature of the problems here clearly comprehensible
to me.  And if I was befogged, I'll wager other people were too.

Regards,
Branden


signature.asc
Description: PGP signature

booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Helmut Grohne]

Hi,

This bootstrap aspect got me and I discussed this with a number of
people and did some research.

On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> I don't think this is true? At least not in the broader sense: if you
> compile something on Debian, it will obviously get linked against
> libraries and dependencies as they are in Debian.
> Perhaps what you mean is that, given an entire separate sysroot-like
> tree, passing the appropriate compiler and linker flags and
> environment variables, you can use the local compiler we ship to build
> 'foreign' programs. That is true, but again it requires to set up the
> environment appropriately, including linker flags. And the caller
> needs to ensure the environment, including linker flags, is
> appropriate for the target environment (I guess 'host' environment, in
> GNU parlance). Therefore, I don't think it would be unreasonable to
> require that if the target environment is split-usr, then the caller
> also needs to specify an appropriate
> '-Wl,--dynamic-linker=/lib/ld-whatever' option.

Given the feedback, I am convinced that changing PT_INTERP is a stupid
idea regardless of whether it is technically feasible. There must be a
better way. Let's step back a bit.

The underlying problem here is performing the initial filesystem
bootstrap. The semantics of this are a bit vague as they are not spelled
out in policy, so we will have to derive them from implementations.

I think the major players are (in descending popularity):
 * debootstrap
 * mmdebstrap
 * cdebootstrap
 * multistrap

multistrap predates mmdebstrap and when there was no mmdebstrap, I used
it a lot. When attempting to test it, I totally couldn't convince it to
bootstrap from an unsigned or locally signed repository.  The patch in
#908451 didn't cut it. I also note that it creates a /lib64 -> /lib
symbolic link which feels quite incompatible with merged-/usr.  For
these reasons, I am dropping multistrap from the tools under
consideration and recommend removing it from the archive. If you happen
to use multistrap, now would be a good moment to tell me. Personally,
all of my use cases of multistrap have been converted to mmdebstrap and
that made a lot of things simpler.

cdebootstrap vaguely works though unsigned operation seems dysfunctional
as it runs apt-get update during cdebootstrap-helper-apt.postinst and
that fails. I happen to not have figured out why and treat this failure
as a success.

So the most popular implementations quite evidently are debootstrap and
mmdebstrap and both "just work". I note though that they work quite
differently:
 * debootstrap (depending on flags including --variant) pre-merges its
   chroot while mmdebstrap relies on packages doing it.

   I think that the question whether a distribution is merged is a
   property of the distribution and not the bootstrap tool, so I
   strongly recommend following mmdebstrap's view on this. The
   debootstrap way means that we have to include patches for every
   derivative, which is a process that does not scale well.

 * mmdebstrap operates in two phases. It first unpacks and configures a
   rather minimal set of packages and then proceeds to adding packages
   passed to --include in a second phase once essential is fully
   configured while debootstrap immediately unpacks everything.

   I think the debootstrap approach is slightly worse here, because it
   means that preinst scripts of non-essential packages cannot rely on
   essential packages having been configured.

In any case, we have to deal with both behaviours.

After this little excursion into bootstrap technology, let's go back to
the /usr-merge and its effects.

I think at this point, we have quite universal consensus about the goal
of moving files to their canonical location (i.e. from / to /usr) as a
solution to the aliasing problems while we do not have consensus on
precisely how to do this (i.e. with changing dpkg or without). If you
believe that this is not consensus, please speak up.

So in a distant future our packages will not contain any files in /bin
or /lib. In particular, this affects /bin/sh and the dynamic loader,
both of which are required to run maintainer scripts, which are
currently required for creating the symbolic links. Boom.

Solutions have been proposed to this and I think they all fall into one
of the following four categories.

 1. Don't move. We just keep those files that require a particular
location (such as /bin/sh or the dynamic loader) in their
non-canonical location. As such, maintainer scripts will be able to
run and perform the conversion to symbolic links afterwards.

 2. Move and ship links. Since we unpack all essential data.tar before
running the first maintainer script, having one package contain the
compatibility symlinks is enough to fix the problem.

 3. Move and avoid using non-canonical locations. This is the approach
where we write maintainer scripts as #!/usr/bin/sh and consider

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 13 May 2023 at 14:59, RL  wrote:
>
> Luca Boccassi  writes:
>
> > I think documentation is fundamental for dealing with local
> > changes. When I say that I don't think we should worry about strange
> > local-only changes I mean exactly as you said, that I don't think we
> > should start shipping complicated code that tries to deal with people
> > diverting /sbin/init to /var/lolcat or whatever. Clear documentation
> > that tells "if you did X look at Y and do Z" is the bare minimum I'd
> > think.
>
> Hi,
>
> I have been trying to follow this discussion because I agree with the
> above, and we have been trying to get exactly this added to bookworm's
> release notes.
>
> Regardless of what happens in the future, users need to know what might
> break when the upgrade from a non-usr-merged system to bookworm. So far
> there isnt really anything clear out there for users that i can find
> that clearly lists the problems, but (as far as i can follow), the same
> issues in this thread may cause issues for people with more ususual
> setups.
>
> the merge request is here:
>   https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/155/diffs
>
> (We are trying to give users a full list while also making it clear that
> most people wont have an issue on the upgrade)
>
> Would be great to confirm we've captured everything and given the right
> advice (to the extent that this is possible) on avoiding issues.

Thanks, looks good, left a couple of comments on the MR.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[RL]

Luca Boccassi  writes:

> I think documentation is fundamental for dealing with local
> changes. When I say that I don't think we should worry about strange
> local-only changes I mean exactly as you said, that I don't think we
> should start shipping complicated code that tries to deal with people
> diverting /sbin/init to /var/lolcat or whatever. Clear documentation
> that tells "if you did X look at Y and do Z" is the bare minimum I'd
> think.

Hi,

I have been trying to follow this discussion because I agree with the
above, and we have been trying to get exactly this added to bookworm's
release notes.

Regardless of what happens in the future, users need to know what might
break when the upgrade from a non-usr-merged system to bookworm. So far
there isnt really anything clear out there for users that i can find
that clearly lists the problems, but (as far as i can follow), the same
issues in this thread may cause issues for people with more ususual
setups.

the merge request is here:
  https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/155/diffs

(We are trying to give users a full list while also making it clear that
most people wont have an issue on the upgrade)

Would be great to confirm we've captured everything and given the right
advice (to the extent that this is possible) on avoiding issues.


Thanks
Richard

Bug#1035904: dpkg currently warning about merged-usr systems (revisited) (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Ansgar]

Package: tech-ctte
X-Debbugs-Cc: Russ Allbery , Sean Whitton 
, Helmut Grohne , Luca Boccassi 
, debian-d...@lists.debian.org, debian-devel@lists.debian.org

On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
> Ansgar  writes:
> > Debian going out of its way to tell derivative users to switch back from
> > merged-/usr to split-/usr is the *opposite* of trying to make things as
> > smooth for them as possible.
> 
> Yes, I agree with that part and I think I objected to that at the time.
> Nonetheless, one bad decision doesn't mean that it is Debian policy that
> we don't care about derivatives or their users.  I think we made a mistake
> there which is not in alignment with our ideals or our goals.  We should
> try to reverse that mistake, not double down on it.

Cool, then let's ask tech-ctte.

Dear ctte, please consider overruling the dpkg maintainer to include
the patch from #994388[1].

Thanks,
Ansgar

  [1]: https://bugs.debian.org/994388#397

Re: DEP 17: Improve support for directory aliasing in dpkg

[Russ Allbery]

Ansgar  writes:
> On Wed, 2023-05-10 at 13:50 -0700, Russ Allbery wrote:

>> Caring about them isn't the same thing as doing everything they want. 
>> We can both try to make things as smooth for them as possible and still
>> make design decisions about Debian that they may disagree with or that
>> may make some property they want to maintain difficult or impossible. 
>> It's the sort of decision we have to make on a case-by-case basis.

> Debian going out of its way to tell derivative users to switch back from
> merged-/usr to split-/usr is the *opposite* of trying to make things as
> smooth for them as possible.

Yes, I agree with that part and I think I objected to that at the time.
Nonetheless, one bad decision doesn't mean that it is Debian policy that
we don't care about derivatives or their users.  I think we made a mistake
there which is not in alignment with our ideals or our goals.  We should
try to reverse that mistake, not double down on it.

-- 
Russ Allbery (r...@debian.org)  

Re: DEP 17: Improve support for directory aliasing in dpkg

[Ansgar]

Hi Russ,

On Wed, 2023-05-10 at 13:50 -0700, Russ Allbery wrote:
> Ansgar  writes:
> > As far as I understand, we do explicitly *not* care about our
> > derivatives with regard to merged-/usr as some packages in Debian
> > recommend users to move *away* from merged-/usr to split-/usr on
> > derivatives, i.e., to an unsupported fs layout.
> 
> Caring about them isn't the same thing as doing everything they want.  We
> can both try to make things as smooth for them as possible and still make
> design decisions about Debian that they may disagree with or that may make
> some property they want to maintain difficult or impossible.  It's the
> sort of decision we have to make on a case-by-case basis.

Debian going out of its way to tell derivative users to switch back
from merged-/usr to split-/usr is the *opposite* of trying to make
things as smooth for them as possible.

I asked the ctte to consider not telling derivative users to revert
from merged-/usr and was told me that "we [ctte] would not consider
this [change] to be in line with our existing decisions" (informally).

I take that as explicitly not caring that we break derivative users'
systems.

Ansgar

Re: DEP 17: Improve support for directory aliasing in dpkg

[Russ Allbery]

Ansgar  writes:

> So why do we allow changes that require listing all reverse dependencies
> in Breaks then? This is known to be wrong for all non- listed packages,
> e.g., all local/vendor/derivative-specific packages.

Because it's a balance; we don't want to stop making changes, and never
making a backward-compatible change doesn't work, so we do the best we can
with the tools we have.  However, if someone with an out-of-Debian package
tells us that a change breaks it, historically we did add them to Breaks.
We just don't have a good way of discovering this.

> As far as I understand, we do explicitly *not* care about our
> derivatives with regard to merged-/usr as some packages in Debian
> recommend users to move *away* from merged-/usr to split-/usr on
> derivatives, i.e., to an unsupported fs layout.

Caring about them isn't the same thing as doing everything they want.  We
can both try to make things as smooth for them as possible and still make
design decisions about Debian that they may disagree with or that may make
some property they want to maintain difficult or impossible.  It's the
sort of decision we have to make on a case-by-case basis.

-- 
Russ Allbery (r...@debian.org)  

Re: DEP 17: Improve support for directory aliasing in dpkg

[Ansgar]

On Wed, 2023-05-10 at 08:35 -0700, Sean Whitton wrote:
> On Sun 07 May 2023 at 11:14AM +02, Ansgar wrote:
> > Debian's dependency system requires to explicitly declare
> > Depends/Conflicts/Replaces/Breaks, but for obvious reasons we
> > cannot do
> > that for packages outside Debian's ecosystem.
> > 
> > The same is true for diversions/alternatives/* or anything else
> > requiring coordination among all users: the dpkg ecosystem has too
> > many
> > practical limitations to support non-Debian packages on anything
> > but a
> > "it might work" basis (which is usually "good enough").  (This is
> > even
> > true for packages within the Debian ecosystem, especially when one
> > considers partially implemented features like multi-arch.)
> 
> I don't think this is the consensus view.

So why do we allow changes that require listing all reverse
dependencies in Breaks then? This is known to be wrong for all non-
listed packages, e.g., all local/vendor/derivative-specific packages.

> Our derivatives are among our users, for example, and we care about
> them being able to add packages in appropriate ways.

As far as I understand, we do explicitly *not* care about our
derivatives with regard to merged-/usr as some packages in Debian
recommend users to move *away* from merged-/usr to split-/usr on
derivatives, i.e., to an unsupported fs layout.

AFAIR the ctte felt that doing so on derivatives is fine for packages
in Debian (w/o an explicit formal ruling).

Ansgar

Re: DEP 17: Improve support for directory aliasing in dpkg

[Sean Whitton]

Hello,

On Tue 09 May 2023 at 02:07AM +01, Luca Boccassi wrote:

> But again, I do think we need to try and define what it is that we
> want to support here. If we are serious about it, then we should
> codify it, and hold any future changes to the same standards, wherever
> they may come from. If we are not willing to do this, then I have to
> ask why.

If we can come to some specific conclusions about what we are and are
not going to support that have consensus, as part of this work, we can
and should write those down in Policy.  There's probably too much
complexity to achieve something more general than that, however.

It is completely reasonable, as you wrote in another message, to want
this transition to be held to the same standards as others.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Sean Whitton]

Hello,

On Sun 07 May 2023 at 11:14AM +02, Ansgar wrote:

> Debian's dependency system requires to explicitly declare
> Depends/Conflicts/Replaces/Breaks, but for obvious reasons we cannot do
> that for packages outside Debian's ecosystem.
>
> The same is true for diversions/alternatives/* or anything else
> requiring coordination among all users: the dpkg ecosystem has too many
> practical limitations to support non-Debian packages on anything but a
> "it might work" basis (which is usually "good enough").  (This is even
> true for packages within the Debian ecosystem, especially when one
> considers partially implemented features like multi-arch.)

I don't think this is the consensus view.

Our derivatives are among our users, for example, and we care about them
being able to add packages in appropriate ways.

Our policy documents and best practices contain various provisions with
user's own packages in mind.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Tue, 9 May 2023 at 05:12, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Tue, May 09, 2023 at 01:56:53AM +0100, Luca Boccassi wrote:
> > On Mon, 8 May 2023 at 19:06, Sean Whitton  wrote:
> > > It's designed to stop as-yet-unknown problems happening, too.
> >
> > Well, sure, but we've been at this for years, any such problems should
> > really be known by now. This is with Bookworm as it stands of course,
> > when we go in and make more changes then we obviously have to be
> > careful, but that's the entire reason this thread exists and is still
> > going on.
>
> This actually feels rather worrying to me. On one hand, you say that
> problems should be know. On the other hand, you proposed a simple
> transition with quite a number of problems that you apparently didn't
> see coming. Even relatively simple mechanisms, such as just repacking
> all the .debs to ship files in their canonical location and then trying
> to install them, revealed a dpkg unpack error in zutils. This
> combination of claiming that problems should be known while at the same
> time apparently not knowing them makes me uneasy to move forward here.
>
> So while I want to see the moratorium lifted, it all makes a lot more
> sense to me given what we've seen in this thread. The worst of outcomes
> I see here is the one where we cause problems that don't have a good
> solution as any way forward would break someone's use case (with
> someone's use case often being smooth upgrades in one way or another).
> It's those where we cannot move forward nor revert.

No need to get worried! With "Bookworm as it stands" I mean that
literally. What we ship in Bookworm has been stable for years, so
while something new could be lurking somewhere, it seems vanishingly
unlikely at this stage.

Once we start changing things in Trixie, in whichever way we decide,
that's of course all new and all bets are off.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Luca,

On Tue, May 09, 2023 at 01:56:53AM +0100, Luca Boccassi wrote:
> On Mon, 8 May 2023 at 19:06, Sean Whitton  wrote:
> > It's designed to stop as-yet-unknown problems happening, too.
> 
> Well, sure, but we've been at this for years, any such problems should
> really be known by now. This is with Bookworm as it stands of course,
> when we go in and make more changes then we obviously have to be
> careful, but that's the entire reason this thread exists and is still
> going on.

This actually feels rather worrying to me. On one hand, you say that
problems should be know. On the other hand, you proposed a simple
transition with quite a number of problems that you apparently didn't
see coming. Even relatively simple mechanisms, such as just repacking
all the .debs to ship files in their canonical location and then trying
to install them, revealed a dpkg unpack error in zutils. This
combination of claiming that problems should be known while at the same
time apparently not knowing them makes me uneasy to move forward here.

So while I want to see the moratorium lifted, it all makes a lot more
sense to me given what we've seen in this thread. The worst of outcomes
I see here is the one where we cause problems that don't have a good
solution as any way forward would break someone's use case (with
someone's use case often being smooth upgrades in one way or another).
It's those where we cannot move forward nor revert.

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 21:09, Russ Allbery  wrote:
>
> Sam Hartman  writes:
>
> > As someone who has been following this, I support the work Helmut and
> > Simon Richter have been doing.
>
> > I have more confidence in that view than the one Luca is proposing.
> > I also support Shawn's interpretation that being conservative here is
> > good.
>
> > I think even with my support we have no consensus.  However hopefully we
> > can get a few more people who have been reading the whole thread to
> > chime in and a consensus will appear.
>
> I've also been following this.
>
> I appreciate Luca's questioning of the necessity of parts of the approach
> and looking for simpler solutions; I think that's valuable feedback, and
> we should avoid assuming that every conceivable edge case is supported in
> Debian.  There are unsupported edge cases in Debian already and likely
> always will be because distributions are complex.
>
> That said, I find Helmut and Simon's analysis to be more persuasive so
> far.  I do think we should try to find a fairly robust solution, because
> the feature we're trying to support here (smooth upgrades) is a core and
> defining feature of what makes Debian Debian.  That doesn't mean we need
> to support literally anything someone might have done; that won't be
> possible.  But I think there are going to be enough unanticipated problems
> that we should try to cover the anticipated problems, and that includes at
> least the relatively obvious or known outside-of-Debian uses of things
> like diversions.
>
> I would like to stay open to addressing some of those problems via
> documentation or explicit upgrade instructions where that makes sense.  If
> we have places where there's a choice between writing extremely tricky and
> complicated code versus providing people with simple instructions for how
> to locate problematic diversions on their system, remove them before the
> upgrade, and then put them back afterwards (or accomplish their goal in
> some other way), we should consider taking the documentation approach
> instead.  But that still requires being able to enumerate at least the
> most likely problems and understand them.
>
> For example, if local system administrators have been deactivating systemd
> units by diverting them, at first glance I think it would be better to
> clearly tell them that they should stop doing this and instead use
> masking rather than writing code to try to ensure this continues working.

Obviously agree with your last point, I think documentation is
fundamental for dealing with local changes. When I say that I don't
think we should worry about strange local-only changes I mean exactly
as you said, that I don't think we should start shipping complicated
code that tries to deal with people diverting /sbin/init to
/var/lolcat or whatever. Clear documentation that tells "if you did X
look at Y and do Z" is the bare minimum I'd think.

But again, I do think we need to try and define what it is that we
want to support here. If we are serious about it, then we should
codify it, and hold any future changes to the same standards, wherever
they may come from. If we are not willing to do this, then I have to
ask why.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 19:06, Sean Whitton  wrote:
>
> Hello,
>
> On Sun 07 May 2023 at 12:03PM +01, Luca Boccassi wrote:
>
> > Sure, this is in the context of the ongoing discussion in the TC about
> > revising their side of the advice.
>
> I think it's highly unlikely that we revise it rather than just reissue
> it, at the present time, because too many details are unsettled.
>
> > Also, we shouldn't lose sight of the reason why this was issued in the
> > first place: it is designed to stop a problem from happening, and that
> > problem can only happen when both conditions are true. I can't read
> > minds obviously, but I imagine that's the reason the RT advice was
> > worded as it was.
>
> It's designed to stop as-yet-unknown problems happening, too.

Well, sure, but we've been at this for years, any such problems should
really be known by now. This is with Bookworm as it stands of course,
when we go in and make more changes then we obviously have to be
careful, but that's the entire reason this thread exists and is still
going on.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 09:58, Helmut Grohne  wrote:
>
> On Mon, May 08, 2023 at 02:07:08AM +0100, Luca Boccassi wrote:
> > I can see we don't agree on this matter, of course, that is clear. And
> > I hope we can find common ground. But let me provocatively ask this
> > first: is the same rule going to be enforced for all other changes
> > that happen in the project that might affect external packages? If
> > anybody points out past changes, recent or less recent, that caused
> > issues for third party packages, will the TC ask for those changes to
> > be reverted or otherwise modified accordingly? Will a change to Policy
> > be proposed that spells out that third party packages cannot ever be
> > broken, no matter what they do, and must always work?
>
> I'm not sure about the TC's role in this. For the record, I am doing all
> of the analysis (and design work) in this thread without a TC hat. I
> also cannot comment on what the TC is going to rule this matter. Can we
> leave that aside or formally file it there if you see a need?

Sure, it was just to keep it impersonal. Substitute TC with
"interested party" or so.

> I agree that what we support is vague at best and we can readily see
> from earlier conflicts that this is a recurring matter. We still
> disagree over how much maintainers should support sysvinit. I've also
> quite recently failed at properly preparing a transition (non-essential
> adduser) and while we could write about it in release-notes, what is
> going to happen is that we'll revert it for bookworm and then I can
> retry properly.
>
> You may also have noticed that my analysis of possible problems in this
> thread very much reasons about packages shipped in Debian releases. I
> would actually like to call external packages and local diversions
> unsupported, but I was rightfully criticised that this is falling short.
>
> So no, I cannot tell you where the boundary of our support is. I
> initially assumed it to be closer to where you paint it and am now
> trying to adapt to meet the expectations of others.
>
> For instance, I've also reached out to DSA and inquired on their use.
> While I haven't found local diversions or local statoverrides in
> dsa-puppet.git, it seems that a number of external packages ship files
> in /sbin or /lib (including udev rules and systemd units).

I think the question of what do we want to support is an important
one, and I care greatly that for this particular endeavour we do not
impose a higher burden on ourselves that would otherwise be expected
in different situations. If expectations are shifting considerably, we
should codify it, so that everyone is held up to the same standards.

> > The more pre-depends, the more constraints we put on apt. I do not
> > have a specific scenario in mind as we don't even have a full set of
> > changes to look at, but it seems clear to me it will have _some_
> > effect, no?
>
> We've been there with multiarch-support and my experience with that
> suggests that the primary effect is increasing the size of Packages
> files. Though given that you are obviously worried here, I suppose more
> research is warranted.

If that's the only thing to worry about, then I'm not worried at all!

> > Sure that's a legitimate concern, however, wouldn't it fall into the
> > "needs special handling" bucket? It is a case where the file is moving
> > both in location and package, so it is covered by the blank statement
> > "either don't do that or implement the required workaround via
> > diversion/conflict/etc". What am I missing?
>
> You are missing the distribution of responsibility. Quite commonly,
> backports are performed by someone else than the package maintainer.
> Yet, an uncoordinated backport can now render the package in unstable
> rc-buggy.

Well, sure, but once again any backport changes can break in
interesting and novel ways. The simplest of them all is getting the
version wrong so that updates do not work... and yet we have very
little if anything in the way to stop that, no? You can just ignore
Lintian screaming at you and upload...

To me that's just another case to be solved with good documentation
and communication.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

Will get to the rest later tonight, two quick points:

On Mon, 8 May 2023 at 09:58, Helmut Grohne  wrote:
> > But the more I think about it, the more I am convinced that the
> > default option working best for Debian is the one that matches the
> > project's choice of a filesystem layout. After all, this is
> > configurable in the toolchain for a reason.
> > And the vast majority of the rest of the world has long since finished
> > this transition, so I struggle to think where software built with this
> > default wouldn't work. Bullseye will be oldoldstable at that point,
> > and even that was default merged for new installations, and really old
> > ones (oldoldoldoldstable at that point? I lost count) will be long
> > EOL. I suppose they could still be around unmaintained, but who uses a
> > toolchain from 8 years in the future to build software for an EOL
> > distribution 8 years in the past? Normally it's the other way around,
> > as even glibc adds new symbols and is not forward compatible.
>
> This seems somewhat convincing to me. Would you reach out to toolchain
> maintainers to discuss this as an early change after the release of
> bookworm?

Have done so now via the gcc mailing list.

> > On the ELF interpreter, as long as we can reasonably ensure it works,
> > I do believe we should switch it, regardless of what we do with the
> > symlinks, how we ship/add/build/package/create/manage them, as a
> > desired final state. Again, we should make the default in Debian work
> > for Debian. And given the default for Debian from Bookworm onward is
> > that the loader is in /usr/lib/, it seems perfectly reasonable to me
> > that it software built for Debian and shipped in Debian should look
> > there for it.
>
> I suppose that we've been confusing the different approaches here. The
> question of what links base-files should contain mostly arises if you
> start from the assumption that we do not modify the ELF interpreter
> location. Once changing its (and /bin/sh's) location, the question of
> how to install those symlinks can indeed be done in base-files.postinst
> or at some other place where dpkg doesn't have to know much about it
> indeed. Would you agree to examine the approach where we don't modify
> the ELF interpreter location in parallel as a backup plan?

Yeah we definitely should do that. I think we should separate a bit
long-term vs short-term on that front, as it will help reach a
conclusion more quickly. I think that aspect is easy to revise, and
shouldn't lock us in a particular position one way or the other.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Russ Allbery]

Sam Hartman  writes:

> As someone who has been following this, I support the work Helmut and
> Simon Richter have been doing.

> I have more confidence in that view than the one Luca is proposing.
> I also support Shawn's interpretation that being conservative here is
> good.

> I think even with my support we have no consensus.  However hopefully we
> can get a few more people who have been reading the whole thread to
> chime in and a consensus will appear.

I've also been following this.

I appreciate Luca's questioning of the necessity of parts of the approach
and looking for simpler solutions; I think that's valuable feedback, and
we should avoid assuming that every conceivable edge case is supported in
Debian.  There are unsupported edge cases in Debian already and likely
always will be because distributions are complex.

That said, I find Helmut and Simon's analysis to be more persuasive so
far.  I do think we should try to find a fairly robust solution, because
the feature we're trying to support here (smooth upgrades) is a core and
defining feature of what makes Debian Debian.  That doesn't mean we need
to support literally anything someone might have done; that won't be
possible.  But I think there are going to be enough unanticipated problems
that we should try to cover the anticipated problems, and that includes at
least the relatively obvious or known outside-of-Debian uses of things
like diversions.

I would like to stay open to addressing some of those problems via
documentation or explicit upgrade instructions where that makes sense.  If
we have places where there's a choice between writing extremely tricky and
complicated code versus providing people with simple instructions for how
to locate problematic diversions on their system, remove them before the
upgrade, and then put them back afterwards (or accomplish their goal in
some other way), we should consider taking the documentation approach
instead.  But that still requires being able to enumerate at least the
most likely problems and understand them.

For example, if local system administrators have been deactivating systemd
units by diverting them, at first glance I think it would be better to
clearly tell them that they should stop doing this and instead use
masking rather than writing code to try to ensure this continues working.

-- 
Russ Allbery (r...@debian.org)  

Re: DEP 17: Improve support for directory aliasing in dpkg

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

Helmut> Hi Luca,
Helmut> On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
>> The local/external aspect is already covered in Ansgar's reply
>> and subthread.

Helmut> I hope that we can at least agree that we don't have
Helmut> consensus on this view. And the more I think about it, the
Helmut> more it becomes clear to me that this non-consensus is part
Helmut> of the larger disagreement we have about this whole
Helmut> transition. Do you see any way towards getting to common
Helmut> ground here?

As someone who has been following this, I support the work Helmut and
Simon Richter have been doing.
I have more confidence in that view than the one Luca is proposing.
I also support Shawn's interpretation that being conservative here is
good.

I think even with my support we have no consensus.  However hopefully we
can get a few more people who have been reading the whole thread to
chime in and a consensus will appear.


signature.asc
Description: PGP signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Sean Whitton]

Hello,

On Sun 07 May 2023 at 12:03PM +01, Luca Boccassi wrote:

> Sure, this is in the context of the ongoing discussion in the TC about
> revising their side of the advice.

I think it's highly unlikely that we revise it rather than just reissue
it, at the present time, because too many details are unsettled.

> Also, we shouldn't lose sight of the reason why this was issued in the
> first place: it is designed to stop a problem from happening, and that
> problem can only happen when both conditions are true. I can't read
> minds obviously, but I imagine that's the reason the RT advice was
> worded as it was.

It's designed to stop as-yet-unknown problems happening, too.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 5/8/23 20:38, Luca Boccassi wrote:

[local diversions]


Sure, they are supported in the sense that they can be enabled, and
then you get to keep the pieces.


They are supported in the sense that someone actually added an explicit 
flag for dpkg-divert for specifically this feature and documented it in 
the manual page as an interface.


Maintainer scripts don't need to work around the admin installing 
arbitrary incompatible tools, because we generally expect admins to know 
what they are doing -- however requiring admins to perform multiple 
diversion registrations to have them count is a change of the interface.


The /bin/sh diversion is a bit ...special. This should have been an 
alternative, but cannot be because /bin/sh needs to be functional after 
unpacking Essential packages, so maintainer scripts work during bootstrap.


There is an interesting use case here as well, bootstrapping a foreign 
architecture, where we only unpack the Essential set, divert /sbin/init 
to /sbin/init.real and place a shell script as /sbin/init that completes 
the installation once the resulting tree is mounted as a root filesystem 
on the target platform. It has been a while since I used that, in 
principle this should even still work with systemd.


   Simon

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 03:57, Simon Richter  wrote:
>
> Hi,
>
> On 5/7/23 18:14, Ansgar wrote:
>
> > Is there any specific reason why specifically diversions are a problem
> > where "it might work" is not sufficient? That is, why should we divert
> > from the usual standard for dealing with packages outside the Debian
> > ecosystem here?
>
> Locally created diversions are a supported feature, and the only way for
> admins to replace single files in a way that is safe for installing updates.
>
> Even within Debian, it is not sufficient to just coordinate uploads of
> packages that divert each others' files, because the new diversion needs
> to be in place before a newly-canonicalized package is unpacked, a
> Breaks relationship does not enforce that ordering, and while a
> Conflicts without a Replaces does, this adds a lot of constraints to the
> solver.

Sure, they are supported in the sense that they can be enabled, and
then you get to keep the pieces. We ship thousands of maintainer
scripts, and I have never seen one that takes into account completely
arbitrary and random possible local diversion, apart from dash for
/bin/sh (and we are about to nuke most of it!), when
moving/adjusting/fixing and whatnot. Do you have any such
counter-example in mind?

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

On Mon, May 08, 2023 at 02:07:08AM +0100, Luca Boccassi wrote:
> I can see we don't agree on this matter, of course, that is clear. And
> I hope we can find common ground. But let me provocatively ask this
> first: is the same rule going to be enforced for all other changes
> that happen in the project that might affect external packages? If
> anybody points out past changes, recent or less recent, that caused
> issues for third party packages, will the TC ask for those changes to
> be reverted or otherwise modified accordingly? Will a change to Policy
> be proposed that spells out that third party packages cannot ever be
> broken, no matter what they do, and must always work?

I'm not sure about the TC's role in this. For the record, I am doing all
of the analysis (and design work) in this thread without a TC hat. I
also cannot comment on what the TC is going to rule this matter. Can we
leave that aside or formally file it there if you see a need?

I agree that what we support is vague at best and we can readily see
from earlier conflicts that this is a recurring matter. We still
disagree over how much maintainers should support sysvinit. I've also
quite recently failed at properly preparing a transition (non-essential
adduser) and while we could write about it in release-notes, what is
going to happen is that we'll revert it for bookworm and then I can
retry properly.

You may also have noticed that my analysis of possible problems in this
thread very much reasons about packages shipped in Debian releases. I
would actually like to call external packages and local diversions
unsupported, but I was rightfully criticised that this is falling short.

So no, I cannot tell you where the boundary of our support is. I
initially assumed it to be closer to where you paint it and am now
trying to adapt to meet the expectations of others.

For instance, I've also reached out to DSA and inquired on their use.
While I haven't found local diversions or local statoverrides in
dsa-puppet.git, it seems that a number of external packages ship files
in /sbin or /lib (including udev rules and systemd units).

> The more pre-depends, the more constraints we put on apt. I do not
> have a specific scenario in mind as we don't even have a full set of
> changes to look at, but it seems clear to me it will have _some_
> effect, no?

We've been there with multiarch-support and my experience with that
suggests that the primary effect is increasing the size of Packages
files. Though given that you are obviously worried here, I suppose more
research is warranted.

> Sure that's a legitimate concern, however, wouldn't it fall into the
> "needs special handling" bucket? It is a case where the file is moving
> both in location and package, so it is covered by the blank statement
> "either don't do that or implement the required workaround via
> diversion/conflict/etc". What am I missing?

You are missing the distribution of responsibility. Quite commonly,
backports are performed by someone else than the package maintainer.
Yet, an uncoordinated backport can now render the package in unstable
rc-buggy.

> But the more I think about it, the more I am convinced that the
> default option working best for Debian is the one that matches the
> project's choice of a filesystem layout. After all, this is
> configurable in the toolchain for a reason.
> And the vast majority of the rest of the world has long since finished
> this transition, so I struggle to think where software built with this
> default wouldn't work. Bullseye will be oldoldstable at that point,
> and even that was default merged for new installations, and really old
> ones (oldoldoldoldstable at that point? I lost count) will be long
> EOL. I suppose they could still be around unmaintained, but who uses a
> toolchain from 8 years in the future to build software for an EOL
> distribution 8 years in the past? Normally it's the other way around,
> as even glibc adds new symbols and is not forward compatible.

This seems somewhat convincing to me. Would you reach out to toolchain
maintainers to discuss this as an early change after the release of
bookworm?

> On the ELF interpreter, as long as we can reasonably ensure it works,
> I do believe we should switch it, regardless of what we do with the
> symlinks, how we ship/add/build/package/create/manage them, as a
> desired final state. Again, we should make the default in Debian work
> for Debian. And given the default for Debian from Bookworm onward is
> that the loader is in /usr/lib/, it seems perfectly reasonable to me
> that it software built for Debian and shipped in Debian should look
> there for it.

I suppose that we've been confusing the different approaches here. The
question of what links base-files should contain mostly arises if you
start from the assumption that we do not modify the ELF interpreter
location. Once changing its (and /bin/sh's) location, the question of
how to install those symlinks can indeed be done in b

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 5/7/23 18:14, Ansgar wrote:


Is there any specific reason why specifically diversions are a problem
where "it might work" is not sufficient? That is, why should we divert
from the usual standard for dealing with packages outside the Debian
ecosystem here?


Locally created diversions are a supported feature, and the only way for 
admins to replace single files in a way that is safe for installing updates.


Even within Debian, it is not sufficient to just coordinate uploads of 
packages that divert each others' files, because the new diversion needs 
to be in place before a newly-canonicalized package is unpacked, a 
Breaks relationship does not enforce that ordering, and while a 
Conflicts without a Replaces does, this adds a lot of constraints to the 
solver.


   Simon

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 22:10, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> > The local/external aspect is already covered in Ansgar's reply and 
> > subthread.
>
> I hope that we can at least agree that we don't have consensus on this
> view. And the more I think about it, the more it becomes clear to me
> that this non-consensus is part of the larger disagreement we have about
> this whole transition. Do you see any way towards getting to common
> ground here?

I can see we don't agree on this matter, of course, that is clear. And
I hope we can find common ground. But let me provocatively ask this
first: is the same rule going to be enforced for all other changes
that happen in the project that might affect external packages? If
anybody points out past changes, recent or less recent, that caused
issues for third party packages, will the TC ask for those changes to
be reverted or otherwise modified accordingly? Will a change to Policy
be proposed that spells out that third party packages cannot ever be
broken, no matter what they do, and must always work?

> > Sure, but adding changes that are (seemingly) unnecessary for a large
> > percentage of affected packages also brings uncertainty. Every
> > software has bugs, thus it follows that injecting more software in the
> > way of a package being installed will likely also inject bugs. Which
> > doesn't mean we shouldn't consider it, however, it should be weighted
> > appropriately.
>
> Let me put this into perspective. In this scenario, we will have a few
> packages with versioned Pre-Depends on usr-is-merged. The seemingly
> unnecessary change here is adding more Pre-Depends of the same kind to
> many more packages. It seems very likely to me that one of the few
> Pre-Depends will cause usr-is-merged to be upgraded early and thus those
> possibly unnecessary Pre-Dependencies will be harmless. Do you actually
> have some scenario in mind that would warrant judging this as risky
> beyond suspicion? (Which is not to say that there is no risk as the
> whole affair bears quite some risk.)

The more pre-depends, the more constraints we put on apt. I do not
have a specific scenario in mind as we don't even have a full set of
changes to look at, but it seems clear to me it will have _some_
effect, no?

> > Packages that need special handling will need special handling for
> > backporting too. This is nothing new, there was never a project-wide
> > guarantee that a package uploaded to testing can apply 1:1 to
> > backports, it is common enough to require changes/reverts/adjustments,
> > and if it's fine to require that in other cases, it's fine for this
> > case too.
>
> It seems that you missed my argument and it likely wasn't spelled out
> explicitly enough, so let me retry. Yes, you may need to adapt packages
> that are being backported. We don't disagree about that (and hope people
> get it right, which they won't, but so be it). The really bad thing here
> is that a backports upload may require changes to the package in
> unstable!
>
> Say we packaged foo version 1 in stable and it puts everything in /bin.
> Then we update foo to version 2 in unstable and foo gains a new
> /bin/bar. Due to the debhelper addon, this is actually shipped as
> /usr/bin/bar. Great. Then we backport foo version 2 to stable. Given
> that debhelper no longer moves, it'll be /bin/bar. Then we notice that
> foo is not laid out nicely and we split a bar package from it in version
> 3 and move /usr/bin/bar into bar. Now a user may install stable, install
> foo version 1, install the foo version 2 backport and then update to
> nextstable. In that stable upgrade, bar version 3 may be unpacked before
> foo version 3 and as a result /usr/bin/bar goes missing when the
> backported foo version 2 gets upgraded to the regular foo version 3 as
> this deletes /bin/bar.
>
> So when we backport a package, the unstable package may need to be
> modified to avoid such unpack file loss scenarios. In a simple case, we
> may be able to just add Conflicts, but the takeaway is that backporting
> a package may now break upgrades to nextstable in a way that requires
> fixes in nextstable to accommodate for such upgrades.

Sure that's a legitimate concern, however, wouldn't it fall into the
"needs special handling" bucket? It is a case where the file is moving
both in location and package, so it is covered by the blank statement
"either don't do that or implement the required workaround via
diversion/conflict/etc". What am I missing?

> > If the majority of packages are simply converted, with no manual
> > handling and no diversion, then it should be simple to handle: the
> > debhelper in stable will not perform the conversion by definition as
> > the logic won't be present, and any dh upload to backports will have
> > such logic disabled, so that other packages that get uploaded to
> > backports and built with either the stable or the backports debhelpe

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Luca,

On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> The local/external aspect is already covered in Ansgar's reply and subthread.

I hope that we can at least agree that we don't have consensus on this
view. And the more I think about it, the more it becomes clear to me
that this non-consensus is part of the larger disagreement we have about
this whole transition. Do you see any way towards getting to common
ground here?

> Sure, but adding changes that are (seemingly) unnecessary for a large
> percentage of affected packages also brings uncertainty. Every
> software has bugs, thus it follows that injecting more software in the
> way of a package being installed will likely also inject bugs. Which
> doesn't mean we shouldn't consider it, however, it should be weighted
> appropriately.

Let me put this into perspective. In this scenario, we will have a few
packages with versioned Pre-Depends on usr-is-merged. The seemingly
unnecessary change here is adding more Pre-Depends of the same kind to
many more packages. It seems very likely to me that one of the few
Pre-Depends will cause usr-is-merged to be upgraded early and thus those
possibly unnecessary Pre-Dependencies will be harmless. Do you actually
have some scenario in mind that would warrant judging this as risky
beyond suspicion? (Which is not to say that there is no risk as the
whole affair bears quite some risk.)

> Packages that need special handling will need special handling for
> backporting too. This is nothing new, there was never a project-wide
> guarantee that a package uploaded to testing can apply 1:1 to
> backports, it is common enough to require changes/reverts/adjustments,
> and if it's fine to require that in other cases, it's fine for this
> case too.

It seems that you missed my argument and it likely wasn't spelled out
explicitly enough, so let me retry. Yes, you may need to adapt packages
that are being backported. We don't disagree about that (and hope people
get it right, which they won't, but so be it). The really bad thing here
is that a backports upload may require changes to the package in
unstable!

Say we packaged foo version 1 in stable and it puts everything in /bin.
Then we update foo to version 2 in unstable and foo gains a new
/bin/bar. Due to the debhelper addon, this is actually shipped as
/usr/bin/bar. Great. Then we backport foo version 2 to stable. Given
that debhelper no longer moves, it'll be /bin/bar. Then we notice that
foo is not laid out nicely and we split a bar package from it in version
3 and move /usr/bin/bar into bar. Now a user may install stable, install
foo version 1, install the foo version 2 backport and then update to
nextstable. In that stable upgrade, bar version 3 may be unpacked before
foo version 3 and as a result /usr/bin/bar goes missing when the
backported foo version 2 gets upgraded to the regular foo version 3 as
this deletes /bin/bar.

So when we backport a package, the unstable package may need to be
modified to avoid such unpack file loss scenarios. In a simple case, we
may be able to just add Conflicts, but the takeaway is that backporting
a package may now break upgrades to nextstable in a way that requires
fixes in nextstable to accommodate for such upgrades.

> If the majority of packages are simply converted, with no manual
> handling and no diversion, then it should be simple to handle: the
> debhelper in stable will not perform the conversion by definition as
> the logic won't be present, and any dh upload to backports will have
> such logic disabled, so that other packages that get uploaded to
> backports and built with either the stable or the backports debhelper
> won't have any change performed on them.

As much a I'd like to trust you on things actually being simple, we've
seen over and over again that the simple approaches have non-trivial
flaws. If you were to highlight resulting problems (and propose
solutions), that would be more convincing to me than continuously
labeling it simple.

> Or to put it in another way: I think our defaults should prioritize
> the Debian native use case. Given we ship our loader in /usr/lib/ld*
> now, it makes sense to me that the default in GCC is to point to
> /usr/lib/ld*. Callers can override that as needed for
> third-party/external/foreign use cases.

I guess you'll be having a hard time convincing the toolchain
maintainers of this change, but my other point was that this is
unnecessary when we can use patchelf after the fact.

> > How about the long-term vision of this? Elsewhere you indicated that
> > you'd like the aliasing symlinks to not be shipped by any data.tar. Does
> > that imply that we'd keep patching the interpreter and using /usr/bin/sh
> > forever in the essential set? If adding the links to base-files, it
> > would be of temporary nature only.
> >
> > If adding the symlinks to base-files, how about /lib64? Would we ship it
> > for all architectures or just for those that need it (e.g. amd64,
> > l

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 12:51, Luca Boccassi  wrote:
>
> On Sun, 7 May 2023 at 06:55, Helmut Grohne  wrote:
> >
> > Hi Luca,
> >
> > On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
> > > Sure, there are some things that need special handling, as you have
> > > pointed out. What I meant is that I don't think we need special
> > > handling for _all_ affected packages. AFAIK nothing is using
> > > diversions for unit files or udev rules, for example (I mean if any
> > > package is, please point it out, because I would like a word...). I
> >
> > I've posted a list in
> > https://lists.debian.org/20230428080516.ga203...@subdivi.de and indeed,
> > udev rules are being diverted in one case.
>
> *fetching sledgehammer*

Filed https://bugs.debian.org/1035667 with extreme prejudice, and MR
up to fix it too.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 06:55, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
> > Sure, there are some things that need special handling, as you have
> > pointed out. What I meant is that I don't think we need special
> > handling for _all_ affected packages. AFAIK nothing is using
> > diversions for unit files or udev rules, for example (I mean if any
> > package is, please point it out, because I would like a word...). I
>
> I've posted a list in
> https://lists.debian.org/20230428080516.ga203...@subdivi.de and indeed,
> udev rules are being diverted in one case.

*fetching sledgehammer*

> But then, you only capture diversions inside Debian's ecosystem and miss
> out on other kinds of diversions such as local diversions. We currently
> support imposing local diversions on pretty much arbitrary files
> including unit files. And while we've occasionally moved files between /
> and /usr before the transition, doing it for 2000 packages significantly
> grows the risk of it affecting someone. So really, we want all such
> diversions duplicated before unpacking a package the has moved its
> files. The way to achieve that is Pre-Depending on usr-is-merged. To me,
> this sounds like we really want some special handling for all affected
> packages.

The local/external aspect is already covered in Ansgar's reply and subthread.

> I also caution that we've started from a very simple approach and tried
> fixing it up to address the problems that we recognized in analyzing it.
> My impression is that we are not finished with our discovery here and
> won't be for quite some time. This uncertainty of what else we might
> break is the most significant downside I see with your approach.

Sure, but adding changes that are (seemingly) unnecessary for a large
percentage of affected packages also brings uncertainty. Every
software has bugs, thus it follows that injecting more software in the
way of a package being installed will likely also inject bugs. Which
doesn't mean we shouldn't consider it, however, it should be weighted
appropriately.

> > very strongly suspect this will be a small minority out of the total
> > of ~2k would need this treatment, and the vast majority would not. Do
> > you disagree with this gut feeling?
>
> I do disagree indeed. While the special handling may be mostly
> mechanical for the majority of packages, it still see it as required.
>
> Worse, we also need to discuss how this affects backporting of packages.
> Any package enabling the addon needs to have the addon removed for a
> backport to undo the move. Worse, when backporting debhelper, any
> package that uses the new compat level must explicitly disable the
> addon. And then we may need to fix upgrade paths from backports to
> stable.

Packages that need special handling will need special handling for
backporting too. This is nothing new, there was never a project-wide
guarantee that a package uploaded to testing can apply 1:1 to
backports, it is common enough to require changes/reverts/adjustments,
and if it's fine to require that in other cases, it's fine for this
case too.
If the majority of packages are simply converted, with no manual
handling and no diversion, then it should be simple to handle: the
debhelper in stable will not perform the conversion by definition as
the logic won't be present, and any dh upload to backports will have
such logic disabled, so that other packages that get uploaded to
backports and built with either the stable or the backports debhelper
won't have any change performed on them.

> > Curve ball question: is there anything blocking us from canonicalizing
> > PT_INTERP at the source in gcc so that the essential set (and
> > everything else, really) can work without the lib -> usr/lib symlink?
> > Would anything obvious break?
> > This would be in trixie, so even when unpacked on bookworm for the
> > upgrade case, the loader is guaranteed to be accessible from the
> > canonical path. Heck, we could even consider canonicalizing shebangs
> > in scripts shipped in essential packages? I'd imagine /bin/sh would
> > have the same issue as the loader?
>
> Heh! As much as this initially sounds insanely crazy,

I know, right? And yet...

> I was unable to
> spot a reason for why this cannot work at all. It definitely sounds like
> fixing the bootstrap protocol to me. On the flip side, I'd assume the
> loader location (for the purpose of examining ELF objects) to be hard
> coded in quite some places.  Changing gcc seems out of question, because
> we also expect binaries built on Debian to run on non-Debian systems.

I don't think this is true? At least not in the broader sense: if you
compile something on Debian, it will obviously get linked against
libraries and dependencies as they are in Debian.
Perhaps what you mean is that, given an entire separate sysroot-like
tree, passing the appropriate compiler and linker flags and
environment variables, you can use the l

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 00:44, Sean Whitton  wrote:
>
> Hello,
>
> On Sat 06 May 2023 at 09:47PM +01, Luca Boccassi wrote:
>
> > On Sat, 6 May 2023 at 19:51, Helmut Grohne  wrote:
> >>
> >> > - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
> >> > packages at the same time is maintained from bookworm till trixie, and
> >> > will lifted after trixie ships, and applies implicitly to all the
> >> > ~2000 binary pkgs that are affected by the above
> >>
> >> While the CTTE placed the moratorium until the release of bookworm, the
> >> release team extended it beyond, see
> >> https://lists.debian.org/e1ocdqk-0005ge...@respighi.debian.org. So you
> >> need explicit agreement from the release team on your plan. Failing
> >> that, any package that has been forcefully moved is immediately
> >> rc-buggy due to failing a release team requirement.
> >
> > Of course the release team needs to be on board, no questions about
> > that. But given the idea is to maintain their decision exactly as it
> > stands I wouldn't imagine it would be an issue? Once again, the
> > moratorium is explicitly about moving between locations _and_
> > packages, in combination, not either/or. From that same email you
> > linked:
> >
> > "Files moving their canonical location between / and /usr (details in
> > [1]) *and* from one binary package to another binary package within
> > one release cycle remain an RC issue unless dpkg supports it."
> >
> > I'm proposing to keep this in place as a general rule, with the new
> > escape hatch that you devised as the only addition.
>
> Actually, the morotorium is not explicitly only about moving between
> both locations and packages.  Firstly, the TC's morotorium does not have
> the qualification, restricting *any* movements in data.tar.*:
>
> The Technical Committee recommends that during the Debian 12
> development cycle, the maintainers of individual packages should not
> proactively move files from the root filesystem to the corresponding
> locations in /usr in the data.tar.* of packages. Files that were in
> /usr in the Debian 11 release should remain in /usr, while files
> that were in /bin, /lib* or /sbin in the Debian 11 release should
> remain in those directories.  If files were moved from /bin, /lib*
> or /sbin into /usr since the Debian 11 release, they should be moved
> back to their Debian 11 locations.
>
> Then secondly, the RT's message is ambiguous, because it says both that
> they want the /TC's/ morotorium to remain in place, and also that files
> moving between both locations and packages is an RC issue.
>
> Until the RT's position is clarified, I think we should treat the
> broader prohibition as what they require.  The TC are going to discuss
> this issue at our meeting on Tuesday, and one possible outcome is that
> we reissue our version of the broader prohibition.

Sure, this is in the context of the ongoing discussion in the TC about
revising their side of the advice. Also, we shouldn't lose sight of
the reason why this was issued in the first place: it is designed to
stop a problem from happening, and that problem can only happen when
both conditions are true. I can't read minds obviously, but I imagine
that's the reason the RT advice was worded as it was.

> Stepping back:
>
> I am far from being an expert on the details of merged-/usr.  But one
> thing I've noticed is that among the people who have spent the most time
> looking into it, the majority think that simple fixes are not going to
> be sufficient.  Only a few people who have spent a lot of time on it
> still think that the fixes that are required are relatively simple ones.
>
> If the former group are wrong then the transition takes longer than it
> needs to, but we don't lose any confidence in the state of our users'
> installations.
>
> If the latter group are wrong then we'll probably end up with a longer
> transition anyway, and a worse situation for both our maintainers and
> our users.  And it's within one of the areas of Debian that we're most
> proud of -- completely smooth upgrades between stable releases.
>
> So, I think we should assume the people who are more worried are right,
> for the time being.  We lose little in doing so.

Pondering about things is all well and good, however, I think the
'worry' needs to be realistic (eg: a conflict due to /bin/foo and
/usr/bin/foo both existing is not something that needs to cause any
worrying, as it is already disallowed by policy and unsupported), and
congruent with usual practices that are applied for all other changes
and processes. IE, the appropriate weighting should be given to
potential problems spotted.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 10:14, Ansgar  wrote:
>
> Hi,
>
> On Sun, 2023-05-07 at 07:50 +0200, Helmut Grohne wrote:
> > But then, you only capture diversions inside Debian's ecosystem
>
> It's unreasonable to support stuff outside Debian's ecosystem: even
> basic dependency relations do not work for this.
>
> Debian's dependency system requires to explicitly declare
> Depends/Conflicts/Replaces/Breaks, but for obvious reasons we cannot do
> that for packages outside Debian's ecosystem.
>
> The same is true for diversions/alternatives/* or anything else
> requiring coordination among all users: the dpkg ecosystem has too many
> practical limitations to support non-Debian packages on anything but a
> "it might work" basis (which is usually "good enough").  (This is even
> true for packages within the Debian ecosystem, especially when one
> considers partially implemented features like multi-arch.)
>
> Is there any specific reason why specifically diversions are a problem
> where "it might work" is not sufficient? That is, why should we divert
> from the usual standard for dealing with packages outside the Debian
> ecosystem here?
>
> > I also caution that we've started from a very simple approach and tried
> > fixing it up to address the problems that we recognized in analyzing it.
> > My impression is that we are not finished with our discovery here and
> > won't be for quite some time.
>
> Well, we find limitations in dpkg that we in all other contexts usually
> ignore. If we used similar expectations in other cases, we would need
> to very much restrict when Breaks/Conflicts/Replaces might be used at
> all: it's totally unrealistic to list all (possibly local) packages
> that ship conflicting files, possibly only created by maintainer
> scripts. Or to explicitly list all reverse dependencies that might be
> broken by a particular change. We also would not have multi-arch yet as
> the dependency system doesn't support it fully (some of which is
> already known, but probably discovery isn't finished yet).
>
> (Of course in some cases explicitly listing reverse dependencies can be
> avoided: just always introduce something like
>
> Provides: ${foo}-compat (= 1)
>
> for *all* dependencies and forbid `>=` in `Depends`; this allows to
> stop providing that in cases where one would have to declare explicit
> `Breaks` before. But only the direct provider can use this, so it's
> already too limited... Alternatively forbid *all* changes that would
> require this, i.e., require stable interfaces. However we do not do
> this.)
>
> But for all these issues we just say "meh, you are out of luck".

Indeed, if we don't worry about local random changes or random third
party packages beyond documenting what needs attention, we shouldn't
worry about them for this either. As already mentioned lots of third
party packages don't even use Debian's toolchain to build packages, so
there's nothing that we can do anyway.

Kind regards, Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Ansgar]

Hi,

On Sun, 2023-05-07 at 07:50 +0200, Helmut Grohne wrote:
> But then, you only capture diversions inside Debian's ecosystem

It's unreasonable to support stuff outside Debian's ecosystem: even
basic dependency relations do not work for this.

Debian's dependency system requires to explicitly declare
Depends/Conflicts/Replaces/Breaks, but for obvious reasons we cannot do
that for packages outside Debian's ecosystem.

The same is true for diversions/alternatives/* or anything else
requiring coordination among all users: the dpkg ecosystem has too many
practical limitations to support non-Debian packages on anything but a
"it might work" basis (which is usually "good enough").  (This is even
true for packages within the Debian ecosystem, especially when one
considers partially implemented features like multi-arch.)

Is there any specific reason why specifically diversions are a problem
where "it might work" is not sufficient? That is, why should we divert
from the usual standard for dealing with packages outside the Debian
ecosystem here?

> I also caution that we've started from a very simple approach and tried
> fixing it up to address the problems that we recognized in analyzing it.
> My impression is that we are not finished with our discovery here and
> won't be for quite some time.

Well, we find limitations in dpkg that we in all other contexts usually
ignore. If we used similar expectations in other cases, we would need
to very much restrict when Breaks/Conflicts/Replaces might be used at
all: it's totally unrealistic to list all (possibly local) packages
that ship conflicting files, possibly only created by maintainer
scripts. Or to explicitly list all reverse dependencies that might be
broken by a particular change. We also would not have multi-arch yet as
the dependency system doesn't support it fully (some of which is
already known, but probably discovery isn't finished yet).

(Of course in some cases explicitly listing reverse dependencies can be
avoided: just always introduce something like

Provides: ${foo}-compat (= 1)

for *all* dependencies and forbid `>=` in `Depends`; this allows to
stop providing that in cases where one would have to declare explicit
`Breaks` before. But only the direct provider can use this, so it's
already too limited... Alternatively forbid *all* changes that would
require this, i.e., require stable interfaces. However we do not do
this.)

But for all these issues we just say "meh, you are out of luck".

Ansgar

Re: DEP 17: Improve support for directory aliasing in dpkg

[Ansgar]

Hi,

On Sun, 2023-05-07 at 07:50 +0200, Helmut Grohne wrote:
> But then, you only capture diversions inside Debian's ecosystem and miss
> out on other kinds of diversions such as local diversions. We currently
> support imposing local diversions on pretty much arbitrary files
> including unit files.

No, we do not really support diversions. Once you are divert a file, it
is luck whether this works or not.

Once you divert files, maintainer scripts and other parts would have to
be aware of this and chose whether to use the original file included in
the file or the diverted file.

Not handling diversions can lead to files disappearing, data loss or
other breakage, but it's very rare a package considers this.

Ansgar

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Luca,

On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
> Sure, there are some things that need special handling, as you have
> pointed out. What I meant is that I don't think we need special
> handling for _all_ affected packages. AFAIK nothing is using
> diversions for unit files or udev rules, for example (I mean if any
> package is, please point it out, because I would like a word...). I

I've posted a list in
https://lists.debian.org/20230428080516.ga203...@subdivi.de and indeed,
udev rules are being diverted in one case.

But then, you only capture diversions inside Debian's ecosystem and miss
out on other kinds of diversions such as local diversions. We currently
support imposing local diversions on pretty much arbitrary files
including unit files. And while we've occasionally moved files between /
and /usr before the transition, doing it for 2000 packages significantly
grows the risk of it affecting someone. So really, we want all such
diversions duplicated before unpacking a package the has moved its
files. The way to achieve that is Pre-Depending on usr-is-merged. To me,
this sounds like we really want some special handling for all affected
packages.

I also caution that we've started from a very simple approach and tried
fixing it up to address the problems that we recognized in analyzing it.
My impression is that we are not finished with our discovery here and
won't be for quite some time. This uncertainty of what else we might
break is the most significant downside I see with your approach.

> very strongly suspect this will be a small minority out of the total
> of ~2k would need this treatment, and the vast majority would not. Do
> you disagree with this gut feeling?

I do disagree indeed. While the special handling may be mostly
mechanical for the majority of packages, it still see it as required.

Worse, we also need to discuss how this affects backporting of packages.
Any package enabling the addon needs to have the addon removed for a
backport to undo the move. Worse, when backporting debhelper, any
package that uses the new compat level must explicitly disable the
addon. And then we may need to fix upgrade paths from backports to
stable.

> Of course, it goes without saying, we should check this before going
> forward in any direction.

The more I try, the more I have the impression that we enumerate the
ways this can go wrong and the more we poke, the more we find.

> Of course the release team needs to be on board, no questions about
> that. But given the idea is to maintain their decision exactly as it
> stands I wouldn't imagine it would be an issue? Once again, the
> moratorium is explicitly about moving between locations _and_
> packages, in combination, not either/or. From that same email you
> linked:

This is evidently ambiguous as RT also reference the CTTE moratorium,
which includes

"Files that were in /usr in the Debian 11 release should remain in /usr,
while files that were in /bin, /lib* or /sbin in the Debian 11 release
should remain in those directories."

Quite evidently, clarification is needed.

> There are already distro-wide upgrade piuparts checks run occasionally
> IIRC, at least I've seen a bug from one being reported this week, so
> we should be most of the way there already?

I examined the piuparts check in
https://lists.debian.org/20230425190728.ga1471...@subdivi.de already, so
no, not at all.

> To be clear, this would be very nice and welcome to have obviously,
> but I don't think it needs to be a blocker. We don't have such checks

Actually, getting the service seems to be the least of our problems.
It's fairly simple to implement and I have written a PoC style
implementation for parts of it already as part of my analysis.

> for vast parts of Policy, including moving files without
> Breaks+Replaces as evidenced by the recent MBF, and yet we managed to
> survive thus far. I don't think it's fair that this workstream should
> be held to higher standards than the rest of the project.

Given the expected breakage and its latency ahead, I think minimizing
risk is prudent.

> > 4. Add canonicalization support to debhelper as an (internal) addon.
> >Enable this addon in the next compat level. This will again populate
> >${misc:Pre-Depends} with "usr-is-merged (>= fixed version)" as needed.
> >Note that usrmerge's provides is unversioned and doesn't satisfy such
> >a dependency.
> 
> As already mentioned, I do not believe this is necessary for _all_
> cases. It is necessary for a certain number (that we should ascertain
> beforehand!) of cases, and we need the machinery implemented for them,
> but I don't think we should impose this workflow with pre-depends and
> diversions for all affected packages. I think it should be mandatory
> for problematic packages such as those you already pointed out, _and_
> for cases where the maintainer wants to move files also between binary
> packages.

Given local diversions, I am now convinced 

Re: DEP 17: Improve support for directory aliasing in dpkg

[Sean Whitton]

Hello,

On Sat 06 May 2023 at 09:47PM +01, Luca Boccassi wrote:

> On Sat, 6 May 2023 at 19:51, Helmut Grohne  wrote:
>>
>> > - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
>> > packages at the same time is maintained from bookworm till trixie, and
>> > will lifted after trixie ships, and applies implicitly to all the
>> > ~2000 binary pkgs that are affected by the above
>>
>> While the CTTE placed the moratorium until the release of bookworm, the
>> release team extended it beyond, see
>> https://lists.debian.org/e1ocdqk-0005ge...@respighi.debian.org. So you
>> need explicit agreement from the release team on your plan. Failing
>> that, any package that has been forcefully moved is immediately
>> rc-buggy due to failing a release team requirement.
>
> Of course the release team needs to be on board, no questions about
> that. But given the idea is to maintain their decision exactly as it
> stands I wouldn't imagine it would be an issue? Once again, the
> moratorium is explicitly about moving between locations _and_
> packages, in combination, not either/or. From that same email you
> linked:
>
> "Files moving their canonical location between / and /usr (details in
> [1]) *and* from one binary package to another binary package within
> one release cycle remain an RC issue unless dpkg supports it."
>
> I'm proposing to keep this in place as a general rule, with the new
> escape hatch that you devised as the only addition.

Actually, the morotorium is not explicitly only about moving between
both locations and packages.  Firstly, the TC's morotorium does not have
the qualification, restricting *any* movements in data.tar.*:

The Technical Committee recommends that during the Debian 12
development cycle, the maintainers of individual packages should not
proactively move files from the root filesystem to the corresponding
locations in /usr in the data.tar.* of packages. Files that were in
/usr in the Debian 11 release should remain in /usr, while files
that were in /bin, /lib* or /sbin in the Debian 11 release should
remain in those directories.  If files were moved from /bin, /lib*
or /sbin into /usr since the Debian 11 release, they should be moved
back to their Debian 11 locations.

Then secondly, the RT's message is ambiguous, because it says both that
they want the /TC's/ morotorium to remain in place, and also that files
moving between both locations and packages is an RC issue.

Until the RT's position is clarified, I think we should treat the
broader prohibition as what they require.  The TC are going to discuss
this issue at our meeting on Tuesday, and one possible outcome is that
we reissue our version of the broader prohibition.

Stepping back:

I am far from being an expert on the details of merged-/usr.  But one
thing I've noticed is that among the people who have spent the most time
looking into it, the majority think that simple fixes are not going to
be sufficient.  Only a few people who have spent a lot of time on it
still think that the fixes that are required are relatively simple ones.

If the former group are wrong then the transition takes longer than it
needs to, but we don't lose any confidence in the state of our users'
installations.

If the latter group are wrong then we'll probably end up with a longer
transition anyway, and a worse situation for both our maintainers and
our users.  And it's within one of the areas of Debian that we're most
proud of -- completely smooth upgrades between stable releases.

So, I think we should assume the people who are more worried are right,
for the time being.  We lose little in doing so.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 19:51, Helmut Grohne  wrote:
>
> Hi Luca,
>
> I fear you are still missing too many relevant details. What sounds like
> a simple plan is severely broken if carried out in this simple form. It
> disappoints me that you continue to present it as this simple given the
> earlier evidence.
>
> On Fri, May 05, 2023 at 11:11:54PM +0100, Luca Boccassi wrote:
> > - every package is forcefully canonicalized as soon as trixie is open
> > for business
>
> I gave you scripts to prototype this (via binary patching) and
> demonstrated that this immediately makes a fair number of packages rc
> buggy due to causing unpack errors. I used zutils as an example. It
> really is not that simple and it absolutely cannot be done forcefully,
> but must be done in an opt-in way. I won't repeat the rationale here.

Sure, there are some things that need special handling, as you have
pointed out. What I meant is that I don't think we need special
handling for _all_ affected packages. AFAIK nothing is using
diversions for unit files or udev rules, for example (I mean if any
package is, please point it out, because I would like a word...). I
very strongly suspect this will be a small minority out of the total
of ~2k would need this treatment, and the vast majority would not. Do
you disagree with this gut feeling?
Of course, it goes without saying, we should check this before going
forward in any direction.

> > - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
> > packages at the same time is maintained from bookworm till trixie, and
> > will lifted after trixie ships, and applies implicitly to all the
> > ~2000 binary pkgs that are affected by the above
>
> While the CTTE placed the moratorium until the release of bookworm, the
> release team extended it beyond, see
> https://lists.debian.org/e1ocdqk-0005ge...@respighi.debian.org. So you
> need explicit agreement from the release team on your plan. Failing
> that, any package that has been forcefully moved is immediately
> rc-buggy due to failing a release team requirement.

Of course the release team needs to be on board, no questions about
that. But given the idea is to maintain their decision exactly as it
stands I wouldn't imagine it would be an issue? Once again, the
moratorium is explicitly about moving between locations _and_
packages, in combination, not either/or. From that same email you
linked:

"Files moving their canonical location between / and /usr (details in
[1]) *and* from one binary package to another binary package within
one release cycle remain an RC issue unless dpkg supports it."

I'm proposing to keep this in place as a general rule, with the new
escape hatch that you devised as the only addition.

> Again, let me try to fix up your plan:
>
> 1. Implement a service that reliably notices missing Breaks + Replaces
>even in the presence of aliasing. Augment it to notice
>canonicalization changes and have it report that Conflicts (or other
>measures) are needed in such cases. This should operate for upgrades
>from stable to testing, stable to unstable and testing to unstable.

There are already distro-wide upgrade piuparts checks run occasionally
IIRC, at least I've seen a bug from one being reported this week, so
we should be most of the way there already?
To be clear, this would be very nice and welcome to have obviously,
but I don't think it needs to be a blocker. We don't have such checks
for vast parts of Policy, including moving files without
Breaks+Replaces as evidenced by the recent MBF, and yet we managed to
survive thus far. I don't think it's fair that this workstream should
be held to higher standards than the rest of the project.

> 2. Add a usr-merge-helper script to usr-is-merged that helps with
>instating temporary diversions for the purpose of avoiding file loss.
>
> 3. Add a dpkg-divert wrapper for duplicating diversions according to
>aliasing to usr-is-merged. Add postinst scripts that duplicate
>existing diversions.

Yep, sounds good

> 4. Add canonicalization support to debhelper as an (internal) addon.
>Enable this addon in the next compat level. This will again populate
>${misc:Pre-Depends} with "usr-is-merged (>= fixed version)" as needed.
>Note that usrmerge's provides is unversioned and doesn't satisfy such
>a dependency.

As already mentioned, I do not believe this is necessary for _all_
cases. It is necessary for a certain number (that we should ascertain
beforehand!) of cases, and we need the machinery implemented for them,
but I don't think we should impose this workflow with pre-depends and
diversions for all affected packages. I think it should be mandatory
for problematic packages such as those you already pointed out, _and_
for cases where the maintainer wants to move files also between binary
packages.

> 5. Write a policy on how to perform changes and how to move files. Simon
>Richter suggested a policy for dpkg-divert already. This needs to 

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 19:51, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Sat, May 06, 2023 at 04:52:30PM +0100, Luca Boccassi wrote:
> > To have a working system you need several more steps that are
> > performed by the instantiator/image builder, such as providing working
> > and populated proc/sys/dev, writable tmp/var, possibly etc. And it
> > needs to be instantiated with user/password/ssh certs/locale/timezone.
> > And if it needs to be bootable on baremetal/vm, it needs an ESP. And
> > then if you have an ESP and want to run in a VM with SB, you'll need
> > self-enrolling certs on first use or ensuring the 3rd party CA is
> > provisioned. And then...
>
> You paint it this way, but it really used to just work until we got the
> /usr-merge. Indeed, debvm creates virtual machine images effectively by
> bootstrapping a filesystem from packages and turning the resulting tree
> into a file system image.
>
>  * /proc, /sys, /dev are mounted by systemd. All you need to do here is
>create the directories and base-files does so.
>  * /tmp is shipped by base-files.
>  * user and password creation is not handled yet, but can be handled by
>something similar to systemd-firstboot.
>  * Not sure what you mean with certs, locale and timezone. You can just
>install ca-certificates, locales and tzdata as part of the bootstrap.
>  * The bootloader part for baremetal is kinda out of scope for
>bootstrap, which is why debvm side-steps this. You can also skip it
>for containers and build chroots. So it is one out of multiple use
>cases that needs extra work here.
>
> In a good chunk of situations, you can get just by without messing
> around. Well that is until we broke it via usr-is-merged. I concur with
> Simon Richter, that restoring this property is a primary concern.
>
> > You get the point. Going from a bunch of packages to a running system
> > necessarily has many steps in between, some that are already done and
> > taken for granted, for example when you say "works as a container" I'm
> > pretty sure the "container" engine is taking care of at the very least
> > proc/dev/sys for you, and it's just expected to work. bin -> usr/bin,
> > sbin -> usr/sbin and lib -> usr/lib should get the same treatment: if
> > they are not there, the invoked engine should prepare them. systemd
> > and nspawn have been able to do this for a while now.
>
> No, this misses the point. You can configure essential in a very limited
> environment. However, you cannot do so without the lib or lib64 symlink
> (depending on the architecture) and the bin symlink. This is so
> critical, that it cannot be deferred to some external entity. It must be
> part of the bootstrap protocol. There are some suggested ways to fix
> this (such as adding separate bootstrap scripts next to maintainer
> scripts), but nothing implemented.
>
> > Not having those hard coded means that the use case of / on a tmpfs
> > with the rest instantiated on the fly, assembled with the vendor's
> > /usr and /etc trees, becomes possible, which is neat. And said trees
> > can pass the checksum/full integrity muster.
>
> It's neat that you can solve your use case by breaking other people's
> use cases. This is not constructive interaction however. This kind of
> behaviour is precisely what caused so much conflict around the
> /usr-merge. What if I gave a shit for your use case? Denying the
> /usr-merge and just continuing unmerged as long as possible (as merging
> would break my use case) would be my strategy of choice. You can make a
> difference here by starting to recognize other people's use cases and
> proposing solutions in that merged world. And no, it's not "add duct
> tape to every bootstrap tool".
>
> So I really want to see a solution for the bootstrap protocol before
> moving the dynamic linker and /bin/sh to its canonical location. The
> current bootstrap protocol is kept on life-support by installing the
> usrmerge package by default. Dropping usrmerge from the
> init-system-helpers dependency as first alternative or moving the
> dynamic linker would break it. If I had a solution in mind, I'd
> definitely post it right here, but unfortunately I have not.
>
> Helmut

There is absolutely no need to be so aggressive. I care about both use
cases, but evidently you have formed a different view in your mind, so
nothing productive is going to come out of this subthread right now,
so I'll stop replying. It's off-topic anyway.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Luca,

On Sat, May 06, 2023 at 04:52:30PM +0100, Luca Boccassi wrote:
> To have a working system you need several more steps that are
> performed by the instantiator/image builder, such as providing working
> and populated proc/sys/dev, writable tmp/var, possibly etc. And it
> needs to be instantiated with user/password/ssh certs/locale/timezone.
> And if it needs to be bootable on baremetal/vm, it needs an ESP. And
> then if you have an ESP and want to run in a VM with SB, you'll need
> self-enrolling certs on first use or ensuring the 3rd party CA is
> provisioned. And then...

You paint it this way, but it really used to just work until we got the
/usr-merge. Indeed, debvm creates virtual machine images effectively by
bootstrapping a filesystem from packages and turning the resulting tree
into a file system image.

 * /proc, /sys, /dev are mounted by systemd. All you need to do here is
   create the directories and base-files does so.
 * /tmp is shipped by base-files.
 * user and password creation is not handled yet, but can be handled by
   something similar to systemd-firstboot.
 * Not sure what you mean with certs, locale and timezone. You can just
   install ca-certificates, locales and tzdata as part of the bootstrap.
 * The bootloader part for baremetal is kinda out of scope for
   bootstrap, which is why debvm side-steps this. You can also skip it
   for containers and build chroots. So it is one out of multiple use
   cases that needs extra work here.

In a good chunk of situations, you can get just by without messing
around. Well that is until we broke it via usr-is-merged. I concur with
Simon Richter, that restoring this property is a primary concern.

> You get the point. Going from a bunch of packages to a running system
> necessarily has many steps in between, some that are already done and
> taken for granted, for example when you say "works as a container" I'm
> pretty sure the "container" engine is taking care of at the very least
> proc/dev/sys for you, and it's just expected to work. bin -> usr/bin,
> sbin -> usr/sbin and lib -> usr/lib should get the same treatment: if
> they are not there, the invoked engine should prepare them. systemd
> and nspawn have been able to do this for a while now.

No, this misses the point. You can configure essential in a very limited
environment. However, you cannot do so without the lib or lib64 symlink
(depending on the architecture) and the bin symlink. This is so
critical, that it cannot be deferred to some external entity. It must be
part of the bootstrap protocol. There are some suggested ways to fix
this (such as adding separate bootstrap scripts next to maintainer
scripts), but nothing implemented.

> Not having those hard coded means that the use case of / on a tmpfs
> with the rest instantiated on the fly, assembled with the vendor's
> /usr and /etc trees, becomes possible, which is neat. And said trees
> can pass the checksum/full integrity muster.

It's neat that you can solve your use case by breaking other people's
use cases. This is not constructive interaction however. This kind of
behaviour is precisely what caused so much conflict around the
/usr-merge. What if I gave a shit for your use case? Denying the
/usr-merge and just continuing unmerged as long as possible (as merging
would break my use case) would be my strategy of choice. You can make a
difference here by starting to recognize other people's use cases and
proposing solutions in that merged world. And no, it's not "add duct
tape to every bootstrap tool".

So I really want to see a solution for the bootstrap protocol before
moving the dynamic linker and /bin/sh to its canonical location. The
current bootstrap protocol is kept on life-support by installing the
usrmerge package by default. Dropping usrmerge from the
init-system-helpers dependency as first alternative or moving the
dynamic linker would break it. If I had a solution in mind, I'd
definitely post it right here, but unfortunately I have not.

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 16:11, Simon Richter  wrote:
>
> Hi,
>
> On 06.05.23 21:28, Luca Boccassi wrote:
>
> [shipping usrmerge symlinks in packages]
>
> > In the far future I'd like for these details to be owned by image
> > builders/launchers/setup processes rather than a package, but this can
> > be discussed separately and independently, no need to be tied to this
> > effort.
>
> Ideally I'd like to have this information in a single package rather
> than distributed over ten different tools, especially as this is also
> release and platform dependent.
>
> If possible, I'd like to go back to the gold standard of
>   - download Essential: yes packages and their dependencies
>   - unpack them using dpkg --fsys-tarfile | tar -x
>   - install over this directory with dpkg --root=... -i *.deb
>
> to get something that works as a container. Right now, that only works
> if I remove "init" from the package list, which is fair since a
> container doesn't need an init system anyway.
>
> The less an image builder needs to deviate from this approach, the
> better for our users.

To have a working system you need several more steps that are
performed by the instantiator/image builder, such as providing working
and populated proc/sys/dev, writable tmp/var, possibly etc. And it
needs to be instantiated with user/password/ssh certs/locale/timezone.
And if it needs to be bootable on baremetal/vm, it needs an ESP. And
then if you have an ESP and want to run in a VM with SB, you'll need
self-enrolling certs on first use or ensuring the 3rd party CA is
provisioned. And then...

You get the point. Going from a bunch of packages to a running system
necessarily has many steps in between, some that are already done and
taken for granted, for example when you say "works as a container" I'm
pretty sure the "container" engine is taking care of at the very least
proc/dev/sys for you, and it's just expected to work. bin -> usr/bin,
sbin -> usr/sbin and lib -> usr/lib should get the same treatment: if
they are not there, the invoked engine should prepare them. systemd
and nspawn have been able to do this for a while now.

Not having those hard coded means that the use case of / on a tmpfs
with the rest instantiated on the fly, assembled with the vendor's
/usr and /etc trees, becomes possible, which is neat. And said trees
can pass the checksum/full integrity muster.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 06.05.23 21:28, Luca Boccassi wrote:

[shipping usrmerge symlinks in packages]


In the far future I'd like for these details to be owned by image
builders/launchers/setup processes rather than a package, but this can
be discussed separately and independently, no need to be tied to this
effort.


Ideally I'd like to have this information in a single package rather 
than distributed over ten different tools, especially as this is also 
release and platform dependent.


If possible, I'd like to go back to the gold standard of
 - download Essential: yes packages and their dependencies
 - unpack them using dpkg --fsys-tarfile | tar -x
 - install over this directory with dpkg --root=... -i *.deb

to get something that works as a container. Right now, that only works 
if I remove "init" from the package list, which is fair since a 
container doesn't need an init system anyway.


The less an image builder needs to deviate from this approach, the 
better for our users.


Simon


OpenPGP_signature
Description: OpenPGP digital signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 11:00, Simon McVittie  wrote:
>
> On Fri, 05 May 2023 at 23:11:54 +0100, Luca Boccassi wrote:
> > In practice, I suspect that out of ~2000 packages shipping bin/ sbin/
> > lib*/, only a small fraction would end up needing to further move
> > files out to other packages
>
> I think the most common case for this is likely to be systemd system
> units, which are currently in /lib/systemd and are sometimes moved between
> binary packages. Splitting out dbus-system-bus-common during the bookworm
> release cycle is a recent example, but it also seems reasonably common
> to move systemd units around as part of having a distinction between
> "the ready-made system service" and "the binary you can run by hand"
> (see apache2 vs. apache2-bin).
>
> udev rules are in a similar situation: consumed by an important system
> service, but shipped by any random package that wants to adjust its
> behaviour.
>
> Two things about systemd units make them a relatively difficult case
> for distro-wide changes like this:
>
> For /bin, /sbin and to a lesser extent /lib/TUPLE, we can often assume
> that only "core" packages (whose maintainers should be paying attention to
> subtleties like this) are affected by any change, because the historical
> definition of those directories was "just enough to boot the system or
> do disaster recovery", minimizing what should be there; but the number
> of packages that touch /lib/systemd is rather large, and a lot of them
> are leaf or near-leaf packages.
>
> Also, they're managed (and sometimes installed) by debhelper, which
> needs to be able to do the right thing relatively automatically. For
> example, if maintainers need to be able to take some special action at
> the point at which their /lib/systemd units move to /usr/lib/systemd,
> then I think debhelper installing into /usr/lib/systemd would have to
> be gated by a compat level change.

Yes, I suspect you are right that it would be the bulk of the leaf
packages. But still shouldn't invalidate what I mentioned? As long as
they are kept in the same package, there shouldn't be a need for
diversions. And if they need to be moved in this cycle, the diversion
dance will be needed. Do you see any problems with this approach?

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 06:21, Simon Richter  wrote:
>
> Hi,
>
> On 06.05.23 07:11, Luca Boccassi wrote:
>
> > - every package is forcefully canonicalized as soon as trixie is open
> > for business
>
> You will also need to ship at least
>
>   - /lib -> usr/lib (on 32 bit)
>   - /lib64 -> usr/lib64 (on 64 bit)
>
> as a symlink either in the libc-bin package or any other Essential
> package, to fulfill the requirement that unpacked Essential packages are
> operational.

Sure, that doesn't sound problematic? We'll need the same for bin/ and
sbin/ for at least a cycle as you already pointed out. Sounds like a
job for base-files.

In the far future I'd like for these details to be owned by image
builders/launchers/setup processes rather than a package, but this can
be discussed separately and independently, no need to be tied to this
effort.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon McVittie]

On Fri, 05 May 2023 at 23:11:54 +0100, Luca Boccassi wrote:
> In practice, I suspect that out of ~2000 packages shipping bin/ sbin/
> lib*/, only a small fraction would end up needing to further move
> files out to other packages

I think the most common case for this is likely to be systemd system
units, which are currently in /lib/systemd and are sometimes moved between
binary packages. Splitting out dbus-system-bus-common during the bookworm
release cycle is a recent example, but it also seems reasonably common
to move systemd units around as part of having a distinction between
"the ready-made system service" and "the binary you can run by hand"
(see apache2 vs. apache2-bin).

udev rules are in a similar situation: consumed by an important system
service, but shipped by any random package that wants to adjust its
behaviour.

Two things about systemd units make them a relatively difficult case
for distro-wide changes like this:

For /bin, /sbin and to a lesser extent /lib/TUPLE, we can often assume
that only "core" packages (whose maintainers should be paying attention to
subtleties like this) are affected by any change, because the historical
definition of those directories was "just enough to boot the system or
do disaster recovery", minimizing what should be there; but the number
of packages that touch /lib/systemd is rather large, and a lot of them
are leaf or near-leaf packages.

Also, they're managed (and sometimes installed) by debhelper, which
needs to be able to do the right thing relatively automatically. For
example, if maintainers need to be able to take some special action at
the point at which their /lib/systemd units move to /usr/lib/systemd,
then I think debhelper installing into /usr/lib/systemd would have to
be gated by a compat level change.

smcv

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 06.05.23 07:11, Luca Boccassi wrote:


- every package is forcefully canonicalized as soon as trixie is open
for business


You will also need to ship at least

 - /lib -> usr/lib (on 32 bit)
 - /lib64 -> usr/lib64 (on 64 bit)

as a symlink either in the libc-bin package or any other Essential 
package, to fulfill the requirement that unpacked Essential packages are 
operational.


   Simon


OpenPGP_signature
Description: OpenPGP digital signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Fri, 5 May 2023 at 17:38, Andreas Metzler  wrote:
>
> On 2023-05-05 Simon Richter  wrote:
> [...]
> > My proposal would be to put the onus on the client registering the
> > diversion:
> [...]
> >  - packages are encouraged to register both diversions
>
> Hello,
>
> That seems to be a rather ugly user interface, ("There is dpkg-divert on
> Debian, but because the usrmerge you need to invoke it twce to be
> sure"). Will we need to have a meta-transition years from now trying to
> get get rid of the double diversions?

It is not the prettiest thing but it is a very clever solution.
Perhaps it could be mitigated with an addendum that makes it optional,
and to be used only when strictly needed, after all moving files
within the same package is fine, it's only the combination of moving
location _and_ package that causes problems. In other words:

- every package is forcefully canonicalized as soon as trixie is open
for business
- the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
packages at the same time is maintained from bookworm till trixie, and
will lifted after trixie ships, and applies implicitly to all the
~2000 binary pkgs that are affected by the above
- the moratorium can be bypassed by a maintainer if and only if the
appropriate conflicts/replaces/diverts as discussed in this thread are
put in place and left there for as long as needed (TBD whether this
means for trixie's cycle or for trixie+1)

In practice, I suspect that out of ~2000 packages shipping bin/ sbin/
lib*/, only a small fraction would end up needing to further move
files out to other packages, so the divert dance requirement can be
restricted only to those. This way impact is minimized, required
testing is smaller, and we get in the final state on day one of trixie
dev cycle.
Moving files between packages already requires busywork anyway, so a
bit more won't hurt that much, especially if we figure out a way to
provide the functionality with a dh addon or such to do the heavy
lifting.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Andreas Metzler]

On 2023-05-05 Simon Richter  wrote:
[...]
> My proposal would be to put the onus on the client registering the
> diversion:
[...]
>  - packages are encouraged to register both diversions

Hello,

That seems to be a rather ugly user interface, ("There is dpkg-divert on
Debian, but because the usrmerge you need to invoke it twce to be
sure"). Will we need to have a meta-transition years from now trying to
get get rid of the double diversions?

cu Andreas

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 05.05.23 18:36, Timo Röhling wrote:

- it is not an error to register a diversion for an alias of an 
existing diversion, provided the package and target matches, this is a 
no-op
- it is not an error to unregister a diversion for an alias of a path 
that has been unregistered previously, that is a no-op as well



How do you distinguish between aliased diversions and "real" ones?
Because if you allow the registration of duplicate diversions, the
following can happen:



- Package A is installed, preinst creates a diversion
- Package B is installed, preinst creates the same diversion
- Package A is uninstalled, postrm removes the diversion


That's why the package and target need to match -- so by extension 
package B may not divert a path that package A already diverts.


With dpkg-divert as it is, this works because no aliasing support 
exists. A diverted dpkg-divert that generates additional diversions 
would need to catch this case, reject it and restore the previous state, 
same as an aliasing aware dpkg-divert.


   Simon


OpenPGP_signature
Description: OpenPGP digital signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Timo Röhling]

Hi,

* Simon Richter  [2023-05-05 17:59]:
- it is not an error to register a diversion for an alias of an 
existing diversion, provided the package and target matches, this is a 
no-op
- it is not an error to unregister a diversion for an alias of a path 
that has been unregistered previously, that is a no-op as well

How do you distinguish between aliased diversions and "real" ones?
Because if you allow the registration of duplicate diversions, the
following can happen:

- Package A is installed, preinst creates a diversion
- Package B is installed, preinst creates the same diversion
- Package A is uninstalled, postrm removes the diversion

Now package B has lost its diversion. 



Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 04.05.23 20:26, Helmut Grohne wrote:


From my point of view, the ultimate goal here should be moving all files

to their canonical location and thereby make aliasing effects
irrelevant. Do you confirm?


Yes, that would solve the problem for the current transition without any 
changes in dpkg.


There will be some small aliasing effects still, e.g. ldd will probably 
output the linker path from the ELF ABI instead of the canonical linker 
path, but nothing major.


What I'm concerned about is that, whether we like it or not, we're still 
(d|r)efining interfaces here, and that affects future development, 
including future transitions.



As such, I do not see aliasing support in dpkg as complementing the
forced file move approach with lots of workarounds such as diverting
dpkg-divert. Rather, I see them as exclusive strategies.


Yes, absolutely. No one can coordinate two groups that don't even agree 
on the scope of the work or whether a definition of the scope is even 
necessary before beginning.



Each of these
strategies has significant downsides. In combining the different
strategies, we combine their downsides, but since their benefit is
shared, we do not gain anything in return but increase the price to pay.


For this transition, yes, but we need to think beyond that. Any 
workaround we introduce here will need one release to be rolled out, one 
release to be stable, and one release to disappear. We can, in many but 
not all cases, introduce dependencies to enforce a particular ordering 
and reduce that by one release cycle.


We're too late to roll out workarounds in the bookworm cycle, so we can 
deploy them with trixie at the earliest. Essential packages cannot 
contain diversions, so we need support from all installers (debootstrap, 
cdebootstrap, mmdebootstrap) or leave file paths for Essential packages 
as they are, delaying the end of the transition until forky.


Making the workarounds obsolete before that will allow us to finish the 
transition earlier and thus reduce the effort of future transitions. For 
the users, nothing changes, they have had the files moved forcefully 
years ago.


At the same time, there is little overlap in the groups that would 
implement the different strategies, so there is no person who would need 
to decide what they are working on to the detriment of the other option.



On the flip side, if dpkg (and thus dpkg-divert) is to gain aliasing
support, I see no reason (and benefit) to diverting dpkg-divert.


Me neither. The problem is, we have no commitment from anyone to 
implement aliasing support. Volunteering Guillem hasn't worked, I don't 
really have time to do more than "best effort", and I don't know many 
other people even looking into that, so we cannot decide on that strategy.



Can you explain why you see combining these strategies as something
worth exploring?


It will get something done on the way to finishing the transition until 
forky, even if dpkg changes don't manifest, but can provide a solution 
in trixie if they do.



then a package containing /bin/foo and a package containing /usr/bin/foo now
have a file conflict in dpkg. Not sure if that is a problem, or exactly the



This case already is prohibited by policy section 10.1. It can only
happen as a temporary state during a file move (from / to /usr and from
one package to another).


Yes, but from a technical point of view we cannot rely on Policy, it 
only allows us to assign blame, but that will not fix the error message 
a user is experiencing.


With the status quo, dpkg will not detect the file conflict, which may 
or may not be the desired result, but it will not flag an error during 
execution.


With two file names being diverted to the same name, it is likely 
(again, I haven't tested) that installation of the conflicting package 
will be aborted unless there is a Replaces: in effect, which is 
something that whatever tool is calling dpkg will need to deal with.



behaviour we want. Probably the latter, which would allow us to define a
policy "if aliased paths are diverted, the diversion needs to match", which
in turn would allow the conflict checker during alias registration to verify
that the aliased diversions are not in conflict.



If we do not modify dpkg to improve aliasing support, then yes, such a
scenario will require a Conflicts declaration or a different measure
averting this problem.


No, that would require the package declaring the diversion to conflict 
with itself, that makes no sense. We need to clarify the interface for 
dpkg-divert here (i.e. create Policy), and that interface will need to 
be supported by both the diverted dpkg-divert and the one shipped with dpkg.


As of now, the one in dpkg will happily register diversions for aliased 
paths, but to my knowledge, none have been registered, and the diverted 
dpkg-divert does not exist yet, so we have full freedom to define what 
the policy should be.


My proposal would be to put the onus on the

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Simon,

On Thu, May 04, 2023 at 03:37:49AM +0900, Simon Richter wrote:
> For aliasing support in dpkg, that means we need a safe policy of dealing
> with diversions that conflict through aliasing that isn't "reject with
> error", because the magic dpkg-divert would always generate conflicts.

I think we still have that misunderstanding I mentioned earlier, so let
me try to resolve that again.

>From my point of view, the ultimate goal here should be moving all files
to their canonical location and thereby make aliasing effects
irrelevant. Do you confirm?

As such, I do not see aliasing support in dpkg as complementing the
forced file move approach with lots of workarounds such as diverting
dpkg-divert. Rather, I see them as exclusive strategies. Each of these
strategies has significant downsides. In combining the different
strategies, we combine their downsides, but since their benefit is
shared, we do not gain anything in return but increase the price to pay.
Why should we do that?

So when we discuss diverting dpkg-divert, I imply that we do not change
the implementation of dpkg wrt. aliasing. So this branch of discussion
that you raise here, seems irrelevant to me.

On the flip side, if dpkg (and thus dpkg-divert) is to gain aliasing
support, I see no reason (and benefit) to diverting dpkg-divert.

Can you explain why you see combining these strategies as something
worth exploring?

> then a package containing /bin/foo and a package containing /usr/bin/foo now
> have a file conflict in dpkg. Not sure if that is a problem, or exactly the

This case already is prohibited by policy section 10.1. It can only
happen as a temporary state during a file move (from / to /usr and from
one package to another).

> behaviour we want. Probably the latter, which would allow us to define a
> policy "if aliased paths are diverted, the diversion needs to match", which
> in turn would allow the conflict checker during alias registration to verify
> that the aliased diversions are not in conflict.

If we do not modify dpkg to improve aliasing support, then yes, such a
scenario will require a Conflicts declaration or a different measure
averting this problem.

> The diverted dpkg-divert would probably generate extra register/unregister
> calls as soon as dpkg-divert itself is aliasing aware, but all that does is
> generate warning messages about existing diversions being added again, or
> nonexistent diversions being deleted -- these are harmless anyway, because
> maintainer scripts are supposed to be idempotent, and dpkg-divert supports
> that by not requiring scripts to check before they register/unregister.

Again, the premise seems unreasonable to me. Also note that such a
diversion of dpkg-divert certainly is meant as a temporary measure
facilitating the transition. I'd hope we could delete it in forky
already and failing that thereafter.

> We get to draw this card exactly once, and any package that would need the
> same diversion would need to conflict with usr-is-merged, which would make
> it basically uninstallable.

I don't think the case of packages wanting to divert update-alternatives
is all that common. Please elaborate on the use case. Also note that
this suggestion already is to be considered a plan B. My current
understanding is that as long as we do not canonicalize alternatives at
all, we don't run into problems with them. This kinda is ugly, but the
number of affected packages is small.

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 03.05.23 19:19, Helmut Grohne wrote:


What still applies here is that we can have usr-is-merged divert
/usr/bin/dpkg-divert and have it automatically duplicate any possibly
aliased diversion and then the diverter may Pre-Depends: usr-is-merged
(>=...) to have its diversions duplicated. Of course, doing so will make
usr-is-merged very hard to remove, but we have experience here from
multiarch-support.


For aliasing support in dpkg, that means we need a safe policy of 
dealing with diversions that conflict through aliasing that isn't 
"reject with error", because the magic dpkg-divert would always generate 
conflicts.


One thing we need to check is whether diversions to the same target 
cause file conflicts -- I think they should.


So if you divert

/bin/foo -> /usr/bin/foo.dontdelete
/usr/bin/foo -> /usr/bin/foo.dontdelete

then a package containing /bin/foo and a package containing /usr/bin/foo 
now have a file conflict in dpkg. Not sure if that is a problem, or 
exactly the behaviour we want. Probably the latter, which would allow us 
to define a policy "if aliased paths are diverted, the diversion needs 
to match", which in turn would allow the conflict checker during alias 
registration to verify that the aliased diversions are not in conflict.


The diverted dpkg-divert would probably generate extra 
register/unregister calls as soon as dpkg-divert itself is aliasing 
aware, but all that does is generate warning messages about existing 
diversions being added again, or nonexistent diversions being deleted -- 
these are harmless anyway, because maintainer scripts are supposed to be 
idempotent, and dpkg-divert supports that by not requiring scripts to 
check before they register/unregister.



And of course, we can always draw the diversion card and have
usr-is-merged divert /usr/bin/update-alternatives to have it
canonicalize paths as required to be able to migrate alternatives in a
sane way (from a consumer point of view).


We get to draw this card exactly once, and any package that would need 
the same diversion would need to conflict with usr-is-merged, which 
would make it basically uninstallable.


   Simon

Re: DEP 17: Improve support for directory aliasing in dpkg

[David Kalnischkies]

On Wed, May 03, 2023 at 10:31:14AM +0200, Raphael Hertzog wrote:
> On Tue, 02 May 2023, Helmut Grohne wrote:
> > I think there is a caveat (whose severity I am unsure about): In order
> > to rely on this (and on DEP 17), we will likely have versioned
> > Pre-Depends on dpkg. Can we reasonably rule out the case where and old
> > dpkg is running, unpacking a fixed dpkg, configuring the fixed dpkg and
> > then unpacking an affected package still running the unfixed dpkg
> > process?

APT instructs dpkg to --unpack and to --configure in different calls,
you can't mix and match those in the same call and apt never does the
(combining) --install (not that it would really matter here).
Also, dpkg is essential and as such has to work unpacked aka unpacking
a fixed dpkg means that this fixed dpkg will (later) configure itself.

Now, given dpkg is essential, it also means it gets the essential
treatment from APT (by default) which means it will try to unpack it as
soon as possible while trying to keep the time it remains unconfigured
at a minimum. Give it a try, you usually see essential packages being
interacted with first and in their own calls if you look close enough.
That isn't an accident, the idea is that some random 'optional' package
failing to install in some way should not leave you in a situation where
essentials are in a state of limbo.

If you increase the complexity of (pre-)requirements through APT will
end up being forced to hand multiple packages in one go. Just pull up
the last time you upgraded libc6: You will see a bunch of -dev packages
and MultiArch siblings being unpacked alongside libc6 and libc-bin. You
will only see those two being configured right after through. The
dependencies will it is… so we might have to be a bit careful about the
dependencies dpkg carries if such a route is taken.


That said, there is always the 'stretch' horror story of APT installing
all of KDE before touching dpkg because of the install-info transition…
Although that was avoided before the release by removing from dpkg the
Breaks leading us into this dark alley… (just to be sure: APT wasn't
wrong, the dependencies weren't – but the idea to manually upgrade dpkg
first to avoid some pitfalls was suggested which turned out to be wrong).

Also, I wonder if we run into Pre-Depends loops and similar nasties
given that the essential set is somewhat likely to pre-depend on
things which use(d) to be in /lib which would in turn Pre-Depend on
dpkg.

(I haven't tried and memory is sketchy about those finer more
 complicated matters, but dpkg certainly can produce working orders
 for loops by inspecting which maintainer scripts exist or not, so
 upgrades involving those might or might not work. All bets are off
 which version of dpkg would be dealing with those through)


> I don't know APT well enough to answer that question but from my point of
> view it's perfectly acceptable to document in the release notes that you
> need to upgrade dpkg first.

Those never work in practice through. Nobody logs in on their buildd
chroots and upgrades them "properly", we all just hope for the best.

Even on systems we care more about people are regularity caught red
handed by bothering support with questions whose answers are spelled
out in detail in the release notes. Case in point: "Changed security
archive layout" last time or "Non-free firmware moved to its own
component in the archive" this time around…

And those are easy to diagnose and fix. 'You "might" have some "random"
files not present on disk. So your system might not even boot or spawns
interdimensional portals. You better reinstall…' is not the type of
thing you wanna here from support.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Raphaël,

On Wed, May 03, 2023 at 10:31:14AM +0200, Raphael Hertzog wrote:
> I don't know APT well enough to answer that question but from my point of
> view it's perfectly acceptable to document in the release notes that you
> need to upgrade dpkg first.

Yes, this issue seems vaguely solvable one way or another. It also
affects other approaches modifying dpkg in the very same way.

> Are you sure that we need anything for diversions except some documented
> policy on how to deal with it?

Yes! There is a hard ordering constraint involved here. Failure to do so
results in unpack errors and or file loss in much the same way.

> AFAIK the following sequence performs no filesystem changes and should
> be sufficient to move a diversion to its new location (I only consider the
> case of an upgrade, not of a new installation that should just work
> "normally" on the new location):
> 
> dpkg-divert --package $package --remove /bin/foo --no-rename
> dpkg-divert --package $package --add /usr/bin/foo --divert 
> /usr/bin/foo.diverted --no-rename

This is insufficient. Either we modify dpkg to consider aliasing when
managing diversions (i.e. Simon Richter's branch or DEP17) or there is a
more complex ordering requirement involved:

 * We must not remove the aliased diversion (/bin/foo) before the
   diverted package has moved its files to the canonical location
   (/bin/foo -> /usr/bin/foo).
 * We must add the canonical diversion (/usr/bin/foo) before the
   diverted package update that moves its files to canonical locations
   can be unpacked.

Say we currently have

Package: diverter
Version: 1
Files: /bin/foo
preinst: diverts /bin/foo

Package: diverted
Version: 1
Files: /bin/foo

We must first update the diverter.

Package: diverter
Version: 2
Files: /usr/bin/foo
preinst: diverts both /bin/foo and /usr/bin/foo

Since we divert both locations, diverter can now deal with an old
diverted and a canonicalized diverted.

Package: diverted
Version: 2
Conflicts: diverter (<< 2~)
Files: /usr/bin/foo

At the time of unpacking the updated diverted, we must ensure that no
diverter versioned 1 is unpacked. Breaks does not help here as it allows
concurrent unpacks. Neither does Replaces since dpkg thinks that
/bin/foo is different from /usr/bin/foo and thus no replacing happens.

Package: diverter
Version: 3
Conflicts: diverted (<< 2~)
Files: /usr/bin/foo
preinst: diverts /usr/bin/foo

When unpacking the updated diverter, we must ensure that no diverted
version 1 is unpacked. Again, Breaks and Replaces does not suffice.
Therefore an upgrade from stable to nextstable containing both diverter
and diverted must temporarily remove either package, which is known to
annoy apt.

What still applies here is that we can have usr-is-merged divert
/usr/bin/dpkg-divert and have it automatically duplicate any possibly
aliased diversion and then the diverter may Pre-Depends: usr-is-merged
(>=...) to have its diversions duplicated. Of course, doing so will make
usr-is-merged very hard to remove, but we have experience here from
multiarch-support.

Hope this clarifies.

> The case of update-alternatives is likely more tricky. You already looked
> into it. That's a place where it will be harder to get things right
> without some changes.

As detailed in
https://lists.debian.org/debian-devel/2023/04/msg00169.html I believe
that update-alternatives really are not tricky at all as long as we do
not attempt to migrate them to canonical paths in any way. For instance,
elvis-tiny needs to continue to name the editor alternative
/bin/elvis-tiny even when it actually moves that file to /usr/bin. The
reason that this does not hurt is that we never attempted to move
alternatives (unlike regular files in packages).

If we really want to migrate alternatives to canonical paths, we do get
into the tricky area of preserving the user configuration and we also
break custom scripts, ansible's community.general.alternatives, uses of
puppet's alternatives modules and probably a lot more.

And of course, we can always draw the diversion card and have
usr-is-merged divert /usr/bin/update-alternatives to have it
canonicalize paths as required to be able to migrate alternatives in a
sane way (from a consumer point of view).

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Raphael Hertzog]

Hello,

On Tue, 02 May 2023, Helmut Grohne wrote:
> I think there is a caveat (whose severity I am unsure about): In order
> to rely on this (and on DEP 17), we will likely have versioned
> Pre-Depends on dpkg. Can we reasonably rule out the case where and old
> dpkg is running, unpacking a fixed dpkg, configuring the fixed dpkg and
> then unpacking an affected package still running the unfixed dpkg
> process?

I don't know APT well enough to answer that question but from my point of
view it's perfectly acceptable to document in the release notes that you
need to upgrade dpkg first.

> I think the file loss problem is one sufficient reason to have the
> moratorium. We didn't need other reasons once we knew this one. Now that
> we look into dropping the moratorium, we need to ensure that there are
> no reasons anymore and we learned that diversions are affected in a
> non-trivial way. So even if we were to fix just the file loss problem,
> the diversion problems would still be sufficient reason to keep the
> moratorium unless they were also fixed by the approach. Here you need
> both directions a) diverting a non-canonical location would have to
> divert a canonical file and b) diverting a canonical location would have
> to divert a non-canonical file. This is breaking the initial assumption.

Are you sure that we need anything for diversions except some documented
policy on how to deal with it?

AFAIK the following sequence performs no filesystem changes and should
be sufficient to move a diversion to its new location (I only consider the
case of an upgrade, not of a new installation that should just work
"normally" on the new location):

dpkg-divert --package $package --remove /bin/foo --no-rename
dpkg-divert --package $package --add /usr/bin/foo --divert 
/usr/bin/foo.diverted --no-rename

The case of update-alternatives is likely more tricky. You already looked
into it. That's a place where it will be harder to get things right
without some changes.

> In any case, this train of thought is definitely widening the solution
> space. Thank you very much.

You are welcome.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: DEP 17: Improve support for directory aliasing in dpkg

[Sam Hartman]

> "Helmut" == Helmut Grohne  writes:

Helmut> Luca Boccassi kindly pointed me at config-package-dev
Helmut> though. This is a tool for generating local packages and it
Helmut> also employs dpkg-divert.  There is a significant risk of
Helmut> breaking this use case. If we were to divert dpkg-divert and
Helmut> automatically duplicate diversions, this use case were
Helmut> automatically covered.

Most of the uses of config-package-dev are typically within /etc.
Also, people who use config-package-dev tend to be managing
infrastructure and have ways to handle upgrades of their infrastructure.

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

On Tue, May 02, 2023 at 02:09:32PM +0200, Helmut Grohne wrote:
> This is problems we know about now, but it likely is not an exhaustive
> list. This list was mostly guided by Guillem's intuition of what could
> break at https://wiki.debian.org/Teams/Dpkg/MergedUsr and I have to say
> that his intuition was quite precise thus far. Notably missing in the
> investigation are statoverrides. However, we should also look for a more
> generic approach that tries capturing unexpected breakage.

I mentioned statoverrides as missing. I think we can split statoverrides
into the two classes "package changes" and "admin changes". Quite
obviously, moving files, will break admin changes. I see little ways
around this, we can partially mitigate this by detecting common
statoverrides and migrating them automatically, but in the end, we'll
probably have to explain issues with admin-initiated statoverrides in
the release notes.

For package changes, the good thing is that statoverrides usually change
stats of files owned by the package initiating them. Thus a package
moving files can also move statoverrides (though this again means that
automatic moves e.g. by debhelper must be opt-in in order to avoid
breaking stuff). For getting an idea of the scope, we can use
https://binarycontrol.debian.net/?q=dpkg-statoverride.*+%2F%28bin%7Csbin%7Clib%7Clib32%7Clib64%7Clibo32%7Clibx32%29&path=%2Funstable%2F

* fuse and fuse3 adapt to an admin initiated statoverride of
  /bin/fusermount.
* nfs-common cleans an obsolete dpkg-statoverride of /sbin/mount.nfs
* systemd-cron adds a statoverride for /lib/systemd-cron/crontab_setgid
  and needs to migrate it with its files.
* yp-tools adds a statoverride for /sbin/unix_chkpwd and needs to
  migrate it with its files.

I also tried installing all packages that contain dpkg-statoverride in
any of their maintainer scripts and capturing the resulting statoverride
file. That doesn't yield anything unexpected thus far, but it also
hasn't completed yet. I'll reply to this message with findings if
there are any beyond the ones above.

So statoverides seem quite similar to the diversions induced by dash:
Mostly harmless if handled correctly while moving the files, but we
cannot just move the files in an opt-out fashion. Beyond that we need to
augment release notes to ask admins to carefully update their local
statoverrides (and local diversions).

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

On Tue, May 02, 2023 at 02:09:32PM +0200, Helmut Grohne wrote:
> I noticed that the number of packages shipping non-canonical files is
> relatively small. It's less than 2000 binary packages in unstable and
> their total size is about 2GB. So I looked into binary-patching them and
> attach the resulting scripts.

Sorry for missing the attachments.

Helmut


repackdeb.sh
Description: Bourne shell script


autorepack.sh
Description: Bourne shell script


createchroot.sh
Description: Bourne shell script

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Luca,

On Fri, Apr 21, 2023 at 03:29:33PM +0100, Luca Boccassi wrote:
> After Bookworm ships I plan to propose a policy change to the CTTE and
> policy maintainers to forbid shipping files in the legacy directories
> altogether, followed by a debhelper change to adjust any stragglers
> automatically at build time and a mass rebuild, plus MBF for the small
> % that does not use dh and a piuparts test to stop migration for
> anything that is uploaded and doesn't comply. That should bring the
> matter to an end, without needing to modify dpkg.

I think we now learned that this is quite oversimplified, but possibly
fixable. At this time we know about the following problem areas:
 * file loss during upgrades
   + The main reason for having the moratorium
   + Possible workaround is using Conflicts
 * diversion mismatches
   + Possibly fixable by duplicating affected diversions
 * alternatives
   + Only become a problem if we try migrating them to canonical paths
 * triggers
   + Possible fix is duplicating trigger interest

This is problems we know about now, but it likely is not an exhaustive
list. This list was mostly guided by Guillem's intuition of what could
break at https://wiki.debian.org/Teams/Dpkg/MergedUsr and I have to say
that his intuition was quite precise thus far. Notably missing in the
investigation are statoverrides. However, we should also look for a more
generic approach that tries capturing unexpected breakage.

I noticed that the number of packages shipping non-canonical files is
relatively small. It's less than 2000 binary packages in unstable and
their total size is about 2GB. So I looked into binary-patching them and
attach the resulting scripts.
 * repackdeb.sh is a helper script for repacking an individual binary
   package and canonicalizing the contained paths.
 * autorepack.sh creates and apt repository of repacked .debs and calls
   repackdeb.sh repeatedly in that process. Should complete in half an
   hour.
 * createchroot.sh consumes the apt repository to create a chroot using
   mmdebstrap. In there, all paths should be canonical.
 * Depends: curl, devscripts, dpkg-dev, fakeroot, mmdebstrap, moreutils
 * You are expected to place these scripts in an empty directory and run
   them from there. Please read them before running them. You'll likely
   have to edit at least createchroot.sh to make it useful.

I'm not sure how to devise test cases from this yet.

 * I tried createchroot.sh with --include=zutils, because that's one of
   the packages where I expect breakage due to mismatching diversions.
   Sure enough, I got the expected unpack error from dpkg!

 * I tried createchroot.sh with --include=gnome to see it in action with
   a few more packages and that appeared successful. So in this scenario
   there were no unforeseen and readily visible problems.

If you end up performing tests using these or similar scripts, please
report your results. If you can think of relevant scenarios, please
tell.

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Raphaël,

On Tue, May 02, 2023 at 12:30:21PM +0200, Raphael Hertzog wrote:
> We don't want to stat all the files in all packages but we could do better:
> if we are about to remove an old file that is available through a
> symlinked directory, we could check the new name of the file and see if
> it's available in some package... and if yes just forget the file without
> removing it.

Indeed, this is a neat idea. You are effectively implying that we only
ever move files from non-canonical locations to canonical locations and
never the other way round. I think this is a reasonable assumption and
this assumption is the one that makes your variant simpler.

I think there is a caveat (whose severity I am unsure about): In order
to rely on this (and on DEP 17), we will likely have versioned
Pre-Depends on dpkg. Can we reasonably rule out the case where and old
dpkg is running, unpacking a fixed dpkg, configuring the fixed dpkg and
then unpacking an affected package still running the unfixed dpkg
process?

> This file removal is the reason of the moratorium and incuring some extra
> cost in some specific cases (installation through directory symlinks which
> is not the default case, and would not affect us after the migration is
> complete) seems certainly fair.

I think the file loss problem is one sufficient reason to have the
moratorium. We didn't need other reasons once we knew this one. Now that
we look into dropping the moratorium, we need to ensure that there are
no reasons anymore and we learned that diversions are affected in a
non-trivial way. So even if we were to fix just the file loss problem,
the diversion problems would still be sufficient reason to keep the
moratorium unless they were also fixed by the approach. Here you need
both directions a) diverting a non-canonical location would have to
divert a canonical file and b) diverting a canonical location would have
to divert a non-canonical file. This is breaking the initial assumption.

In any case, this train of thought is definitely widening the solution
space. Thank you very much.

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Raphael Hertzog]

Hello,

On Wed, 26 Apr 2023, Simon Richter wrote:
> > This and /bin/sh is the kind of files I'd consider important. And then
> > upon thinking further it became more and more difficult for me to make
> > sense of the objection. On a merged system, we can just move that file
> > to its canonical location without having any trouble even with an
> > unmodified dpkg. So from my pov, the question about important files can
> > be disregarded. I hope Simon Richter agrees.
> 
> Yes, the relevant code at
> 
> https://github.com/guillemj/dpkg/blob/main/src/main/unpack.c#L749
> 
> already handles moving a file inside the same package, and that has
> existed for some time, that's why I use two packages for the PoC.

Hum... why aren't we improving this part of the code?

We don't want to stat all the files in all packages but we could do better:
if we are about to remove an old file that is available through a
symlinked directory, we could check the new name of the file and see if
it's available in some package... and if yes just forget the file without
removing it.

This file removal is the reason of the moratorium and incuring some extra
cost in some specific cases (installation through directory symlinks which
is not the default case, and would not affect us after the migration is
complete) seems certainly fair.

The cost of analyzing directory components is a cost that we will have on
all dpkg invocations but it doesn't seem unreasonable to me. We could also
restrict it to the top-level directories to make it negligible as this
is the only transition that we care about here.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Hi Marvin,

On Sat, Apr 29, 2023 at 02:08:37PM -0400, Marvin Renich wrote:
> My understanding from following this thread (and others) is that dpkg
> has a bug that can easily be triggered by a sysadmin replacing a
> directory with a symlink (and some other necessary conditions that don't
> happen very often), which is explicitly allowed by policy.  This bug is

I fear that this is more nuanced than it initially looks. While we kinda
support symlinking parts of the directory hierarchy, the implicit
assumption has always been that this never introduces aliasing. In other
words, the target of such a symlink needs to be a location that is not
used by any package.

The problem here arises from introducing aliasing symlinks - which has
never been supported. In that sense, we do not have consensus on whether
this is a bug in dpkg or merely a missing feature.

> the one that is causing the problem with the approach that was chosen by
> the people implementing usrmerge, even though they were aware of this
> problem and a different approach that would have taken two release
> cycles and would not have triggered this bug was considered and
> rejected.

In all fairness, my understanding is that the different approach would
have had different semantics. Not all binaries would have been available
via both / and /usr and that would have resulted in continuous
incompatibility with other Linux distributions.

> If this is correct, then Luca's approach may fix the problem for
> usrmerge, but does not fix the general dpkg bug.  (And, IIUC, is going
> to take two _more_ release cycles to fix the problems with usrmerge as
> implemented!  Hmm...)

That general dpkg bug is only observable when aliasing happens. Once no
package ships anything inside non-canonical paths, no relevant aliasing
is happening anymore, so we will not experience any symptoms of that
bug. As such, we could call it unsupported again once Luca's approach
has completed. So yeah, I think the beauty of that approach would be
getting out of this mess without patching dpkg. I've not yet fully
convinced myself that this is indeed viable (though I'm trying to).

> The --add-alias solution that has been suggested in this thread seems
> like it would fix the general problem iff policy was changed to require
> sysadmins to use it if they replaced a directory with a symlink.

I don't think we want to support aliasing as a general mechanism
available to administrators.

> I do not understand why the dpkg maintainer has rejected this solution;
> it would still be a fix for the general bug after the usrmerge
> transition has completed.  And it would be at least one order of
> magnitude more performant than scanning the filesystem for directory
> symlinks.

The --add-alias mechanism certainly adds significant complexity to the
dpkg code base. Once introduced, it would become API and cannot be
removed anymore, so it introduces a continuous maintenance cost. And if
you disagree with the bug being a bug (and think of it as rejecting to
implement a new feature), that rejection vaguely makes sense. We may
disagree with that reasoning, but it is far from baseless.

So the problem kinda is that aliasing is happening and causes dpkg's
undefined behaviour wrt aliasing to cause problems. We basically have
two ways to fix this:
 A. Ensure that no aliasing is happening anymore. (e.g. by
canonicalizing all paths)
 B. Make dpkg cope with such aliasing.

And while some seem to see these solutions as complementary, I think we
should build consensus around either option as combining them gets us
the downsides of both without adding benefit.

Helmut

Re: DEP 17: Improve support for directory aliasing in dpkg

[Simon Richter]

Hi,

On 30.04.23 03:08, Marvin Renich wrote:


My understanding from following this thread (and others) is that dpkg
has a bug that can easily be triggered by a sysadmin replacing a
directory with a symlink (and some other necessary conditions that don't
happen very often), which is explicitly allowed by policy.


No, it becomes "easily" triggered because usrmerge aims to move the 
files in the packages, which would cause /bin to become empty at some 
point in the future, at which point dpkg correctly needs to remove it.


As long as /bin/ls exists in coreutils, the /bin -> usr/bin symlink will 
be safe from deletion.



The --add-alias solution that has been suggested in this thread seems
like it would fix the general problem iff policy was changed to require
sysadmins to use it if they replaced a directory with a symlink.


Yes, we want something that is basically --add-alias that returns us to 
a state where dpkg's model is an accurate representation of reality.



I do not understand why the dpkg maintainer has rejected this solution;


I've understood this to be mostly an issue with the quality of the 
patches submitted, not the general idea. Dpkg has a really extensive 
test suite, I have not so far seen any patch being submitted that even 
adds test cases there. My branch adds six, that is still ridiculously low.



it would still be a fix for the general bug after the usrmerge
transition has completed.  And it would be at least one order of
magnitude more performant than scanning the filesystem for directory
symlinks.


That, and as far as I can see, fixing the general problem will also be a 
prerequisite to finishing the transition in the first place because 
there are lots of subtle little things caused by the model mismatch, and 
the workarounds solve the problem at hand but further entrench the mismatch.


   Simon

PS: Current state of my branch:

Selecting previously unselected package pkg-usrmerge.
(Reading database ... 6 files and directories currently installed.)
Preparing to unpack pkg-usrmerge.deb ...
D01: process_archive oldversionstatus=not installed
/home/geier/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb/tmp.ci/preinst 
install

D01: ensure_canonical: new, (re)loading
D01: ensure_canonical: found mapping /bin -> /usr/bin
D01: ensure_canonical: mapping /bin -> /usr/bin seems to be new
D01: create_alias: alias /bin/test-conflict -> 
/usr/bin/test-conflict conflicts with pkg-b(0)


OpenPGP_signature
Description: OpenPGP digital signature

Re: DEP 17: Improve support for directory aliasing in dpkg

[Marvin Renich]

* Helmut Grohne  [230428 09:50]:
> I think we have a misunderstanding here. As far as I understand it, the
> core idea of Luca's approach is that we move all files to their
> canonical locations and then - when nothing is left in directories such
> as /bin or /lib - there is no aliasing anymore, which is why we do not
> have to teach dpkg about aliasing and never patch it.

My understanding from following this thread (and others) is that dpkg
has a bug that can easily be triggered by a sysadmin replacing a
directory with a symlink (and some other necessary conditions that don't
happen very often), which is explicitly allowed by policy.  This bug is
the one that is causing the problem with the approach that was chosen by
the people implementing usrmerge, even though they were aware of this
problem and a different approach that would have taken two release
cycles and would not have triggered this bug was considered and
rejected.

If this is correct, then Luca's approach may fix the problem for
usrmerge, but does not fix the general dpkg bug.  (And, IIUC, is going
to take two _more_ release cycles to fix the problems with usrmerge as
implemented!  Hmm...)

The --add-alias solution that has been suggested in this thread seems
like it would fix the general problem iff policy was changed to require
sysadmins to use it if they replaced a directory with a symlink.

I do not understand why the dpkg maintainer has rejected this solution;
it would still be a fix for the general bug after the usrmerge
transition has completed.  And it would be at least one order of
magnitude more performant than scanning the filesystem for directory
symlinks.

...Marvin

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Fri, 28 Apr 2023 at 10:12, Luca Boccassi  wrote:
>
> On Fri, 28 Apr 2023 at 09:09, Helmut Grohne  wrote:
> > So yeah, with the exception of dash, this looks fairly good. Let me also
> > dive into dash. Unlike the majority of diverters, it diverts in postinst
> > rather than preinst to allow controlling /bin/sh via debconf. A similar
> > technique is in effect by gpr. In any case, this is special, because
> > dash diverts its own files, so when moving dash's file, its diversions
> > can be migrated at the same time. It merely means, that we cannot have
> > debhelper just move files (as that would horribly break dash) and
> > instead have to move files on a package-by-package way. We could also
> > opt for removing dash's diversion in the default case and there even is
> > a patch for doing so (#989632) since almost two years. Too bad we didn't
> > apply it. In any case, as long as the file moving is not forced via
> > debhelper, dash should be harmless.
>
> If I understand correctly, by "forced via debhelper" you mean the
> proposal of fixing the paths at build time, right? But if not via
> that, it means having to fix all of them by hand, which is a lot - is
> it possible to fix dash instead? Or else, we could add an opt-out via
> one of the usual dh mechanisms, and use it only in dash perhaps?

Also as pointed out by the maintainer the debconf machinery was
dropped from dash, last year:
https://salsa.debian.org/debian/dash/-/commit/c322a1c9fc6be11d7eb4439

This should hopefully simplify things?

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne]

Please excuse the volume of mails I am producing at this time, but I
still think this adds to the discussion.

On Thu, Apr 27, 2023 at 12:34:06AM +0200, Helmut Grohne wrote:
> I have a bad feeling about this. I think some dpkg maintainer warned us
> that diversions would break. Let's peek at his list again. He also said
> update-alternatives would be broken. I admit not having dug into this
> yet, but my gut feeling already is that update-alternatives will become
> "funny" as well though I guess we cannot fix update-alternatives by
> adding alternatives.

Sometimes, feelings can be wrong. I admit not having done much with
alternatives, so let me roughly summarize the relevant aspects and then
evaluate the aliasing effects.

Every alternative has an "alternative name". This name is the one inside
/etc/alternatives. When it refers to a binary in $PATH, usually that
binary name also is the name of the alternative.  Consequently, it is
very unlikely for us to have a conflict on this name due to aliasing.

Every alternative also has "generic name", which is the primary path of
the symlink installed. This can be /bin/something or /usr/bin/something
or something else entirely. I looked through the various paths I could
find in alternatives and never found two distinct alternatives using
aliased paths, so this much will very likely go without problems.

I looked around for possible non-canonical paths in alternatives (by
installing lots of packages and discovered these:
 * /bin/bsd-csh is shipped by csh and is provided as primary target for
   the csh alternative
 * /bin/csh is a generic name for the csh alternative
 * /bin/ed is shipped by ed and is provided as primary target for the
   editor alternative
 * /bin/elvis-tiny is shipped by elvis-tiny and is provided as primary
   target for the editor alternative
 * /bin/more is shipped by util-linux and is provided as primary target
   for the pager alternative
 * /bin/ksh is a generic name for the ksh alternative
 * /bin/ksh93 is shipped by ksh93u+m and is provided as primary target
   for the ksh alternative
 * /bin/mksh is shipped by mksh and is provided both as primary and as
   slave target for the ksh alternative
 * /bin/mksh-static is shipped by mksh and is provided both as primary
   and as slave target for the ksh alternative
 * /bin/mt is a generic name for the mt alternative
 * /bin/mt-gnu is shipped by cpio and is provided as primary target
   for the mt alternative
 * /bin/mt-st is shipped by mt-st and is provided as primary target
   for the mt alternative
 * /bin/nano is shipped by nano and is provided as primary target
   for the editor alternative
 * /bin/nano-tiny is shipped by nano-tiny and is provided as primary
   target for the editor alternative
 * /bin/nc is a generic name for the nc alternative
 * /bin/nc.openbsd is shipped by netcat-openbsd and is provided as
   primary target and slave target for the nc alternative
 * /bin/nc.traditional is shipped by netcat-traditional and is provided
   as primary target and slave target for the nc alternative
 * /bin/netcat is a slave link for the nc alternative
 * /bin/rc is a generic name for the rc alternative and installed by
   9base
 * /bin/rksh is a slave link for the ksh alternative
 * /bin/rksh93 is shipped by ksh93u+m and is provided as slave target
   for the ksh alternative
 * /bin/tcsh is shipped by tcsh and is provided as a primary target for
   the csh alternative
 * /bin/true is not shipped by uim, but installed as primary target for
   the uim-toolbar alternative
 * /lib/cpp is a generic name for the cpp alternative and installed by
   cpp
 * /lib/systemd/system/ipset.service is a generic name for the
   ipset.service alternative
 * /lib/systemd/system/iptables.service is a generic name for the
   iptables.service alternative
 * /lib/systemd/system/netfilter-persistent.service is shipped by
   iptables-persistent and provided as primary target and slave target
   for the iptables.service and ipset.service alternatives by
   iptables-persistent and ipset-persistent respectively
 * /lib/systemd/system/ip6tables.service is a slave link for the
   iptables.service alternative
 * /lib/firmware/regulatory.db is a generic name for the regulartory.db
   alternative
 * /lib/firmware/regulatory.db.p7s is a slave link for the
   regulatory.db alternative
 * /lib/firmware/regulatory.db.p7s-debian is shipped by wireless-regdb
   and is provided as slave target for the regulatory.db alternative
 * /lib/firmware/regulatory.db.p7s-upstream is shipped by wireless-regdb
   and is provided as slave target for the regulatory.db alternative
 * /lib/firmware/regulatory.db-debian is shipped by wireless-regdb and
   is provided as primary alternative for the regulatory.db alternative
 * /lib/firmware/regulatory.db-upstream is shipped by wireless-regdb and
   is provided as primary alternative for the regulatory.db alternative

I think ideally, we'd want these to become canonicalized eventually. And
this is w

Re: DEP 17: Improve support for directory aliasing in dpkg

[Russ Allbery]

Helmut Grohne  writes:
> On Thu, Apr 27, 2023 at 10:58:46AM +0200, Marc Haber wrote:

>> My gut feeling is that we are wasting prescious time of numerous
>> skilled Debian Developers to find ugly workarounds to something that
>> should be done in dpkg, but isnt being done because one dpkg maintainer
>> has decided to not go the way the project has decided to go.

> I find this mail of yours very disappointing and possibly even failing
> our Code of Conduct on multiple accounts.

I am unhappy to see the Code of Conduct used in this way.

Marc's message was not a personal attack.  It did not assume bad faith, or
indeed make any statements about motives at all.  He expressed his opinion
about project priorities and put it in the context of his personal
judgment of the facts of the situation as he sees them.

You may disagree with his summary of facts, or his opinion about or
evaluation of the current situation, or even the usefulness of him raising
this point due to lack of resources.  It is certainly appropriate to raise
those disagreements in response, or even to ignore the message if you
don't think it's a constructive line of discussion.  (In particular, I
think Marc assumes that a solution in dpkg would be more straightforward,
something that I think is debatable on technical grounds.)

But to say that this is possibly a violation of the Code of Conduct is to
say that this message doesn't meet the bar for civil discussion on our
lists, and I think it is unreasonable to expect anyone to be more civil or
even-handed than Marc was in his summary of behavior that he strongly
disagrees with.  (And, to state the obvious, I don't believe that message
was a violation of our Code of Conduct.)  Trying to set the bar higher
than this would have the effect of forbidding particular types of hard
conversations, which is not healthy for the project.

We have to be able to talk about interpersonal disagreements and problems
of alignment of motives and goals among the people working on the project.
Sometimes those discussions are going to be uncomfortable, but we can't
ignore them and never discuss them because they're uncomfortable.  We are
a collection of humans working together collaboratively, which means there
will be tension and conflict and we have to deal with that, constructively
but honestly and forthrightly.

Part of working collaboratively with other people is that those people get
to criticize how you are doing your work, as long as they do so
respectfully and assuming good faith.  Sometimes that includes saying that
one believes the actions of another developer are causing a misallocation
of project resources or time.  Whether or not we end up agreeing that is
true, this is a valid topic for discussion, and sometimes it is feedback
that other developers need to hear so that they can do some introspection
and evaluate whether that may indeed be the case.

-- 
Russ Allbery (r...@debian.org)  

Re: DEP 17: Improve support for directory aliasing in dpkg

[James Addison]

On Fri, 28 Apr 2023 at 17:52, Jochen Sprickerhof  wrote:
>
> Hi James,
>
> * James Addison  [2023-04-28 14:54]:
> >To make sure we don't miss any packages out accidentally: could you
> >confirm that those hundred-or-so errors occurred from 27 or so
> >distinct packages?
> >
> >(looking at RC bugs created within the past week, I currently find 27
> >bugs with 'Breaks+Replaces' in the title)
> >
> >https://udd.debian.org/bugs.cgi?release=na&merged=ign&keypackages=only&fnewer=only&fnewerval=7&flastmodval=7&rc=1&cpopcon=1&chints=1&ckeypackage=1&ctags=1&cdeferred=1&caffected=1&sortby=last_modified&sorto=asc&format=html#results
>
> That's only key packages. Here is the full list:
>
> https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;include=subject%3Amissing+Breaks%2BReplaces;submitter=helmut%40subdivi.de
>
> Cheers Jochen

Ah, typical user error from me.  Anyway - glad to find that they've
all been filed.  Thank you, Jochen!

Re: DEP 17: Improve support for directory aliasing in dpkg

[Jochen Sprickerhof]

Hi James,

* James Addison  [2023-04-28 14:54]:

To make sure we don't miss any packages out accidentally: could you
confirm that those hundred-or-so errors occurred from 27 or so
distinct packages?

(looking at RC bugs created within the past week, I currently find 27
bugs with 'Breaks+Replaces' in the title)

https://udd.debian.org/bugs.cgi?release=na&merged=ign&keypackages=only&fnewer=only&fnewerval=7&flastmodval=7&rc=1&cpopcon=1&chints=1&ckeypackage=1&ctags=1&cdeferred=1&caffected=1&sortby=last_modified&sorto=asc&format=html#results


That's only key packages. Here is the full list:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;include=subject%3Amissing+Breaks%2BReplaces;submitter=helmut%40subdivi.de

Cheers Jochen


signature.asc
Description: PGP signature