Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Fri, 2021-11-12 at 04:57 +0100, Guillem Jover wrote:
> 
> On Sun, 2021-08-22 at 11:21:38 +0100, Luca Boccassi wrote:
> > On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> > > On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> > > > The bug is real, nobody doubts that - it has been filed on dpkg 20
> > > > years ago.
> > > 
> > > You keep repeating this, but I have no idea what bug you refer to.
> > > 
> > > There's #148258 (from 2002), which is conffile related, and not
> > > actionable and should probably just be closed.
> > > 
> > > There's #182747 (from 2003), which while apparently similar is
> > > something else completely. This is about the (IMO) misfeature of
> > > supporting a local admin to redirect (not alias) a directory using a
> > > local symlink (mainly for space management reasons). For an
> > > explanation
> > > see .
> > > 
> > > There's #406715 (from 2007) which is related to the above misfeature.
> > 
> > I am referring to #134758 since it's linked as the root cause from
> > usrmerge's #848622.
> 
> Well, that's a bogus block then, because that's obviously not the root
> cause. I see I was CCed when that block was set, I guess I missed it.
> :/ Fixed it now…
> 
> > "dpkg-query: Make -S handle unowned symlinks resolving to owned
> > pathnames" filed in February 2002 - 19 years and a half ago. I refer to
> > that because it's linked as the root cause in the BTS of the relevant
> > issue with usrmerge we are discussing.
> 
> Even if the wishlist from that report got implemented, it would still
> not fully solve all the problems, where among them «dpkg-query -S»
> is probably the lesser one, which would not work in the other direction
> anyway (querying a path under /usr/ known to dpkg as being under /).
> 
> And then I'm not convinced this should even be implemented at all,
> as it would introduce behavior differences between literal pathnames
> and patterns, and making them slower (for the first case) or potentially
> extremely slower (for the second case), in addition to making the queries
> dependent on the on-disk layout (so unreliable from the packaging PoV,
> as it would invent on the spot, pathnames not truly coming from any
> package nor otherwise known to dpkg).
> 
> This for a misfeature in dpkg (supporting redirecting symlinks) that
> allowed the current mess anyway. So I'm inclined to wontfix and close
> that one.
> 
> 
> Not looking forward to further interactions…
> Guillem

Thanks for following up! There are currently 469 open bugs against
src:dpkg, it's of course entirely up to you which ones you choose to
fix and which one you close+wontfix. As far as workarounds go, this one
is really really trivial to deal with, so I personally wouldn't mind at
all.

Thank you for your work!

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Guillem Jover]

unblock 848622 by 134758
thanks

On Sun, 2021-08-22 at 11:21:38 +0100, Luca Boccassi wrote:
> On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> > On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> > > The bug is real, nobody doubts that - it has been filed on dpkg 20
> > > years ago.
> > 
> > You keep repeating this, but I have no idea what bug you refer to.
> > 
> > There's #148258 (from 2002), which is conffile related, and not
> > actionable and should probably just be closed.
> > 
> > There's #182747 (from 2003), which while apparently similar is
> > something else completely. This is about the (IMO) misfeature of
> > supporting a local admin to redirect (not alias) a directory using a
> > local symlink (mainly for space management reasons). For an
> > explanation
> > see .
> > 
> > There's #406715 (from 2007) which is related to the above misfeature.
> 
> I am referring to #134758 since it's linked as the root cause from
> usrmerge's #848622.

Well, that's a bogus block then, because that's obviously not the root
cause. I see I was CCed when that block was set, I guess I missed it.
:/ Fixed it now…

> "dpkg-query: Make -S handle unowned symlinks resolving to owned
> pathnames" filed in February 2002 - 19 years and a half ago. I refer to
> that because it's linked as the root cause in the BTS of the relevant
> issue with usrmerge we are discussing.

Even if the wishlist from that report got implemented, it would still
not fully solve all the problems, where among them «dpkg-query -S»
is probably the lesser one, which would not work in the other direction
anyway (querying a path under /usr/ known to dpkg as being under /).

And then I'm not convinced this should even be implemented at all,
as it would introduce behavior differences between literal pathnames
and patterns, and making them slower (for the first case) or potentially
extremely slower (for the second case), in addition to making the queries
dependent on the on-disk layout (so unreliable from the packaging PoV,
as it would invent on the spot, pathnames not truly coming from any
package nor otherwise known to dpkg).

This for a misfeature in dpkg (supporting redirecting symlinks) that
allowed the current mess anyway. So I'm inclined to wontfix and close
that one.


Not looking forward to further interactions…
Guillem

Re: merged /usr vs. symlink farms

[Andreas Metzler]

On 2021-08-26 Timo Röhling  wrote:
[...]
> However, Guillem also seems to think that dpkg can manage file
> symlinks in a real directory better than an directory symlinks in /,
> which is why he proposed symlink farms in the first place.

Hello,

Afaiui, the symlink farm would just work from dpkg's point of view, even
with dpkg from oldoldoldoldstable. It can handle symlinks to files just
fine.

usrmerge is conceptually a lot harder because dpkg needs to get to handle
the fact that /bin/foo and /usr/bin/foo conflict.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: merged /usr vs. symlink farms

[Timo Röhling]

Hi,

* Sam Hartman  [2021-08-26 08:56]:

That may not matter so much for Debian (at least in some people's minds)
but being compatible with the rest of the world has enough value in this
instance that I consider the issue moot.

I agree that this is a very convincing argument. A considerably
weaker supportive argument would also be that these symlinks are
here to stay for years, possibly even decades, which makes four or
five static symlinks much simpler to maintain long-term.

Still, I would prefer to know in advance if there are additional
technical concerns which need to be fixed or worked around.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Sam Hartman]

> "Timo" == Timo Röhling  writes:
Timo> However, Guillem also seems to think that dpkg can manage file
Timo> symlinks in a real directory better than an directory symlinks
Timo> in /, which is why he proposed symlink farms in the first
Timo> place.  Assuming dpkg implements the proposed
Timo> canonicalization, is there a scenario where the aliased dirs
Timo> break, but would not break with a symlink farm?

Quite possibly.
However, I haven't been examining that part of the design space because
the symlink farms approach misses key features that are important to
some usrmerge use cases.

In particular, if you're sharing a /usr between containers or similar,
and moving toward the empty /etc at boot ostree style approach, symlink
farms don't work and base level symlinks do work.

That may not matter so much for Debian (at least in some people's minds)
but being compatible with the rest of the world has enough value in this
instance that I consider the issue moot.

Symlink farms are a different thing, and that's not what we're trying to
get here.

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Thu, 2021-08-26 at 14:51 +0200, Simon Richter wrote:
> Hi,
> 
> On 8/26/21 1:17 PM, Luca Boccassi wrote:
> 
> > > Ideally the question whether a system works correctly would be a
> > > technical, not a political one that is based on a majority vote of
> > > people who do not look behind the facade.
> 
> > Precisely - and the correct technical question is, how many bug reports
> > were opened by users?
> 
> That is the majority vote by people who do not look behind the facade.

A vote is a conscious decision and arbitrary act. Your system breaking
is not.

> > Strawmen can always be constructed, for example
> > you can completely brick a system by running 'apt install bash:armhf'
> > but that doesn't mean multiarch should be scrapped and reverted, it
> > would be silly to suggest that - and yet here we are.
> 
> This is precisely the debate we would be having if there was a package 
> that enables armhf as a foreign architecture whenever it is installed: 
> it would not break systems on its own, but we'd have to be a lot more 
> careful in other packages about expressing dependencies.
> 
> I have armhf enabled on my laptop, and the resolver has suggested 
> replacing bash:amd64 with bash:armhf a few times -- but I know that my 
> setup is unusual.

It's exactly the same. This dpkg bug manifests itself via a carefully
crafted sequence of package upgrades/downgrades with a specific set of
metadata between them. If I uploaded a new version of iproute2 which
enabled armhf and installed bash:armhf the suggestion wouldn't be to
revert the debian multiarch implementation.

> > Conversely this
> > doesn't mean the dpkg database bug shouldn't be fixed,
> 
> Keep in mind that this isn't a dpkg bug, because dpkg never created a 
> filesystem layout that was inconsistent with its database; usrmerge did.
> 
> What we are talking about is adding a workaround for the usrmerge 
> package to dpkg, and how to make sure that it works, regardless of 
> whether the system has already been converted or not, if it has, which 
> version of usrmerge was used, whether the package is currently installed 
> or not, whether it will be installed as part of an upgrade and if so, 
> which state the dpkg package is in at the time it is installed.
> 
> For example, a possible scenario could be that the system was converted 
> with usrmerge version 9 (which modified dpkg.cfg), then the usrmerge 
> package was deinstalled as a later version declared a conflict that the 
> user resolved in favour of the other package, usrmerge hasn't been 
> installed since, then during the upgrade to bookworm, a new version of 
> dpkg is unpacked (but not configured yet), then a new version of 
> usrmerge is unpacked, and then dpkg --configure -a is run.
> 
> So we need a list of things that need to happen to get back into a 
> consistent state and a way of sequencing that inside the package 
> dependency DAG.

This is 100% a dpkg bug. rpm doesn't have it. Other package managers
don't have it. And it's fine - all software has bugs.

> > so that the
> > replace+move bug can't happen in the future, but it simply makes
> > overblown and hyperbolic ideas such as "all systems are broken, revert
> > everything" and "let's cancel the TC decision" appear counter-
> > productive and deeply unhelpful toward finding an actual, realistic
> > solution.
> 
> The TC decision does not specify technical details, which was a grave 
> mistake because the implementation was so shoddy that I doubt the TC 
> would have signed off on that, had they been consulted on the details.

You are missing a gigantic 'IMHO'. For me the implementation was great
- simple, to the point, matching other distros so that we don't get
yet-another-incompatibility for the sake of being 'special', and
working just fine across millions of installations for 2+ years and
counting with hardly a hiccup. Of course it has bugs, because it's
software, or more precisely yet it makes old bugs in other packages
(dpkg) come to light, which happens and it's not the end of the world -
we'll fix them or work around them, like the tens of thousands of other
bugs affecting debian. Also, from what we have heard "unofficially"
here and elsewhere, it appears that the TC has always indeed preferred
the current approach of matching other distros and symlinking the top
dirs, rather than the objectively failed symlinks farm approach.

> Pretty much no one is interested in rolling this back, but a lot of 
> people, me included, are indifferent and cannot see the benefits, 
> especially as they consist mainly of things that used to work in the 
> past, were broken during the systemd rollout and subsequently declared 
> out of scope when people complained, so all users of these features have 
> already left anyway.
> 
> An actual, realistic solution will work for all users, including those 
> who have now installed or upgraded to buster from DVD, run offline and 
> will upgrade to bookworm from DVD when it comes out.

You say 

Re: merged /usr vs. symlink farms

[Timo Röhling]

* Simon Richter  [2021-08-26 14:51]:
It makes a lot more sense to have a dpkg-internal tool that can 
investigate the differences between the file system and the dpkg 
database, resolve conflicts (possibly in an interactive process when 
required by a corner case like the one I mentioned earlier -- luckily, 
these should be really rare) and then leave a consistent state again 
that allows package maintainers to use all dpkg features again after 
the bookworm release.

Agreed. Having yet another tool messing with dpkg "behind its back"
will only exacerbate the problems we're trying to solve.

Also, I wouldn't say it is the *only* possible way to move forward,
but it is *a* possible way. I am also still trying to understand
Guillem position and his objections better.

This proposal addresses one half on Guillem's objections by making
dpkg's point of view consistent with the actual filesystem reality. 


However, Guillem also seems to think that dpkg can manage file
symlinks in a real directory better than an directory symlinks in /,
which is why he proposed symlink farms in the first place.  Assuming
dpkg implements the proposed canonicalization, is there a scenario
where the aliased dirs break, but would not break with a symlink
farm?

My apologies if this has been answered already. Just point me to the
relevant emails/wiki pages then.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 8/26/21 1:17 PM, Luca Boccassi wrote:


Ideally the question whether a system works correctly would be a
technical, not a political one that is based on a majority vote of
people who do not look behind the facade.



Precisely - and the correct technical question is, how many bug reports
were opened by users?


That is the majority vote by people who do not look behind the facade.


Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are.


This is precisely the debate we would be having if there was a package 
that enables armhf as a foreign architecture whenever it is installed: 
it would not break systems on its own, but we'd have to be a lot more 
careful in other packages about expressing dependencies.


I have armhf enabled on my laptop, and the resolver has suggested 
replacing bash:amd64 with bash:armhf a few times -- but I know that my 
setup is unusual.



Conversely this
doesn't mean the dpkg database bug shouldn't be fixed,


Keep in mind that this isn't a dpkg bug, because dpkg never created a 
filesystem layout that was inconsistent with its database; usrmerge did.


What we are talking about is adding a workaround for the usrmerge 
package to dpkg, and how to make sure that it works, regardless of 
whether the system has already been converted or not, if it has, which 
version of usrmerge was used, whether the package is currently installed 
or not, whether it will be installed as part of an upgrade and if so, 
which state the dpkg package is in at the time it is installed.


For example, a possible scenario could be that the system was converted 
with usrmerge version 9 (which modified dpkg.cfg), then the usrmerge 
package was deinstalled as a later version declared a conflict that the 
user resolved in favour of the other package, usrmerge hasn't been 
installed since, then during the upgrade to bookworm, a new version of 
dpkg is unpacked (but not configured yet), then a new version of 
usrmerge is unpacked, and then dpkg --configure -a is run.


So we need a list of things that need to happen to get back into a 
consistent state and a way of sequencing that inside the package 
dependency DAG.



so that the
replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken, revert
everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic
solution.


The TC decision does not specify technical details, which was a grave 
mistake because the implementation was so shoddy that I doubt the TC 
would have signed off on that, had they been consulted on the details.


Pretty much no one is interested in rolling this back, but a lot of 
people, me included, are indifferent and cannot see the benefits, 
especially as they consist mainly of things that used to work in the 
past, were broken during the systemd rollout and subsequently declared 
out of scope when people complained, so all users of these features have 
already left anyway.


An actual, realistic solution will work for all users, including those 
who have now installed or upgraded to buster from DVD, run offline and 
will upgrade to bookworm from DVD when it comes out.



Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other day,
namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?


No. We couldn't sequence that correctly, we'd have to somehow make 
installation of the usrmerge package mandatory, and that tool would need 
to touch the dpkg database at a point where dpkg is active in the 
background.


It makes a lot more sense to have a dpkg-internal tool that can 
investigate the differences between the file system and the dpkg 
database, resolve conflicts (possibly in an interactive process when 
required by a corner case like the one I mentioned earlier -- luckily, 
these should be really rare) and then leave a consistent state again 
that allows package maintainers to use all dpkg features again after the 
bookworm release.


That will be a lot of work, and we cannot speed it up anyway because 
we're tied to the release cycle here, so I'd rather do this properly 
than deploy another paper mâché thing that then turns out to have 
another weird bug that will delay the transition to bookworm+2.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Thu, 2021-08-26 at 13:50 +0200, Philip Hands wrote:
> Luca Boccassi  writes:
> 
> > On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:
> > > Hi,
> > > 
> > > On 8/26/21 8:38 AM, Marco d'Itri wrote:
> > > 
> > > > > By my definition, these have never been working correctly, but
> > > > > semantics I guess.
> > > 
> > > > It is not semantics. You keep saying that countless Debian and Ubuntu
> > > > systems are not working correctly, but since this obviously does not
> > > > reflect the experience of the owners of these systems then just about
> > > > everybody will believe that you are wrong about merged-/usr.
> > > 
> > > Ideally the question whether a system works correctly would be a 
> > > technical, not a political one that is based on a majority vote of 
> > > people who do not look behind the facade.
> > > 
> > > Simon
> > 
> > Precisely - and the correct technical question is, how many bug reports
> > were opened by users?
> 
> While I'm sympathetic with the argument that a lack of bug reports
> suggests that the theoretical problems (that I hope we all agree, do
> exist) seem not to be exhibiting themselves in the wild, it is very far
> from proof.
> 
> If some random file disappears on your system one day, what happens?
> 
> Most likely, nothing, and you never notice.
> 
> Possibly, your system starts to misbehave.  What do you do then?
> 
>   If you're a naive user, such as a recent arrival from Windows, then
>   misbehaviour is something that you've been trained to expect and you
>   install from scratch.  The idea of reporting a bug never enters your
>   head.
> 
>   If you're aware that this sort of thing really should not happen, then
>   what happens next is probably down to how busy you are.  If you're
>   busy, you probably check your SMART stats, and having convinced
>   yourself that the disks are OK, either restore from backup and check
>   for what ealse is missing, or use debsums to see the extent of the
>   damage and reinstall a package or two.  Again, since you're only
>   trying to get back to work, and didn't track down the cause, no bug
>   report.
> 
> The only bug reports you're going to get about this are either going to
> be the useless "Something didn't work" bugs, that I doubt would ever get
> tied to this cause, or the ones submitted by experienced, diligent, and
> time-rich bug reporters (which is a rather rare combination).
> 
> The fact that we don't see bug reports should surprise no one.
> 
> Cheers, Phil.

Actually in unstable (not in a release) this bug was observed, caught,
properly reported and fixed: https://bugs.debian.org/953562 which would
suggest these instances can and have been indeed detected when they
happened - but anyway, that's why the full quote without the cut is:

> Precisely - and the correct technical question is, how many bug
> reports
> were opened by users? Strawmen can always be constructed, for example
> you can completely brick a system by running 'apt install bash:armhf'
> but that doesn't mean multiarch should be scrapped and reverted, it
> would be silly to suggest that - and yet here we are. Conversely this
> doesn't mean the dpkg database bug shouldn't be fixed, so that the
> replace+move bug can't happen in the future, but it simply makes
> overblown and hyperbolic ideas such as "all systems are broken,
> revert
> everything" and "let's cancel the TC decision" appear counter-
> productive and deeply unhelpful toward finding an actual, realistic
> solution.

IE, it's fine to think about all the possible ways things can go wrong,
but the solutions need to be proportionate. So proposing to trash
everything and going against the TC decision in absence of any user
report and any knowledge from anybody of an actual breakage in
buster/bullseye/19.10/20.04/20.10/21.04 is as far from proportionate as
one can possibly be, just as suggesting to revert multiarch because
'apt install bash:ppc64el' breaks an amd64 machine beyond repair would
be completely unreasonable. Am I suggesting the replaces+move issue
should be ignored? No, hence the second part of the quote that was cut:

> Given this and the last few mails, it seems to me at this point the
> only possible way forward is what Timo and Ted suggested the other
> day,
> namely to have an external tool, possibly part of src:usrmerge, scan
> the dpkg database and fix it? Possibly on a trigger?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Philip Hands]

Luca Boccassi  writes:

> On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:
>> Hi,
>> 
>> On 8/26/21 8:38 AM, Marco d'Itri wrote:
>> 
>> > > By my definition, these have never been working correctly, but
>> > > semantics I guess.
>> 
>> > It is not semantics. You keep saying that countless Debian and Ubuntu
>> > systems are not working correctly, but since this obviously does not
>> > reflect the experience of the owners of these systems then just about
>> > everybody will believe that you are wrong about merged-/usr.
>> 
>> Ideally the question whether a system works correctly would be a 
>> technical, not a political one that is based on a majority vote of 
>> people who do not look behind the facade.
>> 
>> Simon
>
> Precisely - and the correct technical question is, how many bug reports
> were opened by users?

While I'm sympathetic with the argument that a lack of bug reports
suggests that the theoretical problems (that I hope we all agree, do
exist) seem not to be exhibiting themselves in the wild, it is very far
from proof.

If some random file disappears on your system one day, what happens?

Most likely, nothing, and you never notice.

Possibly, your system starts to misbehave.  What do you do then?

  If you're a naive user, such as a recent arrival from Windows, then
  misbehaviour is something that you've been trained to expect and you
  install from scratch.  The idea of reporting a bug never enters your
  head.

  If you're aware that this sort of thing really should not happen, then
  what happens next is probably down to how busy you are.  If you're
  busy, you probably check your SMART stats, and having convinced
  yourself that the disks are OK, either restore from backup and check
  for what ealse is missing, or use debsums to see the extent of the
  damage and reinstall a package or two.  Again, since you're only
  trying to get back to work, and didn't track down the cause, no bug
  report.

The only bug reports you're going to get about this are either going to
be the useless "Something didn't work" bugs, that I doubt would ever get
tied to this cause, or the ones submitted by experienced, diligent, and
time-rich bug reporters (which is a rather rare combination).

The fact that we don't see bug reports should surprise no one.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:
> Hi,
> 
> On 8/26/21 8:38 AM, Marco d'Itri wrote:
> 
> > > By my definition, these have never been working correctly, but
> > > semantics I guess.
> 
> > It is not semantics. You keep saying that countless Debian and Ubuntu
> > systems are not working correctly, but since this obviously does not
> > reflect the experience of the owners of these systems then just about
> > everybody will believe that you are wrong about merged-/usr.
> 
> Ideally the question whether a system works correctly would be a 
> technical, not a political one that is based on a majority vote of 
> people who do not look behind the facade.
> 
> Simon

Precisely - and the correct technical question is, how many bug reports
were opened by users? Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are. Conversely this
doesn't mean the dpkg database bug shouldn't be fixed, so that the
replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken, revert
everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic
solution.

Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other day,
namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 8/26/21 8:38 AM, Marco d'Itri wrote:


By my definition, these have never been working correctly, but
semantics I guess.



It is not semantics. You keep saying that countless Debian and Ubuntu
systems are not working correctly, but since this obviously does not
reflect the experience of the owners of these systems then just about
everybody will believe that you are wrong about merged-/usr.


Ideally the question whether a system works correctly would be a 
technical, not a political one that is based on a majority vote of 
people who do not look behind the facade.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Marco d'Itri]

On Aug 26, Guillem Jover  wrote:

> By my definition, these have never been working correctly, but
> semantics I guess.
It is not semantics. You keep saying that countless Debian and Ubuntu 
systems are not working correctly, but since this obviously does not 
reflect the experience of the owners of these systems then just about 
everybody will believe that you are wrong about merged-/usr.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Danilo Santos]

Today's update, Debian test can't read my windows partition. I fix it
inside the bios configuration.

El mié, 25 de ago. de 2021 a la(s) 15:27, Aurelien Jarno (
aurel...@aurel32.net) escribió:

> On 2021-08-20 23:15, Simon Richter wrote:
> > I think that one of the release goals should be that any freshly
> installed
> > or upgraded system should have a dpkg database that is consistent with
> > reality, and I'd prioritize that higher than actually finishing the
> > transition, because as long as we can have files vanish from under us, we
> > can't expect that the transition will be reliable for bullseye->bookworm
> > updates.
> >
> > Basically, we've been lucky so far.
>
> I am not sure we have been so lucky. Problems have been found during the
> release cycles, some of them got fixed (#953562), some other are still
> there (#926699).
>
> --
> Aurelien Jarno  GPG: 4096R/1DDD8C9B
> aurel...@aurel32.net http://www.aurel32.net
>


-- 
*Grupo Santos Frias S.R.L*
*Danilo Alejandro Santos Frias*
*Cel: (809) 708-7460*

Re: merged /usr vs. symlink farms

[Guillem Jover]

Hi!

On Sun, 2021-08-22 at 09:18:25 +0200, Andreas Metzler wrote:
> Afaict we have still no idea on how to move on.
> 
> 1 I think you agree that there is a significant number of usrmerged Debian
>   installations out there. It does not really matter whether there are 7% or
>   40%. They exist and should (keep) working.

By my definition, these have never been working correctly, but
semantics I guess. My wish would be to indeed salvage those systems,
that's why I implemented dpkg-fsys-usrunmess. But if people refuse,
then at some point I'd personally consider them lost, TBH. :/

> 2 As you have stated there are known issues with dpkg and usrmerged
>   systems. Some of them are are triggered by moving files from / to /usr.
> 
> Starting from here it is hard to see a way forward.

Well, in my mind the first and most immediate action that would be
done, is to stop the bleeding, by:

  - reverting the changes in deboostrap in sid, bullseye (and ideally
in buster too),
  - reverting the notion that split-/usr is unsupported (which includes
the extremely confusing interpretation about this applying to
sid/testing too), and update documentation such as release-notes,
etc.,
  - adding a Conflicts against usrmerge somewhere (dpkg, base-files or
similar) for extra caution,
  - (optionally) removing usrmerge from buster, bullseye and sid,

I'd be ready to prepare patches or file reports for any of these.

But then, my expectations regarding Debian have plummeted so…

> Is it a realistic option to require something like
> -
> dpkg --purge usrmerge
> dpkg-fsys-usrunmess
> -
> pre dist-upgrade to bookworm to get back all systems to a sane (from
> dpkg POV) state and start fresh?

If by requiring you mean documenting the requirement, then I don't
think that would be enough.

But I could see introducing a preinst somewhere (perhaps base-files,
to permit upgrading dpkg itself) which would check for those unsupported
systems and emit a big fat message explaining the situation and
instructions to follow to be able to proceed, then abort. Similar in vein
with how other preventions (AFAIR) have been implemented for example on
glibc or udev for environment, multi-arch or kernel requirements.

Sadly, even introducing that into bullseye would not guarantee that the
bookworm upgrade would already be in a sane state, as there's no
guarantee people do not skip point releases. But I assume it would cover
most of the installations. So the problem would be greatly minimized.
Only after the bookworm upgrade has been completed this would be
guaranteed.

> Is this robust (except for crash as stated in the manpage), or would
> you consider it beta-quality?

There's a known problematic interaction with a systemd's emergency mode
misbehavior/bug, which I've worked around, plus some other robustness
improvements I've prepared that will be included in 1.21.0, and targeted
for 1.20.10. I should probably also add a temporary policy-rc.d to
batch and defer all service actions for after the mass reconfiguration,
which I'll try to code up soonish to further improve robustness.

Otherwise I've taken great care and consideration on trying to make it
as robust as possible, but I guess out of cautiousness and given the
nature of what the script is doing (which is still a hack, although a
self-contained one at least), I'd be hard pressed to probably ever
declare it 100% safe. :/ Of course more testing is always helpful!

Thanks,
Guillem

Re: merged /usr vs. symlink farms

[Russ Allbery]

Guillem Jover  writes:

> The fact that the supporters of a *filesystem layout* have been happy to
> dismiss and ignore this and have been pushing for what I think can be
> easily described as the worst ever "transition" done in Debian, very
> sadly, for me this whole topic marks a before and after in Debian, and
> has put my trust in the technical side of the project into question.

I agree that you've been pointing out potential problems for years and
that your warnings have been at least partly vindicated by the problems we
are indeed seeing.

My understanding (which may be entirely incorrect, in which case please
let me know what your preferred solution actually is!) is that your
preferred solution was to migrate individual packages.  What I'm hearing
from the frequent threads in debian-devel is that a sufficient number of
Debian contributors have rejected that approach (for various reasons) as
to make that solution not viable.  In other words, my reading of the
consensus is that this option has effectively been vetoed by the rest of
the project (noting that this veto was certainly not unanimous).

Please note that I'm not taking a position on the merits of that decision,
simply noting that, based on my reading of debian-devel, this is has what
has happened, and I do not believe it will be possible at this point to
convince the project as a whole to unwind usrmerge and go back to doing
individual package migrations.

Given that as a design constraint (we will not be doing this transition
via one-by-one changes to each package), what would you support as a good
architectural solution to this transition?  Even ruling out that approach,
the design space seems large and flexible; surely there must be some
coherent way of doing this transition that does not require individual
action for each affected package and preserves some of the "be done with
it" design goals of the current usrmerge approach while avoiding the
problems you have pointed out.

I think it's obvious that any such design will require support from dpkg.

You're very understandably upset that people are insisting on a solution
that you feel is clearly incorrect, but I think that's also how the people
who are disagreeing with you are feeling, and as long as that's true on
both sides it's hard to see how we're going to arrive at a solution that
isn't going to make you even more unhappy.  In order to break this
deadlock, I think we have to have a design discussion in the shared space
of mutually agreeable solutions and not (on all sides) retreat back to a
single preferred architectural decision and only point out the problems
with any other approach.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Russ Allbery]

Simon Richter  writes:

> I'd expand the definition of Conflicts/Replaces though: packages that
> use names that conflict because of usrmerge would need to declare it,
> because as soon as we teach dpkg to recognize these conflicts, the
> packages would fail to install on stable.

Yes, that's probably the most appropriate place to make a change to Policy
to clarify this.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 25.08.21 18:57, Russ Allbery wrote:


The problem here is also that if there are two packages like that, on an
usrmerge system, we would not know this is happening.



I agree, of course, but I don't see a way in which Policy can help with
that problem unless this packaging decision was intentional and the person
who made that decision would have chosen otherwise if Policy had said to
not do it.


I'd expand the definition of Conflicts/Replaces though: packages that 
use names that conflict because of usrmerge would need to declare it, 
because as soon as we teach dpkg to recognize these conflicts, the 
packages would fail to install on stable.



This seems more like an appropriate check for an archive-wide QA tool
looking for cross-package problems.


We have a tool that looks for missing Conflicts/Replaces, and that tool 
needs to be extended as well.


   Simon

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Wed, Aug 25, 2021 at 09:57:09AM -0700, Russ Allbery wrote:
> Wouter Verhelst  writes:
> > On Mon, Aug 23, 2021 at 08:23:50AM -0700, Russ Allbery wrote:
> 
> >> If we tried to document every random bit of buggy packaging behavior
> >> anyone thought of in Policy, Policy would become unwieldy, so I want to
> >> verify here that someone really thought having one package containing a
> >> file in /bin and another package containing the same file in /usr/bin
> >> was was a reasonable thing to do (as opposed to accidental).  Are there
> >> packages in the archive like this?  Or could you point me at the
> >> message in the thread that said this was non-buggy?  I think I missed
> >> it.
> 
> > The problem here is also that if there are two packages like that, on an
> > usrmerge system, we would not know this is happening.
> 
> I agree, of course, but I don't see a way in which Policy can help with
> that problem unless this packaging decision was intentional and the person
> who made that decision would have chosen otherwise if Policy had said to
> not do it.
>
> This seems more like an appropriate check for an archive-wide QA tool
> looking for cross-package problems.

Indeed, and that was the point I was trying to make: it's not something
Policy can help with, unless it is intentional (and it is my belief that
this is not likely to be the case).

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: merged /usr vs. symlink farms

[Aurelien Jarno]

On 2021-08-20 23:15, Simon Richter wrote:
> I think that one of the release goals should be that any freshly installed
> or upgraded system should have a dpkg database that is consistent with
> reality, and I'd prioritize that higher than actually finishing the
> transition, because as long as we can have files vanish from under us, we
> can't expect that the transition will be reliable for bullseye->bookworm
> updates.
> 
> Basically, we've been lucky so far.

I am not sure we have been so lucky. Problems have been found during the
release cycles, some of them got fixed (#953562), some other are still
there (#926699).

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Wed, 2021-08-25 at 09:57:09 -0700, Russ Allbery wrote:
> Wouter Verhelst  writes:
> > The problem here is also that if there are two packages like that, on an
> > usrmerge system, we would not know this is happening.

Also this does not need to come from "buggy" packaging practices.

> I agree, of course, but I don't see a way in which Policy can help with
> that problem unless this packaging decision was intentional and the person
> who made that decision would have chosen otherwise if Policy had said to
> not do it.

> This seems more like an appropriate check for an archive-wide QA tool
> looking for cross-package problems.

I've said this many times over the months (years!?), that while this
can easily affect stuff from the archive, which we do control, where
policy applies and where we could try to poorly reimplement the checks
that dpkg does to detect them in some QA checker, even though the
following problems would still apply:

  - the checks cannot be performed (only) over static snapshots of
the archive, as this would affect partial upgrades too,
  - the checks would need to take into account packages we have
stopped shipping way in the past as those can linger around
installed,
  - the checks would have a hard time with the non-declarative side
of the packaging stack,

is that something else I expect to have a non-zero chance to bite users
are all those common practices that people give for granted and that we
have supposedly supported in the past, as guaranteed by our packaging
system, such as:

  - keeping installed packages that have stopped shipping in Debian,
  - installation of packages from third-parties, or from local
overlays or similar, or even rebuilt forks of packages from Debian,
  - installation or holding of Debian packages from older releases,
  - local diversions or alternatives,

which are out of our QA reach.


The fact that the supporters of a *filesystem layout* have been happy
to dismiss and ignore this and have been pushing for what I think can
be easily described as the worst ever "transition" done in Debian, very
sadly, for me this whole topic marks a before and after in Debian, and
has put my trust in the technical side of the project into question.

Of course some of those supporters are now agreeing these problems can
be insidious, progress I guess. And at the same time others are claiming
that acknowledging those problems would hold back the entire distribution,
while others are now saying we should stop doing packaging changes because,
well, these problems perhaps are potentially problematic, the irony.

And then we get people raving over what's the worst and most atrocious
hacks to pile on into dpkg to try to workaround the actual root cause
(turtles^Wnasty hacks and kludges all the way down I guess). Which I
obviously expect to eventually be coerced into merging into dpkg at
some point or another via our esteemed Authority.

From where I'm sitting Debian is the project that lost its way…

Sadly,
Guillem

Re: merged /usr vs. symlink farms

[Russ Allbery]

Wouter Verhelst  writes:
> On Mon, Aug 23, 2021 at 08:23:50AM -0700, Russ Allbery wrote:

>> If we tried to document every random bit of buggy packaging behavior
>> anyone thought of in Policy, Policy would become unwieldy, so I want to
>> verify here that someone really thought having one package containing a
>> file in /bin and another package containing the same file in /usr/bin
>> was was a reasonable thing to do (as opposed to accidental).  Are there
>> packages in the archive like this?  Or could you point me at the
>> message in the thread that said this was non-buggy?  I think I missed
>> it.

> The problem here is also that if there are two packages like that, on an
> usrmerge system, we would not know this is happening.

I agree, of course, but I don't see a way in which Policy can help with
that problem unless this packaging decision was intentional and the person
who made that decision would have chosen otherwise if Policy had said to
not do it.

This seems more like an appropriate check for an archive-wide QA tool
looking for cross-package problems.

(That said, the molly-guard example does seem to indicate that at least
one packager in the past did not realize this would be a problem.)

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Mon, Aug 23, 2021 at 08:23:50AM -0700, Russ Allbery wrote:
> Luca Boccassi  writes:
> 
> > Thank you - it has been brought up in this thread as an example of a
> > valid setup, so if it is not, I think it could be good to be extra clear
> > in the policy? How about the following:
> 
> If we tried to document every random bit of buggy packaging behavior
> anyone thought of in Policy, Policy would become unwieldy, so I want to
> verify here that someone really thought having one package containing a
> file in /bin and another package containing the same file in /usr/bin was
> was a reasonable thing to do (as opposed to accidental).  Are there
> packages in the archive like this?  Or could you point me at the message
> in the thread that said this was non-buggy?  I think I missed it.

The problem here is also that if there are two packages like that, on an
usrmerge system, we would not know this is happening.

Let's say there's a package "foo" which installs /usr/bin/foo, and an
package "binfoo" which installs "/bin/foo". In the current situation,
dpkg would not know that the two files are equivalent, and would happily
overwrite /usr/bin/foo with /bin/foo if "binfoo" was installed after
"foo".

Then when the user notices the "foo" program is not doing what they
believe it should be doing and runs "reportbug /usr/bin/foo", reportbug
will file the bug against the package "foo" rather than the package
"binfoo" which is the actual package whose binary they are trying to
use.

In contrast, if foo and binfoo both install "/bin/foo" (or both install
"/usr/bin/foo", either way works), then dpkg will complain at
installation time that one of the two packages tries to overwrite a file
from the other and refuse to continue.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Simon McVittie]

On Tue, 24 Aug 2021 at 10:46:45 -0400, Theodore Ts'o wrote:
> the definition of usrmerged is
> relatively well understood (symlinks for /{bin,lib,sbin} to
> /usr/{bin,lib,sbin})

For completeness: also /libQUAL to usr/libQUAL, for each libQUAL
that either participates in multilib or contains ld.so(8). See
setup_merged_usr() in debootstrap/functions or directories_to_merge()
in usrmerge/convert-usrmerge.

(For x86_64 users this means lib32, lib64 and libx32.)

smcv

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Theodore Ts'o]

On Tue, Aug 24, 2021 at 11:57:27AM +0200, Simon Richter wrote:
> Hi,
> 
> On 8/24/21 2:48 AM, Theodore Ts'o wrote:
> 
> > So in theory, if we had a program which looked for the top-level
> > symlinks /{bin,lib,sbin} -> /usr/{bin,lib,sbin}, and if they exist,
> > scans dpkg database is scanned looking for of the form
> > /{bin,lib,sbin}/$1, and updates them with /usr/{bin,lib,sbin}/$1, and
> > then in the future, if dpkg sees the top-level symlink, canonicalizes
> > any files referenced in the packages to /usr/{bin,lib,sbin}/$1, with a
> > fallback searching for /{bin,lib,sbin}/$1 in the file system, this
> > would solve the problem.
> 
> Yes. To apply the transformation, this would likely have to happen in the
> dpkg package itself, so the one-time transformation is applied only when
> dpkg can maintain the workaround from that point on.

Sure, since we're talking about dpkg database surgery, it's better
that the sources of such a program be located in the dpkg sources, and
regardless of who does the work, the dpkg maintainers should be
involved in the code review of any such change.

> That is the half that is missing from my proposal, as I was focusing on how
> to transition non-usrmerged systems from within dpkg.

I've been more focused on the usrmerged systems since nearly all of my
personal systems are already usrmerged, since I tend to reinstall my
systems whenver I upgrade my hardware, and deboostrap has installing
systems with the usrmerge top-level symlinks since Buster.
Furthermore, people have been arguing strenuously that the possibility
of file loss is real for these usrmerged systems, so fixing this
seemed to be high priority, regardless of how many systems use the
usrmerge setup.

So protecting against lost files was higher priority in my mind,
whether the people arguing that it's rare because Ubuntu users having
been losing files, or not, I accept the argument that if it can happen
(and you've demonstrated via adversarial test cases that it *can*), we
should fix that bug, whether it's considered an RC bug or not.

(Although my personal belief is that we should be a lot more open to
fixing less critical bugs in stable releases, so if we have a fix, I'd
be all for rolling it out early to Bullseye regardless of whether it's
considered "RC" or not.)

> > Let's ignore how we would deploy this helper program and the update
> > dpkg from a stable upgrade perspective, but in terms of preventing
> > potential problems during the testing window, getting an update to
> > dpkg which included the database fixup program and which ran from the
> > maintainer script would be a potential solution path.
> 
> Yes-ish. We'd also need to make sure that installing the usrmerge package
> after that dpkg upgrade does not make things inconsistent again, so it would
> make sense for the dpkg source package to provide a "usrmerge" package that
> is well integrated, and for dpkg to conflict with older versions of
> usrmerge.

I'm less worried about whether usrmerge is part of dpkg or not, since
most of my usrmerged systems are due to them being reinstalled since
Debian Buster has been around, and the definition of usrmerged is
relatively well understood (symlinks for /{bin,lib,sbin} to
/usr/{bin,lib,sbin}).  But it wouldn't hurt for dpkg to provide its
own "usrmerge" functionality, and I certainly wouldn't argue against it.

> > Furthermore, if dpkg knew to always canonicalize filenames from
> > /{bin,lib,sbin}/XXX to /usr/{bin,lib,sbin}/XXX when adding to the
> > database, and when it looked for files in the file system looked first
> > in /usr/{bin,lib,sbin}/XXX, with a fallback to /usr/{bin,lib,sbin}/XXX
> > the file names in the package would not need to change all, since all
> > of the magic fixup work would be happening inside dpkg.
> 
> Yes. In an ideal world, we wouldn't hardcode the list of symlinks in dpkg
> though, that's why I proposed shipping symlinks in base-files and having
> dpkg recognize the symlink-vs-directory conflict as intent to move a
> filesystem tree around.

That's certainly the more general solution, although again, I think
the definition of usrmerge is well understood, especially since Debian
has been on the trailing edge of the usrmerge transition across the
Linux ecosystem, and so the high priority should be for fixing the
specific case of moving the contents to /{bin,lib,sbin} to
/usr/{bin,lib,sbin} and leaving symlinks behind for the directories.

If we want to support people who want to say, move /usr/bin to
/u1/bin, and /usr/lib to /u2/lib, or even /usr/lib/X11 to
/funky/dir/X11, great!  Personally, I wouldn't consider that a high
priority item on the requirements list, though, unless it comes
essentially for free.

Cheers,

- Ted

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Simon Richter]

Hi,

On 8/24/21 2:48 AM, Theodore Ts'o wrote:


So in theory, if we had a program which looked for the top-level
symlinks /{bin,lib,sbin} -> /usr/{bin,lib,sbin}, and if they exist,
scans dpkg database is scanned looking for of the form
/{bin,lib,sbin}/$1, and updates them with /usr/{bin,lib,sbin}/$1, and
then in the future, if dpkg sees the top-level symlink, canonicalizes
any files referenced in the packages to /usr/{bin,lib,sbin}/$1, with a
fallback searching for /{bin,lib,sbin}/$1 in the file system, this
would solve the problem.


Yes. To apply the transformation, this would likely have to happen in 
the dpkg package itself, so the one-time transformation is applied only 
when dpkg can maintain the workaround from that point on.


That is the half that is missing from my proposal, as I was focusing on 
how to transition non-usrmerged systems from within dpkg.



Let's ignore how we would deploy this helper program and the update
dpkg from a stable upgrade perspective, but in terms of preventing
potential problems during the testing window, getting an update to
dpkg which included the database fixup program and which ran from the
maintainer script would be a potential solution path.


Yes-ish. We'd also need to make sure that installing the usrmerge 
package after that dpkg upgrade does not make things inconsistent again, 
so it would make sense for the dpkg source package to provide a 
"usrmerge" package that is well integrated, and for dpkg to conflict 
with older versions of usrmerge.



Furthermore, if dpkg knew to always canonicalize filenames from
/{bin,lib,sbin}/XXX to /usr/{bin,lib,sbin}XXX when adding to the
database, and when it looked for files in the file system looked first
in /usr/{bin,lib,sbin}/XXX, with a fallback to /usr/{bin,lib,sbin}/XXX
the file names in the package would not need to change all, since all
of the magic fixup work would be happening inside dpkg.


Yes. In an ideal world, we wouldn't hardcode the list of symlinks in 
dpkg though, that's why I proposed shipping symlinks in base-files and 
having dpkg recognize the symlink-vs-directory conflict as intent to 
move a filesystem tree around.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Timo Röhling]

Hi,

* Theodore Ts'o  [2021-08-23 20:48]:

I want to ask a potentially stupid question.
[...]

This is pretty much what I was wondering about in
https://lists.debian.org/debian-devel/2021/08/msg00372.html
You, however, phrased it much more eloquently than I could.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Making the dpkg database correspond with reality (Was Re: merged /usr vs. symlink farms)

[Theodore Ts'o]

I want to ask a potentially stupid question.

As I understand things, the problem is that in a usrmerge'd file
system where we have the top-level symlinks /{bin,lib,sbin} which
point at /usr/{bin,lib,sbin}, the problem is if we have a package
which contains the file in /sbin/blart, it gets installed in
/usr/sbin/blart thanks to the symlink, but the dpkg database has an
entry for /sbin/blart, and that is the conflict which is the problem.

So in theory, if we had a program which looked for the top-level
symlinks /{bin,lib,sbin} -> /usr/{bin,lib,sbin}, and if they exist,
scans dpkg database is scanned looking for of the form
/{bin,lib,sbin}/$1, and updates them with /usr/{bin,lib,sbin}/$1, and
then in the future, if dpkg sees the top-level symlink, canonicalizes
any files referenced in the packages to /usr/{bin,lib,sbin}/$1, with a
fallback searching for /{bin,lib,sbin}/$1 in the file system, this
would solve the problem.

Let's ignore how we would deploy this helper program and the update
dpkg from a stable upgrade perspective, but in terms of preventing
potential problems during the testing window, getting an update to
dpkg which included the database fixup program and which ran from the
maintainer script would be a potential solution path.

Furthermore, if dpkg knew to always canonicalize filenames from
/{bin,lib,sbin}/XXX to /usr/{bin,lib,sbin}XXX when adding to the
database, and when it looked for files in the file system looked first
in /usr/{bin,lib,sbin}/XXX, with a fallback to /usr/{bin,lib,sbin}/XXX
the file names in the package would not need to change all, since all
of the magic fixup work would be happening inside dpkg.

Is this a viable path forward, or am I missing something?

Thanks,

- Ted

Re: merged /usr vs. symlink farms

[Russ Allbery]

Ansgar  writes:

> Different, non-conflicting packages shipping binaries with the same name
> in /bin and /usr/bin (or similar) should be resolved for a while
> now. That as looked at when usrmerge was first introduced. I'm aware of
> one instance where this was intentional to prefer one program over the
> other (molly-guard), but it since uses diversions.

> (Actually molly-guard ships several /sbin/pm-* binaries, while pm-utils
> ships them as /usr/sbin/pm-*, but I think this is fine as the ones in
> pm-utils are diverted by molly-guard's preinst script.)

> For what it is worth I could find the following instances in Debian
> bookworm (amd64 + all), not taking into account package conflicts,
> covering /bin, /sbin and /lib*:

Thanks for the data!

> sbin/pm-hibernate: root=admin/molly-guard 
> usr=admin/pm-utils,metapackages/progress-linux-container
> sbin/pm-suspend: root=admin/molly-guard 
> usr=admin/pm-utils,metapackages/progress-linux-container
> sbin/pm-suspend-hybrid: root=admin/molly-guard 
> usr=admin/pm-utils,metapackages/progress-linux-container

I will assume this uses diversions, but probably the strongest argument
for saying something in Policy was this confusion.  At least one Debian
maintainer didn't realize that diversions were the best way of doing this
rather than shipping binaries in a different PATH.

> sbin/syslogd: root=utils/busybox-syslogd usr=net/inetutils-syslogd
> bin/systemctl: root=admin/systemd usr=admin/systemctl
> sbin/update-service: root=admin/runit usr=admin/daemontools-run
> lib/x86_64-linux-gnu/libsystemd.so.0: root=libs/libelogind0 
> usr=libs/libsystemd0
> sbin/exfatlabel: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
> sbin/fsck.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
> sbin/mkfs.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs

> And one additional instance in unstable:

> lib/systemd/system/ifup@.service: root=admin/ifupdown2 usr=admin/ifupdown
> lib/systemd/system/networking.service: root=admin/ifupdown2 usr=admin/ifupdown

I've chcked these and they all declare explicit Conflicts.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Ansgar]

Hi Russ,

On Mon, 2021-08-23 at 13:41 -0700, Russ Allbery wrote:
> Right now, in the absence of such a plan, it's obvious that having
> two
> unrelated packages (that do not Conflict) ship a binary with the same
> name
> in /bin and /usr/bin is not sensible, yes?  (I believe that's the
> topic
> under discussion in this thread.)  I'm trying to understand if enough
> people thought this was a sensible, non-buggy thing to do today that
> it's
> worthwhile adding something to Policy explicitly saying that it's
> not.

Different, non-conflicting packages shipping binaries with the same
name in /bin and /usr/bin (or similar) should be resolved for a while
now. That as looked at when usrmerge was first introduced. I'm aware of
one instance where this was intentional to prefer one program over the
other (molly-guard), but it since uses diversions.

(Actually molly-guard ships several /sbin/pm-* binaries, while pm-utils
ships them as /usr/sbin/pm-*, but I think this is fine as the ones in
pm-utils are diverted by molly-guard's preinst script.)

For what it is worth I could find the following instances in Debian
bookworm (amd64 + all), not taking into account package conflicts,
covering /bin, /sbin and /lib*:

sbin/pm-hibernate: root=admin/molly-guard 
usr=admin/pm-utils,metapackages/progress-linux-container
sbin/pm-suspend: root=admin/molly-guard 
usr=admin/pm-utils,metapackages/progress-linux-container
sbin/pm-suspend-hybrid: root=admin/molly-guard 
usr=admin/pm-utils,metapackages/progress-linux-container
sbin/syslogd: root=utils/busybox-syslogd usr=net/inetutils-syslogd
bin/systemctl: root=admin/systemd usr=admin/systemctl
sbin/update-service: root=admin/runit usr=admin/daemontools-run
lib/x86_64-linux-gnu/libsystemd.so.0: root=libs/libelogind0 usr=libs/libsystemd0
sbin/exfatlabel: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
sbin/fsck.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs
sbin/mkfs.exfat: root=otherosfs/exfat-utils usr=otherosfs/exfatprogs

And one additional instance in unstable:

lib/systemd/system/ifup@.service: root=admin/ifupdown2 usr=admin/ifupdown
lib/systemd/system/networking.service: root=admin/ifupdown2 usr=admin/ifupdown

Ansgar

Re: merged /usr vs. symlink farms

[Russ Allbery]

Simon Richter  writes:

> It is less nonsensical because usrmerge exists, since we presumably
> don't want to keep the /bin paths in the packages, so at some point we
> need to move /bin/foo to /usr/bin/foo inside a package.  That is safe
> with current dpkg, as dpkg will not delete /bin/foo if it has the same
> inode as a just-unpacked file.

[...]

I think this implies that writing something in Policy about this would be
premature.  The issues you raise are related to something new that is
being discussed and would be part of a migration plan to a merged /usr
world.  The appropriate time to document those details in Policy would be
after we agreed on a plan, not now when they're just tentative ideas.

Right now, in the absence of such a plan, it's obvious that having two
unrelated packages (that do not Conflict) ship a binary with the same name
in /bin and /usr/bin is not sensible, yes?  (I believe that's the topic
under discussion in this thread.)  I'm trying to understand if enough
people thought this was a sensible, non-buggy thing to do today that it's
worthwhile adding something to Policy explicitly saying that it's not.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 23.08.21 17:23, Russ Allbery wrote:

[one package with /bin/foo, another with /usr/bin/foo]


This seems clearly nonsensical to me even if usrmerge was never on the
horizon, since which binary you got would randomly depend on the PATH
ordering and the order of /bin vs. /usr/bin in user-set PATHs is not fixed
and has never mattered.


It is less nonsensical because usrmerge exists, since we presumably 
don't want to keep the /bin paths in the packages, so at some point we 
need to move /bin/foo to /usr/bin/foo inside a package. That is safe 
with current dpkg, as dpkg will not delete /bin/foo if it has the same 
inode as a just-unpacked file.


We have another kind of common transition: moving files between packages 
with a Replaces: relation.


If a package undergoes both transitions in the same release cycle, then 
dpkg would indeed see package A containing /bin/foo, and package B with 
Replaces: A containing /usr/bin/foo.


And in this case, /bin/foo can be removed after /usr/bin/foo is 
unpacked, and then the file vanishes because dpkg did not register the 
ownership transfer.


   Simon

Re: merged /usr vs. symlink farms

[Zack Weinberg]

Tomas Pospisek  wrote:
> On 22.08.21 00:11, Guillem Jover wrote:
>> I'm personally just not seeing such consensus, despite the attempts of
>> some to make it pass as so. My perception is that this topic has become
>> such a black hole of despair, that people that take issue with it, are
>>  simply stepping away.
...
> I do not really care which solution will be chosen. I hope it will
> be one that doesn't break my system(s) too hard so I'll be able to
> ask a search engine and follow the hints and instructions.

I'm just a user, but this seems like the right moment for me to speak up:

Whether or not / and /usr ever get merged *doesn't affect me as a
user.*   All the stuff I use will be in my $PATH either way.  The
benefits, AIUI, are all for people developing or packaging software
that has to be compatible with many different Linux distributions, but
does not care about portability to non-Linux environments.  That
simply isn't me.  So I don't care if the transition happens or not,
nor about the timeframe, and I'd *like* to not have to care about how
it gets done.

What I *do* very much care about is whether I can trust package
installation (of official, stable packages) to not break my systems,
and the way this discussion is going, it seems like I might not be
able to, and yes, that is disheartening.

The chief dpkg maintainer has given clear technical reasons why the
approach taken by usrmerge risks breaking people's systems.  There is
a proof-of-concept in this very thread, demonstrating that the bugs
are real.  From where I'm sitting, that should have been the end of
the argument.  Hard stop on further merge-related changes in unstable
until a transition plan has been worked out that *won't* tickle dpkg
bugs; if that means waiting another release cycle in order to ship
dpkg bugfixes, *so be it*.  (I reiterate that as a user I don't care
whether this transition ever actually happens.)  Maybe even revert to
non-merged mode in the installer and drop usrmerge from testing and
stable, as a precaution.

But somehow what I see happening instead, is that Guillem's concerns
are being brushed aside, demoralizing him to the point where, if it
were me in his shoes, I would have resigned from the entire project;
and half the posters in this thread are raring to push ahead with a
transition that has a nonzero chance of wrecking one of my servers the
next time unattended-upgrades does its thing.  (If I understand the
issue correctly, it *could* bite on a point upgrade or even a security
patch of any system that has already been transitioned the way
usrmerge/d-i do it, including all fresh bullseye and now also bookworm
installations.)  That's a horrifying failure of both technical and
social project management.

I used the word "nonzero" in the last paragraph intentionally.
I don't want to hear any probabilistic arguments why everything should
be fine.  I want a transition plan that *guarantees* no breakage.
That's what Debian has given us end-users for 20+ years and I hate
that I might have to worry about not getting it again.

And I think several people here owe Guillem an apology.

zw

Re: merged /usr vs. symlink farms

[Russ Allbery]

Luca Boccassi  writes:

> Thank you - it has been brought up in this thread as an example of a
> valid setup, so if it is not, I think it could be good to be extra clear
> in the policy? How about the following:

If we tried to document every random bit of buggy packaging behavior
anyone thought of in Policy, Policy would become unwieldy, so I want to
verify here that someone really thought having one package containing a
file in /bin and another package containing the same file in /usr/bin was
was a reasonable thing to do (as opposed to accidental).  Are there
packages in the archive like this?  Or could you point me at the message
in the thread that said this was non-buggy?  I think I missed it.

This seems clearly nonsensical to me even if usrmerge was never on the
horizon, since which binary you got would randomly depend on the PATH
ordering and the order of /bin vs. /usr/bin in user-set PATHs is not fixed
and has never mattered.

(It may be that someone has done this *accidentally* and thus created an
edge case that the package management system has to cope with, but that's
a question of finding buggy packages, which is not something Policy can
really help with.)

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Sam Hartman]

> "Simon" == Simon Richter  writes:

>> I can see two arguments why we might need a dpkg update:
>> 
>> 1) To fix bugs related to directory aliasing.
>> 
>> I don't think that there is a consensus those bugs need to be
>> fixed to move forward.  (Put another way it's not clear the
>> community agrees they are RC).

Simon> dpkg as it is now will detect only the case when a file is
Simon> moved to the usrmerge path inside a package, or when a file
Simon> is moved from one package to another (with Replaces), but not
Simon> both at the same time.

I confirm I do not consider this RC nor do I believe it should block the
transition.
[explanation removed from quoted material]
Simon> We can work around that by introducing a policy that no files
Simon> that are currently registered as living outside /usr may be
Simon> moved between packages, but... just no.

When combined with a policy that you should not move files from
/bin|/lib to /usr/bin|/usr/lib in the bookworm cycle, that seems
workable to me.
In other words, individual packages should not transition to usrmerged
paths on their own during this cycle.
There may be some corner case where we need this.
If so, we'll have to consider the replaces issues.
Presumably we could replace and explicitly conflict rather than
breaking.
And yes that's not ideal but it will be sufficient to make sure that
both packages are not unpacked at the same time.


>> Yes, there are bugs.  Yes, it would be good to get them fixed in
>> the bookworm cycle.  But despite the issue being brought up,
>> there is not strong support for the idea that we must block on a
>> solution to the dpkg directory aliasing bugs.

Simon> I think that one of the release goals should be that any
Simon> freshly installed or upgraded system should have a dpkg
Simon> database that is consistent with reality, and I'd prioritize
Simon> that higher than actually finishing the transition, because
Simon> as long as we can have files vanish from under us, we can't
Simon> expect that the transition will be reliable for
bullseye-> bookworm updates.

I think you are in the rough (not part of the consensus) on this desire.



signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Sam Hartman]

> "Simon" == Simon Richter  writes:
Simon> Current dpkg already has handling code so that /bin/foo ->
Simon> /usr/bin/foo is not a problematic move even on usrmerge'd
Simon> systems, so a possible policy would be to allow those and
Simon> disallow package splits, that way we could continue
Simon> transitioning packages.

Why would we need to transition packages?
If we're going to have something like usrmerge run eventually,
It seems like things will be much simpler if we forbid or at least
strongly discourage package level transitions in the bookworm cycle.
Doing that seems to be closer to the consensus of what we're discussing
here than spending effort transitioning packages.


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sun, 2021-08-22 at 19:10 -0700, Russ Allbery wrote:
> Luca Boccassi  writes:
> > On Sun, 2021-08-22 at 07:45 -0700, Russ Allbery wrote:
> 
> > > This is already the case.  Policy 10.1:
> 
> > >    To support merged-/usr systems, packages must not install files in
> > >    both /path and /usr/path. For example, a package must not install both
> > >    /bin/example and /usr/bin/example.
> 
> > Thank you - is that intended to mean "the same package", or "any two
> > packages"? Ie, is foo2 allowed to install /bin/foo if foo1 installs
> > /usr/bin/foo or is that RC-buggy too?
> 
> I don't think we have an explicit statement that you can't do this because
> I'm not sure it's come up, but it's obviously not a safe thing to do (and
> that was true long before usrmerge was even considered).

Thank you - it has been brought up in this thread as an example of a
valid setup, so if it is not, I think it could be good to be extra
clear in the policy? How about the following:

 To support merged-\ ``/usr`` systems, packages must not install files in
 both ``/path`` and ``/usr/path``. For example, a package must not install
-both ``/bin/example`` and ``/usr/bin/example``.
+both ``/bin/example`` and ``/usr/bin/example``. Also, package ``example-b``
+cannot install ``/bin/example`` if package ``example-a`` already installs
+``/usr/bin/example``, and viceversa.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Russ Allbery]

Luca Boccassi  writes:
> On Sun, 2021-08-22 at 07:45 -0700, Russ Allbery wrote:

>> This is already the case.  Policy 10.1:

>>To support merged-/usr systems, packages must not install files in
>>both /path and /usr/path. For example, a package must not install both
>>/bin/example and /usr/bin/example.

> Thank you - is that intended to mean "the same package", or "any two
> packages"? Ie, is foo2 allowed to install /bin/foo if foo1 installs
> /usr/bin/foo or is that RC-buggy too?

I don't think we have an explicit statement that you can't do this because
I'm not sure it's come up, but it's obviously not a safe thing to do (and
that was true long before usrmerge was even considered).

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[gregor herrmann]

On Sun, 22 Aug 2021 19:02:18 -0400, Theodore Ts'o wrote:

> On Sun, Aug 22, 2021 at 12:26:46PM +0200, David Kalnischkies wrote:
> > So, when did you last log into your build chroot to upgrade dpkg and
> > apt first?
> Personally, I never upgrade build chroots between major versions.  I
> just use a tool like sbuild-createchroot to create them from scratch.

That's one option, the other (and faster) is, quoting from
~/.bash_history from a week ago:

#v+
cp -ar amd64/bullseye-base.cow amd64/bookworm-base.cow
cp -ar i386/bullseye-base.cow i386/bookworm-base.cow
sed -i -e 's/bullseye/bookworm/g' amd64/bookworm-base.cow/etc/apt/sources.list 
i386/bookworm-base.cow/etc/apt/sources.list
./update_all_cow.sh # wrapper around `cowbuilder --update --basepath …' for all 
cowbuilder chroots
#v-


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   


signature.asc
Description: Digital Signature

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Sun, Aug 22, 2021 at 12:26:46PM +0200, David Kalnischkies wrote:
> 
> So, when did you last log into your build chroot to upgrade dpkg and
> apt first? And while at that, did you follow the release notes – from
> the future, as they have yet to be written for the release you are
> arguably upgrading to already?

Personally, I never upgrade build chroots between major versions.  I
just use a tool like sbuild-createchroot to create them from scratch.
That's mainly because I need to keep the Buster chroot around for
backports, so I'll just create a new Bullseye chroot when I need it.

And of course my unstable chroot is continuously being upgraded but
that avoids most of the concerns people have about the Debian N to N+1
stable upgrade path.

> But okay, lets assume you actually do: apt and dpkg tend not to be
> statically linked, so they end up having dependencies

So on my "pet" production systems (on my "cattle" systems I just
recreate them from scratch, e.g. "gce-xfstests create-image"[1] which
uses deboostrap), what I do is update /etc/apt/sources.list, and then
do "apt update ; apt-get install dpkg ; apt-get install apt" before I
run "apt-get dist-upgrade".  This handles the dependencies for me, and
while a few packages will get upgraded using the old dpkg and apt, the
vast majority of the packages will get upgraded using the latest
stable version of dpkg and apt.  This seems like relatively cheap
insurance; it doesn't hurt, and it might help avoid some nasty corner
cases that got overlooked during testing.

[1] https://thunk.org/gce-xfstests

> Don't get me wrong, as an apt dev I would love if we could do that. It
> is kinda annoying to work around issues you have fixed years ago, but
> aren't available in (soon) oldstable.

Well, I'll observe that if we told people to upgrade dpkg and apt
first using "apt-get install ...", this automatically handles the
dependencies, and while it doesn't make *all* of the potential issues
go away, I would think it would reduce the potential corner cases and
hence make it easier to test to make sure the right thing happens on
an update from Debian vN to vN+1, would it not?


> P.S.: As someone will ask: Ubuntu splits the user base in two: Those who
> run their release upgrader which runs outside of the packaging system and
> largely can do whatever (including bring in a standalone apt/dpkg just
> dealing with an upgrade – they usually resign to much simpler things
> through) and those who don't like for example chroots and containers who
> effectively use whatever an upgrade path 'apt dist-upgrade' gives you.

... and in my opinion, that's a *fine* strategy.  Sure, it would be
nice if "apt dist-upgrade" always worked smoothly, and that's a great
aspirational goal, but if we can reduce risks to users ---
particularly non-technical/non-expert users --- by using a release
upgrader which upgrades apt and dpkg first, we should do that.  So
let's take a page from Ubuntu, by all means!

- Ted

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sun, 2021-08-22 at 12:42 +0200, Steve Cotton wrote:
> Am Sun, Aug 22, 2021 at 11:21:38AM +0100 schrieb Luca Boccassi:
> > On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> > > Just like no one had detected the database corruption in Ubuntu
> > > before
> > > I spotted the problem via code review and analysis (which I guess
> > > in
> > > your world translates to "opinion"). I'd expect the problems with
> > > aliased directories to be that kind of insidious issue that
> > > people
> > > have a very hard time trying to pin point, and which will be
> > > getting
> > > worse as time passes.
> > 
> > If I understand correctly, what has been stated as a potential
> > theoretical consequence is files disappearing and upgrades failing.
> > Why
> > would these be hard to detect? It would seem to be a pretty visible
> > consequence, no?
> 
> How would you know which package to log a bug on? Would you feel able
> to log a
> useful bug report at all, given that all you've detected is that
> something is
> losing data from the filesystem?
> 
> * How do you know it's not a kernel filesystem bug?
> * How do you know it's not a kernel caching bug?
> * How do you know it wasn't just a typo by the administrator?
> * How long ago did it happen anyway, when did you last use this
> utility?
> * Could it have been an accidental power-off?
> * Hardware bug?
> * Something installed from experimental?
> 
> Guillem didn't say it was hard to detect, the text you quoted says
> "very hard
> time trying to pin point".
> 
> Steve

Of course I can't know for sure, but if an upgrade "lost" a file, I
would imagine a user would file a bug against either the kernel
(blaming the filesystem) or dpkg (blaming the package manager). That's
what I assume I would do in that situation. Failing that, I suppose the
fallback is reaching out to the usual support channels - generalist
mailing list, irc. Has there been an increase (or any at all) in bug
reports about files disappearing on the kernel/dpkg/apt? Has there been
an increase (or any at all) of support request about disappeared files
on debian-devel or debian-user? I mean, from my experience our users
are very vocal when things break badly, even if they don't know exactly
where the root cause is - I am used to see bugs filed against
src:nvidia-graphics-driver pretty much anytime anything remotely
related to "show stuff on screen" goes wrong, if there happens to be an
nvidia card in use :-)

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sun, 2021-08-22 at 07:45 -0700, Russ Allbery wrote:
> Luca Boccassi  writes:
> 
> > I've asked this before - I might be very wrong, but I was under the
> > impression that having both /bin/foo and /usr/bin/foo (which is the
> > example mentioned) was already considered RC-buggy and needed
> > fixing?
> > Is that not the case?
> 
> This is already the case.  Policy 10.1:
> 
>     To support merged-/usr systems, packages must not install files
> in
>     both /path and /usr/path. For example, a package must not install
> both
>     /bin/example and /usr/bin/example.

Thank you - is that intended to mean "the same package", or "any two
packages"? Ie, is foo2 allowed to install /bin/foo if foo1 installs
/usr/bin/foo or is that RC-buggy too?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Tomas Pospisek]

On 22.08.21 00:11, Guillem Jover wrote:


I'm personally just not seeing such consensus, despite the attempts of
some to make it pass as so. My perception is that this topic has become
such a black hole of despair, that people that take issue with it, are
simply stepping away.


Possibly. But for me as one datapoint of N the amount and level of 
tricky technical detail is way beyond me, so I am standing on the 
sideline and watching the discussion and leaving the arguments to the 
experts. And I do not really care which solution will be chosen. I hope 
it will be one that doesn't break my system(s) too hard so I'll be able 
to ask a search engine and follow the hints and instructions.


Wrt not caring much about which solution will be chosen: Debian has now 
passed through transitions where people were extremely (emotionally) 
involved. I wasn't consistently on the winning side: my preference would 
have been to keep it simple and to be able to fire up `vim` to debug 
some aspect of the boot system or services failing to start, but here we 
are, we've got systemd instead. So I had to adapt, to learn new stuff 
and the world did not end in fact I am OK with it.


Could be that I am that single one DD with this perspective, or maybe not.

In the end I think Debian is a do-o-cracy: we can decide whatever we 
want via TC or GR but unless there's someone willing to do the work all 
decisions won't matter. So it is my hope - in fact my conviction - that 
whoever does the work for the merged /usr will find a solution that 
won't trash my system(s) too badly and I'll find a way to fix'em. And if 
the scratch will itch too badly I might eventually even file a patch 
that will give others relief as well.


From this same perspective:


Exhausted,
Guillem


seeing you exhausted is sad. You do not need to carry the weight 
of this problem, of dpkg or of Debian. I believe Debian *will* find a 
way. Some way. Debian *does* have many capable and interested people. 
*Do* take care of yourself. Do make sure that working on dpkg is still 
enjoyable for you or whatever your motivation is. There's strictly no 
need you burning yourself out on this. You *can* take a step back an 
deinvest *yourself* from this. It is a technical and maybe a social 
problem but it is *not* *your* problem unless you let it be so.


Huge respect and thanks for all your work this far!
*t

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 22.08.21 16:52, Simon Richter wrote:

The most generic approach would be to have a symlink farming mode in 
dpkg, where it has a goal (as defined by a package) to create a symlink 
/lib -> usr/lib, but while another package declares /lib to be a 
directory, the directory has precedence and dpkg generates the minimal 
set of symlinks that create the same effect -- so if /usr/lib/foo exists 
and /lib/foo doesn't, it generates a symlink /lib/foo -> ../usr/lib/foo.


Hmm, this could work with a Pre-Depends from base-files/bookworm to 
dpkg/bookworm, and base-files/bookworm providing the /bin, /lib and 
/sbin symlinks to indicate to dpkg that this is desired.


base-files has sufficiently few rdepends that it can be updated rather 
late, but it is Essential: yes, so it is guaranteed to be available on 
new installations (which would also have a dpkg that understands it).


So a rule to resolve symlink-vs-directory conflicts would change from 
"ignore the symlink" to "build a symlink farm that tries to make the 
same paths available", and the symlink farm would be optimized as 
packages drop directories.


So e.g. if I have two kernel packages with

/lib/modules/1.2.3

and

/usr/lib/modules/1.2.4

then the desired symlink in base-files

/lib -> usr/lib

would conflict with /lib in the kernel package, and dpkg resolves this 
to /lib/modules -> ../usr/lib/modules (still a conflict), then to 
/lib/modules/1.2.4 -> ../../usr/lib/modules/1.2.4, which can be created 
to make the modules in /usr/lib available from /lib as if /lib was 
symlinked.


When the package providing /lib/modules/1.2.3 is removed, the last 
package providing /lib/modules is gone, so we can replace this with a 
symlink /lib/modules -> ../usr/lib/modules.


The transition is finished when the last package that uses /lib as a 
directory is gone. That will be libc6, which still has to provide ld.so 
in the usual path even in bookworm.


For systems that are usrmerged, the symlink is already as dpkg expects, 
so it doesn't start symlink farming there. We can use the bookworm cycle 
to move files to /usr as long as we don't move them to other packages, 
so we'd be getting fairly close to getting the database and the 
filesystem consistent after the upgrade -- basically everything but 
libc6 would be finished in bookworm.


The second thing is that we cannot deprecate scanning old paths during 
the bookworm cycle -- so systemd needs to continue looking into 
/lib/systemd, so the transition will not be finished during the bookworm 
cycle anyway.


   Simon

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 22.08.21 05:10, Theodore Ts'o wrote:


So with the goal of trying to enumerate possible solutions, it sounds
some combination of:



(a) disallowing moving problematic files between packages, with possibly some
 QA tools to enforce this
(b) keeping the next release cycle *short*, say only a year
(c) requiring that dpkg be upgraded first, and having dpkg and
 related tools understand the concept of usrmerge and the
 fact that /{bin,lib,sbin} and /usr/{bin,lib,sbin} are identical
 for usrmerged systems



might be possible paths forward.  Do you agree?


Yes. I'd even say that if we don't have any problematic moves, we can 
get away with updating dpkg at any point during the upgrade -- the 
current state is more "fragile" than "broken", which would help 
immensely because we don't have to expend manpower on updating 
autobuilders and explaining to people that dist-upgrade is not to be used.


> What are other possible solutions?

Current dpkg already has handling code so that /bin/foo -> /usr/bin/foo 
is not a problematic move even on usrmerge'd systems, so a possible 
policy would be to allow those and disallow package splits, that way we 
could continue transitioning packages. If we manage to move all files 
out, the discrepancy between the dpkg database and the filesystem will 
be minimal (but we can't transition libc6 that way, because usrmerge 
left us with a state where we can't unpack a libc6 that provides ld.so 
as a symlink in /lib).


The approach that will take the least amount of work but is a nasty 
horrible hack would be to integrate usrmerge into dpkg, forcibly 
converting any system that isn't yet, updating the dpkg database in 
either case, and then filtering file lists during writing going forward.


I'd rather not try to make the conflict checking code itself understand 
aliasing through symlinks, that looks as if it would be a source of 
bugs, but transforming file lists to canonical paths before checking 
should be doable, and any symlink-vs-directory conflict outside those we 
explicitly handle then become fatal instead of silently ignored.


The most generic approach would be to have a symlink farming mode in 
dpkg, where it has a goal (as defined by a package) to create a symlink 
/lib -> usr/lib, but while another package declares /lib to be a 
directory, the directory has precedence and dpkg generates the minimal 
set of symlinks that create the same effect -- so if /usr/lib/foo exists 
and /lib/foo doesn't, it generates a symlink /lib/foo -> ../usr/lib/foo.


This would allow us to transition the package contents to match reality 
on usrmerged systems, and dpkg will finish the transition as the last 
package that declares /lib to be a directory is gone, and it could be 
used for any future transitions that need old paths to work for a while.


Without an extra hack, this would not allow us to transition ld.so out 
of /lib though because


/usr/lib/ld-linux.so.2
/lib -> usr/lib

would be unpacked without the symlink because the file conflict causes 
the symlink to be dropped.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Russ Allbery]

Luca Boccassi  writes:

> I've asked this before - I might be very wrong, but I was under the
> impression that having both /bin/foo and /usr/bin/foo (which is the
> example mentioned) was already considered RC-buggy and needed fixing?
> Is that not the case?

This is already the case.  Policy 10.1:

To support merged-/usr systems, packages must not install files in
both /path and /usr/path. For example, a package must not install both
/bin/example and /usr/bin/example.

-- 
Russ Allbery (r...@debian.org)  

Re: merged /usr vs. symlink farms

[Steve Cotton]

Am Sun, Aug 22, 2021 at 11:21:38AM +0100 schrieb Luca Boccassi:
> On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> > Just like no one had detected the database corruption in Ubuntu before
> > I spotted the problem via code review and analysis (which I guess in
> > your world translates to "opinion"). I'd expect the problems with
> > aliased directories to be that kind of insidious issue that people
> > have a very hard time trying to pin point, and which will be getting
> > worse as time passes.
> 
> If I understand correctly, what has been stated as a potential
> theoretical consequence is files disappearing and upgrades failing. Why
> would these be hard to detect? It would seem to be a pretty visible
> consequence, no?

How would you know which package to log a bug on? Would you feel able to log a
useful bug report at all, given that all you've detected is that something is
losing data from the filesystem?

* How do you know it's not a kernel filesystem bug?
* How do you know it's not a kernel caching bug?
* How do you know it wasn't just a typo by the administrator?
* How long ago did it happen anyway, when did you last use this utility?
* Could it have been an accidental power-off?
* Hardware bug?
* Something installed from experimental?

Guillem didn't say it was hard to detect, the text you quoted says "very hard
time trying to pin point".

Steve

Re: merged /usr vs. symlink farms

[David Kalnischkies]

On Sat, Aug 21, 2021 at 12:47:51PM -0400, Theodore Ts'o wrote:
> Personally, I *don't* have a problem about telling people to manually
> update dpkg, apt, and/or apt-get before they do the next major stable
> release (maybe it's because this is something I do as a matter of
> course; it's not that much extra effort, and I'm a paranoid s.o.b.,

So, when did you last log into your build chroot to upgrade dpkg and
apt first? And while at that, did you follow the release notes – from
the future, as they have yet to be written for the release you are
arguably upgrading to already?

But okay, lets assume you actually do: apt and dpkg tend not to be
statically linked, so they end up having dependencies. Some of them even
surprising. In bullseye you e.g. have to upgrade cryptsetup first before
apt can be upgraded (apt → libgcc-s1 → cryptsetup-initramfs → …).
And that is just the first and most obvious chain.
But it would never happen that you would e.g. need to upgrade all of the
KDE desktop environment before upgrading dpkg, right? Well… in Debian
squeeze that nearly happened, but we had to break that chain because it
was actually forming a 500+ loop apt/lenny had trouble dealing with.
Those chains are only investigated if they lead to major problems:
I e.g. never looked at the C++ v5 (copy on write) transition which
likely entangled apt with half of the universe in the general case.

But okay, lets assume we form a team who actually looks into all these.
It is easy, right? No. Dependencies and their version constrains (can)
differ by architecture and mostly propagate by negative dependencies
meaning the individual system state is hugely important. The result is
that hardly any upgrade is the same even if we subsume it all under
"upgrade to bullseye". And regardless of how hard we try, there will
always be other packages which have to go before it is dpkgs turn, so
at least all these have to make due with what was released previously
anyhow.


> and I know that's the most tested path given how Debian testing
> works).

Not really as you can see e.g. with libgcc-s1, as most upgrade problems
aren't due to dpkg or apt in practice, but because the intermediate
steps of a package making testing upgrades a smooth sail aren't visible
for the stable updates. If you want a better tested path, I suggest
stable updates by jumping from week to week via snapshot.d.o. You may
want to skip weeks with actual bugs through.



In the end, it is just simpler to assume that every release+1 package is
installed by dpkg/release (and of course apt/release) than trying to
reason if and how we can make it happen that dpkg is handled before some
other package (without forming loops) or even can be sanely upgraded
ahead of the upgrade entirely. Or are you perhaps volunteering?


Don't get me wrong, as an apt dev I would love if we could do that. It
is kinda annoying to work around issues you have fixed years ago, but
aren't available in (soon) oldstable. We would need a more aggressive
stable-updates strategy but in reality we tend to be held to even higher
standards than all other packages because we are native key packages…
(just look when we froze dpkg… apt at least has a tiny loophole for now
 by not being build-essential in a strict sense)

Not that it really matters. It would just add more moving parts to the
upgrade process – a process, if this entire thread is any indication,
which is hardly understood by anyone in Debian and considered entirely
optional by many. That alone makes me very sad on many levels.


(I wrote this before reading Guillems replies which end on a similar
 note even though he comes from the opposite end – dpkg worried about
 the finer file-level details and apt about the general package-level
 picture meeting halfway as usual… kinda funny)


Best regards

David Kalnischkies

P.S.: As someone will ask: Ubuntu splits the user base in two: Those who
run their release upgrader which runs outside of the packaging system and
largely can do whatever (including bring in a standalone apt/dpkg just
dealing with an upgrade – they usually resign to much simpler things
through) and those who don't like for example chroots and containers who
effectively use whatever an upgrade path 'apt dist-upgrade' gives you.
Which also explains why Ubuntu hasn't fully /usr-merged yet more or less
waiting for Debian to figure that one out. Or, well, they spearhead even
here now as it is apparently too much to ask for an upgrade path in
Debian nowadays.


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sat, 2021-08-21 at 23:10 -0400, Theodore Ts'o wrote:
> On Sun, Aug 22, 2021 at 02:15:31AM +0200, Simon Richter wrote:
> > 
> > The latter is what brought us into a situation where it is no
> > longer safe to
> > move files between packages and between aliased directories in the
> > same
> > upgrade, and because users will be expected to upgrade in a single
> > step
> > between stable releases, that means these two types of changes are
> > mutually
> > exclusive for the entire release cycle.
> 
> So with the goal of trying to enumerate possible solutions, it sounds
> some combination of:
> 
> (a) disallowing moving problematic files between packages, with
> possibly some
>     QA tools to enforce this
> (b) keeping the next release cycle *short*, say only a year
> (c) requiring that dpkg be upgraded first, and having dpkg and
>     related tools understand the concept of usrmerge and the
>     fact that /{bin,lib,sbin} and /usr/{bin,lib,sbin} are identical
>     for usrmerged systems
> 
> might be possible paths forward.  Do you agree?  What are other
> possible solutions?
> 
> - Ted

I've asked this before - I might be very wrong, but I was under the
impression that having both /bin/foo and /usr/bin/foo (which is the
example mentioned) was already considered RC-buggy and needed fixing?
Is that not the case?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:
> On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> > On Sat, 2021-08-21 at 16:20 +0200, Wouter Verhelst wrote:
> > > I'm not saying the solution which the dpkg maintainers are
> > > proposing
> > > is the only valid solution, but if you go and tell them "ah the
> > > real
> > > problems you point out are irrelevant" then You! Are! Doing! It!
> > > Wrong!
> > 
> > Again, if the magnitude of this dpkg bug was really that serious
> > there
> > would be visible consequences after almost 3 years of deployments
> > across two distributions with who knows how many million instances,
> > and
> > yet "having to run dpkg -S again" is all we can see. Where are the
> > bug
> > reports? Where are the enraged users with unusable broken system
> > and
> > lost data? Where are the reports of Canonical going out of business
> > because Ubuntu is unusable?
> 
> Just like no one had detected the database corruption in Ubuntu
> before
> I spotted the problem via code review and analysis (which I guess in
> your world translates to "opinion"). I'd expect the problems with
> aliased directories to be that kind of insidious issue that people
> have a very hard time trying to pin point, and which will be getting
> worse as time passes.

If I understand correctly, what has been stated as a potential
theoretical consequence is files disappearing and upgrades failing. Why
would these be hard to detect? It would seem to be a pretty visible
consequence, no?

> > The bug is real, nobody doubts that - it has been filed on dpkg 20
> > years ago.
> 
> You keep repeating this, but I have no idea what bug you refer to.
> 
> There's #148258 (from 2002), which is conffile related, and not
> actionable and should probably just be closed.
> 
> There's #182747 (from 2003), which while apparently similar is
> something else completely. This is about the (IMO) misfeature of
> supporting a local admin to redirect (not alias) a directory using a
> local symlink (mainly for space management reasons). For an
> explanation
> see .
> 
> There's #406715 (from 2007) which is related to the above misfeature.

I am referring to #134758 since it's linked as the root cause from
usrmerge's #848622.

"dpkg-query: Make -S handle unowned symlinks resolving to owned
pathnames" filed in February 2002 - 19 years and a half ago. I refer to
that because it's linked as the root cause in the BTS of the relevant
issue with usrmerge we are discussing.

> > What I am taking issues with is the representation of its
> > actual, real effects, and thus its severity and the consequences
> > for
> > the project. There are a lot of words being spent on how terrible
> > and
> > broken and unacceptable the status quo is, and yet not a single
> > link
> > to a bug report.
> 
> What I'm appalled at is the sloppiness and dogma shown in the name of
> a filesystem layout that will have very minimal benefit for final
> users
> (in contrast to some use cases for some admins or installations that
> should already know what they are doing, and can manage all potential
> downsides in a controlled way through a hack like usrmerge),
> knowingly
> in detriment of robustness and stability.

**in your opinion** it will have minimal benefits. It is a legitimate
opinion, but still an opinion with which others disagree. Seeing how
every popular distro has moved or is moving, general consensus appear
to be going in the other direction. On the other hand, robustness and
stability are measurable: number of crashes, number of lost systems,
number of systems with lost data. The total count in the past 3 years
where it has been default in Debian and Ubuntu puts the grand total of
reports of crashes, lost systems and lost data at zero. So again, the
evidence that this change decreases stability is nowhere to be seen.

> > By all means, go and fix it, make it a top priority for dpkg to
> > sort
> > out, all hands on deck, whatever needed -
> 
> To even consider the possibility to support this missing feature in
> dpkg would require for it to get support for at least tracking
> filesystem metadata (which is has *never* *ever* supported), which is
> currently not deployable:
> 
>   
> 
> Even with that support in place, that just would not give automatic
> aliased directory support. It would need no package to ship anything
> inside those directories, and it would also need for some package to
> ship those aliased symlinks. And then many corner cases would need to
> be considered as then dpkg will need to reconcile what's on the
> filesystem, on its database and on the various .deb (given that these
> have been a shared resource, that it cannot possibly and safely
> switch
> type by itself), w/o also breaking previous expectations. Not to
> mention
> that the general aliasing issues still would not disappear.

I make no claim

Re: merged /usr vs. symlink farms

[Timo Röhling]

Hi,

* Simon Richter  [2021-08-22 02:15]:
There are two issues here: dpkg not handling certain corner cases, and 
the usemerge package modifying the file system, bypassing dpkg.

Maybe this question has been answered elsewhere, but I keep
wondering: What prevents dpkg from updating/reparing its database
to match what usrmerge did? It's not like files got shifted around
arbitrarily.

The database rewrite would be irreversable, but switching back and
forth is not a required feature anyway. And the rewrite is
idempotent, so it might even be possible to run this unconditionally
as part of some future release upgrade.

Of course, dpkg would need a path canonicalization that
transparently maps /{bin,lib*,sbin} to /usr/{bin,lib*,sbin} on *all*
file operations and is enabled together with the database
conversion.  This might take some effort to code, mostly because it
must be applied consistently. Still, it's best to have this done
right in dpkg once and for all, and I'm probably not the only one
who would be willing to help.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sat, 2021-08-21 at 20:45 +0100, Colin Watson wrote:
> On Sat, Aug 21, 2021 at 06:47:50PM +0100, Luca Boccassi wrote:
> > My recollection (which might be wrong, but a quick look at release
> > notes seems to support it with 11.04 having multiarch 2 years
> > before
> > Wheezy) is that Canonical led the way with the multiarch effort in
> > Ubuntu, and Debian followed with lots of huffing, puffing and
> > grumbling.
> 
> As a Canonical employee who was involved in the multiarch design work
> at
> the time, this is a pretty unfair-to-Debian version of history.  Yes,
> some things took a bit longer to get organized on the Debian side for
> various reasons, but the design and implementation work was done in
> collaboration with key people in Debian and was definitely better for
> it; Guillem and Raphaël in particular did a lot of hard work on dpkg
> and
> dpkg-dev respectively.  (Also, several of us on the Canonical side
> regarded ourselves as having one foot firmly in each camp; I
> certainly
> didn't see it as a confrontational sort of thing where we were having
> to
> drag Debian along with us - rather the contrary, there was a lot of
> enthusiasm in Debian for it.)

So my recollection was indeed wrong - I'll happily retract the comment
with apologies, thank you for correcting me.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Andreas Metzler]

On 2021-08-22 Guillem Jover  wrote:
[...]
> The huge majority of files under /lib* (which is the actual bulk of them)
> should require no symlink farms. Many of the ones under /bin and /sbin
> (we are talking about around 240 packages here) might be switchable w/o
> compat symlinks after careful consideration (or not…), many of the ones
> that require symlinks would need these just for a period of time, and
> only a handful would remain (the ones that are part of standard
> interfaces, which I'd expect would be mostly shells?). I think the amount
> of symlinks currently provided by f.ex. lvm2 and e2fsprogs combined would
> amount to more symlinks than what we would eventually end up with TBH.
[...]
> I've also tried to stop replying to all misconceptions, inaccuracies
> and plain wrong stuff on this thread (f.ex. people vocal in this thread
> apparently do not exactly seem to know how upgrades or our package
> manager works in Debian?), first to avoid dominating it with replies
> (like some kind of deranged person, where I think I've probably already
> surpassed my quota), and because it just all seems like a monumental
> waste of time, energy and motivation. Which I'd rather spend elsewhere.

Hello Guillem,

Afaict we have still no idea on how to move on.

1 I think you agree that there is a significant number of usrmerged Debian
  installations out there. It does not really matter whether there are 7% or
  40%. They exist and should (keep) working.

2 As you have stated there are known issues with dpkg and usrmerged
  systems. Some of them are are triggered by moving files from / to /usr.

Starting from here it is hard to see a way forward. Is it a
realistic option to require something like
-
dpkg --purge usrmerge
dpkg-fsys-usrunmess
-
pre dist-upgrade to bookworm to get back all systems to a sane (from
dpkg POV) state and start fresh? Is this robust (except for crash as
stated in the manpage), or would you consider it beta-quality?

TIA, cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Sun, Aug 22, 2021 at 02:15:31AM +0200, Simon Richter wrote:
> 
> The latter is what brought us into a situation where it is no longer safe to
> move files between packages and between aliased directories in the same
> upgrade, and because users will be expected to upgrade in a single step
> between stable releases, that means these two types of changes are mutually
> exclusive for the entire release cycle.

So with the goal of trying to enumerate possible solutions, it sounds
some combination of:

(a) disallowing moving problematic files between packages, with possibly some
QA tools to enforce this
(b) keeping the next release cycle *short*, say only a year
(c) requiring that dpkg be upgraded first, and having dpkg and
related tools understand the concept of usrmerge and the
fact that /{bin,lib,sbin} and /usr/{bin,lib,sbin} are identical
for usrmerged systems

might be possible paths forward.  Do you agree?  What are other
possible solutions?

- Ted

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 21.08.21 19:47, Luca Boccassi wrote:


By all means, go and fix it, make it a top priority for dpkg to sort
out, all hands on deck, whatever needed - but to demand the entire
project has to stand still, and to de-facto derail the effort put in to
catch up with the rest of the world by imposing an unworkable,
demonstrably failed solution (symlinks farm) to work around a dpkg bug
instead of fixing it internally, to me does not seem acceptable in any
way, shape or form without some real, serious evidence that the sky has
indeed fallen.


There are two issues here: dpkg not handling certain corner cases, and 
the usemerge package modifying the file system, bypassing dpkg.


The latter is what brought us into a situation where it is no longer 
safe to move files between packages and between aliased directories in 
the same upgrade, and because users will be expected to upgrade in a 
single step between stable releases, that means these two types of 
changes are mutually exclusive for the entire release cycle.


This happened precisely because people "put in the effort" to implement 
this change without coordinating. This thing derailed on its own, and 
accusing the people pointing that out of ill will is not going to fix that.


The fact that the sky has not yet fallen is not a reason for inaction, 
but instead gives us the breathing room to implement a proper solution 
instead of another hack that will add yet another possible upgrade 
scenario we have to anticipate in the future.


We already have inconsistent package builds depending on whether a 
package was built on a pre- or post-transition autobuilder, with the 
exact same packages installed otherwise.


We have no piuparts test coverage for these scenarios, we have no QA 
tools to verify that installing the new packages will always lead to 
predictable outcomes no matter in which order you install them in -- 
such tools were never necessary before as dpkg resolves these problems 
in a deterministic manner provided its installation database is 
consistent with reality.


This is the situation we're in: the millions of systems out there work, 
but we cannot guarantee that they will continue to work through the next 
update because the QA infrastructure now has massive blind spots. This 
is what needs to be fixed before further progress can be made.


If the bookworm upgrade does break systems on upgrade, then the sky will 
indeed have fallen, only then it will be too late to do anything about it.



"It works for me" is anecdote, bugs count is not, it is a key metric of
this industry, I am quite surprised this needs to be specified. It's
how we decide whether a release is ready or not.


I see two problems here:

First, we're not the "industry." We're a Free Software movement. The 
industry just happens to embrace us at the moment.


Second, I wonder why it doesn't give you pause if a group of senior 
software people that uses a particular metric in one instance suddenly 
seems to be utterly unaware of its existence in another.



What you call analytical
evidence on the other hand is a fancy word for "opinion".


Analytical evidence happens when you read the dpkg source code and the 
usrmerge perl script and find a glaring obvious problem in the way they 
interact, then construct a simple adversarial example in 11 lines of 
text and two dpkg-deb calls and verify that it indeed loses files.


The scenario I posted will apply to any systemd package split between 
bullseye and bookworm. If systemd moves unit files to /usr, then it 
cannot move these to another package for the entirety of the release, or 
risk the units disappearing on upgrade. It also applies to kernel module 
and firmware packages, and several core system utilities.


We got lucky with the "which" command as that was in /usr already.


Opinions are
useful and interesting and important, but saying "everything is broken"
when there is a surprising lack of evidence of that being the case, is
not very useful or constructive.


Absence of evidence is not evidence of absence.

   Simon

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Fri, 2021-08-20 at 07:56:33 -0600, Sam Hartman wrote:
> > "Theodore" == Theodore Ts'o  writes:
> Theodore> FWIW, from following the discussion, I've become more and
> Theodore> more convinced that a symlink farm is *not* the right
> Theodore> answer, regardless of whether it is done centrally or via
> Theodore> individual packages moving files and created symlinks if
> Theodore> necessary in individual maintainer scripts.

There was talk about the huge amount of symlinks required in a symlink
farm setting, but that might have been true for a scenario where those
symlinks would have been handled automatically and transparently.

The huge majority of files under /lib* (which is the actual bulk of them)
should require no symlink farms. Many of the ones under /bin and /sbin
(we are talking about around 240 packages here) might be switchable w/o
compat symlinks after careful consideration (or not…), many of the ones
that require symlinks would need these just for a period of time, and
only a handful would remain (the ones that are part of standard
interfaces, which I'd expect would be mostly shells?). I think the amount
of symlinks currently provided by f.ex. lvm2 and e2fsprogs combined would
amount to more symlinks than what we would eventually end up with TBH.

> Ted, in terms of judging consensus, I'd like to explicitly ACK your
> summary.
> I think that you have accurately described where we are in the
> discussion.
> 
> As you know, one of the ways we can see  how close we are on consensus
> is to look at what happens when someone proposes a summary like you did.
> 
> Simon Richter tried to challenge your summary effectively saying that we
> couldn't have an informed consensus because there were open technical
> issues that had not been addressed.  This was roundly rejected by
> yourself, Philip Hands and Luca Boccassi.

Well then that's a very underwhelming way to judge consensus TBH.

Philip remarked on the falling-sky theme (which I covered in a reply to
Luca elsewhere), and then used a crystal ball to predict the future,
where a particular filesystem layout would pass the test of time in
contrast to dpkg, which apparently (obviously a bit biased here) is such
a trivial detail in our distribution that it can be easily replaced w/o
requiring to scrap everything and starting a new distribution from
scratch.

Luca posted a mess of disinformation.

Ted's reply I've partially covered at the top of this mail. The rest
I've covered in the aforementioned reply to Luca. And some of the
remaining parts I've heard the "dpkg team" think are wild conjectures
on motivations and similar.

Or there's the following for example:

> IN particular, most systems are usrmerged today, and while these bugs
> are annoying, many people get along just fine.

That would imply that most people have either gone out of their way to
explicitly install and run usrmerge or they have reinstalled from
scratch. I cannot believe the first to be of any significance, the
second would put into question why we bother supporting release upgrades
(after all many other distros do not support that, we perhaps should
catch up with what the rest of the world is doing!).


I've also tried to stop replying to all misconceptions, inaccuracies
and plain wrong stuff on this thread (f.ex. people vocal in this thread
apparently do not exactly seem to know how upgrades or our package
manager works in Debian?), first to avoid dominating it with replies
(like some kind of deranged person, where I think I've probably already
surpassed my quota), and because it just all seems like a monumental
waste of time, energy and motivation. Which I'd rather spend elsewhere.



I'm personally just not seeing such consensus, despite the attempts of
some to make it pass as so. My perception is that this topic has become
such a black hole of despair, that people that take issue with it, are
simply stepping away.

Exhausted,
Guillem

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> My recollection (which might be wrong, but a quick look at release
> notes seems to support it with 11.04 having multiarch 2 years before
> Wheezy) is that Canonical led the way with the multiarch effort in
> Ubuntu, and Debian followed with lots of huffing, puffing and
> grumbling.

You mean that time when Ubuntu merged an implementation based on a broken
design with broken interfaces, that was causing dpkg database damage, where
the tech-ctte also tried to force through, in all its wisdom? Right, great
example. (And let's ignore release cadences for more spectacular effect.)

  

But just wow, such a mischaracterization and deformation of the events.
Sadly, at this point I'm not surprised. This goes along comments such as
that the intersection of packages not using debhelper and shipping
split-/usr files are in the "thousands" (way less than the actual number
of packages shipping those pathnames), or the ones in the paragraphs
below about that mythical bug report, and the single failed attempt to
symlink farm, etc, etc…

> On Sat, 2021-08-21 at 16:20 +0200, Wouter Verhelst wrote:
> > I'm not saying the solution which the dpkg maintainers are proposing
> > is the only valid solution, but if you go and tell them "ah the real
> > problems you point out are irrelevant" then You! Are! Doing! It!
> > Wrong!
> 
> Again, if the magnitude of this dpkg bug was really that serious there
> would be visible consequences after almost 3 years of deployments
> across two distributions with who knows how many million instances, and
> yet "having to run dpkg -S again" is all we can see. Where are the bug
> reports? Where are the enraged users with unusable broken system and
> lost data? Where are the reports of Canonical going out of business
> because Ubuntu is unusable?

Just like no one had detected the database corruption in Ubuntu before
I spotted the problem via code review and analysis (which I guess in
your world translates to "opinion"). I'd expect the problems with
aliased directories to be that kind of insidious issue that people
have a very hard time trying to pin point, and which will be getting
worse as time passes.

> The bug is real, nobody doubts that - it has been filed on dpkg 20
> years ago.

You keep repeating this, but I have no idea what bug you refer to.

There's #148258 (from 2002), which is conffile related, and not
actionable and should probably just be closed.

There's #182747 (from 2003), which while apparently similar is
something else completely. This is about the (IMO) misfeature of
supporting a local admin to redirect (not alias) a directory using a
local symlink (mainly for space management reasons). For an explanation
see .

There's #406715 (from 2007) which is related to the above misfeature.

> What I am taking issues with is the representation of its
> actual, real effects, and thus its severity and the consequences for
> the project. There are a lot of words being spent on how terrible and
> broken and unacceptable the status quo is, and yet not a single link
> to a bug report.

What I'm appalled at is the sloppiness and dogma shown in the name of
a filesystem layout that will have very minimal benefit for final users
(in contrast to some use cases for some admins or installations that
should already know what they are doing, and can manage all potential
downsides in a controlled way through a hack like usrmerge), knowingly
in detriment of robustness and stability.

> By all means, go and fix it, make it a top priority for dpkg to sort
> out, all hands on deck, whatever needed -

To even consider the possibility to support this missing feature in
dpkg would require for it to get support for at least tracking
filesystem metadata (which is has *never* *ever* supported), which is
currently not deployable:

  

Even with that support in place, that just would not give automatic
aliased directory support. It would need no package to ship anything
inside those directories, and it would also need for some package to
ship those aliased symlinks. And then many corner cases would need to
be considered as then dpkg will need to reconcile what's on the
filesystem, on its database and on the various .deb (given that these
have been a shared resource, that it cannot possibly and safely switch
type by itself), w/o also breaking previous expectations. Not to mention
that the general aliasing issues still would not disappear.

> but to demand the entire project has to stand still,

So wait, when it suits you the "entire project" is involved and cannot
do stuff, but when it does not the "entire project" is not required to
do anything because the proposed solution magically solves stuff for
free with no effort involved… right.

> and to de-facto derail the 

Re: merged /usr vs. symlink farms

[Colin Watson]

On Sat, Aug 21, 2021 at 06:47:50PM +0100, Luca Boccassi wrote:
> My recollection (which might be wrong, but a quick look at release
> notes seems to support it with 11.04 having multiarch 2 years before
> Wheezy) is that Canonical led the way with the multiarch effort in
> Ubuntu, and Debian followed with lots of huffing, puffing and
> grumbling.

As a Canonical employee who was involved in the multiarch design work at
the time, this is a pretty unfair-to-Debian version of history.  Yes,
some things took a bit longer to get organized on the Debian side for
various reasons, but the design and implementation work was done in
collaboration with key people in Debian and was definitely better for
it; Guillem and Raphaël in particular did a lot of hard work on dpkg and
dpkg-dev respectively.  (Also, several of us on the Canonical side
regarded ourselves as having one foot firmly in each camp; I certainly
didn't see it as a confrontational sort of thing where we were having to
drag Debian along with us - rather the contrary, there was a lot of
enthusiasm in Debian for it.)

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sat, 2021-08-21 at 16:20 +0200, Wouter Verhelst wrote:
> On Sat, Aug 21, 2021 at 02:40:02PM +0100, Luca Boccassi wrote:
> > On Sat, 2021-08-21 at 10:26 +0200, Wouter Verhelst wrote:
> > > It bothers me that you believe "we've been doing this for a while
> > > and it didn't cause any problems, so let's just continue doing
> > > things that way even if the people who actually wrote the damn
> > > code
> > > say that path is littered with minefields and they're scared of
> > > what
> > > could happen when we finish the tranition this way" is a valid
> > > strategy. It goes against everything I was taught to do to write
> > > reliable software.
> > 
> > Many people are bothered by many things - such is life. For
> > example, I
> > am very bothered that it appears impossible to do any kind of
> > project-
> > wide innovation in Debian,
> 
>   "I don't deny the benefits.  I do think that in the current
>   implementation, the drawbacks outweigh those benefits.  That's not
> to
>   say it couldn't be done.  But if it is done, we should do it
> *right*.
>   We're Debian.  That's what we do."
> 
>   -- Colin Walters,  
> https://lists.debian.org/debian-devel/2003/06/msg00475.html
> 
> It's true that there are other distributions out there who go for the
> quick-and-dirty solution, who want the feature before the benefits,
> downsides, and risks have all been fleshed out. There's a reason why
> I'm
> not contributing to those distributions; there's a reason why I don't
> use those distributions.
> 
> "Doing it right", even if that takes time, has proven benefits.
> 
> When the RPM world implemented "multiarch", they only supported
> installing 32-bit binaries on 64-bit versions of the same
> architecture.
> They did have that feature implemented and functioning in a few
> months
> or so, but the functionality of it was very limited -- and even today
> it
> has problems, in that the way in which RPM checks that packages are
> correct has some inherent heuristics that can make mistakes. Yes,
> I've
> encountered those in practice on CentOS systems that my customers at
> the
> time really really wanted to get up and running again pretty quickly.
> 
> When Debian and Ubuntu implemented multiarch (look ma, no quotes),
> the
> time from concept to tests to implementation to public availability
> was
> *much* longer than it was in the RPM world; and while this work was
> unfinished, there was a lot of angry nagging about the lack of this
> feature and why can they do it in the RPM world you guys are idiots,
> but
> eventually it was implemented; and I think you'll agree that the dpkg
> implementation of multiarch is far superior to the RPM one: it's
> possible to use multiarch not just for compatibility with 32-bit
> versions of your 64-bit platform, as in the RPM world, but *also* for
> running arm binaries on x86 with qemu user emulation, or for
> cross-compiling, or for various other features that the RPM world can
> only dream of.

My recollection (which might be wrong, but a quick look at release
notes seems to support it with 11.04 having multiarch 2 years before
Wheezy) is that Canonical led the way with the multiarch effort in
Ubuntu, and Debian followed with lots of huffing, puffing and
grumbling.

> To get back to the point: I'm not saying we shouldn't merge /usr. We
> should; the benefits of a properly merged /usr far outweigh any
> disadvantages it may bring.  However, having an inconsistent dpkg
> database is far more serious than just "oh dpkg -S won't work as
> expected". It means dpkg isn't properly keeping track of which files
> belong to which package anymore, which means you will have issues
> with a
> package that Replaces: another, or with removing packages (especially
> with security-conscious binaries), or with diversions, or with
> alternatives, or with file conflicts, or with basically anything that
> asks dpkg about locations of files; and just dismissing it with a
> handwavy "ah well just run dpkg -S again" is so far removed from
> reality
> that it's not even funny. I think the dpkg maintainers are 100%
> correct
> to point out that that *is* a problem for which currently no viable
> solution seems to exist, and that any way forward *must* include a
> solution to that problem.
> 
> I'm not saying the solution which the dpkg maintainers are proposing
> is
> the only valid solution, but if you go and tell them "ah the real
> problems you point out are irrelevant" then You! Are! Doing! It!
> Wrong!

Again, if the magnitude of this dpkg bug was really that serious there
would be visible consequences after almost 3 years of deployments
across two distributions with who knows how many million instances, and
yet "having to run dpkg -S again" is all we can see. Where are the bug
reports? Where are the enraged users with unusable broken system and
lost data? Where are the reports of Canonical going out of business
because Ubuntu is unusable? The bug is real, nobody doubts that - it
has been filed on d

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Sat, Aug 21, 2021 at 10:26:13AM +0200, Wouter Verhelst wrote:
> It bothers me that you believe "we've been doing this for a while and it
> didn't cause any problems, so let's just continue doing things that way
> even if the people who actually wrote the damn code say that path is
> littered with minefields and they're scared of what could happen when we
> finish the tranition this way" is a valid strategy. It goes against
> everything I was taught to do to write reliable software.

So as an expert, what's your recommendation about what is to be done?
Personally, I *don't* have a problem about telling people to manually
update dpkg, apt, and/or apt-get before they do the next major stable
release (maybe it's because this is something I do as a matter of
course; it's not that much extra effort, and I'm a paranoid s.o.b.,
and I know that's the most tested path given how Debian testing
works).

Other people think that is a terrible idea, to be avoided at all
costs.  I don't understand why that is such a terrible outcome, since
I do it already, but perhaps I can be educated on that point.

In any case, I believe downsides of the symlink farm alternative are
greater than the downsides of other options:

* just living with the risk of potential corner cases which
  might affect users when they do the bullseye -> bookworm
  upgrade, since aparently the sky hasn't fallen with Ubuntu, or

* advise users to upgrade dpkg/apt/apt-get first when they do
  the next release, with the strength of that advice depending
  on how likely users are to suffer from a "mine" causing
  their system to lose a limb or two.

Heck, if the minefield is that dangerous (and if so, why the *heck*
aren't Ubuntu users screaming from data loss, system instability,
etc?), perhaps you should advise the release managers that dpkg needs
to have fixes pushed out to Bullseye NOW! NOW! NOW! to eliminate the
potential imminent damage that you seem to be so fearful of our users
might get hit with.

Can you give more details about real life scenarios which is
triggering your fears, and whether there are ways we can mitigate
against those scenarios?

Best regards,

- Ted

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Sat, Aug 21, 2021 at 02:40:02PM +0100, Luca Boccassi wrote:
> On Sat, 2021-08-21 at 10:26 +0200, Wouter Verhelst wrote:
> > It bothers me that you believe "we've been doing this for a while
> > and it didn't cause any problems, so let's just continue doing
> > things that way even if the people who actually wrote the damn code
> > say that path is littered with minefields and they're scared of what
> > could happen when we finish the tranition this way" is a valid
> > strategy. It goes against everything I was taught to do to write
> > reliable software.
> 
> Many people are bothered by many things - such is life. For example, I
> am very bothered that it appears impossible to do any kind of project-
> wide innovation in Debian,

  "I don't deny the benefits.  I do think that in the current
  implementation, the drawbacks outweigh those benefits.  That's not to
  say it couldn't be done.  But if it is done, we should do it *right*.
  We're Debian.  That's what we do."

  -- Colin Walters, https://lists.debian.org/debian-devel/2003/06/msg00475.html

It's true that there are other distributions out there who go for the
quick-and-dirty solution, who want the feature before the benefits,
downsides, and risks have all been fleshed out. There's a reason why I'm
not contributing to those distributions; there's a reason why I don't
use those distributions.

"Doing it right", even if that takes time, has proven benefits.

When the RPM world implemented "multiarch", they only supported
installing 32-bit binaries on 64-bit versions of the same architecture.
They did have that feature implemented and functioning in a few months
or so, but the functionality of it was very limited -- and even today it
has problems, in that the way in which RPM checks that packages are
correct has some inherent heuristics that can make mistakes. Yes, I've
encountered those in practice on CentOS systems that my customers at the
time really really wanted to get up and running again pretty quickly.

When Debian and Ubuntu implemented multiarch (look ma, no quotes), the
time from concept to tests to implementation to public availability was
*much* longer than it was in the RPM world; and while this work was
unfinished, there was a lot of angry nagging about the lack of this
feature and why can they do it in the RPM world you guys are idiots, but
eventually it was implemented; and I think you'll agree that the dpkg
implementation of multiarch is far superior to the RPM one: it's
possible to use multiarch not just for compatibility with 32-bit
versions of your 64-bit platform, as in the RPM world, but *also* for
running arm binaries on x86 with qemu user emulation, or for
cross-compiling, or for various other features that the RPM world can
only dream of.

To get back to the point: I'm not saying we shouldn't merge /usr. We
should; the benefits of a properly merged /usr far outweigh any
disadvantages it may bring.  However, having an inconsistent dpkg
database is far more serious than just "oh dpkg -S won't work as
expected". It means dpkg isn't properly keeping track of which files
belong to which package anymore, which means you will have issues with a
package that Replaces: another, or with removing packages (especially
with security-conscious binaries), or with diversions, or with
alternatives, or with file conflicts, or with basically anything that
asks dpkg about locations of files; and just dismissing it with a
handwavy "ah well just run dpkg -S again" is so far removed from reality
that it's not even funny. I think the dpkg maintainers are 100% correct
to point out that that *is* a problem for which currently no viable
solution seems to exist, and that any way forward *must* include a
solution to that problem.

I'm not saying the solution which the dpkg maintainers are proposing is
the only valid solution, but if you go and tell them "ah the real
problems you point out are irrelevant" then You! Are! Doing! It! Wrong!

[...]
> The main point is that of course the insights of experts are extremely
> important, incredibly valuable and worth careful consideration,
> especially when making decisions about an unknown future and events yet
> to unfold. But in this case these are predictions about the past, a
> past that already exists and is lived experience for many users here,
> and for all users in Ubuntu.

What that is, is anecdotal evidence. "We've been doing X for a while and
it seems to not kill everything". Cool, great, awesome data points, but
not likely to convince me that there won't ever be any problems. You
can't prove the absense of bugs by anecdotal evidence; you can only
prove the existence of them that way.

What the dpkg maintainers are providing is analytical evidence. "There's
some corner cases here which need to be catered to". You just can't say
that corner cases don't happen because "anecdotal evidence". That's just
not how any of that works.

[...]
> The reality of this industry is that reliable software is an oxymo

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sat, 2021-08-21 at 10:26 +0200, Wouter Verhelst wrote:
> On Fri, Aug 20, 2021 at 11:21:55AM +0100, Luca Boccassi wrote:
> > On Thu, 2021-08-19 at 19:55 -0400, Theodore Ts'o wrote:
> > > On Thu, Aug 19, 2021 at 10:39:45PM +0200, Simon Richter wrote:
> > > > 
> > > > I think no one likes that idea, but it's the only solution that
> > > > doesn't
> > > > immediately fail because it requires a dpkg update that hasn't
> > > > shipped with
> > > > the current stable release, breaks local packages (kernel
> > > > modules, firmware,
> > > > site-wide systemd configuration), or both.
> > > 
> > > This could be solved if we could somehow require dpkg to be
> > > updated
> > > before any other packages during the the next update, no?
> > > 
> > > Breaking this constraint means that we can't make "apt-get
> > > dist-update" work seemlessly --- but what if we were to change
> > > the
> > > documented procedure for doing a major update?
> > > 
> > > That's not ideal, granted, but how does that compare against the
> > > other
> > > alternatives?
> > > 
> > > - Ted
> > > 
> > > P.S.  I had a vague memory that there was some update in the long
> > > distant past where we did require a manual upgrade of dpkg
> > > first.  Or
> > > is my memory playing tricks on me?  I do know that a manual
> > > update of
> > > dpkg is the first step in a crossgrade
> > 
> > An update to dpkg is not _required_. It might be very strongly
> > _desired_ which is a perfectly legitimate stance to take, but it is
> > not
> > technically required, otherwise we couldn't have been shipping with
> > merged-usr as default in new installations of Buster and Bullseye
> > for
> > 2+ years, we could not have been installing usrmerge in older
> > installations for 2+ years, and Ubuntu would not exist anymore
> > since
> > legacy split-usr is discontinued and even older installations are
> > being
> > forcibly converted. So continuing to live with this minor ~20 years
> > old
> > dpkg bug as we've been doing for years is a valid option - one that
> > some might very, very strongly dislike and argue against which is
> > again
> > perfectly legitimate, but it is de-facto an option nonetheless,
> > because
> > it's the actual status quo for 2+ years.
> 
> It bothers me that you believe "we've been doing this for a while and
> it
> didn't cause any problems, so let's just continue doing things that
> way
> even if the people who actually wrote the damn code say that path is
> littered with minefields and they're scared of what could happen when
> we
> finish the tranition this way" is a valid strategy. It goes against
> everything I was taught to do to write reliable software.

Many people are bothered by many things - such is life. For example, I
am very bothered that it appears impossible to do any kind of project-
wide innovation in Debian, and that we have been delegating that to
other distros since forever, and seem condemned to, at best,
frantically play catch-up, and at worst be dragged, kicking and
screaming, into what has been normal everywhere else for a decade, by
upstreams tired and frustrated of having to maintain legacy code paths
for the sole and exclusive benefit of this project. But I digress.

The main point is that of course the insights of experts are extremely
important, incredibly valuable and worth careful consideration,
especially when making decisions about an unknown future and events yet
to unfold. But in this case these are predictions about the past, a
past that already exists and is lived experience for many users here,
and for all users in Ubuntu. So there need to be _very_ convincing
explanations on why these predictions do not seem to match reality at
all, and so far these explanations have failed to materialize, I'm
afraid. We have been told everything is broken and the sky is about to
fall any minute now, and yet we have not been inundated with new bugs
since Buster made merged-usr the default, we have not been inundated
with new bugs by users upgrading from Buster to Bullseye or installing
usrmerge (despite the yet unsubstantiated claim in this thread that apt
dist-upgrade would stop working and require abandoning as a way to
upgrade the distro), and Canonical is not drowning in bug reports for
making usrmerge the mandated reality of 100% of its user base. The
usrmerge Launchpad has 2 (two) bugs [1]. The usrmerge Debian page has 4
(four) bugs, two of which are feature requests [2]. This is hardly the
stuff of nightmares. If one's expert viewpoints and predictions do not
match reality, they are not entitled to gloss over it because they are
the recognized and widely appreciated foremost expert in the field.

The reality of this industry is that reliable software is an oxymoron:
the only bug-free software is the one that doesn't exist. So the
question becomes, what is the measurable impact and magnitude of known
bugs and what is the likelihood and projected appearance rate of
unkno

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Fri, 2021-08-20 at 23:15 +0200, Simon Richter wrote:
> Hi,
> 
> On 8/20/21 3:56 PM, Sam Hartman wrote:
> 
> > Simon's position seemed to be that we need a dpkg update  in order
> > to
> > move forward and that we cannot depend on that mid-release.
> 
> Yes, except if we give up "apt dist-upgrade" as the interface for the
> upgrade to the next stable release.

Forgive me, but you have made this statement a few times now, and each
time you have been asked for a reference to what usrmerge-specific
measures were adopted in Ubuntu's distro updater to justify it, but so
far I have not seen any (I might have simply missed it, in which case I
apologize in advance). Could you please share such reference(s) so that
we can understand what you are referring to?

> > I can see two arguments why we might need a dpkg update:
> > 
> > 1)  To fix bugs related to directory aliasing.
> > 
> > I don't think that there is a consensus those bugs need to be fixed
> > to
> > move forward.  (Put another way it's not clear the community agrees
> > they
> > are RC).
> 
> dpkg as it is now will detect only the case when a file is moved to
> the 
> usrmerge path inside a package, or when a file is moved from one
> package 
> to another (with Replaces), but not both at the same time.

Your example is essentially having /bin/foo and /usr/bin/foo at the
same time, while not being bit-by-bit identical. I was under the
impression that this was already considered RC-buggy and has been for a
long while? Am I mistaken and is that not the case? If so, how common
is it in the archive, do we have numbers?

> > IN particular, most systems are usrmerged today, and while these
> > bugs
> > are annoying, many people get along just fine.
> 
> Yes, and on all of these systems, dpkg does not have an accurate view
> of 
> what is installed, so an upgrade to bookworm that moves files between
> packages is likely to have files vanish, depending on the order in
> which 
> packages are installed.

Why has this not happened upgrading from Buster to Bullseye? Or 19.10
to 20.04? Or 20.04 to 20.10? Etc. What's different in Bookworm that
would cause these issues to suddenly appear?

> > Yes, there are bugs.
> > Yes, it would be good to get them fixed in the bookworm cycle.
> > But despite the issue being brought up, there is not strong support
> > for
> > the idea that we must block on a solution to the dpkg directory
> > aliasing
> > bugs.
> 
> I think that one of the release goals should be that any freshly 
> installed or upgraded system should have a dpkg database that is 
> consistent with reality, and I'd prioritize that higher than actually
> finishing the transition, because as long as we can have files vanish
> from under us, we can't expect that the transition will be reliable
> for 
> bullseye->bookworm updates.
> 
> Basically, we've been lucky so far.

Sorry but I for one strongly disagree that this should be a release
goal at all. It's an internal implementation detail caused by a 20
years old bug in dpkg - the maintainers of which are free to fix such
bug at any time if they like (as they were in the past 20 years since
this has been open and remained unaddressed), or to keep ignoring it.
What is very relevant is which real consequences this does bring to the
surface for users, and so far the only proven, reported and unsolved
issue I have seen is the dpkg -S inconvenience, despite multiple years
and an install base approaching 100% of a very popular distro like
Ubuntu - of course, it is entirely possible that I simply missed them,
in which case links to relevant Launchpad/bugs.d.o would be more than
welcome.

> > 2)
> > We might need a dpkg update to actually do the transition.
> 
> Yes, that was my symlink farmer idea, but that doesn't work: 
> special-case replacing a directory that only contains symlinks with a
> symlink, the same way GNU Stow tries to create its symlinks on the 
> highest possible level, but since dpkg refuses to even unpack a
> package 
> that contains a compatibility symlink and the file itself on an 
> usrmerged system, this won't work.
> 
> Since we need a mechanism to clean up the mess the usrmerge package
> left 
> us with, adding this mechanism to dpkg itself might still be a good
> idea 
> -- while that isn't the first package to be upgraded, it is among the
> first, so we can create a safe environment for later packages and
> thus 
> minimize the number of packages we have to leave in "reinstreq"
> state.
> 
>     Simon

Sorry, but so far it is not proven at all that we _need_ to clean up
anything. Some people would very strongly like to do so, and that's a
fine and legitimate opinion to hold. But it is factual that it is not
objectively _needed_, otherwise Ubuntu wouldn't ship and default-usr-
merged Debian installations since Buster wouldn't exist, and older-and-
migrated-via-usrmerge installations since Buster wouldn't exist either.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitall

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Fri, Aug 20, 2021 at 11:21:55AM +0100, Luca Boccassi wrote:
> On Thu, 2021-08-19 at 19:55 -0400, Theodore Ts'o wrote:
> > On Thu, Aug 19, 2021 at 10:39:45PM +0200, Simon Richter wrote:
> > > 
> > > I think no one likes that idea, but it's the only solution that doesn't
> > > immediately fail because it requires a dpkg update that hasn't shipped 
> > > with
> > > the current stable release, breaks local packages (kernel modules, 
> > > firmware,
> > > site-wide systemd configuration), or both.
> > 
> > This could be solved if we could somehow require dpkg to be updated
> > before any other packages during the the next update, no?
> > 
> > Breaking this constraint means that we can't make "apt-get
> > dist-update" work seemlessly --- but what if we were to change the
> > documented procedure for doing a major update?
> > 
> > That's not ideal, granted, but how does that compare against the other
> > alternatives?
> > 
> > - Ted
> > 
> > P.S.  I had a vague memory that there was some update in the long
> > distant past where we did require a manual upgrade of dpkg first.  Or
> > is my memory playing tricks on me?  I do know that a manual update of
> > dpkg is the first step in a crossgrade
> 
> An update to dpkg is not _required_. It might be very strongly
> _desired_ which is a perfectly legitimate stance to take, but it is not
> technically required, otherwise we couldn't have been shipping with
> merged-usr as default in new installations of Buster and Bullseye for
> 2+ years, we could not have been installing usrmerge in older
> installations for 2+ years, and Ubuntu would not exist anymore since
> legacy split-usr is discontinued and even older installations are being
> forcibly converted. So continuing to live with this minor ~20 years old
> dpkg bug as we've been doing for years is a valid option - one that
> some might very, very strongly dislike and argue against which is again
> perfectly legitimate, but it is de-facto an option nonetheless, because
> it's the actual status quo for 2+ years.

It bothers me that you believe "we've been doing this for a while and it
didn't cause any problems, so let's just continue doing things that way
even if the people who actually wrote the damn code say that path is
littered with minefields and they're scared of what could happen when we
finish the tranition this way" is a valid strategy. It goes against
everything I was taught to do to write reliable software.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 8/20/21 3:56 PM, Sam Hartman wrote:


Simon's position seemed to be that we need a dpkg update  in order to
move forward and that we cannot depend on that mid-release.


Yes, except if we give up "apt dist-upgrade" as the interface for the 
upgrade to the next stable release.



I can see two arguments why we might need a dpkg update:

1)  To fix bugs related to directory aliasing.

I don't think that there is a consensus those bugs need to be fixed to
move forward.  (Put another way it's not clear the community agrees they
are RC).


dpkg as it is now will detect only the case when a file is moved to the 
usrmerge path inside a package, or when a file is moved from one package 
to another (with Replaces), but not both at the same time.


So when I build a test package

Package: test1
Version: 1
Architecture: all
Maintainer: foo
Description: foo

containing

./lib
./lib/test

and a second package

Package: test2
Version: 1
Architecture: all
Replaces: test1 (= 1)
Maintainer: foo
Description: foo

containing

./usr
./usr/lib
./usr/lib/test

and I install both, then dpkg will have test1 registered as the owner of 
/lib/test and test2 as the owner of /usr/lib/test. Uninstalling test1 
then deletes /lib/test, so /usr/lib/test is also gone.


We can work around that by introducing a policy that no files that are 
currently registered as living outside /usr may be moved between 
packages, but... just no.


Good thing /usr/bin/which doesn't need to be moved from /bin, or we'd 
have our first counterexample already. :/


The current dpkg is also unusable for symlink farming, so I will have to 
retract my earlier statement about that being a viable transition 
method: it is not.


Package: test
Version: 1
Architecture: all
Maintainer: foo
Description: foo

containing

./lib
./usr
./usr/lib
./usr/lib/test
./lib/test -> ../usr/lib/test

fails to unpack on usrmerged systems with

 unable to open '/usr/lib/test.dpkg-new': No such file or directory

This is the version of dpkg that will handle the beginning of the "apt 
dist-upgrade" process for bullseye->bookworm updates, and the point at 
which dpkg itself will be upgraded is determined by the current version 
of apt, so our hands are kind of tied here.



IN particular, most systems are usrmerged today, and while these bugs
are annoying, many people get along just fine.


Yes, and on all of these systems, dpkg does not have an accurate view of 
what is installed, so an upgrade to bookworm that moves files between 
packages is likely to have files vanish, depending on the order in which 
packages are installed.



Yes, there are bugs.
Yes, it would be good to get them fixed in the bookworm cycle.
But despite the issue being brought up, there is not strong support for
the idea that we must block on a solution to the dpkg directory aliasing
bugs.


I think that one of the release goals should be that any freshly 
installed or upgraded system should have a dpkg database that is 
consistent with reality, and I'd prioritize that higher than actually 
finishing the transition, because as long as we can have files vanish 
from under us, we can't expect that the transition will be reliable for 
bullseye->bookworm updates.


Basically, we've been lucky so far.


2)
We might need a dpkg update to actually do the transition.


Yes, that was my symlink farmer idea, but that doesn't work: 
special-case replacing a directory that only contains symlinks with a 
symlink, the same way GNU Stow tries to create its symlinks on the 
highest possible level, but since dpkg refuses to even unpack a package 
that contains a compatibility symlink and the file itself on an 
usrmerged system, this won't work.


Since we need a mechanism to clean up the mess the usrmerge package left 
us with, adding this mechanism to dpkg itself might still be a good idea 
-- while that isn't the first package to be upgraded, it is among the 
first, so we can create a safe environment for later packages and thus 
minimize the number of packages we have to leave in "reinstreq" state.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Fri, Aug 20, 2021 at 07:56:33AM -0600, Sam Hartman wrote:
> As you know, one of the ways we can see  how close we are on consensus
> is to look at what happens when someone proposes a summary like you did.

Thanks, that was my goal: trying to see if we could move the
discussion towards some kind of community consensus.

> Simon Richter tried to challenge your summary effectively saying that we
> couldn't have an informed consensus because there were open technical
> issues that had not been addressed.  This was roundly rejected by
> yourself, Philip Hands and Luca Boccassi.
> 
> Simon's position seemed to be that we need a dpkg update  in order to
> move forward and that we cannot depend on that mid-release.
> 
> You talked about ways we could get a dpkg update at the beginning of the
> release process.
> Luca and Philip made more structural objections.
> 
> Simon did not clearly explain *why* we need a dpkg update.

Fair enough.  I am unclear on whether we need a dpkg update; I do
believe, though, that even if we needed a dpkg update for some reason,
though, we should explore options other than /usr unification via
symlink farms, which I continue to believe is highly undesirable
choice.

> I can see two arguments why we might need a dpkg update:
> 
> 1)  To fix bugs related to directory aliasing.
> 
> I don't think that there is a consensus those bugs need to be fixed to
> move forward.  (Put another way it's not clear the community agrees they
> are RC).

Actually, I think we can make a stronger statement than that.  Even if
the dpkg bugs relating to directory aliasing which are release
critical in terms of severity (and it's unclear to me whether they
rise to that level), the specifically germane question is whether they
have to be fixed *before* proceeding with the bullseye->bookworm
update.  After all, it might be that say, "dpkg-query -S" can fail[1],
but even *if* this were considered priority "serious" --- which I do
not believe --- if it doesn't break upgrading systems, then it can be
fixed when dpkg is upgraded, and we don't have to upgrade dpkg first.

(And again, is it *really* that bad if we tell users that it is
advisable to upgrade dpkg first?  TBH, I very often will upgrade dpkg
and apt by hand, before running "apt dist-upgrade" to this day,
because of past experience from upgrades long, long ago...  Maybe it's
a cargo cult practice, like typing "sync" three times, but it's not
that hard!)

So perhaps one path forwarwd is to example the breakages listed in
[1], and consider how likely any of them are likely to break upgrades.
I will admit that concerns around update-alternatives and dpkg-divert
sound the scariest --- but I've been using a usrmerged system for a
while, and nothing as broken for me.  And as others have pointed out,
Ubuntu has been using usrmerged systems exclusively for a while now,
"going behind dpkg's back", and there doesn't seem to have been a lot
of reports of disaster befalling Ubuntu.  Is there things they are
doing to mitigate the potential problems around dpkg-divert and
update-alternatives?  What can we learn from the Ubuntu experience?

[1] https://wiki.debian.org/Teams/Dpkg/MergedUsr

> However, there are proposed solutions under development that in terms of
> being favored in a consensus discussion  are preferable to
> usr-merge-via-symlink farms:
> 
> A) An extraordinary upgrade process.  For example requiring that if you
> are running on a system that is not usrmerged already, you need to
> install usrmerge at the beginning of the upgrade.
> (it could still be transitively essential, but explicitly asking people
> where it matters to install early).
> 
> b) Require that bookworm packages work on non-usrmerged systems and
> support non-usrmerged build chroots in the bookworm cycle.
> 
> None of these solutions are ideal.
> There is still technical work to do, and  there a absolutely are open
> technical issues.

Agreed.  And if the folks who are working can let us know how we can
help determine how we can come to some kind of resolution on those
open technical issues, that would be great.  Letting things drag out
isn't going to be helpful, so once we have mature options to consider,
hopefully we can weigh the pros and the cons and come to some kind of
group "hum" about the best path forward.

(And I'm having flashbacks to my days as an IETF working group chair.  :-)

  - Ted

Re: merged /usr vs. symlink farms

[Sam Hartman]

> "Theodore" == Theodore Ts'o  writes:

Theodore> On Thu, Aug 19, 2021 at 11:17:17AM +0100, Simon McVittie wrote:
>> In this specific case, I think the thing you're having a problem
>> with is the gradual, file-by-file migration of executables into
>> /usr by individual packages and individual packages'
>> maintainers. That's not merged-/usr: merged-/usr does the
>> migration all at once, by creating the aliasing symlinks (and
>> then we can clean up the contents of data.tar.* to put all
>> /usr-like files below /usr at our leisure, during the next
>> release cycle, without needing maintainer script glue).

Theodore> FWIW, from following the discussion, I've become more and
Theodore> more convinced that a symlink farm is *not* the right
Theodore> answer, regardless of whether it is done centrally or via
Theodore> individual packages moving files and created symlinks if
Theodore> necessary in individual maintainer scripts.

Ted, in terms of judging consensus, I'd like to explicitly ACK your
summary.
I think that you have accurately described where we are in the
discussion.

As you know, one of the ways we can see  how close we are on consensus
is to look at what happens when someone proposes a summary like you did.

Simon Richter tried to challenge your summary effectively saying that we
couldn't have an informed consensus because there were open technical
issues that had not been addressed.  This was roundly rejected by
yourself, Philip Hands and Luca Boccassi.

Simon's position seemed to be that we need a dpkg update  in order to
move forward and that we cannot depend on that mid-release.

You talked about ways we could get a dpkg update at the beginning of the
release process.
Luca and Philip made more structural objections.

Simon did not clearly explain *why* we need a dpkg update.
I can see two arguments why we might need a dpkg update:

1)  To fix bugs related to directory aliasing.

I don't think that there is a consensus those bugs need to be fixed to
move forward.  (Put another way it's not clear the community agrees they
are RC).

IN particular, most systems are usrmerged today, and while these bugs
are annoying, many people get along just fine.
Yes, there are bugs.
Yes, it would be good to get them fixed in the bookworm cycle.
But despite the issue being brought up, there is not strong support for
the idea that we must block on a solution to the dpkg directory aliasing
bugs.


2)
We might need a dpkg update to actually do the transition.
Thatt is somehow dpkg mechanisms replace what the usrmerge package
does.  Or alternatively dpkg mechanisms give us a facility to run
usrmerge early enough.

It's absolutely true that there is an open technical issue of how to
trigger the transition.
However, there are proposed solutions under development that in terms of
being favored in a consensus discussion  are preferable to
usr-merge-via-symlink farms:

A) An extraordinary upgrade process.  For example requiring that if you
are running on a system that is not usrmerged already, you need to
install usrmerge at the beginning of the upgrade.
(it could still be transitively essential, but explicitly asking people
where it matters to install early).

b) Require that bookworm packages work on non-usrmerged systems and
support non-usrmerged build chroots in the bookworm cycle.

None of these solutions are ideal.
There is still technical work to do, and  there a absolutely are open
technical issues.

My reading of the discussion is the same as yours though.  We have an
informed consensus that usrmerge-via-symlinks is not the direction we
should take.
I recognize that there are people who are not part of that consensus:
Simon Richter and Guillem Jover  among them.
I think the project has given people who are in the rough an opportunity
to present their objections and has taken the time to understood
technical objections that were presented.


I also agree with your reading of the history that there were process
problems in how we interacted with the dpkg team.
I agree with you that is water under the bridge in terms of our
technical decision today.
I hope we choose to learn from that for better future decision making,
but I do not think either our users or the free software community would
be served by letting those process concerns stop technical progress.


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Philip Hands]

Luca Boccassi  writes:

> On Thu, 2021-08-19 at 19:55 -0400, Theodore Ts'o wrote:
>> On Thu, Aug 19, 2021 at 10:39:45PM +0200, Simon Richter wrote:
>> > 
>> > I think no one likes that idea, but it's the only solution that doesn't
>> > immediately fail because it requires a dpkg update that hasn't shipped with
>> > the current stable release, breaks local packages (kernel modules, 
>> > firmware,
>> > site-wide systemd configuration), or both.
>> 
>> This could be solved if we could somehow require dpkg to be updated
>> before any other packages during the the next update, no?
>> 
>> Breaking this constraint means that we can't make "apt-get
>> dist-update" work seemlessly --- but what if we were to change the
>> documented procedure for doing a major update?
>> 
>> That's not ideal, granted, but how does that compare against the other
>> alternatives?
>> 
>>  - Ted
>> 
>> P.S.  I had a vague memory that there was some update in the long
>> distant past where we did require a manual upgrade of dpkg first.  Or
>> is my memory playing tricks on me?  I do know that a manual update of
>> dpkg is the first step in a crossgrade
>
> An update to dpkg is not _required_. It might be very strongly
> _desired_ which is a perfectly legitimate stance to take, but it is not
> technically required, otherwise we couldn't have been shipping with
> merged-usr as default in new installations of Buster and Bullseye for
> 2+ years, we could not have been installing usrmerge in older
> installations for 2+ years, and Ubuntu would not exist anymore since
> legacy split-usr is discontinued and even older installations are being
> forcibly converted. So continuing to live with this minor ~20 years old
> dpkg bug as we've been doing for years is a valid option - one that
> some might very, very strongly dislike and argue against which is again
> perfectly legitimate, but it is de-facto an option nonetheless, because
> it's the actual status quo for 2+ years.

Well, quite -- we seem to have had predictions of the sky falling as a
result of usrmerge and/or merged-/usr, but if it is falling it's doing
it remarkably slowly.

I've been paying attention to this for a while, what with being on the
TC up until just before the unanimous decision (which I fully support).

I'm as sure as one can be about the future that if in 20 years I am
still around to type this into a supported Debian system:

  [ $(stat -Lc %i /bin) != $(stat -Lc %i /usr/bin) ] && cowsay Surprise!

I will not see a talking cow[1].

I'm also pretty sure that the 2041 version of dpkg will either be
completely relaxed about having /bin be a symlink, or we'll be using
something else by then.

That being the case, we might as well get on with it rather than trying
to pretend that filling everyone's disks with shedloads of symlinks for
a while in-between now and then is a useful thing to do.

Cheers, Phil.

[1] unless of course there's a reason to use some sort of bind-mount or
other filesystem trickery that's invented in the interim to achieve the
same result.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Thu, 2021-08-19 at 19:55 -0400, Theodore Ts'o wrote:
> On Thu, Aug 19, 2021 at 10:39:45PM +0200, Simon Richter wrote:
> > 
> > I think no one likes that idea, but it's the only solution that doesn't
> > immediately fail because it requires a dpkg update that hasn't shipped with
> > the current stable release, breaks local packages (kernel modules, firmware,
> > site-wide systemd configuration), or both.
> 
> This could be solved if we could somehow require dpkg to be updated
> before any other packages during the the next update, no?
> 
> Breaking this constraint means that we can't make "apt-get
> dist-update" work seemlessly --- but what if we were to change the
> documented procedure for doing a major update?
> 
> That's not ideal, granted, but how does that compare against the other
> alternatives?
> 
>   - Ted
> 
> P.S.  I had a vague memory that there was some update in the long
> distant past where we did require a manual upgrade of dpkg first.  Or
> is my memory playing tricks on me?  I do know that a manual update of
> dpkg is the first step in a crossgrade

An update to dpkg is not _required_. It might be very strongly
_desired_ which is a perfectly legitimate stance to take, but it is not
technically required, otherwise we couldn't have been shipping with
merged-usr as default in new installations of Buster and Bullseye for
2+ years, we could not have been installing usrmerge in older
installations for 2+ years, and Ubuntu would not exist anymore since
legacy split-usr is discontinued and even older installations are being
forcibly converted. So continuing to live with this minor ~20 years old
dpkg bug as we've been doing for years is a valid option - one that
some might very, very strongly dislike and argue against which is again
perfectly legitimate, but it is de-facto an option nonetheless, because
it's the actual status quo for 2+ years.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: merged /usr vs. symlink farms

[Craig Small]

On Fri, 20 Aug 2021 at 09:56, Theodore Ts'o  wrote:

> P.S.  I had a vague memory that there was some update in the long
> distant past where we did require a manual upgrade of dpkg first.  Or
> is my memory playing tricks on me?  I do know that a manual update of
> dpkg is the first step in a crossgrade
>
There was an instance of this. I cannot remember if it was the libc4/a.out
or something to do with libstdc but it was library-related and a bit of a
pain for end-users.

In any case, it was not an ideal situation or something we should aim for.

 - Craig

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Thu, Aug 19, 2021 at 10:39:45PM +0200, Simon Richter wrote:
> 
> I think no one likes that idea, but it's the only solution that doesn't
> immediately fail because it requires a dpkg update that hasn't shipped with
> the current stable release, breaks local packages (kernel modules, firmware,
> site-wide systemd configuration), or both.

This could be solved if we could somehow require dpkg to be updated
before any other packages during the the next update, no?

Breaking this constraint means that we can't make "apt-get
dist-update" work seemlessly --- but what if we were to change the
documented procedure for doing a major update?

That's not ideal, granted, but how does that compare against the other
alternatives?

- Ted

P.S.  I had a vague memory that there was some update in the long
distant past where we did require a manual upgrade of dpkg first.  Or
is my memory playing tricks on me?  I do know that a manual update of
dpkg is the first step in a crossgrade

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 8/19/21 4:45 PM, Theodore Ts'o wrote:


FWIW, from following the discussion, I've become more and more
convinced that a symlink farm is *not* the right answer, regardless of
whether it is done centrally or via individual packages moving files
and created symlinks if necessary in individual maintainer scripts.


I think no one likes that idea, but it's the only solution that doesn't 
immediately fail because it requires a dpkg update that hasn't shipped 
with the current stable release, breaks local packages (kernel modules, 
firmware, site-wide systemd configuration), or both.


The dpkg team are rightfully skeptical about introducing a policy 
decision into the codebase, especially as the next dist-upgrade that 
users are going to perform will upgrade dpkg somewhere in the middle of 
the process, after several of its dependencies, which are precisely the 
packages affected. You can't change default behavior here, you can't 
introduce a new critical control field because it would create a 
Depends/Pre-Depends loop, ...


What *could* be done in dpkg is to change the policy for replacing a 
directory with a symlink. Currently, those entries are dropped, and the 
comment next to the code responsible for that suggests that the most 
common scenario where that would be seen were broken packages where 
someone swapped source and destination while creating a symlink.


But: a new policy would still have to honor file conflicts. The /lib 
directory cannot be replaced with a symlink until all packages that ship 
a directory here are gone. Because /lib/ld-linux.so.2 is part of the 
ABI, that cannot happen on its own, so we can probably narrow this down 
to "until all packages that ship regular files below /lib are gone."


As I understand it, that is where the symlink farming approach comes 
from: when the other packages move their regular files out of the way 
and replace them by symlinks, we have a criterion by which we can decide 
that the entire hierarchy can be replaced by a symlink.


We still need a mechanism in either dpkg or a package handling the 
transition that actually performs this operation. If we can do this in a 
package without touching dpkg, then IMO that would be preferable, but in 
either case that mechanism needs to be defined first.



Speaking personally, I'm not super excited about /usr unification.
But then again, I don't work on projects such as embedded systems,
containerized systems, etc., which seem to benefit from /usr
unification, and there *is* value in being similar to other Linux
distributions.


In my embedded projects, unification is counterproductive, because the 
initramfs effectively takes on the functionality of the root filesystem, 
except it cannot be modified in-place anymore with dpkg, and instead I 
need a script that copies relevant files, follows dependencies and then 
creates a compressed read-only root filesystem I can use for early boot.


That is about as convenient as using cramfs as the root filesystem, 
except it needs a lot more memory, and I cannot prepare this on the host.


Starting systemd without /usr, /proc and /sys mounted has been broken 
for ages, so booting without an initramfs simply does not work anyway there.


With sysvinit, it still mostly works, although some programs (also 
mostly from the systemd ecosystem) fail before /usr is mounted as they 
are missing libraries.


My expectation is that systemd will take over initramfs generation 
during the next release cycle, that might make the situation a bit more 
bearable as we can then reuse dependency information from units instead 
of having a shell script recreate it using heuristics.


Right now, a kernel update on a 150 MHz NiosII CPU takes several hours 
because the initramfs rebuild is just so slow, so for embedded systems, 
Debian as it is is close to unusable anyway and I believe embedded 
systems vendors have jumped ship a while ago. I have.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Thu, Aug 19, 2021 at 11:17:17AM +0100, Simon McVittie wrote:
> In this specific case, I think the thing you're having a problem with is
> the gradual, file-by-file migration of executables into /usr by individual
> packages and individual packages' maintainers. That's not merged-/usr:
> merged-/usr does the migration all at once, by creating the aliasing
> symlinks (and then we can clean up the contents of data.tar.* to put all
> /usr-like files below /usr at our leisure, during the next release cycle,
> without needing maintainer script glue).

FWIW, from following the discussion, I've become more and more
convinced that a symlink farm is *not* the right answer, regardless of
whether it is done centrally or via individual packages moving files
and created symlinks if necessary in individual maintainer scripts.

The symlink farm idea seems to be pushed by the dpkg team, because
it's clear that supprorting directory aliasing by having /bin ->
usr/bin, /lib -> /usr/lib, etc., top-level symlinks does create more
work for the dpkg team, and they seem to be put off by the fact that
they hadn't agreed to do that work, and they appear to claim that they
weren't consulted in advance.

But if we are going to follow how Fedora, Solaris, etc, have been
moving elimitating the traditional /{bin,sbin,lib} and
/usr/{bin,sbin,lib} split, directory aliasing the way Fedora, Solaris,
etc. have done things is the only way to go.

Perhaps the dpkg team should have been consulted earlier, and if they
could have convincingly argued that this was a show stopper, or they
had demanded that someone else should have provided the engineering
effort to make dpkg handle the directory aliasing *first*, perhaps we
shouldn't have even stated in the /{bin,sbin,lib} ->
/usr/{bin,sbin,lib} unification journey, despite the fact that all of
the other distributions have gone down that path.

Speaking personally, I'm not super excited about /usr unification.
But then again, I don't work on projects such as embedded systems,
containerized systems, etc., which seem to benefit from /usr
unification, and there *is* value in being similar to other Linux
distributions.

In any case, that's water on the bridge.  We are where we are, and
stopping midway through the /usr unification journey would be a far
worse outcome.  And given that we've already lost the benefits of the
split /usr architecture (specifically, the ability to boot without
/usr being mounted, which I recognize is not as useful in the 21st
century) --- we should push on and finish the job.

Given that symlink farms have all sorts of downsides, the best path
foward seems to be to teach dpkg about the top-level directory
aliasing, and simply handling this appropriately.  Issues such as the
/bin/sh vs. /usr/bin/sh unification causing problems with /etc/shells
is an issue all distributions have to deal with anyway, and we can
look to see how they have handled it.

Cheers,

- Ted

Re: merged /usr vs. symlink farms

[Simon McVittie]

On Thu, 19 Aug 2021 at 10:06:27 +0200, Helmut Grohne wrote:
> You keep proposing adding /bin/foo -> /usr/bin/foo symbolic links via
> maintainer scripts.

I'm not proposing this! I'm trying to *not* need to do that in any more
packages, and instead do usrmerge or equivalent, so that individual
packages' maintainers don't have to take per-package action to move
their files from /bin,/sbin,/lib* into /usr while creating compatibility
symlinks for non-merged-/usr systems.

However, I know Guillem and some others object to that strategy, so I'm
trying to also be clear about which of the fixes that are necessary with
usrmerge would be equally necessary for a symlink-farm-based strategy,
if people who prefer a symlink-farm-based strategy gain consensus for
that strategy.

Exactly how the symlinks in a symlink-farm-based strategy are created
is orthogonal to whether packages need to avoid hard-coding (e.g.) /bin/env
or /usr/bin/sh, in filesystem layouts where both paths with and without /usr
exist, which would break the installation of that package onto the traditional
filesystem layout where only /usr/bin/env and /bin/sh exist.

I agree that if a symlink-farm-based strategy is used, then delegating the
creation of the individual symlinks to individual packages' maintainer
scripts has practical problems, and it would be better to centralize the
creation of those symlinks (somehow). I think it's up to the people who
want to generate symlink farms to solve that problem.

Note that what the Technical Committee resolved as the desired state
is merged /usr with aliasing symlinks such as /bin -> usr/bin, not a
symlink farm with individual symlinks such as /bin/sh -> /usr/bin/sh,
precisely because we are concerned about the number of "moving parts"
involved in setting up a symlink farm. (Speaking for myself here, not
for the TC, but I don't think I'm misrepresenting anyone's position on
the symlink farm approach by saying this.)

> The collateral damage of the merged-/usr
> work to the work I'm interested in is huge.

In this specific case, I think the thing you're having a problem with is
the gradual, file-by-file migration of executables into /usr by individual
packages and individual packages' maintainers. That's not merged-/usr:
merged-/usr does the migration all at once, by creating the aliasing
symlinks (and then we can clean up the contents of data.tar.* to put all
/usr-like files below /usr at our leisure, during the next release cycle,
without needing maintainer script glue).

The aliasing symlinks create problems for dpkg, as Guillem has documented
elsewhere, and as a result some people are pursuing a symlink-farm-based
alternative to the aliasing symlinks. If that symlink-farm-based approach
is taken, then yes, we will need either a centralized mechanism to
construct those symlink farms, or a lot of maintainer script glue
(and, again, the Technical Committee's recommendation was to not do that).

The packages that needed maintainer-script changes *before* merged-/usr,
in order to enable merged-/usr, are those that previously shipped files
at both /foo and /usr/foo in their data.tar.*, such as
coreutils (<< 8.24-1) for /{usr/,}bin/touch. The reason they need
maintainer script code is that we still support non-merged-/usr systems;
their maintainer scripts are a no-op on merged-/usr systems, so if the
bookworm release only supported merged-/usr, then their maintainer
script code could disappear during the bookworm+1 cycle.

I agree that *those* maintainer scripts are creating extra work for
people working on DPKG_ROOT and similar things, and would not have been
necessary if we had not gone in the direction of merged-/usr in around
2016. If we can make those unnecessary, or more declarative, then that
would be a good direction.

smcv