Re: merged-/usr vs. partially-symlink-farmed-root
* Ansgar [210823 11:16]:
> Hi Marvin,
>
> On Mon, 2021-08-23 at 10:29 -0400, Marvin Renich wrote:
> > Yet they cannot be counted on to work on Debian now, nor will they on
> > non- or partially-merged systems. You are saying "the end result is
> > thus, so the partially merged system must have this property."
>
> No. I am comparing end results from two different proposals. I am not
> talking about any intermediate state.
>
> There is no replacing /bin with a /bin -> /usr/bin symlink ever in the
> partially-symlink-farmed-root proposal. So you only get the symlinks
> provided explicitly in /bin by packages and, e.g., dash would ship
> forever a /bin/sh -> /usr/bin/sh symlink and this would be present on
> all systems.
>
> Only non-required symlinks like for, say, run-parts could be dropped.
>
> See [1] about "finishing the transition".
>
> [1]: https://lists.debian.org/debian-devel/2019/02/msg00321.html
I did, indeed, miss this. I stand corrected; please accept my
apologies.
While I understand Guillem's rational, I believe that the effort to
clean up once there are only symlinks in /{,s}bin could be easily
automated and would technically be worth the effort (including code in
dpkg to recognize symlinks such as /bin/dash → /usr/bin/dash and mark
them in its database as "intentionally not placed on the filesystem").
However, at this point, I don't believe it is socially feasible to
continue down the symlink-farm path.
Again, I apologize for the misinformed argument.
...Marvin
Re: merged-/usr vs. partially-symlink-farmed-root
Hi Marvin, On Mon, 2021-08-23 at 10:29 -0400, Marvin Renich wrote: > Yet they cannot be counted on to work on Debian now, nor will they on > non- or partially-merged systems. You are saying "the end result is > thus, so the partially merged system must have this property." No. I am comparing end results from two different proposals. I am not talking about any intermediate state. There is no replacing /bin with a /bin -> /usr/bin symlink ever in the partially-symlink-farmed-root proposal. So you only get the symlinks provided explicitly in /bin by packages and, e.g., dash would ship forever a /bin/sh -> /usr/bin/sh symlink and this would be present on all systems. Only non-required symlinks like for, say, run-parts could be dropped. See [1] about "finishing the transition". [1]: https://lists.debian.org/debian-devel/2019/02/msg00321.html > Anyone who has a '#!/bin/python3' script now, must ensure the link is > there themselves, and that would not change in the middle of a > symlink-farm transition, nor would it hinder such transition. Yes, but where it would work once the transition to the new proposed layout differs between merged-/usr and partially-symlink-farmed-root (unless one does fully-symlink-farmed-root including symlinks for all binaries in /usr/bin and /usr/sbin, but that would be yet another different proposal). > However, do not use bogus arguments such as the one above to try to > maken your point. It clutters the discussion with needless > debunking. I think you misunderstand how the partially-symlink-farmed-root proposal is different from the merged-/usr proposal. Exactly to avoid such misunderstandings the partially-symlink-farmed-root proposal should not be named merged-/usr. Ansgar
Re: merged-/usr vs. partially-symlink-farmed-root
* Ansgar [210822 17:29]: > Hi, > > On Sun, 2021-08-22 at 12:29 -0400, Marvin Renich wrote: > > * Ansgar [210822 05:08]: > > > To get a filesystem layout equivalent to merged-/usr via symlinks > > > farming *every* package shipping files in at least /usr/bin, > > > /usr/sbin and possibly some of /usr/lib would need to include > > > symlinks in /bin, /sbin, /lib. This would affect far more > > > packages than updating the packages currently shipping files in > > > /bin, /sbin and /lib* to ship these under /usr instead. > > > > It is true that for a symlink-farm-usr-merge system to be strictly > > equivalent to a symlink-dir-usr-merge system, many packages that > > never had /bin/foo but had /usr/bin/foo would have to add a symlink > > /bin/foo, however this is clearly unnecessary. > > No, it is required if we want users to be able to run scripts that use > `#!/bin/python3` (or similar for other interpreters). Such scripts > exist in the wild, they were not initially written on Debian. > > These would need Debian-specific patches to work on Debian. It is just > the opposite of now having to patch software packaged in Debian to use > /bin/rm instead of /usr/bin/rm (that also happens in the wild for > software not developed on Debian). We also happen to miss such > instances and needlessly make software buggy on Debian. > > Not having to worry about yet another small incompatibility between > distributions is a nice feature. Yet they cannot be counted on to work on Debian now, nor will they on non- or partially-merged systems. You are saying "the end result is thus, so the partially merged system must have this property." That does not follow. It is part of the impatience for a finished product that usually ends up with something less ideal than would be possible with a more patient approach. (And the symlink-dir approach appears to me to be driven primarily by impatience.) Anyone who has a '#!/bin/python3' script now, must ensure the link is there themselves, and that would not change in the middle of a symlink-farm transition, nor would it hinder such transition. I was initially in favor of the symlink-farm approach, because it is the correct technical solution. I have become convinced that the symlink-dir approach is the correct social solution (which is often, and I believe in this case it is, worse than the correct technical solution). I also believe that fixing dpkg to handle the general "symlink dir on file system where .deb has a real dir" case is useful for other cases. However, do not use bogus arguments such as the one above to try to make your point. It clutters the discussion with needless debunking. ...Marvin
Re: merged-/usr vs. partially-symlink-farmed-root
On Sun, Aug 22, 2021 at 10:24:56PM +0100, Luca Boccassi wrote:
> The point of the migration is that /usr/bin will be identical to /bin,
> etc. If they are not identical, then it's not usrmerge as it is
> understood and has been adopted by many upstreams for a decade, it's
> something else that is incompatible with it.
I'll note that people on this thread are using usrmerge in different
senses of the word.
For simplicity's sake, what I've tried to do in my posts is to refer
to "usrmerge" as meaning the creations of top-level symlinks at
/{bin,lib,sbin} pointing /usr//{bin,lib,sbin}. This is the specific
proposal made here:
https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
... and in Debian, see:
https://wiki.debian.org/UsrMerge
So I think there is some justification to using to usrmerge only to
refer to the top-level symlinks approach.
A more general term might be "/usr unification", quoting from a
2012(!) LWN article:
"/usr unification" (or simply "usrmove") is the idea of moving
the contents of /bin, /lib, and related root-level directories
into their equivalents under /usr.
- https://lwn.net/Articles/483921/
After moving the contents of /{bin,lib,sbin}/* to their equivalents
under /usr, the next question is whether we stop things from breaking?
By using top-level sytmlinks, which many call "usrmerge", or creating
symlink farms in the directories /{bin,lib,sbin}, for which I try to
use the more awkward construction, "/usr unification via symlink
farms"?
Admittedly, "/usr unification via symlink farms" is awkward, but I've
been hoping we can declare consensus that using symlink farms is
undersirable way of trying to achieve /usr unification, since it
wastes a lot of on-disk inodes, and there is complexity involved in
needing to keep the symlink farm up-to-date as new files are created
in /usr/{bin,lib,sbin}.
But in any case, perhaps it would simplfy the discussion if we try to
stick with consistent terminology? So what if we were to use usrmerge
to unambiguously mean achieving /usr unification via top-level
/{bin,lib,sbin} to /usr/{bin,lib,sbin} and considering symlink farms
as being another (although IMHO, inferior) way accomplishing the goal
of /usr/ unification?
- Ted
Re: merged-/usr vs. partially-symlink-farmed-root
Hi, On Sun, 2021-08-22 at 12:29 -0400, Marvin Renich wrote: > * Ansgar [210822 05:08]: > > To get a filesystem layout equivalent to merged-/usr via symlinks > > farming *every* package shipping files in at least /usr/bin, > > /usr/sbin > > and possibly some of /usr/lib would need to include symlinks in > > /bin, > > /sbin, /lib. This would affect far more packages than updating the > > packages currently shipping files in /bin, /sbin and /lib* to ship > > these under /usr instead. > > It is true that for a symlink-farm-usr-merge system to be strictly > equivalent to a symlink-dir-usr-merge system, many packages that > never > had /bin/foo but had /usr/bin/foo would have to add a symlink > /bin/foo, > however this is clearly unnecessary. No, it is required if we want users to be able to run scripts that use `#!/bin/python3` (or similar for other interpreters). Such scripts exist in the wild, they were not initially written on Debian. These would need Debian-specific patches to work on Debian. It is just the opposite of now having to patch software packaged in Debian to use /bin/rm instead of /usr/bin/rm (that also happens in the wild for software not developed on Debian). We also happen to miss such instances and needlessly make software buggy on Debian. Not having to worry about yet another small incompatibility between distributions is a nice feature. Ansgar
Re: merged-/usr vs. partially-symlink-farmed-root
On Sun, 2021-08-22 at 12:29 -0400, Marvin Renich wrote: > * Ansgar [210822 05:08]: > > Hi Guillem, > > > > On Sun, 2021-08-22 at 00:11 +0200, Guillem Jover wrote: > > > There was talk about the huge amount of symlinks required in a > > > symlink farm setting, but that might have been true for a > > > scenario > > > where those symlinks would have been handled automatically and > > > transparently. > > > > To get a filesystem layout equivalent to merged-/usr via symlinks > > farming *every* package shipping files in at least /usr/bin, > > /usr/sbin > > and possibly some of /usr/lib would need to include symlinks in > > /bin, > > /sbin, /lib. This would affect far more packages than updating the > > packages currently shipping files in /bin, /sbin and /lib* to ship > > these under /usr instead. > > It is true that for a symlink-farm-usr-merge system to be strictly > equivalent to a symlink-dir-usr-merge system, many packages that > never > had /bin/foo but had /usr/bin/foo would have to add a symlink > /bin/foo, > however this is clearly unnecessary. The problem that the symlink > farm > is solving is that scripts (distributed or user-written) that depend > on > /bin/foo need to continue to work on a partially merged system. A > previous message (already deleted, so I'm not sure from whom) clearly > identified 240 packages (number pulled from memory, so might be off) > that would have to put symlinks in /bin and /sbin for the > symlink-farm-usr-merge strategy to work. > > The amount of work is orders of magnitude less than you are > representing. There is no need for the symlink-farm to exactly match > the symlink-dir solution. The union of systems during the symlink- > farm > merge and systems after the merge is complete can _only_ count on > /bin/foo existing if it existed before the merge was started. There > is > no need for anything else (wrt /bin/foo existing). > > ...Marvin The point of the migration is that /usr/bin will be identical to /bin, etc. If they are not identical, then it's not usrmerge as it is understood and has been adopted by many upstreams for a decade, it's something else that is incompatible with it. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Re: merged-/usr vs. partially-symlink-farmed-root
Hi, On 22.08.21 18:29, Marvin Renich wrote: The amount of work is orders of magnitude less than you are representing. There is no need for the symlink-farm to exactly match the symlink-dir solution. The union of systems during the symlink-farm merge and systems after the merge is complete can _only_ count on /bin/foo existing if it existed before the merge was started. There is no need for anything else (wrt /bin/foo existing). I would expect people to start using /bin/python at some point just because they can. Simon
Re: merged-/usr vs. partially-symlink-farmed-root
* Ansgar [210822 05:08]: > Hi Guillem, > > On Sun, 2021-08-22 at 00:11 +0200, Guillem Jover wrote: > > There was talk about the huge amount of symlinks required in a > > symlink farm setting, but that might have been true for a scenario > > where those symlinks would have been handled automatically and > > transparently. > > To get a filesystem layout equivalent to merged-/usr via symlinks > farming *every* package shipping files in at least /usr/bin, /usr/sbin > and possibly some of /usr/lib would need to include symlinks in /bin, > /sbin, /lib. This would affect far more packages than updating the > packages currently shipping files in /bin, /sbin and /lib* to ship > these under /usr instead. It is true that for a symlink-farm-usr-merge system to be strictly equivalent to a symlink-dir-usr-merge system, many packages that never had /bin/foo but had /usr/bin/foo would have to add a symlink /bin/foo, however this is clearly unnecessary. The problem that the symlink farm is solving is that scripts (distributed or user-written) that depend on /bin/foo need to continue to work on a partially merged system. A previous message (already deleted, so I'm not sure from whom) clearly identified 240 packages (number pulled from memory, so might be off) that would have to put symlinks in /bin and /sbin for the symlink-farm-usr-merge strategy to work. The amount of work is orders of magnitude less than you are representing. There is no need for the symlink-farm to exactly match the symlink-dir solution. The union of systems during the symlink-farm merge and systems after the merge is complete can _only_ count on /bin/foo existing if it existed before the merge was started. There is no need for anything else (wrt /bin/foo existing). ...Marvin
Re: merged-/usr vs. partially-symlink-farmed-root
On Sun, 2021-08-22 at 11:08 +0200, Ansgar wrote: > Hi Guillem, > > On Sun, 2021-08-22 at 00:11 +0200, Guillem Jover wrote: > > There was talk about the huge amount of symlinks required in a > > symlink > > farm setting, but that might have been true for a scenario where > > those > > symlinks would have been handled automatically and transparently. > > To get a filesystem layout equivalent to merged-/usr via symlinks > farming *every* package shipping files in at least /usr/bin, > /usr/sbin > and possibly some of /usr/lib would need to include symlinks in /bin, > /sbin, /lib. This would affect far more packages than updating the > packages currently shipping files in /bin, /sbin and /lib* to ship > these under /usr instead. > > Note that on a merged-/usr system it is fine, and will be fine > forever, > to use /bin/python3 instead of /usr/bin/python3. > > If this is not done, the layouts are even more different. So they > should be named differently to avoid people confusing them as minor > variants of the same: maybe merged-/usr and partially-symlink-farmed- > root (as not all compatibility symlinks exist). > > > The huge majority of files under /lib* (which is the actual bulk of > > them) > > should require no symlink farms. Many of the ones under /bin and > > /sbin > > (we are talking about around 240 packages here) might be switchable > > w/o > > compat symlinks after careful consideration (or not…), many of the > > ones > > that require symlinks would need these just for a period of time, > > Removing any symlink will cause software to break. > > There was recently one such instance with `run-parts` getting moved > to > /usr/bin. Is your proposal to repeat this for every binary in /bin > and > /sbin (except possibly some selected ones) at one point in time? > There seems no realistic way to find/fix all software affected by > this > beforehand, especially software outside of Debian. What is your > proposal to handle this? > > Note that this breakage only happens with your newly proposed > partially-symlink-farmed-root filesystem layout that you propose > instead of merged-/usr. > > > and > > only a handful would remain (the ones that are part of standard > > interfaces, which I'd expect would be mostly shells?). I think the > > amount > > of symlinks currently provided by f.ex. lvm2 and e2fsprogs combined > > would > > amount to more symlinks than what we would eventually end up with > > TBH. > > So your proposal is to adopt a partially-symlink-farmed-root layout > and > never switch to merged-/usr? Sure, in that case SuSE failing to move > from partially-symlink-farmed-root to merged-/usr is not an argument > against that. But it still is a good argument against adopting > partially-symlink-farmed-root as a way to get to merged-/usr. > > What are the advantages to moving to a new filesystem layout used > only > by Debian which introduces incompatibilities with other > distributions? > > Are there further disadvantages we have not yet found? > > > Luca posted a mess of disinformation. > [...] > > Or there's the following for example: > > > > > IN particular, most systems are usrmerged today, and while these > > > bugs > > > are annoying, many people get along just fine. > > > > That would imply that most people have either gone out of their way > > to > > explicitly install and run usrmerge or they have reinstalled from > > scratch. I cannot believe the first to be of any significance, the > > second would put into question why we bother supporting release > > upgrades > > (after all many other distros do not support that, we perhaps > > should > > catch up with what the rest of the world is doing!). > > Various distributions outside of Debian seem to recommend installing > usrmerge on upgrade, for example Ubuntu or Linux Mint, with Ubuntu > also > pulling in the usrmerge package on upgrades by default. So by now > there > likely is a significant population of such users. > > I also don't see people doing new installations as a problem: people > not only using Debian on existing installations that might be a > decade > old, but also deploying it on new systems is a good thing. I don't > think people doing so has any implications to question supporting > upgrades or not. (Besides totally new systems there are of course > other valid reasons to start from a clean slate instead of upgrading > an > existing system such as having a documented state or just automated > deployment.) > > To be honest the conjecture that the number of people having > installed > Debian or Ubuntu or other Debian-based distributions in the last few > years is insignificant and can be totally ignored feels rather far > fetched just to support an outcome you want to see true. We would > have > much, much larger problems for the future of Debian and Debian-based > distributions than merged-/usr if this was true. So if you have any > support for this claim, I'm interested in seeing it. > > Ansgar Indeed it
Re: merged-/usr vs. partially-symlink-farmed-root
Hi Guillem, On Sun, 2021-08-22 at 00:11 +0200, Guillem Jover wrote: > There was talk about the huge amount of symlinks required in a > symlink > farm setting, but that might have been true for a scenario where > those > symlinks would have been handled automatically and transparently. To get a filesystem layout equivalent to merged-/usr via symlinks farming *every* package shipping files in at least /usr/bin, /usr/sbin and possibly some of /usr/lib would need to include symlinks in /bin, /sbin, /lib. This would affect far more packages than updating the packages currently shipping files in /bin, /sbin and /lib* to ship these under /usr instead. Note that on a merged-/usr system it is fine, and will be fine forever, to use /bin/python3 instead of /usr/bin/python3. If this is not done, the layouts are even more different. So they should be named differently to avoid people confusing them as minor variants of the same: maybe merged-/usr and partially-symlink-farmed- root (as not all compatibility symlinks exist). > The huge majority of files under /lib* (which is the actual bulk of > them) > should require no symlink farms. Many of the ones under /bin and > /sbin > (we are talking about around 240 packages here) might be switchable > w/o > compat symlinks after careful consideration (or not…), many of the > ones > that require symlinks would need these just for a period of time, Removing any symlink will cause software to break. There was recently one such instance with `run-parts` getting moved to /usr/bin. Is your proposal to repeat this for every binary in /bin and /sbin (except possibly some selected ones) at one point in time? There seems no realistic way to find/fix all software affected by this beforehand, especially software outside of Debian. What is your proposal to handle this? Note that this breakage only happens with your newly proposed partially-symlink-farmed-root filesystem layout that you propose instead of merged-/usr. > and > only a handful would remain (the ones that are part of standard > interfaces, which I'd expect would be mostly shells?). I think the > amount > of symlinks currently provided by f.ex. lvm2 and e2fsprogs combined > would > amount to more symlinks than what we would eventually end up with > TBH. So your proposal is to adopt a partially-symlink-farmed-root layout and never switch to merged-/usr? Sure, in that case SuSE failing to move from partially-symlink-farmed-root to merged-/usr is not an argument against that. But it still is a good argument against adopting partially-symlink-farmed-root as a way to get to merged-/usr. What are the advantages to moving to a new filesystem layout used only by Debian which introduces incompatibilities with other distributions? Are there further disadvantages we have not yet found? > Luca posted a mess of disinformation. [...] > Or there's the following for example: > > > IN particular, most systems are usrmerged today, and while these > > bugs > > are annoying, many people get along just fine. > > That would imply that most people have either gone out of their way > to > explicitly install and run usrmerge or they have reinstalled from > scratch. I cannot believe the first to be of any significance, the > second would put into question why we bother supporting release > upgrades > (after all many other distros do not support that, we perhaps should > catch up with what the rest of the world is doing!). Various distributions outside of Debian seem to recommend installing usrmerge on upgrade, for example Ubuntu or Linux Mint, with Ubuntu also pulling in the usrmerge package on upgrades by default. So by now there likely is a significant population of such users. I also don't see people doing new installations as a problem: people not only using Debian on existing installations that might be a decade old, but also deploying it on new systems is a good thing. I don't think people doing so has any implications to question supporting upgrades or not. (Besides totally new systems there are of course other valid reasons to start from a clean slate instead of upgrading an existing system such as having a documented state or just automated deployment.) To be honest the conjecture that the number of people having installed Debian or Ubuntu or other Debian-based distributions in the last few years is insignificant and can be totally ignored feels rather far fetched just to support an outcome you want to see true. We would have much, much larger problems for the future of Debian and Debian-based distributions than merged-/usr if this was true. So if you have any support for this claim, I'm interested in seeing it. Ansgar
