*Sigh*, replying only now because I find this topic exhausting, draining, tiresome and I'd rather be doing productive stuff instead of rehashing stuff for the nth time. :( Even though I've also tried to document and summarize this in the dpkg FAQ and the dpkg MergedUsr wiki page, but I assume I'm not explaining myself clearly because this is not getting through… :(
On Fri, 2020-11-20 at 19:35:07 +0000, Simon McVittie wrote: > On Fri, 20 Nov 2020 at 11:12:20 +0100, Ansgar wrote: > > On Fri, 2020-11-20 at 10:19 +0100, Adam Borowski wrote: > > > On Fri, Nov 20, 2020 at 09:35:42AM +0100, Ansgar wrote: > > > > As far as I know nothing broke catastrophically over the last releases > > > > with merged-/usr. > > > > > > Unless you look at dpkg or attempts at speeding up bootstrap. > > > > > > See > > > https://salsa.debian.org/glibc-team/glibc/-/commit/49d137c4392cb1144f2313f78f31466aaa169b75 > > > for an example. > > Trying to transition from /lib to /usr/lib in the way that is advocated > by some people who don't like usrmerge (one package at a time, without > using usrmerge or something equivalent) can unfortunately *also* cause > transitional brokenness: see > <https://salsa.debian.org/gnome-team/glib/-/commit/1f9c505abe2639296615d13c048227a3df378576>. I mentioned this before, this does not look specific to whatever method to do merged-/usr, instead this seems like a general dpkg problem related to missing fsync() on the directories during unpacks: <https://lists.debian.org/debian-devel/2020/02/msg00477.html> I don't see why this could not affect also merged-/usr-via-aliased-dirs systems when moving other pathnames around. I still need to sit down and fix it, I guess during 1.21.x… > > The good news here is that shipping bash as /usr/bin/bash (instead of > > /bin/bash) solves that problems as dpkg will just install it under > > /usr/bin/bash. No aliasing over symlinks involved. Err, no, it does definitely not solve any such problem, the aliasing via directories is still going to be there (the problem will just show up instead in the other direction). This would just imply trying to force onto everyone getting a broken system. > So we need a way to get there from here. The only possibilities I can > see for that within the framework of a dpkg-based OS are: > > * Aliasing via symlinks: > Something magics /bin -> usr/bin, etc., symlinks into existence, > and dpkg continues to tolerate unpacking over them, providing > a per-installation flag day to move to merged /usr. usrmerge is > an implementation of this approach, debootstrap --merged-usr is > another. Individual systems can do this transition at any time (many > have already done so), and they will get the benefit of merged /usr > (in particular, a class of avoidable bugs can no longer affect them) > after they have passed the flag day. Fortunately the breakage stemming from this approach can now be fixed starting with dpkg 1.20.6, which contains the new dpkg-fsys-usrunmess tool. > * Package-by-package migration: > For every package that ships a file in /bin, /sbin, /lib* whose path > might have been hard-coded elsewhere, the maintainer of that package > takes action to move the file into /usr. If the file's full path might > have been hard-coded elsewhere (/bin/bash, etc.), the maintainer > scripts must additionally create an unmanaged symlink > /bin/bash -> /usr/bin/bash, etc. - similar to what was done to move > /usr/bin/chacl in src:acl to /bin, but in reverse. And then, to get the > benefits of merged /usr, we have to wait for *every* package to stop > shipping anything in /bin, /sbin, /lib*, and *then* have a usrmerge-like > per-installation flag day at which the collection of symlinks /bin/bash > -> /usr/bin/bash, etc. are replaced by a single symlink /bin -> usr/bin. We only need the unmanaged symlinks generated by maintscripts due to the merged-/usr-via-aliased-dirs hack. :( A transition could have been done automatically instead, as I've mentioned in the past. See below. But if the intention is to end up with a merged-/usr-via-aliased-dirs then this approach seems rather pointless TBH. Instead what I've mentioned multiple times in the past is that this could have been done (and could still be done), by say a new debhelper compat level that automatically moves all the pathnames from «/» to «/usr», during building, and creates the symlink farms under «/». This would require reverting the merged-/usr-via-aliased-dirs mess (which is now possible) and banning that method completely, so that we can avoid the mess with the required maintainer scripts. Which would imply at least, that: * there are no directory aliasing issues anymore due to this (so f.ex. unpacking different things over the same aliased pathname is not possible, or querying for pathnames in anything that tracks filesystem pathnames does not break or fails silently, be that dpkg*, u-a, or other tools, etc) * let's dpkg track all these pathnames in the system * let's maintainers transition manually to the new paths when they can * let's maintainers remove the «/» compat symlinks when no more users, and for external parties it's not a silent error * let's everyone know by simply looking at the filesystem, what pathnames have already been migrated or not, because some are still required by ABI to be in specific places > In all three cases, Debian-as-a-project does not get the full benefit of > merged /usr until all installations that remain supported have had their > per-installation transition; and in all three cases, we cannot ship > /bin -> usr/bin, etc. in base-files until after every supported system > has undergone that transition. So if Debian 12 is the first to make merged > /usr mandatory during upgrade, as Ansgar proposes, then Debian 13 would be > the first release where base-files could safely ship those symlinks. TBH, I'd consider such base-files package broken, and unsupported from dpkg's perspective, as a breakage inducer. This also obviates anything external to Debian that ships stuff directly under the «/» pathnames or both «/» and «/usr», or does queries or uses any other dpkg command handling pathnames, etc. > I agree that aliasing via symlinks has a complexity cost for low-level > packages like dpkg and debootstrap clones. One way to mitigate this > would be to make the transitional period as short as we can (which is > essentially what Ansgar is proposing, I think), so that some release of > Debian (12 in Ansgar's proposal) can guarantee to be merged-/usr, after > which we have the option to take away the scaffolding that we used to get > there (Ansgar proposes that this should happen between Debian 12 and 13). > There's certainly an argument to be made that since we are *already* > paying this complexity cost, we might as well get the benefit from it > as soon as we can. This still is based on the assumption that the end result is not broken, but that's not the case, this is just a way to get to a broken end state faster. And to restate, I consider the merged-/usr-via-aliased-dirs approach proposed here, broken and unsupported from dpkg's perspective. Regards, Guillem