On Wed, 17 May 2023 at 10:31, Helmut Grohne <hel...@subdivi.de> wrote: > > Hi, > > This bootstrap aspect got me and I discussed this with a number of > people and did some research. > > On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote: > > I don't think this is true? At least not in the broader sense: if you > > compile something on Debian, it will obviously get linked against > > libraries and dependencies as they are in Debian. > > Perhaps what you mean is that, given an entire separate sysroot-like > > tree, passing the appropriate compiler and linker flags and > > environment variables, you can use the local compiler we ship to build > > 'foreign' programs. That is true, but again it requires to set up the > > environment appropriately, including linker flags. And the caller > > needs to ensure the environment, including linker flags, is > > appropriate for the target environment (I guess 'host' environment, in > > GNU parlance). Therefore, I don't think it would be unreasonable to > > require that if the target environment is split-usr, then the caller > > also needs to specify an appropriate > > '-Wl,--dynamic-linker=/lib/ld-whatever' option. > > Given the feedback, I am convinced that changing PT_INTERP is a stupid > idea regardless of whether it is technically feasible. There must be a > better way. Let's step back a bit. > > The underlying problem here is performing the initial filesystem > bootstrap. The semantics of this are a bit vague as they are not spelled > out in policy, so we will have to derive them from implementations. > > I think the major players are (in descending popularity): > * debootstrap > * mmdebstrap > * cdebootstrap > * multistrap > > multistrap predates mmdebstrap and when there was no mmdebstrap, I used > it a lot. When attempting to test it, I totally couldn't convince it to > bootstrap from an unsigned or locally signed repository. The patch in > #908451 didn't cut it. I also note that it creates a /lib64 -> /lib > symbolic link which feels quite incompatible with merged-/usr. For > these reasons, I am dropping multistrap from the tools under > consideration and recommend removing it from the archive. If you happen > to use multistrap, now would be a good moment to tell me. Personally, > all of my use cases of multistrap have been converted to mmdebstrap and > that made a lot of things simpler. > > cdebootstrap vaguely works though unsigned operation seems dysfunctional > as it runs apt-get update during cdebootstrap-helper-apt.postinst and > that fails. I happen to not have figured out why and treat this failure > as a success. > > So the most popular implementations quite evidently are debootstrap and > mmdebstrap and both "just work". I note though that they work quite > differently: > * debootstrap (depending on flags including --variant) pre-merges its > chroot while mmdebstrap relies on packages doing it. > > I think that the question whether a distribution is merged is a > property of the distribution and not the bootstrap tool, so I > strongly recommend following mmdebstrap's view on this. The > debootstrap way means that we have to include patches for every > derivative, which is a process that does not scale well. > > * mmdebstrap operates in two phases. It first unpacks and configures a > rather minimal set of packages and then proceeds to adding packages > passed to --include in a second phase once essential is fully > configured while debootstrap immediately unpacks everything. > > I think the debootstrap approach is slightly worse here, because it > means that preinst scripts of non-essential packages cannot rely on > essential packages having been configured. > > In any case, we have to deal with both behaviours. > > After this little excursion into bootstrap technology, let's go back to > the /usr-merge and its effects. > > I think at this point, we have quite universal consensus about the goal > of moving files to their canonical location (i.e. from / to /usr) as a > solution to the aliasing problems while we do not have consensus on > precisely how to do this (i.e. with changing dpkg or without). If you > believe that this is not consensus, please speak up. > > So in a distant future our packages will not contain any files in /bin > or /lib. In particular, this affects /bin/sh and the dynamic loader, > both of which are required to run maintainer scripts, which are > currently required for creating the symbolic links. Boom. > > Solutions have been proposed to this and I think they all fall into one > of the following four categories. > > 1. Don't move. We just keep those files that require a particular > location (such as /bin/sh or the dynamic loader) in their > non-canonical location. As such, maintainer scripts will be able to > run and perform the conversion to symbolic links afterwards. > > 2. Move and ship links. Since we unpack all essential data.tar before > running the first maintainer script, having one package contain the > compatibility symlinks is enough to fix the problem. > > 3. Move and avoid using non-canonical locations. This is the approach > where we write maintainer scripts as #!/usr/bin/sh and considered > changing PT_INTERP. > > 4. Change the bootstrap protocol. In essence, this has been attempted > in debootstrap by creating these symlinks prior to unpack, but no > consensus has evolved around this approach yet. The category is > wider though and generally requires changes to all bootstrapping > tools. > > To see that it's just these four we can watch an imaginary bootstrap > process on an imaginary distribution. If any package contains the > symlinks, we're in category 2. Otherwise, we see whether these symlinks > exist prior to running the first maintainer script. If they do, we're in > category 4. And finally, we see whether any files are left in aliased > locations (according to the dpkg database). If there are, we're category > 1 and otherwise category 3. > > For instance, Luca proposed changing PT_INTERP in the toolchain, which > is category 3 here. I proposed a minor adaption where we have debhelper > change it post build using patchelf, which also amounts to category 3. > I've started with these to get started with classifying and think we can > disregard both. > > For completeness sake, there is one more entry in category 3: We can run > the dynamic loader from its canonical location explicitly, so we'd > modify maintainer scripts to start with: > > #!/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/sh > > This would only affect preinst scripts participating in bootstrap and > we'd not bother with changing PT_INTERP in the toolchain nor in > packages. Unfortunately, this completely breaks the DPKG_ROOT work and > with that, I see no viable entries in category 3 for moving forward.
Apart from DPKG_ROOT, that is also complicated because of multiarch I suppose? Ie, installing a random $arch package on !$arch. > Moving on to category 4 feels rather obvious, especially because work > has been done there in debootstrap. The approach in debootstrap however > is one that I see as a dead end, because it causes us to maintain this > code multiple times. It's the number of derivatives times the number of > bootstrap tools and that doesn't scale. > > Category 4 is wider though and we also have other prior art at > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap. In particular, > making architecture bootstrap become chrootless would likely be a > generic solution to this and other problems. However, any change needs > to propagate to a stable release in all bootstrapping tools. Therefore > we cannot reasonably finish the transition before forky. This makes > category 4 rather unattractive in a short term, but still worth pursuing > in a long term. I'm not sure this is really a show-stopper, and it would really need to delay till forkie. We are talking about only two packages, needing updates to bootstrap a future release. We have already updated debootstrap in stable and oldstable via proposed-updates for this purpose in the past, so as long as the changes are self-contained and do not affect building older releases, it sounds to me it would be doable to do the same here. https://tracker.debian.org/news/1349768/accepted-debootstrap-10114deb10u1-source-into-oldstable-proposed-updates-oldstable-new-oldstable-proposed-updates/ > Having ruled out categories 3 and 4 maybe category 2 would be good? We > could just ship those symlinks in base-files and be done, right? > Unfortunately, we pass -k to tar in debootstrap, so when it extracts > base-files and tries to unpack the bin -> usr/bin symlink, it sees that > oh no, there already is a symlink at bin (as debootstrap placed it > there) and thus fails. So in order to make this work, we also have to > modify debootstrap (and thus are in a combination of category 3 and 4). As far as gut feelings go, it doesn't feel like to me that modifying debootstrap to deal with this would be too difficult. > Conversely, if we unpack anything else that happens to ship > /bin/something before base-files, then mmdebstrap will fail to unpack > base-files. So we can only add this link to base-files after all other > (essential) packages have moved everything out of bin. That happens to > include /bin/sh. A possible solution to this is doing all of this in the > same dinstall. Then mmdebstrap works before and works afterwards. > > Unfortunately, we're not yet done here. When (for instance) dash is the > last package to move /bin/sh and such to /usr and you upgrade dash, dpkg > notices that no package owns /bin anymore. Thus it helpfully deletes > /bin (the symlink). You're not happy when this happens. We remember the > silver bullet: diversions! So dash.preinst could dpkg-divert --no-rename > --divert /bin.usrmerged -add /bin and dash.postinst could revert that. > Unfortunately, such a diversion affects every package but dash and we > want it exactly the other way round. So what we could do is pass > --package dash-usrmerged (which must not exist). Then it'll actually > keep /bin safe. Unfortunately, we don't know whether dash or bash will > be the last package owning /bin, so both of them need this diversion and > this is a conflict. > > And as if that wasn't enough, we also run into issues around hard links. > As dpkg unpacks a canonicalized gzip, it notices that /bin/gunzip (which > is scheduled for deletion) has the same inode as the new > /usr/bin/uncompress (because gunzip and uncompress are hard linked). As > far as I can see, all we get here is a warning and both files survive > the unpack. It is not clear to me whether this happens by chance or > reliably. > > dpkg: warning: old file '/bin/uncompress' is the same as several new > files! (both '/usr/bin/gunzip' and '/usr/bin/uncompress')^ > dpkg: warning: old file '/bin/gunzip' is the same as several new files! > (both '/usr/bin/gunzip' and '/usr/bin/uncompress') > > Given what I've seen, I'm fairly convinced that I haven't reached the > bottom of it and I'm ready to conclude that this approach is fragile - a > property that is most unwelcome when we deal with the essential set. > > So what's left is category 1. I looked into what the minimum set of > files to be retained could be. To do that end, I moved everything and > then reverted as much as was needed to make bootstrapping work. > * /lib64/ld-linux-x86-64.so.2 (hopefully obvious) > * /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (link target) > * /bin/sh (hopefully obvious) > * /bin/dash (link target) > * /bin/bash (usrmerge runs ldd, which is a #!/bin/bash script) > * /bin/more (update-alternatives doesn't like its absence) > * /bin/cp (unless usrmerge stops hard coding its path) > > So in principle, we may be able to pull this off if we keep bash, > dash, libc6, and util-linux in their original location. In this > scenario, we're also unable to remove usrmerge from the essential set. > The major benefit of category 1 it allows us to move to any other > category at a later time and it removes aliasing effects from all but a > small set of packages. Is it necessary to avoid canonicalizing bash and util-linux? My understanding was that the reason for that was to avoid dpkg from deleting the symlink - wouldn't a single package (eg: dash for /bin, libc6 for /lib* and something else for /sbin) be enough to achieve that? Or did I misread the issue? Overall, this sounds like a fine approach to me - simple, and gets us 99.999% of the way there. It means dash and libc6 cannot move /bin/dash or /lib/ld to other packages in the Trixie cycle, but somehow I suspect that won't be an issue in reality :-) We could start with this, and then update debootstrap and mmdebstrap to also deal with this, and canonicalize dash/libc only after we are confident the bootstrapping issue is really solved. That way we can canonicalize and unblock 99.999% of the distro immediately. Kind regards, Luca Boccassi