Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Sun, 2019-02-24 at 03:23:09 +0100, Guillem Jover wrote:
> On Tue, 2019-02-19 at 05:49:24 +0100, Guillem Jover wrote:
[…]
> >   - file a bug on base-installer to request an option to install
> > non-broken systems due to merged-/usr-via-symlinks.
> 
> Done. 

This got a patch from Colin Watson (thanks!), but it never got applied
before the release, :( I think the commit description does not fully
reflect the current situation and problems, but I'd take that patch as
is any day.

Doing an installation w/o the broken deployment is not too difficult
though, as long as you know what needs doing:

  - Select expert mode.
  - Do all the steps, until installing the base system.
  - Spawn a shell:
+ Edit /var/lib/dpkg/info/bootstrap-base.posinst,
+ Add --no-merged-usr to the debootstrap call.
  - Proceed with the installation.


In addition this deployment method also breaks:

  - dlocate.
  - apt-file.
  - packages.debian.org's search.
  - find /lib (or any of the other symlinked directories).
  - …

> > And I'm probably going to end up writing a unmerge-usr-via-symlinks
> > script so that people with damaged systems can go back somewhat to a
> > sane state, and to open the possibility for a fully automated migration
> > to a proper and correct merged-/usr w/o all the problems above.
> 
> And I might need to start on this soon enough. :(

I'm not sure I can be bothered TBH.

> In addition, given that most probably Debian buster will end up
> installing broken systems by default. I might end up also looking into
> generating fixed minimal netinst images or mini netboot images with a
> fixed debootstrap, or I guess just cdebootstrap present which has
> sane behavior. But I would definitely not be able host the artifacts
> for those. :/

So this happened in buster, but as per the above, one can "easily" avoid
the breakage, if you know about it. :/

Thanks,
Guillem

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Russ Allbery]

Julian Andres Klode  writes:

> Having symlinks in /bin and so on would be unclean: We'd have to maintain
> one symlink per binary in /usr. This is a lot of symlinks.

Does the quantity of symlinks matter?

> We also cannot ever get rid of them - it would break the property.

Well, on any given system, once every file in /bin was a symlink to the
same-named file in /usr/bin and we had some guarantee that no new packages
would break this property, we could in theory replace the entire directory
with a symlink.  Doing that feels like it would require new primitives in
the package management software, though, and it might be hard to do
safely.  (It would also create a break point where packages from before
that flag day could no longer be installed on the system.)

> It also fails to handle subdirectories in lib{exec}. We do want
> /usr/lib/systemd and /lib/systemd to be the same.

You can use the same approach recursively and symlink every file.  This is
an old package manager trick that I think Nix is still using to this day,
and which was used to some success by such things as GNU stow (albeit for
different reasons).

-- 
Russ Allbery (r...@debian.org)   

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Julian Andres Klode]

On Tue, Feb 19, 2019 at 05:49:24AM +0100, Guillem Jover wrote:
> Hi!
> 
> On Tue, 2018-11-20 at 22:16:17 +0100, Adam Borowski wrote:
> > Thus, it seems to me that the plan A for usrmerge has serious downsides for
> > dubious benefits.  What about the plan B I described above?
> 
> So, people still seem to be conflating merged-/usr (the filesystem
> layout) with the different ways to deploy it. Personally I do agree
> that merged-/usr has benefits, and that's why I've f.ex. been switching
> the shared library packages I maintain to do a proper and correct switch
> by installing their files in /usr/lib instead of /lib.
> 
> The underlaying problem with the merged-/usr-via-symlinks is not
> really that it has generated bugs, many transitions we perform incur
> those, for the better. The problem is that it has generated those when
> it was really not necessary in the first place, has made things
> unnecessarily more complicated for everyone, and it's a terrible hack
> trying to reap "quick" benefits at the cost of everyhing else, with
> long lasting consequences even after a full migration to /usr would
> have been finished. And here's why:
> 
> 1) The merged-/usr-via-symlinks deployment method used by both
>debootstrap and usrmerge, means that those systems will get
>permanent filesystem aliasing problems, forever. Even when all
>files might have been moved to their /usr counterparts in the
>packages! This is due to the fact that different pathnames can
>end up pointing (before any canonicalization!) to the same dentry.

This is IMO the most important feature of usrmerge: It does not matter
whether you use /usr/$bar or /$bar, you'll get the same result. Hence
/bin is a symlink to /usr/bin and so on, so you get the same binaries
and libraries in both places.

Having symlinks in /bin and so on would be unclean: We'd have to maintain
one symlink per binary in /usr. This is a lot of symlinks. We also cannot
ever get rid of them - it would break the property.

It also fails to handle subdirectories in lib{exec}. We do want
/usr/lib/systemd and /lib/systemd to be the same.

I don't think there is any other way that sensible handles merging
subdirectories, and therefore, the only reasonable way is to continue
with the current way, and patch up software to normalize /{bin,sbin,lib}
to /usr/bin when they see it on an usrmerged system.

I admit that this is painful, but I think it is worth it.

> 2) It bypasses the packaging system understanding of what is on the
>   filesystem and creates mismatches between what's on binary packages
>   and what's on disk.

I agree that this is a problem. But it seems to me it is a solvable
one.

> 4) Due to having to support the broken merged-/usr-via-symlinks
>deployment, when we want to move the contents of the binary
>packages to the merged-/usr layout, we require now to include tons
>of logic in (probably new) maintainer scripts, when we have been
>trying to remove them altogether. :( With even more files untracked
>by dpkg itself, bypassing the packaging system even more, when the
>compat symlinks could have been shipped in the binary packages.

The solution here seems simple: If you do not want a usrmerged
system, but do want to move binaries to /usr, build a symlink farm
manager into dpkg that automatically creates symlinks in /$foo for
files in /usr/$foo (for some values of $foo), rather than having
packages ship compat themselves.

That would get you support for merged binaries and libraries, but
not for merged lib directories. It's unclear to me how you'd handle
those, hence I mentioned that /lib -> /usr/lib is the only sensible
answer.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Sun, 2019-02-24 at 03:23:09 +0100, Guillem Jover wrote:
> On Tue, 2019-02-19 at 05:49:24 +0100, Guillem Jover wrote:
> > So, as part of damage control I'm going to:
> > 
> >   - include the Build-Tainted-By patches into dpkg 1.19.5.
> 
> Done.
> 
> >   - include a bug-script in dpkg for reportbug to report whether the
> > system has been broken by the merged-/usr-via-symlinks hack.
> 
> Done.
> 
> >   - file a bug on reportbug to request reporting that for all packages.
> 
> Done. 
> 
> >   - file a bug on base-installer to request an option to install
> > non-broken systems due to merged-/usr-via-symlinks.
> 
> Done. 
> 
> > I'm also pondering whether to implement checks for the broken
> > merged-/usr-via-symlinks within «dpkg --audit», even though I don't
> > really want to, as that would imply hardcoding Debian-specific
> > pathnames in there. :(
> 
> I still have to think about this one. :/
> 
> > And I'm probably going to end up writing a unmerge-usr-via-symlinks
> > script so that people with damaged systems can go back somewhat to a
> > sane state, and to open the possibility for a fully automated migration
> > to a proper and correct merged-/usr w/o all the problems above.
> 
> And I might need to start on this soon enough. :(
> 
> In addition, given that most probably Debian buster will end up
> installing broken systems by default. I might end up also looking into
> generating fixed minimal netinst images or mini netboot images with a
> fixed debootstrap, or I guess just cdebootstrap present which has
> sane behavior. But I would definitely not be able host the artifacts
> for those. :/

Oh, also mmdebstrap was defaulting to the broken merged-/usr-via-symlinks,
for which I filed , which has been fixed
now by disabling it completely! I'm planning on switching all my
debootstrap usage to mmdebstrap anyway because it's superior in any
conceivable way. :) I could also see trying to get that forked d-i use
mmdebstrap, also for its speed.

Thanks,
Guillem

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

Hi!

On Tue, 2019-02-19 at 05:49:24 +0100, Guillem Jover wrote:
> So, as part of damage control I'm going to:
> 
>   - include the Build-Tainted-By patches into dpkg 1.19.5.

Done.

>   - include a bug-script in dpkg for reportbug to report whether the
> system has been broken by the merged-/usr-via-symlinks hack.

Done.

>   - file a bug on reportbug to request reporting that for all packages.

Done. 

>   - file a bug on base-installer to request an option to install
> non-broken systems due to merged-/usr-via-symlinks.

Done. 

> I'm also pondering whether to implement checks for the broken
> merged-/usr-via-symlinks within «dpkg --audit», even though I don't
> really want to, as that would imply hardcoding Debian-specific
> pathnames in there. :(

I still have to think about this one. :/

> And I'm probably going to end up writing a unmerge-usr-via-symlinks
> script so that people with damaged systems can go back somewhat to a
> sane state, and to open the possibility for a fully automated migration
> to a proper and correct merged-/usr w/o all the problems above.

And I might need to start on this soon enough. :(

In addition, given that most probably Debian buster will end up
installing broken systems by default. I might end up also looking into
generating fixed minimal netinst images or mini netboot images with a
fixed debootstrap, or I guess just cdebootstrap present which has
sane behavior. But I would definitely not be able host the artifacts
for those. :/

Thanks,
Guillem

merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

Hi!

On Tue, 2018-11-20 at 22:16:17 +0100, Adam Borowski wrote:
> Thus, it seems to me that the plan A for usrmerge has serious downsides for
> dubious benefits.  What about the plan B I described above?

So, people still seem to be conflating merged-/usr (the filesystem
layout) with the different ways to deploy it. Personally I do agree
that merged-/usr has benefits, and that's why I've f.ex. been switching
the shared library packages I maintain to do a proper and correct switch
by installing their files in /usr/lib instead of /lib.

The underlaying problem with the merged-/usr-via-symlinks is not
really that it has generated bugs, many transitions we perform incur
those, for the better. The problem is that it has generated those when
it was really not necessary in the first place, has made things
unnecessarily more complicated for everyone, and it's a terrible hack
trying to reap "quick" benefits at the cost of everyhing else, with
long lasting consequences even after a full migration to /usr would
have been finished. And here's why:

1) The merged-/usr-via-symlinks deployment method used by both
   debootstrap and usrmerge, means that those systems will get
   permanent filesystem aliasing problems, forever. Even when all
   files might have been moved to their /usr counterparts in the
   packages! This is due to the fact that different pathnames can
   end up pointing (before any canonicalization!) to the same dentry.

   This does not only affect dpkg (dpkg, dpkg-divert, dpkg-trigger,
   etc), and update-alternatives (in a worse way), but any program
   that uses pathanmes in the filesystem as keys in databases f.ex.

2) It bypasses the packaging system understanding of what is on the
   filesystem and creates mismatches between what's on binary packages
   and what's on disk.

3) Switching packages to the merged-/usr layout could have been
   accomplished automatically via debhelper for a coverage of around
   99% (?) of the archive. With something along the lines of:

 ,---
 D=debian/tmp
 for d in /bin /sbin /lib; do
   for p in $(find $D/$d ! -type d); do
 b="${p##$D/}"
 m="$D/usr$b"
 if [ ! -e "$m" ]; then
   mkdir -p "$(dirname $m)"
   mv "$p" "$m"
   ln -s "${m##$D}" "$p"
 fi
   done
 done
 `---

   With the property that it would handle gracefully all cases were the
   maintainer has checked that no compat symlinks are required and has
   then progressively moved the pathname installation to their final
   destination under /usr.

4) Due to having to support the broken merged-/usr-via-symlinks
   deployment, when we want to move the contents of the binary
   packages to the merged-/usr layout, we require now to include tons
   of logic in (probably new) maintainer scripts, when we have been
   trying to remove them altogether. :( With even more files untracked
   by dpkg itself, bypassing the packaging system even more, when the
   compat symlinks could have been shipped in the binary packages.

5) Most new installations will not even benefit from this hacky and
   rushed deployment, as long as they do not use a separate parition
   for /usr!

6) The merged-/usr-via-symlinks deployment hack is irreversible right
   now.

This is all really terrible engineering. :(


So, as part of damage control I'm going to:

  - include the Build-Tainted-By patches into dpkg 1.19.5.
  - include a bug-script in dpkg for reportbug to report whether the
system has been broken by the merged-/usr-via-symlinks hack.
  - file a bug on reportbug to request reporting that for all packages.
  - file a bug on base-installer to request an option to install
non-broken systems due to merged-/usr-via-symlinks.

I'm also pondering whether to implement checks for the broken
merged-/usr-via-symlinks within «dpkg --audit», even though I don't
really want to, as that would imply hardcoding Debian-specific
pathnames in there. :(

And I'm probably going to end up writing a unmerge-usr-via-symlinks
script so that people with damaged systems can go back somewhat to a
sane state, and to open the possibility for a fully automated migration
to a proper and correct merged-/usr w/o all the problems above.

Thanks,
Guillem