Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Sun, 2019-02-24 at 03:23:09 +0100, Guillem Jover wrote:
> On Tue, 2019-02-19 at 05:49:24 +0100, Guillem Jover wrote:
[…]
> >   - file a bug on base-installer to request an option to install
> > non-broken systems due to merged-/usr-via-symlinks.
> 
> Done. 

This got a patch from Colin Watson (thanks!), but it never got applied
before the release, :( I think the commit description does not fully
reflect the current situation and problems, but I'd take that patch as
is any day.

Doing an installation w/o the broken deployment is not too difficult
though, as long as you know what needs doing:

  - Select expert mode.
  - Do all the steps, until installing the base system.
  - Spawn a shell:
+ Edit /var/lib/dpkg/info/bootstrap-base.posinst,
+ Add --no-merged-usr to the debootstrap call.
  - Proceed with the installation.


In addition this deployment method also breaks:

  - dlocate.
  - apt-file.
  - packages.debian.org's search.
  - find /lib (or any of the other symlinked directories).
  - …

> > And I'm probably going to end up writing a unmerge-usr-via-symlinks
> > script so that people with damaged systems can go back somewhat to a
> > sane state, and to open the possibility for a fully automated migration
> > to a proper and correct merged-/usr w/o all the problems above.
> 
> And I might need to start on this soon enough. :(

I'm not sure I can be bothered TBH.

> In addition, given that most probably Debian buster will end up
> installing broken systems by default. I might end up also looking into
> generating fixed minimal netinst images or mini netboot images with a
> fixed debootstrap, or I guess just cdebootstrap present which has
> sane behavior. But I would definitely not be able host the artifacts
> for those. :/

So this happened in buster, but as per the above, one can "easily" avoid
the breakage, if you know about it. :/

Thanks,
Guillem

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Russ Allbery]

Julian Andres Klode  writes:

> Having symlinks in /bin and so on would be unclean: We'd have to maintain
> one symlink per binary in /usr. This is a lot of symlinks.

Does the quantity of symlinks matter?

> We also cannot ever get rid of them - it would break the property.

Well, on any given system, once every file in /bin was a symlink to the
same-named file in /usr/bin and we had some guarantee that no new packages
would break this property, we could in theory replace the entire directory
with a symlink.  Doing that feels like it would require new primitives in
the package management software, though, and it might be hard to do
safely.  (It would also create a break point where packages from before
that flag day could no longer be installed on the system.)

> It also fails to handle subdirectories in lib{exec}. We do want
> /usr/lib/systemd and /lib/systemd to be the same.

You can use the same approach recursively and symlink every file.  This is
an old package manager trick that I think Nix is still using to this day,
and which was used to some success by such things as GNU stow (albeit for
different reasons).

-- 
Russ Allbery (r...@debian.org)   

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Julian Andres Klode]

On Tue, Feb 19, 2019 at 05:49:24AM +0100, Guillem Jover wrote:
> Hi!
> 
> On Tue, 2018-11-20 at 22:16:17 +0100, Adam Borowski wrote:
> > Thus, it seems to me that the plan A for usrmerge has serious downsides for
> > dubious benefits.  What about the plan B I described above?
> 
> So, people still seem to be conflating merged-/usr (the filesystem
> layout) with the different ways to deploy it. Personally I do agree
> that merged-/usr has benefits, and that's why I've f.ex. been switching
> the shared library packages I maintain to do a proper and correct switch
> by installing their files in /usr/lib instead of /lib.
> 
> The underlaying problem with the merged-/usr-via-symlinks is not
> really that it has generated bugs, many transitions we perform incur
> those, for the better. The problem is that it has generated those when
> it was really not necessary in the first place, has made things
> unnecessarily more complicated for everyone, and it's a terrible hack
> trying to reap "quick" benefits at the cost of everyhing else, with
> long lasting consequences even after a full migration to /usr would
> have been finished. And here's why:
> 
> 1) The merged-/usr-via-symlinks deployment method used by both
>debootstrap and usrmerge, means that those systems will get
>permanent filesystem aliasing problems, forever. Even when all
>files might have been moved to their /usr counterparts in the
>packages! This is due to the fact that different pathnames can
>end up pointing (before any canonicalization!) to the same dentry.

This is IMO the most important feature of usrmerge: It does not matter
whether you use /usr/$bar or /$bar, you'll get the same result. Hence
/bin is a symlink to /usr/bin and so on, so you get the same binaries
and libraries in both places.

Having symlinks in /bin and so on would be unclean: We'd have to maintain
one symlink per binary in /usr. This is a lot of symlinks. We also cannot
ever get rid of them - it would break the property.

It also fails to handle subdirectories in lib{exec}. We do want
/usr/lib/systemd and /lib/systemd to be the same.

I don't think there is any other way that sensible handles merging
subdirectories, and therefore, the only reasonable way is to continue
with the current way, and patch up software to normalize /{bin,sbin,lib}
to /usr/bin when they see it on an usrmerged system.

I admit that this is painful, but I think it is worth it.

> 2) It bypasses the packaging system understanding of what is on the
>   filesystem and creates mismatches between what's on binary packages
>   and what's on disk.

I agree that this is a problem. But it seems to me it is a solvable
one.

> 4) Due to having to support the broken merged-/usr-via-symlinks
>deployment, when we want to move the contents of the binary
>packages to the merged-/usr layout, we require now to include tons
>of logic in (probably new) maintainer scripts, when we have been
>trying to remove them altogether. :( With even more files untracked
>by dpkg itself, bypassing the packaging system even more, when the
>compat symlinks could have been shipped in the binary packages.

The solution here seems simple: If you do not want a usrmerged
system, but do want to move binaries to /usr, build a symlink farm
manager into dpkg that automatically creates symlinks in /$foo for
files in /usr/$foo (for some values of $foo), rather than having
packages ship compat themselves.

That would get you support for merged binaries and libraries, but
not for merged lib directories. It's unclear to me how you'd handle
those, hence I mentioned that /lib -> /usr/lib is the only sensible
answer.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Sun, 2019-02-24 at 03:23:09 +0100, Guillem Jover wrote:
> On Tue, 2019-02-19 at 05:49:24 +0100, Guillem Jover wrote:
> > So, as part of damage control I'm going to:
> > 
> >   - include the Build-Tainted-By patches into dpkg 1.19.5.
> 
> Done.
> 
> >   - include a bug-script in dpkg for reportbug to report whether the
> > system has been broken by the merged-/usr-via-symlinks hack.
> 
> Done.
> 
> >   - file a bug on reportbug to request reporting that for all packages.
> 
> Done. 
> 
> >   - file a bug on base-installer to request an option to install
> > non-broken systems due to merged-/usr-via-symlinks.
> 
> Done. 
> 
> > I'm also pondering whether to implement checks for the broken
> > merged-/usr-via-symlinks within «dpkg --audit», even though I don't
> > really want to, as that would imply hardcoding Debian-specific
> > pathnames in there. :(
> 
> I still have to think about this one. :/
> 
> > And I'm probably going to end up writing a unmerge-usr-via-symlinks
> > script so that people with damaged systems can go back somewhat to a
> > sane state, and to open the possibility for a fully automated migration
> > to a proper and correct merged-/usr w/o all the problems above.
> 
> And I might need to start on this soon enough. :(
> 
> In addition, given that most probably Debian buster will end up
> installing broken systems by default. I might end up also looking into
> generating fixed minimal netinst images or mini netboot images with a
> fixed debootstrap, or I guess just cdebootstrap present which has
> sane behavior. But I would definitely not be able host the artifacts
> for those. :/

Oh, also mmdebstrap was defaulting to the broken merged-/usr-via-symlinks,
for which I filed , which has been fixed
now by disabling it completely! I'm planning on switching all my
debootstrap usage to mmdebstrap anyway because it's superior in any
conceivable way. :) I could also see trying to get that forked d-i use
mmdebstrap, also for its speed.

Thanks,
Guillem

Re: merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

Hi!

On Tue, 2019-02-19 at 05:49:24 +0100, Guillem Jover wrote:
> So, as part of damage control I'm going to:
> 
>   - include the Build-Tainted-By patches into dpkg 1.19.5.

Done.

>   - include a bug-script in dpkg for reportbug to report whether the
> system has been broken by the merged-/usr-via-symlinks hack.

Done.

>   - file a bug on reportbug to request reporting that for all packages.

Done. 

>   - file a bug on base-installer to request an option to install
> non-broken systems due to merged-/usr-via-symlinks.

Done. 

> I'm also pondering whether to implement checks for the broken
> merged-/usr-via-symlinks within «dpkg --audit», even though I don't
> really want to, as that would imply hardcoding Debian-specific
> pathnames in there. :(

I still have to think about this one. :/

> And I'm probably going to end up writing a unmerge-usr-via-symlinks
> script so that people with damaged systems can go back somewhat to a
> sane state, and to open the possibility for a fully automated migration
> to a proper and correct merged-/usr w/o all the problems above.

And I might need to start on this soon enough. :(

In addition, given that most probably Debian buster will end up
installing broken systems by default. I might end up also looking into
generating fixed minimal netinst images or mini netboot images with a
fixed debootstrap, or I guess just cdebootstrap present which has
sane behavior. But I would definitely not be able host the artifacts
for those. :/

Thanks,
Guillem

merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover]

Hi!

On Tue, 2018-11-20 at 22:16:17 +0100, Adam Borowski wrote:
> Thus, it seems to me that the plan A for usrmerge has serious downsides for
> dubious benefits.  What about the plan B I described above?

So, people still seem to be conflating merged-/usr (the filesystem
layout) with the different ways to deploy it. Personally I do agree
that merged-/usr has benefits, and that's why I've f.ex. been switching
the shared library packages I maintain to do a proper and correct switch
by installing their files in /usr/lib instead of /lib.

The underlaying problem with the merged-/usr-via-symlinks is not
really that it has generated bugs, many transitions we perform incur
those, for the better. The problem is that it has generated those when
it was really not necessary in the first place, has made things
unnecessarily more complicated for everyone, and it's a terrible hack
trying to reap "quick" benefits at the cost of everyhing else, with
long lasting consequences even after a full migration to /usr would
have been finished. And here's why:

1) The merged-/usr-via-symlinks deployment method used by both
   debootstrap and usrmerge, means that those systems will get
   permanent filesystem aliasing problems, forever. Even when all
   files might have been moved to their /usr counterparts in the
   packages! This is due to the fact that different pathnames can
   end up pointing (before any canonicalization!) to the same dentry.

   This does not only affect dpkg (dpkg, dpkg-divert, dpkg-trigger,
   etc), and update-alternatives (in a worse way), but any program
   that uses pathanmes in the filesystem as keys in databases f.ex.

2) It bypasses the packaging system understanding of what is on the
   filesystem and creates mismatches between what's on binary packages
   and what's on disk.

3) Switching packages to the merged-/usr layout could have been
   accomplished automatically via debhelper for a coverage of around
   99% (?) of the archive. With something along the lines of:

 ,---
 D=debian/tmp
 for d in /bin /sbin /lib; do
   for p in $(find $D/$d ! -type d); do
 b="${p##$D/}"
 m="$D/usr$b"
 if [ ! -e "$m" ]; then
   mkdir -p "$(dirname $m)"
   mv "$p" "$m"
   ln -s "${m##$D}" "$p"
 fi
   done
 done
 `---

   With the property that it would handle gracefully all cases were the
   maintainer has checked that no compat symlinks are required and has
   then progressively moved the pathname installation to their final
   destination under /usr.

4) Due to having to support the broken merged-/usr-via-symlinks
   deployment, when we want to move the contents of the binary
   packages to the merged-/usr layout, we require now to include tons
   of logic in (probably new) maintainer scripts, when we have been
   trying to remove them altogether. :( With even more files untracked
   by dpkg itself, bypassing the packaging system even more, when the
   compat symlinks could have been shipped in the binary packages.

5) Most new installations will not even benefit from this hacky and
   rushed deployment, as long as they do not use a separate parition
   for /usr!

6) The merged-/usr-via-symlinks deployment hack is irreversible right
   now.

This is all really terrible engineering. :(


So, as part of damage control I'm going to:

  - include the Build-Tainted-By patches into dpkg 1.19.5.
  - include a bug-script in dpkg for reportbug to report whether the
system has been broken by the merged-/usr-via-symlinks hack.
  - file a bug on reportbug to request reporting that for all packages.
  - file a bug on base-installer to request an option to install
non-broken systems due to merged-/usr-via-symlinks.

I'm also pondering whether to implement checks for the broken
merged-/usr-via-symlinks within «dpkg --audit», even though I don't
really want to, as that would imply hardcoding Debian-specific
pathnames in there. :(

And I'm probably going to end up writing a unmerge-usr-via-symlinks
script so that people with damaged systems can go back somewhat to a
sane state, and to open the possibility for a fully automated migration
to a proper and correct merged-/usr w/o all the problems above.

Thanks,
Guillem

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Russ Allbery]

Guillem Jover  writes:

> So I think I'll go ahead with the current name for now, it's going to be
> an optional field anyway, so if there's a better name proposed that
> conveys a satisfactory meaning, I'll be happy to consider it and do a
> rename, and handle any users of the current name.

Why not some variation of "build environment properties"?

I feel like the general feature here is to capture possibly relevant
information about the build environment, and that's the most general form.
For instance, I could imagine it being interesting in the future to
capture whether the build was done via sbuild, pbuilder, or cowbuilder,
none of which would arguably qualify as "tainted" but which may have
different behavior and may therefore be useful in tracking down a bug.

-- 
Russ Allbery (r...@debian.org)   

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Guillem Jover]

Hi!

On Mon, 2018-12-03 at 18:20:44 -0800, Russ Allbery wrote:
> Guillem Jover  writes:
> > … and then I'm not entirely sure a non-minimal environment should be
> > qualified as tainted? For example contrast using a minimal but outdated
> > installation to a non-minimal, but clean and up-to-date one.
> 
> > I think I'm still of the opinion that a user should be able to build on
> > a normal (clean and up-to-date) system and get a proper result. I guess
> > the problem might be how to define "clean". :)
> 
> Tainted is a loaded term that may make this more confusing.  I think it
> may be better to instead think of it as additional metadata to figure out
> why a package is buggy, if a bug shows up.  Some build states we know are
> more likely to cause problems than others, but if a bug exists only in the
> versions of the package built in a minimal chroot and not in the versions
> built on a regular system, that's a useful clue to what may be causing
> problems.

So, I didn't end up merging the patch for 1.19.[34] due to this, which
was my initial intention. Having thought about this a bit more, I do
really want to include them, and perhaps as a non-native speaker,
tainted looks fine to me. This is also the term I've had in my mind
for a long time to use for tracking f.ex. automatically sanitized
elements in the dpkg database, or when performing dangerous actions
on packages:

  <https://wiki.debian.org/Teams/Dpkg/Spec/TaintedDatabase>

To me tainted does not necessarily mean something is broken in the
resulting artifact, just that there was something unclean about the
environment that might have affected it. There's also precedent in the
Linux kernel with external modules tainting the kernel, but that does
not mean that, say, an oops, was caused by that.

And when it comes to the merged-usr-via-symlinks tag, I actually think
that tainted is really an understatement.

On Wed, 2018-12-05 at 13:35:36 +, Ian Jackson wrote:
> Russ Allbery writes ("Re: Tainted builds (was Re: usrmerge -- plan B?)"):
> > Tainted is a loaded term that may make this more confusing.
> 
> Can we think of a better term before `taint' gets embedded ?
> 
> It's going to be annoying if we have to have an argument every time we
> want to capture some additional data of this kind, because someone
> will pop up and say `my build system is like this and it is fine' or
> whatever.

So I think I'll go ahead with the current name for now, it's going to
be an optional field anyway, so if there's a better name proposed that
conveys a satisfactory meaning, I'll be happy to consider it and do a
rename, and handle any users of the current name.

Thanks,
Guillem

Re: usrmerge -- plan B?

[Sven Hartge]

Martin Steigerwald  wrote:
> Marco d'Itri - 23.12.18, 22:30:
>> On Dec 23, Martin Steigerwald  wrote:

>>> I think I have seen this with either SLES or RHEL that they created
>>> symlinks for every binary in /bin and /sbin, pointing to the binary
>>> in /usr/bin and /usr/sbin. I did not understand why at the time I
>>> have seen this.
 
>> Definitely not RHEL, maybe you are thinking about SCO UNIX?  Anyway,
>> this is just a fantasy: I do not believe that a forest of symlinks
>> could work for the content of /lib*.

> It might have been SLES. During my holidays I am certainly not
> accessing my training VMs. But I can look next year.

SLES12 has this:

server:~ # ls -l /bin | grep usr
lrwxrwxrwx 1 root root  13 Aug 31 16:00 arch -> /usr/bin/arch
lrwxrwxrwx 1 root root  17 Aug 31 16:00 basename -> /usr/bin/basename
lrwxrwxrwx 1 root root  12 Aug 31 16:00 cat -> /usr/bin/cat
lrwxrwxrwx 1 root root  14 Aug 31 16:00 chgrp -> /usr/bin/chgrp
lrwxrwxrwx 1 root root  14 Aug 31 16:00 chmod -> /usr/bin/chmod
lrwxrwxrwx 1 root root  14 Aug 31 16:00 chown -> /usr/bin/chown
lrwxrwxrwx 1 root root  13 Nov  6 10:22 chvt -> /usr/bin/chvt
lrwxrwxrwx 1 root root  18 Nov  6 10:22 clrunimap -> /usr/bin/clrunimap
lrwxrwxrwx 1 root root  11 Aug 31 16:00 cp -> /usr/bin/cp
lrwxrwxrwx 1 root root  13 Nov  6 10:22 cpio -> /usr/bin/cpio
lrwxrwxrwx 1 root root  13 Aug 31 16:00 date -> /usr/bin/date
lrwxrwxrwx 1 root root  11 Aug 31 16:00 dd -> /usr/bin/dd
lrwxrwxrwx 1 root root  18 Nov  6 10:22 deallocvt -> /usr/bin/deallocvt
lrwxrwxrwx 1 root root  11 Aug 31 16:00 df -> /usr/bin/df
lrwxrwxrwx 1 root root  14 Nov 30 09:03 dmesg -> /usr/bin/dmesg
lrwxrwxrwx 1 root root  17 Nov  6 10:22 dumpkeys -> /usr/bin/dumpkeys
lrwxrwxrwx 1 root root  13 Aug 31 16:00 echo -> /usr/bin/echo
lrwxrwxrwx 1 root root  11 Aug 31 16:00 ed -> /usr/bin/ed
lrwxrwxrwx 1 root root  14 Aug 31 16:00 egrep -> /usr/bin/egrep
lrwxrwxrwx 1 root root  14 Aug 31 16:00 false -> /usr/bin/false
[...]

server:~ # ls -l /sbin | grep usr
lrwxrwxrwx 1 root root28 Aug 31 16:03 OCICLI -> 
/usr/sbin/OneClickInstallCLI
lrwxrwxrwx 1 root root28 Aug 31 16:03 OneClickInstallCLI -> 
/usr/sbin/OneClickInstallCLI
lrwxrwxrwx 1 root root35 Aug 31 16:03 OneClickInstallUI -> 
/usr/sbin/OneClickInstallUrlHandler
lrwxrwxrwx 1 root root35 Aug 31 16:03 OneClickInstallUrlHandler -> 
/usr/sbin/OneClickInstallUrlHandler
lrwxrwxrwx 1 root root23 Aug 31 16:03 SuSEfirewall2 -> 
/usr/sbin/SuSEfirewall2
lrwxrwxrwx 1 root root18 Aug 31 16:00 adjtimex -> /usr/sbin/adjtimex
lrwxrwxrwx 1 root root16 Nov 30 09:03 agetty -> /usr/sbin/agetty
lrwxrwxrwx 1 root root16 Aug 31 16:01 arping -> /usr/sbin/arping
lrwxrwxrwx 1 root root17 Aug 31 16:02 audispd -> /usr/sbin/audispd
lrwxrwxrwx 1 root root18 Aug 31 16:02 auditctl -> /usr/sbin/auditctl
lrwxrwxrwx 1 root root16 Aug 31 16:02 auditd -> /usr/sbin/auditd
lrwxrwxrwx 1 root root20 Aug 31 16:02 augenrules -> /usr/sbin/augenrules
lrwxrwxrwx 1 root root18 Aug 31 16:02 aureport -> /usr/sbin/aureport
lrwxrwxrwx 1 root root18 Aug 31 16:02 ausearch -> /usr/sbin/ausearch
lrwxrwxrwx 1 root root17 Aug 31 16:02 autrace -> /usr/sbin/autrace
lrwxrwxrwx 1 root root19 Aug 31 16:00 badblocks -> /usr/sbin/badblocks
lrwxrwxrwx 1 root root15 Nov 30 09:03 blkid -> /usr/sbin/blkid
[...]

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

Re: usrmerge -- plan B?

[Martin Steigerwald]

Marco d'Itri - 23.12.18, 22:30:
> On Dec 23, Martin Steigerwald  wrote:
> > I think I have seen this with either SLES or RHEL that they created
> > symlinks for every binary in /bin and /sbin, pointing to the binary
> > in /usr/bin and /usr/sbin. I did not understand why at the time I
> > have seen this.
> 
> Definitely not RHEL, maybe you are thinking about SCO UNIX?
> Anyway, this is just a fantasy: I do not believe that a forest of
> symlinks could work for the content of /lib*.

It might have been SLES. During my holidays I am certainly not accessing 
my training VMs. But I can look next year.

And no, I am not thinking about SCO UNIX.

-- 
Martin

Re: usrmerge -- plan B?

[Marco d'Itri]

On Dec 23, Martin Steigerwald  wrote:

> I think I have seen this with either SLES or RHEL that they created 
> symlinks for every binary in /bin and /sbin, pointing to the binary in
> /usr/bin and /usr/sbin. I did not understand why at the time I have seen 
> this.
Definitely not RHEL, maybe you are thinking about SCO UNIX?
Anyway, this is just a fantasy: I do not believe that a forest of 
symlinks could work for the content of /lib*.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Martin Steigerwald]

Guillem Jover - 23.12.18, 17:17:
> On Sun, 2018-12-23 at 16:45:28 +0100, Guillem Jover wrote:
> > On Sun, 2018-12-23 at 04:06:14 +0100, Guillem Jover wrote:
> > > […] They also imply to permanently suffer the aliasing problems.
> > 
> > To expand and clarify a bit on this. We have aliasing in general
> > with
> > symlinks and hardlinks, but those tend to not be as problematic when
> > aliasing the last component, as they end up referencing the same
> > object/inode. The problem really comes when the aliasing is
> > performed
> > as part of the pathname (in a "directory" component), because then
> > the aliasing references the dentry, which can very easily cause
> > "action at a distance" issues.
> 
> Hrrmf, clearly not very clear. Let's try again:
> 
>   - Aliasing directory components can create multiple source pathnames
> (before any readlink) that refer to the exact same dentry. - Aliasing
> final components means that the source pathnames are always
> different, and as such they always point to different dentries, even
> if they might end up pointing to the same destination pathname once
> canonicalized.

I think I have seen this with either SLES or RHEL that they created 
symlinks for every binary in /bin and /sbin, pointing to the binary in
/usr/bin and /usr/sbin. I did not understand why at the time I have seen 
this.

-- 
Martin

Re: usrmerge -- plan B?

[Guillem Jover]

On Sun, 2018-12-23 at 16:45:28 +0100, Guillem Jover wrote:
> On Sun, 2018-12-23 at 04:06:14 +0100, Guillem Jover wrote:
> > […] They also imply to permanently suffer the aliasing problems.
> 
> To expand and clarify a bit on this. We have aliasing in general with
> symlinks and hardlinks, but those tend to not be as problematic when
> aliasing the last component, as they end up referencing the same
> object/inode. The problem really comes when the aliasing is performed
> as part of the pathname (in a "directory" component), because then
> the aliasing references the dentry, which can very easily cause
> "action at a distance" issues.

Hrrmf, clearly not very clear. Let's try again:

  - Aliasing directory components can create multiple source pathnames
(before any readlink) that refer to the exact same dentry.
  - Aliasing final components means that the source pathnames are
always different, and as such they always point to different
dentries, even if they might end up pointing to the same
destination pathname once canonicalized.

Thanks,
Guillem

Re: usrmerge -- plan B?

[Guillem Jover]

On Sun, 2018-12-23 at 04:06:14 +0100, Guillem Jover wrote:
> […] They also imply to permanently suffer the aliasing problems.

To expand and clarify a bit on this. We have aliasing in general with
symlinks and hardlinks, but those tend to not be as problematic when
aliasing the last component, as they end up referencing the same
object/inode. The problem really comes when the aliasing is performed
as part of the pathname (in a "directory" component), because then
the aliasing references the dentry, which can very easily cause
"action at a distance" issues.

Thanks,
Guillem

Re: usrmerge -- plan B?

[Marco d'Itri]

On Dec 23, Guillem Jover  wrote:

> To me it's always been very clear the only *proper* way to deploy
> merged-/usr, is to do the changes to the canonical pathnames on
> each relevant package. Even adding compat symlinks, means that dpkg
> would know about them, and they'd be temporary anyway for the most
> part until any remaining user has been made path relative or its
> absolute path changed.
This is impossibile: it will NEVER be practical to change paths like 
/bin/sh and /lib/ld-linux.so.2, so there will always be symlinks in the 
root.

>   - the original dpkg-shlibdeps problems was known way ahead of the
> initial debootstrap switch.
There was an open bug but nobody could reproduce it and the cause of the 
problem was not clear at all, at least to me. The adopted solution was 
also trivial, so if it was so obvious maybe you could have told 
somebody?

>   - the problem with misbuilds was also obvious and mentioned at least
> on that same dpkg-shlibdeps bug report.
Maybe I missed that too?

>   - the dpkg-query issue (and the aliasing problem in general) has
> been also known for a while now.
Yes, but since it does not cause practical problems I think that we can 
live with it.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Guillem Jover]

Hi!

On Mon, 2018-12-03 at 18:22:03 -0800, Russ Allbery wrote:
> Guillem Jover  writes:
> > The current merged-/usr deployment (via usrmerge or the bootstrapping
> > symlink generation before unpack) is a major hack, and bypasses the dpkg
> > understanding of the filesystem, it breaks dpkg-query, and while we
> > could workaround the breakage there, it's still the wrong way to go
> > around this.
> 
> Does this imply that you're considering adding support for merging /usr
> *properly* directly inside dpkg, with all the correct updates to dpkg's
> metadata?

Above I was talking mostly about workarounds, such as the one I
mentioned earlier in the thread, which I've got as a draft patch
for dpkg-query to try to fix #858331:

  


which mostly let's dpkg-query know about the aliasing in the filesystem.
This will still leave out dpkg-divert, dpkg-statoverride, dpkg-trigger,
and update-alternatives. This will also leave out any other software
that uses filesystem paths as keys in their databases.

Some of these other commands will just fail to find the files requested,
because they are not the canonical ones. On a merged-/usr-via-symlinks
u-a in particular can be told to make a mess if called like this f.ex.:

  $ update-alternatives --install /bin/iface iface-a /usr/bin/impl-x 100
  $ update-alternatives --install /usr/bin/iface iface-b /usr/bin/impl-z 200


What you are proposing above would be to do the magical mapping from
the canonical location found in the .deb into the new location on the
filesystem. I don't think this is a generally applicable solution.

Of the top of my head I can think of two potential ways to solve this:

 * Relocatable packages [R], where dpkg would do some kind of global
   diversion on specific pathnames.

   This would require support from the package itself, at least to
   denote that it supports being relocated. This might also require
   major patching on the upstream side in case it does not. And if we
   have to do either, we might as well have modified the canonical
   paths in the package instead, which would be way less work.

   This is still a somewhat interesting theoretical idea, though.

   [R] The mythical dpkg 2.0 spec already talks about this in its
   appendix G, .

 * A variation on relocatable packages, but in addition automatically
   inject the aliased paths into the filesystem and database as if
   they had come from the package, for backwards compatibility, iff
   they are not already present in the package or the database.

   This would also require each injected pathname to be tracked
   specifically, so that something like dpkg-repack can do its magic.

   This would also have the potential to file conflict with random
   stuff in case other packages provide those paths, and might be
   difficult to check centrally, as the generation of those paths
   would be local.


Both of these solutions would at no point convert the /bin, /sbin and
/lib directories into symlinks, I think doing that from within dpkg
would be maddness TBH.

Both would need the dpkg interfaces to be augmented in some way to be
able to tell what are the canonical paths for a file, and in the
second variant, which have been magically injected.

As generalized solutions to the aliasing problem it would mean that
local admins would be able to map files anywhere, which would imply
that the query results for (at least) many dpkg-related things could
vary wildly depending on the system, even when using the exact same
distribution baseline.

So I do not see how the above can be considered proper either in any
way. :)

> Because that would be an awesome way to fully support this.

While I think these are theoretically interesting solutions, they look
to me like huge can of worms, with very problematic consequences on
the global filesystem namespace we use as a shared interface. They
also imply to permanently suffer the aliasing problems.

To me it's always been very clear the only *proper* way to deploy
merged-/usr, is to do the changes to the canonical pathnames on
each relevant package. Even adding compat symlinks, means that dpkg
would know about them, and they'd be temporary anyway for the most
part until any remaining user has been made path relative or its
absolute path changed.



But the way this specific deployment plan has been pushed forward
has been extremely frustrating. There's been continued warnings about
the problems ahead, and they have been consistently ignored or
downplayed:

  - the original dpkg-shlibdeps problems was known way ahead of the
initial debootstrap switch.
  - the problem with misbuilds was also obvious and mentioned at least
on that same dpkg-shlibdeps bug report.
  - the dpkg-query issue (and the aliasing problem in general) has
been also known for a while now.

And still the debootstrap default was switched again, when t

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Ian Jackson]

Russ Allbery writes ("Re: Tainted builds (was Re: usrmerge -- plan B?)"):
> Tainted is a loaded term that may make this more confusing.

Can we think of a better term before `taint' gets embedded ?

It's going to be annoying if we have to have an argument every time we
want to capture some additional data of this kind, because someone
will pop up and say `my build system is like this and it is fine' or
whatever.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Ian Jackson]

Guillem Jover writes ("Re: Tainted builds (was Re: usrmerge -- plan B?)"):
> I think I'm still of the opinion that a user should be able to build on
> a normal (clean and up-to-date) system and get a proper result. I guess
> the problem might be how to define "clean". :)

And how to define "proper".  This is being discussed in more detail in
debian-policy (thread on #824495).

I like your /usr/local taints.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Holger Levsen]

On Tue, Dec 04, 2018 at 01:07:42AM +0100, Guillem Jover wrote:
> These will detect problematic files under /usr/local which can taint
> the current build.
[...]
> +.B usr\-local\-has\-programs

I regularily have stuff in /usr/local/(s)bin/ which does not taint the
system nor my builds, so I think your classification that any file in
/usr/local/(s)bin/ taints the environement or the build is wrong.

(And not sure whether I said this in this thread already, but IMO a 
not /usr merged system is the tainted, stainted, crufted, legacy one...)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Marco d'Itri]

On Dec 04, Russ Allbery  wrote:

> Does this imply that you're considering adding support for merging /usr
> *properly* directly inside dpkg, with all the correct updates to dpkg's
> metadata?
> 
> Because that would be an awesome way to fully support this.
It would make using dpkg -S a bit easier but other than this it would 
not solve any problem: there are files in /bin /sbin /lib which are not 
created by the package manager so something like usrmerge would still be 
needed.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Russ Allbery]

Guillem Jover  writes:

> The current merged-/usr deployment (via usrmerge or the bootstrapping
> symlink generation before unpack) is a major hack, and bypasses the dpkg
> understanding of the filesystem, it breaks dpkg-query, and while we
> could workaround the breakage there, it's still the wrong way to go
> around this.

Does this imply that you're considering adding support for merging /usr
*properly* directly inside dpkg, with all the correct updates to dpkg's
metadata?

Because that would be an awesome way to fully support this.

-- 
Russ Allbery (r...@debian.org)   

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Russ Allbery]

Guillem Jover  writes:

> … and then I'm not entirely sure a non-minimal environment should be
> qualified as tainted? For example contrast using a minimal but outdated
> installation to a non-minimal, but clean and up-to-date one.

> I think I'm still of the opinion that a user should be able to build on
> a normal (clean and up-to-date) system and get a proper result. I guess
> the problem might be how to define "clean". :)

Tainted is a loaded term that may make this more confusing.  I think it
may be better to instead think of it as additional metadata to figure out
why a package is buggy, if a bug shows up.  Some build states we know are
more likely to cause problems than others, but if a bug exists only in the
versions of the package built in a minimal chroot and not in the versions
built on a regular system, that's a useful clue to what may be causing
problems.

But perhaps the reproducible build testing infrastructure is the better
solution to this problem.

-- 
Russ Allbery (r...@debian.org)   

Re: usrmerge -- plan B?

[Guillem Jover]

On Sun, 2018-12-02 at 13:40:05 +0200, Niko Tyni wrote:
> On Fri, Nov 23, 2018 at 04:32:17PM +0100, Guillem Jover wrote:
> > Please, if we decide we want to do merged /usr, let's do it properly.
> 
> I'm toying with the idea of creating a merged-usr package indicating
> 'this system has merged /usr'. It could either fail to install with an
> informational message on unmerged-usr systems, or if we're brave enough,
> interactively or automatically convert the system (possibly utilizing
> the usrmerge package.)
> 
> Then debhelper (or even dpkg-dev) could probe whether the build system
> has merged /usr, and automatically add dependencies on merged-usr to
> resulting binary packages, The dependency injection could be on either
> an opt-out or an opt-in basis.
> 
> The opt-out way would be the more safe and conservative approach,
> assuming everything built on merged /usr is broken on non-merged /usr
> unless the maintainer indicates otherwise.
> 
> On the contrary, the opt-in approach would require that all problematic
> packages (those that will not function fine when built on merged /usr
> systems but installed on non-merged /usr ones) be identified and tagged
> as needing this dependency. This obviously has a chance of missing
> some of the issues.
> 
> Whether opt-out or opt-in, the transition in Debian could then be
> controlled by monitoring dependencies on merged-usr, possibly playing
> whack-a-mole rebuilding things that have gained this dependency in
> maintainer uploads, or letting the gates open at some point and allowing
> even core packages to gain this dependency (effectively forcing /usr
> merge on all systems.)
> 
> All this would make the issues explicit and reflected in the package
> dependency metadata, which seems to me like at least a step forward.

IMO this would make it even worse than what it is right now. And I
think it still keeps missing the point. :/

The current merged-/usr deployment (via usrmerge or the bootstrapping
symlink generation before unpack) is a major hack, and bypasses the
dpkg understanding of the filesystem, it breaks dpkg-query, and while
we could workaround the breakage there, it's still the wrong way to go
around this.

The main reasons I'm reading to justify it are, "because it's faster,
and requires less effort", I mean, seriously… and as a "bonus" we now
have the ctte involved again. Great setup really.

Regards,
Guillem

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Mon, 2018-12-03 at 16:45:15 -0500, Michael Stone wrote:
> On Sun, Dec 02, 2018 at 04:28:46PM -0800, Russ Allbery wrote:
> > Guillem Jover  writes:
> > > Whether a package is being built within a chroot or not, has nothing
> > > to do with how that installation is being managed IMO. It feels a bit
> > > like recording what's the form factor of the machine being run on? :)
> > 
> > I think what people are trying to get at here is "was the package built on
> > a system with packages other than build dependencies plus build-essential
> > plus essential/required packages installed."

I guess, although the request was very specific, so was replying to
that. :) Thanks for extracting what might be the essence of the
request though. But as I mentioned in my earlier reply and as Michael
repeats below, there are way more problematic scenarios; and I don't
think this is the biggest concern when we talk about tainted builds…

> > I do think this would be very useful to track, but it's a bit complicated
> > to work out, and there are probably a few other exceptions that would need
> > to be in place.

… and then I'm not entirely sure a non-minimal environment should be
qualified as tainted? For example contrast using a minimal but outdated
installation to a non-minimal, but clean and up-to-date one.

I think I'm still of the opinion that a user should be able to build on
a normal (clean and up-to-date) system and get a proper result. I guess
the problem might be how to define "clean". :)

But even then, it might indeed be nice to check for this condition,
and record it somehow.

> And you'd still have cases like "someone installed something in
> /usr/local/bin" and such. Might be easier to just track whether it was built
> in a dsa-maintained autobuilder, so a human can identify potential local
> build environment issues as a possible explanation for unexpected behavior
> because that's really the objective. Might also not be worth trying to do
> that vs existing ways to find out where the package was built.

Hah, I just implemented this earlier today! (Attached, although I'm not
happy with the missing early exit, but the alternatives all leave to
be desired.) Also had clarified that the list of emitted tags is
non-exhaustive.

Thanks,
Guillem
From aa5b14369b693e3db3c92f63bf84fb234364 Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Mon, 3 Dec 2018 08:57:14 +0100
Subject: [PATCH] Dpkg::Vendor::Debian: Add support for usr-local-has-* tainted
 tags

These will detect problematic files under /usr/local which can taint
the current build.
---
 man/deb-buildinfo.man | 12 
 scripts/Dpkg/Vendor/Debian.pm | 14 ++
 2 files changed, 26 insertions(+)

diff --git a/man/deb-buildinfo.man b/man/deb-buildinfo.man
index 2248916d6..ba91baac7 100644
--- a/man/deb-buildinfo.man
+++ b/man/deb-buildinfo.man
@@ -166,6 +166,18 @@ of the filesystem that \fBdpkg\fP has recorded in its database.
 For build systems that hardcode pathnames to specific binaries or libraries
 on the resulting artifacts, it can also produce packages that will be
 incompatible with non-/usr-merged filesystems.
+.TP
+.B usr\-local\-has\-configs
+The system has configuration files under \fI/usr/local/etc\fP.
+.TP
+.B usr\-local\-has\-includes
+The system has header files under \fI/usr/local/include\fP.
+.TP
+.B usr\-local\-has\-programs
+The system has programs under \fI/usr/local/bin\fP or \fI/usr/local/sbin\fP.
+.TP
+.B usr\-local\-has\-libraries
+The system has libraries, either static or shared under \fI/usr/local/lib\fP.
 .RE
 .TP
 .BR Installed\-Build\-Depends: " (required)"
diff --git a/scripts/Dpkg/Vendor/Debian.pm b/scripts/Dpkg/Vendor/Debian.pm
index 6948bdc16..a352bbdde 100644
--- a/scripts/Dpkg/Vendor/Debian.pm
+++ b/scripts/Dpkg/Vendor/Debian.pm
@@ -455,6 +455,20 @@ sub _build_tainted_by {
 }
 }
 
+require File::Find;
+my %usr_local_types = (
+configs => [ qw(etc) ],
+includes => [ qw(include) ],
+programs => [ qw(bin sbin) ],
+libraries => [ qw(lib) ],
+);
+foreach my $type (keys %usr_local_types) {
+File::Find::find({
+wanted => sub { $tainted{"usr-local-has-$type"} = 1 if -f },
+no_chdir => 1,
+}, map { "/usr/local/$_" } @{$usr_local_types{$type}});
+}
+
 my @tainted = sort keys %tainted;
 return @tainted;
 }
-- 
2.20.0.rc2.403.gdbc3b29805

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Michael Stone]

On Sun, Dec 02, 2018 at 04:28:46PM -0800, Russ Allbery wrote:

Guillem Jover  writes:


Whether a package is being built within a chroot or not, has nothing
to do with how that installation is being managed IMO. It feels a bit
like recording what's the form factor of the machine being run on? :)


I think what people are trying to get at here is "was the package built on
a system with packages other than build dependencies plus build-essential
plus essential/required packages installed."

I do think this would be very useful to track, but it's a bit complicated
to work out, and there are probably a few other exceptions that would need
to be in place.


And you'd still have cases like "someone installed something in 
/usr/local/bin" and such. Might be easier to just track whether it was 
built in a dsa-maintained autobuilder, so a human can identify potential 
local build environment issues as a possible explanation for unexpected 
behavior because that's really the objective. Might also not be worth 
trying to do that vs existing ways to find out where the package was 
built.

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Russ Allbery]

Guillem Jover  writes:

> Whether a package is being built within a chroot or not, has nothing
> to do with how that installation is being managed IMO. It feels a bit
> like recording what's the form factor of the machine being run on? :)

I think what people are trying to get at here is "was the package built on
a system with packages other than build dependencies plus build-essential
plus essential/required packages installed."

I do think this would be very useful to track, but it's a bit complicated
to work out, and there are probably a few other exceptions that would need
to be in place.

-- 
Russ Allbery (r...@debian.org)   

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Wed, 2018-11-28 at 14:48:32 -0200, Antonio Terceiro wrote:
> On Wed, Nov 28, 2018 at 02:57:52PM +0100, Guillem Jover wrote:
> > This is actually a great idea! I went ahead and implemented this, see
> > attached tentative patch which I'm planning on including in dpkg 1.19.3.
> 
> Would you be willing to also implement
> 
>   Tainted-By: not-built-in-a-chroot
> 
> ?

I also agree with what others have mentioned down-thread, that this is
not really a tainting reason. Tainting would involve say, programs
or libraries under /usr/local/{bin,sbin/lib}/, perhaps even configs
under /usr/local/etc, or even /etc with conffiles diverged from the
defaults, say output from «dpkg --verify», files unowned by a package
(in the sense of either being tracked in dpkg database or whether the
package manually manages the pathnames) being present in directories
in the executable or linker search paths, etc.

While some of these might be reliable, easy to emit and cheap to
compute, doing for example «dpkg --verify» before every build would
be IMO unnacceptably slow, but would be a definitive answer to one
kind of tainted system.

Whether a package is being built within a chroot or not, has nothing
to do with how that installation is being managed IMO. It feels a bit
like recording what's the form factor of the machine being run on? :)

> a few nits in the manpage:

Great! Thanks for the review, I've tried to reword and improve several
of the docs, attached updated patch.

Thanks,
Guillem
From ca2b0f9710867f3d8e8bca3a6654b9dd67db3539 Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Sun, 2 Dec 2018 03:35:49 +0100
Subject: [PATCH] dpkg-genbuildinfo: Include a new Build-Tainted-By field

This field will contain a list of tainting reason tags, which can denote
that the current build has potentially been broken.

On Debian and derivatives it can emit now merged-usr-via-symlinks as a
tainting reason tag.

Suggested-by: Alexander E. Patrakov 
---
 man/deb-buildinfo.man  | 19 +++
 scripts/Dpkg/Control/FieldsCore.pm |  7 ++-
 scripts/Dpkg/Vendor/Debian.pm  | 20 
 scripts/Dpkg/Vendor/Default.pm | 10 ++
 scripts/dpkg-genbuildinfo.pl   |  2 ++
 5 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/man/deb-buildinfo.man b/man/deb-buildinfo.man
index 5013aa047..0708fa319 100644
--- a/man/deb-buildinfo.man
+++ b/man/deb-buildinfo.man
@@ -149,6 +149,25 @@ via some pattern match to avoid leaking possibly sensitive information.
 On Debian and derivatives only build paths starting with \fI/build/\fP
 will emit this field.
 .TP
+.B Build\-Tainted\-By:
+.TQ
+.I " taint-reason-list"
+This folded field contains a space-separated list of reason tags (formed by
+alphanumeric and dash characters) which identify why the current build has
+been tainted (since dpkg 1.19.3).
+.IP
+On Debian and derivatives the following reason tags can be emitted:
+.RS
+.TP
+.B merged\-usr\-via\-symlinks
+The system has a merged \fI/usr\fP via symlinks.
+This will confuse \fBdpkg\-query\fP as it messes with the understanding
+of the filesystem that \fBdpkg\fP has recorded in its database.
+For build systems that hardcode pathnames to specific binaries or libraries
+on the resulting artifacts, it can also produce packages that will be
+incompatible with non-/usr-merged filesystems.
+.RE
+.TP
 .BR Installed\-Build\-Depends: " (required)"
 .TQ
 .I " package-list"
diff --git a/scripts/Dpkg/Control/FieldsCore.pm b/scripts/Dpkg/Control/FieldsCore.pm
index b100366e1..f460433fc 100644
--- a/scripts/Dpkg/Control/FieldsCore.pm
+++ b/scripts/Dpkg/Control/FieldsCore.pm
@@ -176,6 +176,11 @@ our %FIELDS = (
 allowed => CTRL_INFO_PKG,
 separator => FIELD_SEP_SPACE,
 },
+'build-tainted-by' => {
+name => 'Build-Tainted-By',
+allowed => CTRL_FILE_BUILDINFO,
+separator => FIELD_SEP_SPACE,
+},
 'built-for-profiles' => {
 name => 'Built-For-Profiles',
 allowed => ALL_PKG | CTRL_FILE_CHANGES,
@@ -634,7 +639,7 @@ our %FIELD_ORDER = (
 qw(format source binary architecture version binary-only-changes),
 @src_checksums_fields,
 qw(build-origin build-architecture build-kernel-version build-date
-build-path installed-build-depends environment),
+build-path build-tainted-by installed-build-depends environment),
 ],
 CTRL_FILE_CHANGES() => [
 qw(format date source binary binary-only built-for-profiles architecture
diff --git a/scripts/Dpkg/Vendor/Debian.pm b/scripts/Dpkg/Vendor/Debian.pm
index 7d4b6d802..6948bdc16 100644
--- a/scripts/Dpkg/Vendor/Debian.pm
+++ b/scripts/Dpkg/Vendor/Debian.pm
@@ -81,6 +81,8 @@ sub run_hook {
 $self->_add_build_flags(@params);
 } elsif ($hook eq 'builtin-system-build-paths') {
 return qw(/build/);
+} elsif ($hook eq 'build-tainted-by') {
+return $self->_build_tainted_by();
 } else {
 return $self->SUPER::run_hook($ho

Re: usrmerge -- plan B?

[Niko Tyni]

On Fri, Nov 23, 2018 at 04:32:17PM +0100, Guillem Jover wrote:

> Please, if we decide we want to do merged /usr, let's do it properly.

I'm toying with the idea of creating a merged-usr package indicating
'this system has merged /usr'. It could either fail to install with an
informational message on unmerged-usr systems, or if we're brave enough,
interactively or automatically convert the system (possibly utilizing
the usrmerge package.)

Then debhelper (or even dpkg-dev) could probe whether the build system
has merged /usr, and automatically add dependencies on merged-usr to
resulting binary packages, The dependency injection could be on either
an opt-out or an opt-in basis.

The opt-out way would be the more safe and conservative approach,
assuming everything built on merged /usr is broken on non-merged /usr
unless the maintainer indicates otherwise.

On the contrary, the opt-in approach would require that all problematic
packages (those that will not function fine when built on merged /usr
systems but installed on non-merged /usr ones) be identified and tagged
as needing this dependency. This obviously has a chance of missing
some of the issues.

Whether opt-out or opt-in, the transition in Debian could then be
controlled by monitoring dependencies on merged-usr, possibly playing
whack-a-mole rebuilding things that have gained this dependency in
maintainer uploads, or letting the gates open at some point and allowing
even core packages to gain this dependency (effectively forcing /usr
merge on all systems.)

All this would make the issues explicit and reflected in the package
dependency metadata, which seems to me like at least a step forward.
-- 
Niko Tyni   nt...@debian.org

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Fri, 2018-11-30 at 05:51:35 +0900, Mike Hommey wrote:
> "Only Essential: yes and direct build dependencies installed"? Why not
> extend .buildinfo with the list of all packages installed that aren't
> Essential:yes or build dependencies?

Because that'd have the potential to leak privacy and security sensitive
information.

The same reason why neither the kernel version nor the build path are
leaked by default, and require explicit command-line options.

Thanks,
Guillem

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Mike Hommey]

On Thu, Nov 29, 2018 at 09:07:46AM -0200, Antonio Terceiro wrote:
> On Wed, Nov 28, 2018 at 07:02:07PM +0100, Bastian Blank wrote:
> > On Wed, Nov 28, 2018 at 02:48:32PM -0200, Antonio Terceiro wrote:
> > > Would you be willing to also implement
> > >   Tainted-By: not-built-in-a-chroot
> > > ?
> > 
> > What do you want to do with that?  Even our own stuff not always uses
> > chroot, why should it?
> 
> The idea here is to record facts about the system where a package was
> built. Building in a merged-/usr system does not necessarily produce a
> broken package. Not building in a chroot is also not necessarily a
> problem, but still, we want to know that.
> 
> Now, as Andrey points out, nowdays a chroot is not the only type of
> minimal system where one can build packages, so maybe a more
> sophisticated check would be required.

"Only Essential: yes and direct build dependencies installed"? Why not
extend .buildinfo with the list of all packages installed that aren't
Essential:yes or build dependencies?

Mike

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Antonio Terceiro]

On Wed, Nov 28, 2018 at 07:02:07PM +0100, Bastian Blank wrote:
> On Wed, Nov 28, 2018 at 02:48:32PM -0200, Antonio Terceiro wrote:
> > Would you be willing to also implement
> > Tainted-By: not-built-in-a-chroot
> > ?
> 
> What do you want to do with that?  Even our own stuff not always uses
> chroot, why should it?

The idea here is to record facts about the system where a package was
built. Building in a merged-/usr system does not necessarily produce a
broken package. Not building in a chroot is also not necessarily a
problem, but still, we want to know that.

Now, as Andrey points out, nowdays a chroot is not the only type of
minimal system where one can build packages, so maybe a more
sophisticated check would be required.


signature.asc
Description: PGP signature

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Bastian Blank]

On Wed, Nov 28, 2018 at 02:48:32PM -0200, Antonio Terceiro wrote:
> Would you be willing to also implement
>   Tainted-By: not-built-in-a-chroot
> ?

What do you want to do with that?  Even our own stuff not always uses
chroot, why should it?

Bastian

-- 
Ahead warp factor one, Mr. Sulu.

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Andrey Rahmatullin]

On Wed, Nov 28, 2018 at 06:40:46PM +0100, Guillem Jover wrote:
> On Wed, 2018-11-28 at 22:13:41 +0500, Andrey Rahmatullin wrote:
> > On Wed, Nov 28, 2018 at 02:48:32PM -0200, Antonio Terceiro wrote:
> > > (ischroot(1) is from debianutils which is Essential).
> 
> > "On GNU/Linux, chroot detection is not possible when not root."
> 
> I think this was just missed as part of the fix for #670119. Care to
> send an MR/bug?
Nice. I filed a bug.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Guillem Jover]

On Wed, 2018-11-28 at 22:13:41 +0500, Andrey Rahmatullin wrote:
> On Wed, Nov 28, 2018 at 02:48:32PM -0200, Antonio Terceiro wrote:
> > (ischroot(1) is from debianutils which is Essential).

> "On GNU/Linux, chroot detection is not possible when not root."

I think this was just missed as part of the fix for #670119. Care to
send an MR/bug?

Thanks,
Guillem

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Andrey Rahmatullin]

On Wed, Nov 28, 2018 at 02:48:32PM -0200, Antonio Terceiro wrote:
> Would you be willing to also implement
> 
>   Tainted-By: not-built-in-a-chroot
That doesn't mean anything. You can build in a bad chroot and you can
build in a clean minimal sid system which is not a chroot but a VM.

> (ischroot(1) is from debianutils which is Essential).
"On GNU/Linux, chroot detection is not possible when not root."

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Tainted builds (was Re: usrmerge -- plan B?)

[Antonio Terceiro]

On Wed, Nov 28, 2018 at 02:57:52PM +0100, Guillem Jover wrote:
> Hi!
> 
> On Wed, 2018-11-28 at 07:52:08 +0500, Alexander E. Patrakov wrote:
> > Well, the buildd configuration change has been reverted. What worries me now
> > is that there is a risk not yet mitigated, coming from personal systems of
> > Debian developers, and we should also check porter boxes.
> > 
> > As long as there is one Debian Developer (or any other person who has the
> > right to upload binary packages) who has a merged /usr on his system used
> > for building packages, there is a risk of reintroducing the bug through his
> > package. Maybe we should somehow, in the short term, modify dpkg to add
> > something like "Tainted-By: usr-merge" control field to all binary packages
> > produced, if a package is built on a system with merged /usr (detected via
> > /bin being a symlink). And a corresponding automatic check that would
> > auto-reject binary packages with any Tainted-By control field from being
> > uploaded to the Debian archive.
> 
> This is actually a great idea! I went ahead and implemented this, see
> attached tentative patch which I'm planning on including in dpkg 1.19.3.

Would you be willing to also implement

Tainted-By: not-built-in-a-chroot

?

(ischroot(1) is from debianutils which is Essential).

a few nits in the manpage:

> diff --git i/man/deb-buildinfo.man w/man/deb-buildinfo.man
> index 5013aa047..8aa333965 100644
> --- i/man/deb-buildinfo.man
> +++ w/man/deb-buildinfo.man
> @@ -149,6 +149,19 @@ via some pattern match to avoid leaking possibly 
> sensitive information.
>  On Debian and derivatives only build paths starting with \fI/build/\fP
>  will emit this field.
>  .TP
> +.BR Build\-Tainted\-By: " \fItaint-reasons...\fP"
> +The list of reasons in the form of string tags (alphanumeric and dash),
> +that can taint the current build.

Other fields explicitly mention being space-separated, you might want to do the
same here.

> +.RS
> +.TP
> +.B merged-usr-via-symlinks
> +The system has a merged /usr via symlinks.
> +This will confuse \fBdpkg\-query\fP as it messes with the \fBdpkg\fP
> +understanding of the filesystem it manages.
> +It can also produce packages with hardcoded paths that will be incompatible
> +with non-usr-merged filesystems.

I would say here:

For packages that hardcode paths to specific binaries at build time, it can
also produce packages that are incompatible with non-usr-merged systems.


signature.asc
Description: PGP signature

Tainted builds (was Re: usrmerge -- plan B?)

[Guillem Jover]

Hi!

On Wed, 2018-11-28 at 07:52:08 +0500, Alexander E. Patrakov wrote:
> Well, the buildd configuration change has been reverted. What worries me now
> is that there is a risk not yet mitigated, coming from personal systems of
> Debian developers, and we should also check porter boxes.
> 
> As long as there is one Debian Developer (or any other person who has the
> right to upload binary packages) who has a merged /usr on his system used
> for building packages, there is a risk of reintroducing the bug through his
> package. Maybe we should somehow, in the short term, modify dpkg to add
> something like "Tainted-By: usr-merge" control field to all binary packages
> produced, if a package is built on a system with merged /usr (detected via
> /bin being a symlink). And a corresponding automatic check that would
> auto-reject binary packages with any Tainted-By control field from being
> uploaded to the Debian archive.

This is actually a great idea! I went ahead and implemented this, see
attached tentative patch which I'm planning on including in dpkg 1.19.3.

Thanks,
Guillem
diff --git i/man/deb-buildinfo.man w/man/deb-buildinfo.man
index 5013aa047..8aa333965 100644
--- i/man/deb-buildinfo.man
+++ w/man/deb-buildinfo.man
@@ -149,6 +149,19 @@ via some pattern match to avoid leaking possibly sensitive information.
 On Debian and derivatives only build paths starting with \fI/build/\fP
 will emit this field.
 .TP
+.BR Build\-Tainted\-By: " \fItaint-reasons...\fP"
+The list of reasons in the form of string tags (alphanumeric and dash),
+that can taint the current build.
+.RS
+.TP
+.B merged-usr-via-symlinks
+The system has a merged /usr via symlinks.
+This will confuse \fBdpkg\-query\fP as it messes with the \fBdpkg\fP
+understanding of the filesystem it manages.
+It can also produce packages with hardcoded paths that will be incompatible
+with non-usr-merged filesystems.
+.RE
+.TP
 .BR Installed\-Build\-Depends: " (required)"
 .TQ
 .I " package-list"
diff --git i/scripts/Dpkg/Control/FieldsCore.pm w/scripts/Dpkg/Control/FieldsCore.pm
index b100366e1..f460433fc 100644
--- i/scripts/Dpkg/Control/FieldsCore.pm
+++ w/scripts/Dpkg/Control/FieldsCore.pm
@@ -176,6 +176,11 @@ our %FIELDS = (
 allowed => CTRL_INFO_PKG,
 separator => FIELD_SEP_SPACE,
 },
+'build-tainted-by' => {
+name => 'Build-Tainted-By',
+allowed => CTRL_FILE_BUILDINFO,
+separator => FIELD_SEP_SPACE,
+},
 'built-for-profiles' => {
 name => 'Built-For-Profiles',
 allowed => ALL_PKG | CTRL_FILE_CHANGES,
@@ -634,7 +639,7 @@ our %FIELD_ORDER = (
 qw(format source binary architecture version binary-only-changes),
 @src_checksums_fields,
 qw(build-origin build-architecture build-kernel-version build-date
-build-path installed-build-depends environment),
+build-path build-tainted-by installed-build-depends environment),
 ],
 CTRL_FILE_CHANGES() => [
 qw(format date source binary binary-only built-for-profiles architecture
diff --git i/scripts/Dpkg/Vendor/Debian.pm w/scripts/Dpkg/Vendor/Debian.pm
index 7d4b6d802..0a1ad0b50 100644
--- i/scripts/Dpkg/Vendor/Debian.pm
+++ w/scripts/Dpkg/Vendor/Debian.pm
@@ -81,6 +81,8 @@ sub run_hook {
 $self->_add_build_flags(@params);
 } elsif ($hook eq 'builtin-system-build-paths') {
 return qw(/build/);
+} elsif ($hook eq 'build-tainted-by') {
+return $self->_build_tainted_by();
 } else {
 return $self->SUPER::run_hook($hook, @params);
 }
@@ -439,6 +441,24 @@ sub _add_build_flags {
 }
 }
 
+sub _build_tainted_by {
+my $self = shift;
+my %tainted;
+
+foreach my $pathname (qw(/bin /sbin /lib /lib32 /lib64 /libx32 /libo32)) {
+next unless -l $pathname;
+
+my $linkname = readlink $pathname;
+if ($linkname eq "usr$pathname") {
+$tainted{'merged-usr-via-symlinks'} = 1;
+last;
+}
+}
+
+my @tainted = sort keys %tainted;
+return @tainted;
+}
+
 =head1 CHANGES
 
 =head2 Version 0.xx
diff --git i/scripts/Dpkg/Vendor/Default.pm w/scripts/Dpkg/Vendor/Default.pm
index 40815efde..0ad0568df 100644
--- i/scripts/Dpkg/Vendor/Default.pm
+++ w/scripts/Dpkg/Vendor/Default.pm
@@ -140,6 +140,14 @@ field will be created if the current directory is "/build/dpkg-1.18.0". If
 the list contains "/", the path will always be recorded. If the list is
 empty, the current path will never be recorded.
 
+=item build-tainted-by ()
+
+The hook is called by dpkg-genbuildinfo to determine if the current system
+has been tainted in some way that could affect the resulting build, which
+will be recorded in the B field (since dpkg 1.19.3). It
+takes no parameters, but returns a (possibly empty) list of tainted reason
+tags (alphanumeric with dashes).
+
 =back
 
 =cut
@@ -172,6 +180,8 @@ sub run_hook {
 	my $flags = shift @params;
 } elsif ($hook eq 'builtin-system-build-paths') {
 return ();
+   

Re: Deployment of buildd machines [was: Re: usrmerge -- plan B?]

[Julien Cristau]

On 11/28/18 2:26 PM, Daniel Reichelt wrote:
> On 11/22/18 1:56 PM, Ian Jackson wrote:
>> (Bear in mind of course that happily our build machines
>> *can* be reverted because they are frequently re-imaged.[…])
> 
> Using code from a debian package? Some script being hand-knitted using
> hot needles? Anyways, I'm interested in how this is done…"this" meaning
> the imaging/deployment part. Not the setup/configuration of a buildd itself.
> 
The machines are not re-imaged.  The build chroots are regenerated, with
something more or less equivalent to sbuild-createchroot(8).

https://salsa.debian.org/dsa-team/mirror/dsa-puppet/tree/master/modules/schroot/

Cheers,
Julien

Deployment of buildd machines [was: Re: usrmerge -- plan B?]

[Daniel Reichelt]

On 11/22/18 1:56 PM, Ian Jackson wrote:
> (Bear in mind of course that happily our build machines
> *can* be reverted because they are frequently re-imaged.[…])

Using code from a debian package? Some script being hand-knitted using
hot needles? Anyways, I'm interested in how this is done…"this" meaning
the imaging/deployment part. Not the setup/configuration of a buildd itself.

Thanks!

Daniel



signature.asc
Description: OpenPGP digital signature

Re: Re: usrmerge -- plan B?

[Ansgar Burchardt]

On Wed, 2018-11-28 at 09:45 +, Holger Levsen wrote:
> On Wed, Nov 28, 2018 at 07:52:08AM +0500, Alexander E. Patrakov
> wrote:
>  As long as there is one Debian Developer (or any other person who has the
> > right to upload binary packages) who has a merged /usr on his system used
> > for building packages, there is a risk of reintroducing the bug through his
> > package. Maybe we should somehow, in the short term, modify dpkg to add
> > something like "Tainted-By: usr-merge" control field to all binary packages
> > produced, if a package is built on a system with merged /usr (detected via
> > /bin being a symlink). 

There are far more annoying problems that are caused by, for example,
building in a really outdated chroot or a chroot for the wrong
distribution and so on.

(Also the test is wrong, /bin can be a symlink without the system
having merged-/usr when the local admin just moved some files around to
a more preferred location...)

> we have .buildinfo files now which document the packages installed
> during build time. If usrmerge is installed it will be there.

usrmerge being installed doesn't tell you if a system has merged-/usr
or not.  Newly installed systems will have merged-/usr, but no usrmerge
(as debootstrap creates the symlinks), or usrmerge could be removed
after the system has been converted (I did that for my systems).

Ansgar

Re: Re: usrmerge -- plan B?

[Andrey Rahmatullin]

On Wed, Nov 28, 2018 at 07:52:08AM +0500, Alexander E. Patrakov wrote:
> As long as there is one Debian Developer (or any other person who has the
> right to upload binary packages) who has a merged /usr on his system used
> for building packages, there is a risk of reintroducing the bug through his
> package. Maybe we should somehow, in the short term, modify dpkg to add
> something like "Tainted-By: usr-merge" control field to all binary packages
> produced, if a package is built on a system with merged /usr (detected via
> /bin being a symlink). And a corresponding automatic check that would
> auto-reject binary packages with any Tainted-By control field from being
> uploaded to the Debian archive.
Tainted-By: built-not-on-buildd please.
At the very least Tainted-By: built-not-in-clean-uptodate-sid-chroot.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: usrmerge -- plan B?

[Holger Levsen]

On Wed, Nov 28, 2018 at 07:52:08AM +0500, Alexander E. Patrakov wrote:
 As long as there is one Debian Developer (or any other person who has the
> right to upload binary packages) who has a merged /usr on his system used
> for building packages, there is a risk of reintroducing the bug through his
> package. Maybe we should somehow, in the short term, modify dpkg to add
> something like "Tainted-By: usr-merge" control field to all binary packages
> produced, if a package is built on a system with merged /usr (detected via
> /bin being a symlink). 

we have .buildinfo files now which document the packages installed
during build time. If usrmerge is installed it will be there.

> And a corresponding automatic check that would
> auto-reject binary packages with any Tainted-By control field from being
> uploaded to the Debian archive.

we lack that, but thats a simple grep.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Marco d'Itri]

On Nov 28, "Alexander E. Patrakov"  wrote:

> Well, the buildd configuration change has been reverted. What worries me now
> is that there is a risk not yet mitigated, coming from personal systems of
> Debian developers, and we should also check porter boxes.
This is not a new problem at all: in my long Debian career I have 
misbuilt more than one upload by not using a chroot. :-)

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Re: usrmerge -- plan B?

[Alexander E. Patrakov]

Russ Allbery wrote:


We had some things break because of a change to buildd configuration that
caught some people by surprise.


Well, the buildd configuration change has been reverted. What worries me 
now is that there is a risk not yet mitigated, coming from personal 
systems of Debian developers, and we should also check porter boxes.


As long as there is one Debian Developer (or any other person who has 
the right to upload binary packages) who has a merged /usr on his system 
used for building packages, there is a risk of reintroducing the bug 
through his package. Maybe we should somehow, in the short term, modify 
dpkg to add something like "Tainted-By: usr-merge" control field to all 
binary packages produced, if a package is built on a system with merged 
/usr (detected via /bin being a symlink). And a corresponding automatic 
check that would auto-reject binary packages with any Tainted-By control 
field from being uploaded to the Debian archive.


P.S. I am not even a Debian Maintainer, so all of the above may be 
rubbish. Would appreciate a reply that confirms or disproves that my 
thoughts make any sense.


--
Alexander E. Patrakov



smime.p7s
Description: S/MIME Cryptographic Signature

Re: usrmerge -- plan B?

[Alf Gaida]

>> I think it would be good to hear from any derivatives in this
>> position.  We should probably ask them more formally than by having a
>> horrible flamewar on -devel ...

With my siduction dev hat on i want to have usrmerge as soon as
possible. Built the last months with usrmerge activated and can't see
any downsides. Ok, we had to modify some things in our iso build process
and some minor things in our scripts, nothing important.

So we will have indeed the situation that new systems will be usrmerged,
old systems not. We will solve this the nice way: We will strongly
recommend to merge and help with problems. If all things go well -
there will be no reports, if things go south, expect some bugs filed
together with patches.

> I don't mean that I'm unsympathetic to downstreams in that situation, or
> that I wouldn't want to help them; only that their plight /should not/ be an
> obstacle to Debian doing the right thing.

We switched to systemd before debian, i guess we will switch before
debian in the usrmerge case - i don't see any problems doing so.

Cheers Alf

Re: usrmerge -- plan B?

[Josh Triplett]

Ian Jackson wrote:
> Unfortunately that means that while a properly planned and executed
> transition to mandatory merged-/usr might well have offered overall
> technical benefits for the Debian ecosystem, this is not now socially
> possible and pressing on is not worth the social costs of being seen
> to bulldoze opposition.

Prematurely declaring victory for an approach you prefer is a more
subtle tactic, but still an unproductive, escalating, and
heat-generating one. There is no such consensus at this time.

Perhaps the correct transition plan is a years-long transition that will
leave Debian unable to avail itself of any of the advantages until done.
Perhaps it isn't. Perhaps there are ways to accomplish that transition
more quickly. Perhaps there are ways we can provide the benefits in the
short term while accomplishing the transition in the long term. But
regardless, please refrain from declaring one of those positions
authoritatively (rather than as your opinion and position) unless that
becomes the consensus.

Nobody here is looking to *bulldoze* opposition. Rather, people are
mostly seeking to *understand* and *address* opposition. There's a big
difference between "this is going to break things" and "here's something
this is going to break, the current solution doesn't address that".  The
latter is a useful contribution; the former without any signs of the
latter is simply FUD. And it's not unreasonable to identify it as such.

For the record, I'm *not* in this mail suggesting a position on whether
the correct solution is to make usrmerge essential. I do think we should
move to moving merged /usr mandatory. Long-term, I *do* think we should
work on making all packages build to install no files to /bin and /lib
and similar, including lintian warnings for installing in / and
eventually autorejects for that. However, I also think it's unreasonable
for that transition to take years; if we can't reasonably accomplish it
in a few months, then 1) we need better mechanisms for distro-wide
changes, and 2) we should consider a solution like usrmerge in the
interim, until we get to the point that no package in the archive
installs to the directories directly in / and we have rejects in place
to prevent any new packages from doing so.

Here's a thought: we already mount /usr at boot time, could we have a
mechanism at boot time that uses some combination of a tmpfs, symlinks,
and mounts to create a /bin that includes individual symlinks to
corresponding binaries in /usr/bin if and *only* if /bin isn't already a
symlink to /usr/bin? That way, /bin on disk doesn't actually contain any
such symlinks, and we don't need an irreversible process like usrmerge.
Keep that up until the transition completes and the last package
transitions, then ship the symlinks in a new base-files and drop the
transitional startup script. Ditto for /sbin and /lib.

We could also have a dh-style script that automatically moves files and
creates a /usr/usrtransition.d/packagename.conf file read by that
boot-time mechanism. Then we know the transition will be handled
correctly, and we won't have packages getting the details wrong and
ending up with RC bugs.

Sound reasonable? I'd be happy to work on that.

- Josh Triplett

Re: usrmerge -- plan B?

[Steve Langasek]

On Tue, Nov 27, 2018 at 12:57:38PM +, Ian Jackson wrote:
> > In the case of unmerged /usr, the only benefits I'm aware of for the more
> > complex case (unmerged /usr) are circular: existing Debian installations
> > have it, so switching to merged /usr is a change;

> I think this is true for Debian itself now that we have bitten the
> bullet of requiring /usr to be mounted along with /, early during
> boot.  (For the record I think that was a good decision.)

> Unmerged /usr could have continuing benefits for Debian derivatives
> who have avoided requiring early mounting of /usr.  IDK whether such
> derivatives exist.  They could do, if they support a narrower range of
> approaches to storage access than Debian proper.  If such derivatives
> exist then Debian adopting merged /usr would be likely to cause
> problems for them, as we would introduce changes in Debian which would
> be bugs in those derivatives.  I don't know how serious a problem that
> would be.

Support for such a configuration is actively bitrotting as we speak. 
Library dependencies of /bin and /sbin are no longer isolated in /lib; udev
will not reliably set up all devices without access to programs under /usr. 
Even if some derivative based on a recent Debian release has managed to keep
usr-on-separate-partition-without-initramfs working for their purposes, this
is not sensibly maintainable over the longer term, and the existence of such
a derivative should carry very little weight with Debian when deciding
whether to merge /usr.

Example: even *without* merged /usr, an entirely sensible course of action
for any maintainer of a Debian library package is to undo all special casing
of /lib vs. /usr/lib in their debian/rules (.so -dev symlinks vs. runtime
libraries, etc) and ship everything in /usr/lib, because the maintainer can
rely on /usr/lib always being available.

> I think it would be good to hear from any derivatives in this
> position.  We should probably ask them more formally than by having a
> horrible flamewar on -devel: ie, in a way that invites the expression
> of concerns and which reassures people that they will not be flamed or
> dismissed.  That would satisfy what I see as our social duty to
> consult our downstreams.  And if we did that and didn't get replies,
> that might give us confidence that such derivatives don't exist.  So
> we could go ahead with a clear conscience.

I don't agree that there's a social duty to consult downstreams that have
made self-evidently poor engineering decisions, before making a change that
will inconvenience them solely as a result of those same poor decisions.

I don't mean that I'm unsympathetic to downstreams in that situation, or
that I wouldn't want to help them; only that their plight /should not/ be an
obstacle to Debian doing the right thing.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Russ Allbery]

Michael Stone  writes:
> On Tue, Nov 27, 2018 at 08:54:43AM +, Simon McVittie wrote:

>> If I was wrong in assuming good faith and you were being argumentative
>> for the sake of being argumentative, please stop: that is not
>> constructive.
>>
>> Either way, please don't call me stupid. That is not *at all*
>> constructive - especially if you want things you say in future to
>> change my opinion on anything! - and contributes to an atmosphere of
>> hostility that drives away Debian contributors.

> I agree. I wish you were just as busy policing memes like "people who
> have issues with mandatory usrmerge are just scared of change".

There is not enough energy in the world to police all of the unnecessarily
confrontational and counter-productive things people have said in this
thread.  :(

I wholeheartedly agree: the argument here is not about fear of change.
It's primarily about a cost/benefit tradeoff and an orderly transition
that is well-understand and won't surprise anyone, and secondarily about
Debian's ongoing severe social problems around having a constructive
technical discussion.

Thank you so much, Simon, for your incredibly useful and productive
messages in this thread.

I am certain there are technical approaches and good compromises here that
will make everyone happy, but we're not going to be able to reach them if
everyone falls into the trap of loudly repeating their position, getting
defensive, and lashing out at each other.

We had some things break because of a change to buildd configuration that
caught some people by surprise.  It's entirely reasonable that the first
reaction to that was "woah, wait, hold on, slow down, what's going on
here?"  A wonderfully productive next step would be for someone with the
time to do so (which sadly I do not right now) to write up a summary of
what the desired end state should be for Debian (maybe with a few options)
and then a more detailed transition plan about how to get from where we
are now to where we want to be.  That will give us something concrete to
debate and to test against the risks that people perceive, and hopefully
will reduce the hardening of positions on all sides.

-- 
Russ Allbery (r...@debian.org)   

Re: usrmerge -- plan B?

[Michael Stone]

On Tue, Nov 27, 2018 at 08:54:43AM +, Simon McVittie wrote:

If I was wrong in assuming good faith and you were being argumentative for
the sake of being argumentative, please stop: that is not constructive.

Either way, please don't call me stupid. That is not *at all*
constructive - especially if you want things you say in future to change
my opinion on anything! - and contributes to an atmosphere of hostility
that drives away Debian contributors.


I agree. I wish you were just as busy policing memes like "people who 
have issues with mandatory usrmerge are just scared of change".

Re: usrmerge -- plan B?

[Michael Stone]

On Tue, Nov 27, 2018 at 09:50:40AM +0100, Philip Hands wrote:

Michael Stone  writes:

On Mon, Nov 26, 2018 at 03:08:09PM +0100, Marco d'Itri wrote:

I disagree both that simple testing (that you could do with a KVM
snapshot as well) would be hard and I disagree that the benefits of
merged-/usr would be minor.


Nobody has thus far pointed out a single benefit to someone merging usr
on an ordinary system.


I'll bite.

I have systems that were installed ages ago, which now have
insufficiently large root partitions.


So that's not an "ordinary" system, it's a system with an insufficiently 
large root partition, right?



BTW whenever anyone says something like "Nobody" or "Never" in these
discussions, they are just asking to be contradicted.  I'm pretty sure
that people have been pointing out advantages of various sorts for some
time, but that you have your filters turned up high enough that none of
them have managed to lodge in your memory.  (I cannot be bothered to
actually come up with a citation, because it's pretty clear that doing
so would make no difference).


No, I really haven't seen anything. I'm sure there are a lot of great 
advantages for particular systems, but if someone has a system that's 
working perfectly well and they're happy with, I haven't seen any 
explanation for why they'd want this change.

Re: usrmerge -- plan B?

[Marc Haber]

On Tue, 27 Nov 2018 09:50:40 +0100, Philip Hands 
wrote:
>I have systems that were installed ages ago, which now have
>insufficiently large root partitions.

The usrmerge moves stuff from / to /usr, replacing /bin with a symlink
to /usr/bin. This is likely to relax space constraints on small root
file systems.

>Installing usrmerge seems like a way to restore sanity to the latter case.

Indeed.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: usrmerge -- plan B?

[Ian Jackson]

Firstly, thanks for your measured and helpful contributions to this
very unfortunate thread.

Simon McVittie writes ("Re: usrmerge -- plan B?"):
> I hope we can agree that unnecessary complexity is technical debt, but
> necessary complexity is necessary: if complexity exists for a reason,
> then the "cost" of the complexity should be compared with the benefit
> of having it, to decide whether the complexity needs to be kept.

I definitely agree with this.

> In the case of unmerged /usr, the only benefits I'm aware of for the more
> complex case (unmerged /usr) are circular: existing Debian installations
> have it, so switching to merged /usr is a change;

I think this is true for Debian itself now that we have bitten the
bullet of requiring /usr to be mounted along with /, early during
boot.  (For the record I think that was a good decision.)

Unmerged /usr could have continuing benefits for Debian derivatives
who have avoided requiring early mounting of /usr.  IDK whether such
derivatives exist.  They could do, if they support a narrower range of
approaches to storage access than Debian proper.  If such derivatives
exist then Debian adopting merged /usr would be likely to cause
problems for them, as we would introduce changes in Debian which would
be bugs in those derivatives.  I don't know how serious a problem that
would be.

I think it would be good to hear from any derivatives in this
position.  We should probably ask them more formally than by having a
horrible flamewar on -devel: ie, in a way that invites the expression
of concerns and which reassures people that they will not be flamed or
dismissed.  That would satisfy what I see as our social duty to
consult our downstreams.  And if we did that and didn't get replies,
that might give us confidence that such derivatives don't exist.  So
we could go ahead with a clear conscience.

Also, there is a social cost of pressing for change.  That could have
been minimised by taking a slow, measured, and consensual approach.
Other substantial changes in Debian have been achieved successfully
with few people getting upset.

Unfortunately the opportunity to do that for mandatory merged-/usr has
been lost.  Now, that transition would necessarily generate
significant ill-will.  Personally I doubt it is worth it.

> Now, it's entirely valid to trade off long-term complexity (unmerged
> /usr) against short-term complexity (applying the /usr merge); one
> possible answer to whether we should eliminate some unnecessary long-term
> complexity is "yes, but not yet" (and the reason for this entire thread is
> that part of the transition happened in the wrong order, with buildd and
> pbuilder chroots becoming merged-/usr sooner than they should have done).

Another possible answer is "yes but we should achieve this in a
different way".  That seems to be Adam's proposal.

Certainly I hope you agree with me that a transition of this magnitude
ought to be properly planned; the plan should be consulted on, with
real attention paid to feedback.

> If I was wrong in assuming good faith and you were being argumentative for
> the sake of being argumentative, please stop: that is not constructive.
> 
> Either way, please don't call me stupid. That is not *at all*
> constructive - especially if you want things you say in future to change
> my opinion on anything! - and contributes to an atmosphere of hostility
> that drives away Debian contributors.

I absolutely agree with this.  Some of the messages from merged-/usr
sceptics have been very bad (and I have said so in private and public
messages).

But also, right now I'm afraid that the most vigorous proponent of
merged-/usr, here on this list, is being extremely dismissive of
feedback.  The approach Marco is taking is generating ill-will amongst
existing contributors (some of whom will then inevitably lash out, bad
as that is), and it is attracting the attention of undesirables.

If you cannot persuade Marco to let you lead the discussion, I think
you should distance yourself from him.

Unfortunately that means that while a properly planned and executed
transition to mandatory merged-/usr might well have offered overall
technical benefits for the Debian ecosystem, this is not now socially
possible and pressing on is not worth the social costs of being seen
to bulldoze opposition.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: usrmerge -- plan B?

[Scott Leggett]

On 2018-11-27.08:54, Simon McVittie wrote:
> On Tue, 27 Nov 2018 at 09:18:08 +0100, Stephan Seitz wrote:
> > But I don’t want to get the /usr-merge forced upon my systems because this
> > minority is obviously too stupid to install the package and migrate their
> > systems on their own.
> 
> That would be a terrible justification for merged /usr, but it is also
> not a justification that anyone is using. In the interests of assuming
> good faith, I'll assume that you have missed the messages that describe
> why applying merged /usr to all Debian systems might be a good idea, and
> attempt to summarize them.
> 
> I hope we can agree that unnecessary complexity is technical debt, but
> necessary complexity is necessary: if complexity exists for a reason,
> then the "cost" of the complexity should be compared with the benefit
> of having it, to decide whether the complexity needs to be kept.
> 
> Merged /usr is a simplification. It takes a few classes of bugs -
> most obviously "a package refers to a command by its absolute path in
> /usr/bin, but really it's in /bin, so that won't work" and its opposite -
> and makes them disappear.
> 
> In the case of unmerged /usr, the only benefits I'm aware of for the more
> complex case (unmerged /usr) are circular: existing Debian installations
> have it, so switching to merged /usr is a change; and if build systems
> have unmerged /usr, then it's a lot more straightforward for packages to
> find the canonical paths of programs (such as the fact that it's /bin/sh
> and /usr/bin/perl, not the other way round), and packages sometimes
> need to know the canonical paths of programs so that the package will
> continue to work on unmerged /usr systems. If all systems were merged
> /usr, then there would be no reason why packages would need to know that
> sh is in /bin but perl is in /usr/bin, because both executables would
> (effectively) be in both places. So *all* systems being merged-/usr would,
> again, be a simplification.

Thanks for your multiple thoughtful and detailed contributions to this
discussion. They are greatly appreciated.

-- 
Regards,
Scott Leggett.


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Simon McVittie]

On Tue, 27 Nov 2018 at 09:18:08 +0100, Stephan Seitz wrote:
> But I don’t want to get the /usr-merge forced upon my systems because this
> minority is obviously too stupid to install the package and migrate their
> systems on their own.

That would be a terrible justification for merged /usr, but it is also
not a justification that anyone is using. In the interests of assuming
good faith, I'll assume that you have missed the messages that describe
why applying merged /usr to all Debian systems might be a good idea, and
attempt to summarize them.

I hope we can agree that unnecessary complexity is technical debt, but
necessary complexity is necessary: if complexity exists for a reason,
then the "cost" of the complexity should be compared with the benefit
of having it, to decide whether the complexity needs to be kept.

Merged /usr is a simplification. It takes a few classes of bugs -
most obviously "a package refers to a command by its absolute path in
/usr/bin, but really it's in /bin, so that won't work" and its opposite -
and makes them disappear.

In the case of unmerged /usr, the only benefits I'm aware of for the more
complex case (unmerged /usr) are circular: existing Debian installations
have it, so switching to merged /usr is a change; and if build systems
have unmerged /usr, then it's a lot more straightforward for packages to
find the canonical paths of programs (such as the fact that it's /bin/sh
and /usr/bin/perl, not the other way round), and packages sometimes
need to know the canonical paths of programs so that the package will
continue to work on unmerged /usr systems. If all systems were merged
/usr, then there would be no reason why packages would need to know that
sh is in /bin but perl is in /usr/bin, because both executables would
(effectively) be in both places. So *all* systems being merged-/usr would,
again, be a simplification.

Now, it's entirely valid to trade off long-term complexity (unmerged
/usr) against short-term complexity (applying the /usr merge); one
possible answer to whether we should eliminate some unnecessary long-term
complexity is "yes, but not yet" (and the reason for this entire thread is
that part of the transition happened in the wrong order, with buildd and
pbuilder chroots becoming merged-/usr sooner than they should have done).

If I was wrong in assuming good faith and you were being argumentative for
the sake of being argumentative, please stop: that is not constructive.

Either way, please don't call me stupid. That is not *at all*
constructive - especially if you want things you say in future to change
my opinion on anything! - and contributes to an atmosphere of hostility
that drives away Debian contributors.

smcv

Re: usrmerge -- plan B?

[Philip Hands]

Michael Stone  writes:

> On Mon, Nov 26, 2018 at 03:08:09PM +0100, Marco d'Itri wrote:
>>I disagree both that simple testing (that you could do with a KVM
>>snapshot as well) would be hard and I disagree that the benefits of
>>merged-/usr would be minor.
>
> Nobody has thus far pointed out a single benefit to someone merging usr 
> on an ordinary system.

I'll bite.

I have systems that were installed ages ago, which now have
insufficiently large root partitions.

Some of them I've gone through the rather delicate procedure of resizing
partitions, some of which contain LVM, to make things fit.  Some of them
I've been able to move / onto LVM (with the side benefit of being able
to combine / and /boot into one, and thus make it big enough to have a
copy of grml on there).  Some are now populated with a lot of symlinks
to other partitions in a painful attempt to keep them small enough to be
useful.

Installing usrmerge seems like a way to restore sanity to the latter case.

Any of the above fixes requires one to do things that can easily result
in a trashed system if you do things in the wrong order, or get the
details wrong, so are not for the faint-hearted.

BTW whenever anyone says something like "Nobody" or "Never" in these
discussions, they are just asking to be contradicted.  I'm pretty sure
that people have been pointing out advantages of various sorts for some
time, but that you have your filters turned up high enough that none of
them have managed to lodge in your memory.  (I cannot be bothered to
actually come up with a citation, because it's pretty clear that doing
so would make no difference).

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Adam D. Barratt]

On 2018-11-27 08:18, Stephan Seitz wrote:

But I don’t want to get the /usr-merge forced upon my systems because
this minority is obviously too stupid to install the package and
migrate their systems on their own.


Please refrain from posting such messages; they are inappropriate and 
contribute nothing helpful to the discussion.


Regards,

Adam

Re: usrmerge -- plan B?

[Stephan Seitz]

On Mo, Nov 26, 2018 at 11:05:02 +0100, W. Martin Borgert wrote:

I personally don't care about usrmerge, but if it is useful to a
relevant minority, we should not reject it.


Who says we should reject the usrmerge package? The minority who wishes 
for it can install it for years.


But I don’t want to get the /usr-merge forced upon my systems because 
this minority is obviously too stupid to install the package and migrate 
their systems on their own.


Shade and sweet water!

Stephan

--
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |


smime.p7s
Description: S/MIME cryptographic signature

Re: usrmerge -- plan B?

[W. Martin Borgert]

Quoting Ben Hutchings :

Many of the rules in Debian policy are there to support a minority of
users, but we try to follow them anyway.


Hear, hear! I was about to write the same thing.

I personally don't care about usrmerge, but if it is useful to a
relevant minority, we should not reject it.

I'm so thankful, that Debian took the hard and rocky road of
MultiArch, which is probably not needed by most users. But it is
incredibly valuable to me. Same thing for support of e.g. armel
and armhf, which makes my life significantly less miserable.

Re: usrmerge -- plan B?

[Ben Hutchings]

On Mon, 2018-11-26 at 15:34 +0100, Stephan Seitz wrote:
> On Mo, Nov 26, 2018 at 03:08:09 +0100, Marco d'Itri wrote:
> > snapshot as well) would be hard and I disagree that the benefits of
> > merged-/usr would be minor.
> 
> There are no benefits of a merged /usr for users who don’t want to export 
> /usr via NFS or want to use clusters/docker images/etc. And this is the 
> majority of Debian users.
>
> So if a minority of users wants a merged /usr for their special cases 
> they can use your package.

Many of the rules in Debian policy are there to support a minority of
users, but we try to follow them anyway.

Yesterday's special case (SMP, hotplug, mobile, 64-bit, ...) often
becomes tomorrow's common case.

Ben.

-- 
Ben Hutchings
Lowery's Law:
If it jams, force it. If it breaks, it needed replacing anyway.




signature.asc
Description: This is a digitally signed message part

Re: usrmerge -- plan B?

[Simon McVittie]

On Mon, 26 Nov 2018 at 15:00:41 +, Alastair McKinstry wrote:
> Moving config from /etc to below /usr becomes useful for containers, and
> hence clusters.

Both that and merged /usr are particularly useful for containers and
close-to-stateless embedded systems, but they are orthogonal. Please don't
conflate them. I am broadly in favour of both, but I don't want people
to be advocating one for reasons that actually only support the other.

Merged /usr (the rest of this thread) is about taking the static parts
of the root directory (/bin, /sbin, /lib*, /usr) and putting them all in
/usr, so that the only files that are mandatory in the root directory
are /etc, standard compatibility symlinks to keep paths like /bin/sh
working (which can be created during boot by systemd-tmpfiles or similar
if the root filesystem is a tmpfs or otherwise initially empty), and
the standard top-level directories (all of which, except /etc, can be
a separate filesystem if desired).

Moving system integration hooks and configuration defaults from /etc to
/usr is about taking the parts of the root directory for which sysadmin
edits are a supported action, and dividing them into the parts the
sysadmin could conceivably override in future but probably won't[0]
(in /usr or /lib[1]), and the parts the sysadmin has actually overridden
(files in /usr or /lib[1] overridden or "masked" by files in /etc).

If you combine the two, the parts of the root filesystem that cannot be
separated into a /usr or /var filesystem become very small.

smcv

[0] In principle every file in /etc needs to be editable or overridable
by *someone*, some sysadmins edit more than others, and some
files need to be edited in all or nearly all systems; but in
historical/current Debian practice the typical number of edits per
system is a lot smaller than the number of files present in /etc.

[1] If you have unmerged /usr then a minority of these overridable
files are in /lib; the only examples I can think of right now are
/lib/udev/rules.d/* and /lib/systemd/system/*. If you also have
merged /usr then that's the same thing as /usr/lib anyway.

Re: usrmerge -- plan B?

[Marc Haber]

On Mon, 26 Nov 2018 17:12:34 +0100, Marco d'Itri  wrote:
>In other words, you are concerned about failure modes which have never 
>happened and that nobody has been able to explain how they may happen in 
>the future.

s/you/customers looking for an excuse to ditch Debian/

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: usrmerge -- plan B?

[Holger Levsen]

On Mon, Nov 26, 2018 at 05:12:34PM +0100, Marco d'Itri wrote:
> So far the worst case of "usrmerge failure" can be fixed by 
> deleting/moving some files (that were installed by you) and then running 
> the program again. There has never been any case which required 
> reinstalling/restoring a system on which the conversion program failed 
> and I cannot think of why this could ever be needed.
> In other words, you are concerned about failure modes which have never 
> happened and that nobody has been able to explain how they may happen in 
> the future.

that. (should be read by everyone concerned.)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Marco d'Itri]

On Nov 26, Marc Haber  wrote:

> I would be willing to do this on production systems that can easily be
> snapshotted and reverted in no time during an upgrade change, such as
> VMs. Unless forced to, I'd rather not touch systems running on bare
> metal and would need a restore-from-backup should usrmerge fail.
So far the worst case of "usrmerge failure" can be fixed by 
deleting/moving some files (that were installed by you) and then running 
the program again. There has never been any case which required 
reinstalling/restoring a system on which the conversion program failed 
and I cannot think of why this could ever be needed.
In other words, you are concerned about failure modes which have never 
happened and that nobody has been able to explain how they may happen in 
the future.

> I'd really like Debian to take special care to not break my systems
> and to not expose them to unnecessary risk. Customers have been moving
> away from Debian towards Enterprise Linux for cheesy reasons (first we
> didn't update fast enough, now we're updating too fast, and there is
> no support for Debian, Debian uses systemd etc bla foo), the forced
> usrmerge would add to those.
Either they want an enterprise distribution or they want a distribution 
without systemd and merged-/usr, but they cannot have both...
Since merged-/usr makes Debian more similar to enterprise distributions 
it would be a good thing for this kind of customers.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Adam Borowski]

On Mon, Nov 26, 2018 at 03:00:41PM +, Alastair McKinstry wrote:
> 
> On 26/11/2018 14:44, Adam Borowski wrote:
> > On Mon, Nov 26, 2018 at 09:29:50AM -0500, Michael Stone wrote:
> > > On Mon, Nov 26, 2018 at 03:08:09PM +0100, Marco d'Itri wrote:
> > > > I disagree both that simple testing (that you could do with a KVM
> > > > snapshot as well) would be hard and I disagree that the benefits of
> > > > merged-/usr would be minor.
> > > Nobody has thus far pointed out a single benefit to someone merging usr on
> > > an ordinary system.
> > Nor for clusters, for that matter.  I don't get how having the /usr part of
> > the filesystem (no /var, /lib, /etc, etc) would help --
> 
> Moving config from /etc to below /usr becomes useful for containers, and
> hence clusters.

... well, how?

> containers become important for clusters - we are now using Singularity in
> our HPC clusters. Singularity is a development of Docker that allows for
> non-root container execution; we can build containers on our laptops, etc
> (requiring root), and copy them to the cluster, where they will run, even
> connecting with mpiexec / slurm ,etc

You still need all the rest of the files.  There might be cases where making
them volatile and copying from a base state to tmpfs on every boot could be
workable, but that's a pretty odd setup -- which is handled just as well by
more generic solutions.

So why won't you use any of the many ways you pruned from my post you
responded to?

> Now, we can build a container on a laptop, with Debian inside, and run it on
> a 1000-node cluster.  Its realistic to see million-core jobs on Debian in
> the future.

So you propose to make that cluster have every node be more resource
contrained than the weakest Debian-capable SoC / phone available in the
wild?  The vast majority of non-GUI installs I've seen have /usr of below
1GB, with available storage massively larger than that.  So any realistic
HPC cluster will have many orders of magnitude more space, per node --
making even deduplication a waste of human time.  If there's indeed no
storage per-node, please don't insult our intelligence by suggesting having
the central server keep that /usr shared would require a non-laughable
effort.  And if it does... well, please read the list of solutions I
mentioned.

I quite recently worked with multi-container hosting -- there's no problem
with deduplication using already existing tools.  And in hindsight, it was a
waste of my time -- it's customers' data that takes space, not /usr.

> So, while supporting containers may support a minority of users, I suspect
> some will be big users, and as library and app complexity grows, its an
> important Debian use-case.

Yes, it's an important use-case.  And one already solved without merging
/usr.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢰⠒⠀⣿⡁ 
⢿⡄⠘⠷⠚⠋⠀ ... what's the frequency of that 5V DC?
⠈⠳⣄

Re: usrmerge -- plan B?

[Marc Haber]

On Mon, 26 Nov 2018 09:26:44 -0500, Michael Stone 
wrote:
>On Mon, Nov 26, 2018 at 09:24:40AM +0100, Didier 'OdyX' Raboud wrote:
>>usrmerge is in the archive for 3+ years now. What seems to be needed now is
>>for a lot of us to actually _try_ it, find and report bugs, and get this
>>through.
>
>There's no way I'm running it on a production system. I would be willing 
>to run a report mode.

I would be willing to do this on production systems that can easily be
snapshotted and reverted in no time during an upgrade change, such as
VMs. Unless forced to, I'd rather not touch systems running on bare
metal and would need a restore-from-backup should usrmerge fail.

I'd really like Debian to take special care to not break my systems
and to not expose them to unnecessary risk. Customers have been moving
away from Debian towards Enterprise Linux for cheesy reasons (first we
didn't update fast enough, now we're updating too fast, and there is
no support for Debian, Debian uses systemd etc bla foo), the forced
usrmerge would add to those.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: usrmerge -- plan B?

[Alastair McKinstry]

On 26/11/2018 14:44, Adam Borowski wrote:

On Mon, Nov 26, 2018 at 09:29:50AM -0500, Michael Stone wrote:

On Mon, Nov 26, 2018 at 03:08:09PM +0100, Marco d'Itri wrote:

I disagree both that simple testing (that you could do with a KVM
snapshot as well) would be hard and I disagree that the benefits of
merged-/usr would be minor.

Nobody has thus far pointed out a single benefit to someone merging usr on
an ordinary system.

Nor for clusters, for that matter.  I don't get how having the /usr part of
the filesystem (no /var, /lib, /etc, etc) would help --


Moving config from /etc to below /usr becomes useful for containers, and 
hence clusters.


(I suspect this might be the most useful bit of a merged-/usr concept in 
the long run).


containers become important for clusters - we are now using Singularity 
in our HPC clusters. Singularity is a development of Docker that allows 
for non-root container execution; we can build containers on our 
laptops, etc (requiring root), and copy them to the cluster, where they 
will run, even connecting with mpiexec / slurm ,etc


This is important because we've never used Debian on our high-end 
clusters before - they all have parallel filesystems for which we need 
special kernel modules (for DDN or Panasas, or Lustre), supported by the 
vendor. So we needed to use the vendors version of SuSE /Redhat/Centos 
,etc. And building complex applications was then a pain in  the neck.


Now, we can build a container on a laptop, with Debian inside, and run 
it on a 1000-node cluster.  Its realistic to see million-core jobs on 
Debian in the future.


So, while supporting containers may support a minority of users, I 
suspect some will be big users, and as library and app complexity grows, 
its an important Debian use-case.


Alastair

--
Alastair McKinstry, , , 
https://diaspora.sceal.ie/u/amckinstry
Commander Vimes didn’t like the phrase “The innocent have nothing to fear,”
 believing the innocent had everything to fear, mostly from the guilty but in 
the longer term
 even more from those who say things like “The innocent have nothing to fear.”
 - T. Pratchett, Snuff

Re: usrmerge -- plan B?

[Ian Jackson]

Russ Allbery writes ("Re: usrmerge -- plan B?"):
> Ian Jackson  writes:
> > We can then have this discussion again early in the bullseye release
> > cycle.  If we must.
> 
> That idea makes me wince.  This will just result in the same thing
> happening again.  Let's have the discussion *now*, when the problems are
> fresh in our mind, and then defer *action* to early in the bullseye
> release cycle (which I suspect is likely to happen anyway given how long
> it usually takes us to sort through questions of large migrations like
> this).

If we *are* to have this discussion now, we still need a transition
plan.

Adam has presented a sketch of one; it would have to be fleshed out
further, addressing issues that have been raised (eg, dependency
handling) so that we could do a more detailed technical review, before
we could adopt it.

The `install usrmerge package everywhere' transition plan is still
missing.  The next thing its proponents ought to do is to write a
proper transition plan, covering all of the details, so that we can
review it.  The transition plan should cover all aspects, including
for example the implications for dpkg (see Guillem's mail); whether
the transition might be reverted on an individual system; go / no-go
decision points; etc.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: usrmerge -- plan B?

[Adam Borowski]

On Mon, Nov 26, 2018 at 09:29:50AM -0500, Michael Stone wrote:
> On Mon, Nov 26, 2018 at 03:08:09PM +0100, Marco d'Itri wrote:
> > I disagree both that simple testing (that you could do with a KVM
> > snapshot as well) would be hard and I disagree that the benefits of
> > merged-/usr would be minor.
> 
> Nobody has thus far pointed out a single benefit to someone merging usr on
> an ordinary system.

Nor for clusters, for that matter.  I don't get how having the /usr part of
the filesystem (no /var, /lib, /etc, etc) would help -- and there's so many
tools that don't care about individual directories that I'd recommend you
to instead use one of:
 * btrfs
 * overlayfs
 * zfs
 * lvm
 * dm_*
 * even a bunch of mounts
rather than your scheme that requires the rest of the world to adapt to it.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄ agriculture, towns then cities. -- whitroth on /.

Re: usrmerge -- plan B?

[Stephan Seitz]

On Mo, Nov 26, 2018 at 03:08:09 +0100, Marco d'Itri wrote:

snapshot as well) would be hard and I disagree that the benefits of
merged-/usr would be minor.


There are no benefits of a merged /usr for users who don’t want to export 
/usr via NFS or want to use clusters/docker images/etc. And this is the 
majority of Debian users.


So if a minority of users wants a merged /usr for their special cases 
they can use your package.


Shade and sweet water!

Stephan

--
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Michael Stone]

On Mon, Nov 26, 2018 at 03:08:09PM +0100, Marco d'Itri wrote:

I disagree both that simple testing (that you could do with a KVM
snapshot as well) would be hard and I disagree that the benefits of
merged-/usr would be minor.


Nobody has thus far pointed out a single benefit to someone merging usr 
on an ordinary system.

Re: usrmerge -- plan B?

[Michael Stone]

On Mon, Nov 26, 2018 at 09:24:40AM +0100, Didier 'OdyX' Raboud wrote:

usrmerge is in the archive for 3+ years now. What seems to be needed now is
for a lot of us to actually _try_ it, find and report bugs, and get this
through.


There's no way I'm running it on a production system. I would be willing 
to run a report mode.

Re: usrmerge -- plan B?

[Marco d'Itri]

On Nov 26, Ian Jackson  wrote:

> I could do it.  But, frankly, this is quite a lot of work.  I think
> the work (throughout the Debian ecosystem) to test this properly far
> outweighs the minor benefits.
I disagree both that simple testing (that you could do with a KVM 
snapshot as well) would be hard and I disagree that the benefits of 
merged-/usr would be minor.

> And if I do test it and find a lot of bugs then am I going to be
> expected to report them all in detail ?  That is also a ton of work.
> Am I then going to be expected to retest when the bugs are allegedly
> fixed ?  I think this is just outsourcing the pain of bad design
> choices, frankly.
I have not seen any better design proposals so far.

> If we must get to merged-usr on all systems eventually, Adam's
> proposed transition plan is much better.
His plan would not work any better, it would break compatibility with 
every third party software which uses /bin/, /sbin and /lib (and its own 
subdirectories), it would require a huge amount of work by everybody to 
modify most packages (assuming that it were feasible), and meanwhile 
take forever.
His plan would not solve any of your concerns because even if it were 
executed perfectly and quickly then the conversion program would still 
be in the same exact situation as it is now: either everything in /bin/, 
/sbin and /lib (and its own subdirectories) was created by the packaging 
system, and then we already know that it can be converted automatically, 
or it was not, and then we know that there are a few cases when the 
local administrator has to decide what to do about things that were 
installed by himself in the past in the wrong place.
So, how can you seriously propose it as an option?

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Marco d'Itri]

On Nov 26, Bjørn Mork  wrote:

> "Migration is not (easily) reversible" was reported as important in
> 2016: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848626
This is not a bug: nothing is misbehaving, so classifying this as 
wishlist is correct.
I even doubt that it would be technically feasible to implement 
conversion rollback, and since the main concern here was doing the 
conversion by mistake I solved it with the debconf question.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Ian Jackson]

Didier 'OdyX' Raboud writes ("Re: usrmerge -- plan B?"):
> Of course you do have backups.  For the sake of the argument, would you 
> consider trying an install of usrmerge on a restored backup VM somewhere?

I could do it.  But, frankly, this is quite a lot of work.  I think
the work (throughout the Debian ecosystem) to test this properly far
outweighs the minor benefits.

And if I do test it and find a lot of bugs then am I going to be
expected to report them all in detail ?  That is also a ton of work.
Am I then going to be expected to retest when the bugs are allegedly
fixed ?  I think this is just outsourcing the pain of bad design
choices, frankly.

If we must get to merged-usr on all systems eventually, Adam's
proposed transition plan is much better.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: usrmerge -- plan B?

[Bjørn Mork]

Marco d'Itri  writes:
> On Nov 26, Didier 'OdyX' Raboud  wrote:
>
>> Sorry to be blunt about this, but have you reported these? Sniping at (any) 
>
> No, they have not. There is a lot of handwaving in this thread but very 
> few results of actual tests.

"Migration is not (easily) reversible" was reported as important in
2016: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848626

It was immediately demoted to wishlist with this explanation:

"If and when the package will be automatically installed for some reason
 then a warning will be appropriate.
 usrmerge has extra priority, is not a dependency of anything and I am 
 not aware of anybody ever installing it by mistake."

The bug was later closed when a deconf question was added.

The fact that you redefined the topic of the bug does not mean that it
was solved. Your whining about missing bug reports on this subject is
simply arrogant.  Or ignorant.  Or both.


Bjørn

Re: usrmerge -- plan B?

[Marco d'Itri]

On Nov 26, Didier 'OdyX' Raboud  wrote:

> Sorry to be blunt about this, but have you reported these? Sniping at (any) 
No, they have not. There is a lot of handwaving in this thread but very 
few results of actual tests.

After creating again unmerged chroots for the buildds the only bugs 
left, archive-wide, are #860523 (an abandoned package with no reverse 
dependencies, there is even a patch in the BTS) and #913883 (iptables 
mixing manually-created symlinks and diversions).
A few bugs in other packages have been reported and fixed over the years:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=usrmerge;users=m...@linux.it

So we have:
- very few bugs exposed in other packages
- a track record of them being fixed

These are facts. People being worried if it would work or not on their 
own system are just showing emotions, which are usually not relevant in 
engineering decisions.

> Don't forget that a specificity of our bug report system is that the only 
> measure of "it worked without issues" that we have is popcon; we only get a 
> measure of how much things fail, not how good they work:
> 
>   https://qa.debian.org/popcon.php?package=usrmerge
This is only partially relevant because after the conversion there is no 
reason to keep the package around and it can just be removed.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Stephan Seitz]

On Mo, Nov 26, 2018 at 09:24:40 +0100, Didier 'OdyX' Raboud wrote:
usrmerge is in the archive for 3+ years now. What seems to be needed now 
is for a lot of us to actually _try_ it, find and report bugs, and get 
this through.


Why, if it was intended as an optional package for people who want 
a merged /usr? If I don’t need or want a merged /usr I won’t install and 
test the package.


The problem is that people are now talking to forcing the package on 
everyone.


Shade and sweet water!

Stephan

--
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |


smime.p7s
Description: S/MIME cryptographic signature

Re: usrmerge -- plan B?

[Didier 'OdyX' Raboud]

Le jeudi, 22 novembre 2018, 14.56:10 h CET Ian Jackson a écrit :
> There is tradeoff here between risk of breakage, and reduction of
> future work (as most clearly explained by Russ).  The argument that is
> being made is that the risk is low because of a lack of reports of
> breakage.
> 
> Others have observed that systems most likely to experience trouble
> will not have been upgraded.  For example, chiark was first installed
> with Debian 0.93R5 in 1993.  Obviously I haven't usrmerged it.  No-one
> sensible in my position would do so.

Of course you do have backups.  For the sake of the argument, would you 
consider trying an install of usrmerge on a restored backup VM somewhere?

Maybe I'm overlooking something, but it seems that installing usrmerge on a 
production system is going to be a _much_ less demanding task proceeding with 
a stable upgrade.

Cheers,
OdyX

Re: usrmerge -- plan B?

[Didier 'OdyX' Raboud]

Le jeudi, 22 novembre 2018, 00.17:54 h CET Michael Stone a écrit :
> Then this needs to be a very explicit (and much better advertised)
> decision, and it needs a much, much better implementation.

You keep referring to usrmerge as buggy:
> The current usrmerge package has no test mode, will bail with a partially-
> converted system if it runs into problems, and has no way to revert the
> process.

Sorry to be blunt about this, but have you reported these? Sniping at (any) 
package without making the problems you see visible to others (through bugs) 
is not really helpful.

> Pulling in usrmerge during an upgrade isn't going to cut it--we'd need some
> kind of pre-upgrade check that tells people what they need to fix before we
> break it. Designing this in a hurry less than two months before we start
> freezing seems incredibly ambitious.

usrmerge is in the archive for 3+ years now. What seems to be needed now is 
for a lot of us to actually _try_ it, find and report bugs, and get this 
through.

Don't forget that a specificity of our bug report system is that the only 
measure of "it worked without issues" that we have is popcon; we only get a 
measure of how much things fail, not how good they work:

https://qa.debian.org/popcon.php?package=usrmerge

(Funnily enough, it seems to have had a recent spike…)

Cheers,
OdyX

Re: usrmerge -- plan B?

[Theodore Y. Ts'o]

On Thu, Nov 22, 2018 at 11:27:46AM -0800, Russ Allbery wrote:
> > My position as a usrmerge sceptic, of letting them get on with doing
> > their thing, now seems to have been unwise.  The idea that it would be
> > optional mutated, without proper discussion, and without a transition
> > plan, into it being the default for new installs.
> 
> I agree with wanting more discussion and more of a plan before making it
> the default for new installs, and I'm skeptical this is a good idea for
> buster.

Given that one of my packages had a bug filed against it that was
caused by usrmerge, and while I *can* fix it, I am also getting a bit
skeptical about trying to rush the usrmerge for buster --- and
**definitely** if it is a mandatory merge.  I'd be OK usrmerge being
in buster so long as it has a big huge, fat warning of the form, "if
it breaks your system you get to keep both pieces".  So if things
break in usrmerge system on Buster, they would ***not*** be RC, at
least not for the next 3 months.

Post-buster, I agree doing a mandatory usrmerge transition makes
sense, and this should be done very early in the development cycle,
and not at the very end of the development cycle.

> That idea makes me wince.  This will just result in the same thing
> happening again.  Let's have the discussion *now*, when the problems are
> fresh in our mind, and then defer *action* to early in the bullseye
> release cycle (which I suspect is likely to happen anyway given how long
> it usually takes us to sort through questions of large migrations like
> this).

Agreed, completely.  If we leave usrmerge in buster as a "use at your
own risk", then the people who are the most passionate can try it on
their production systems.  That will hopefully give us more feedback,
which will significantely reduce risk for bullseye.

- Ted

Re: usrmerge -- plan B?

[Marco d'Itri]

On Nov 24, "Gabriel F. T. Gomes"  wrote:

> Has this option been given enough attention?  It sounds appealing in
Yes. I would love to implement a --dry-run mode, but I still need to 
figure out how much complex it would be and if it could actually cover 
all cases.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: usrmerge -- plan B?

[Gabriel F. T. Gomes]

On 23 Nov 2018, Michael Stone wrote:

>On Fri, Nov 23, 2018 at 03:14:44PM +0100, Matthias Klumpp wrote:
>>For these cases though maybe the usrmerge script could ask the admin
>>on what to do to handle these particular binaries, instead of
>>failing.  
>
>Maybe, as I suggested upthread, there could be a preview mode in which 
>the admin could be told what would happen, alerted if the process is 
>going to fail, and given some guidance as to what to do--*before* 
>blindly pulling the trigger. We could even ask people to report on 
>whether the preview mode predicts issues on their systems in order to 
>gather data--which I think is far more likely than people letting us 
>know whether the process broke (past tense) their systems (for the
>sake of science).

Has this option been given enough attention?  It sounds appealing in
two ways: 1. it would work as an incentive for people who want to
install usrmerge (be it because they are curious about the results or
because they wish to help), but are not in a mood to reinstall their
systems *if* the merge causes trouble; 2. it would serve as a place to
collect the feedback and summarize the potential problems.

Maybe I'm just asking because I'm one of those not in a mood to
reinstall, but it does sound appealing to me.

Re: usrmerge -- plan B?

[Alexander E. Patrakov]

On 11/24/18 8:51 PM, Alexander E. Patrakov wrote:

Marco d'Itri wrote:


On Nov 21, Michael Stone  wrote:


How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage 
whatsoever

Actually I have quite a lot personally, with exactly zero problems.
On some of them I also enjoy advantages of merged-/usr, like having 
multiple containers share the same /usr.


I agree with the argument in this thread that usrmerge should be either 
Essential or not in Buster at all. I have no Debian systems in 
production, but the company that I have a contract with has some Ubuntu 
systems, and we are considering (as in: there is a successful PoC) the 
use of OSTree for deployment of some production systems. OSTree folks 
recommend merged /usr, so I do install usrmerge in the LXC container 
that is being used as a "golden" tree for deployment. And it works quite 
well for us with just a few extra moves and symlinks, and extlinux as 
the boot loader (to work around removal of GRUB2-related files from the 
Debian package).


So: my opinion is that I would prefer usrmerge in Essential in Buster, 
for the benefit of people who use OSTree for deployment, and to reduce 
divergence between Debian and Ubuntu plans.


Based on reading the other messages in this thread, I changed my 
opinion. The new opinion is that any of the following outcomes is OK, 
with no specific preference:


1. mandatory usrmerge in Buster
2. usrmerge available from somewhere for installation in Buster, but not 
installed on buildds, and dpkg-buildpackage should refuse to build 
anything if /usr is merged and the user does not say "please".


--
Alexander E. Patrakov

Re: Re: usrmerge -- plan B?

[Alexander E. Patrakov]

Russ Allbery wrote:


The reason why I personally jumped into this thread is that I don't think
the initial fix proposed for R was good engineering.  We should not be
doing that sort of fragile machinations in Autoconf configuration.  It
will end in tears.  We will miss some other binary later, or Autoconf will
change flags or how it probes, and we won't catch the breakage.


We could catch some of it automatically by creating /dontuse/bin and 
/dontuse/usr/bin as symlinks to /bin and /usr/bin on all buildds (and 
the sbin equivalents, too), adding that first to the PATH, and 
auto-rejecting any package that has these strings in it. I understand 
that there will be lots of FTBFS from doing that, so it may or may not 
be a good idea, especially because this covers only "we won't catch the 
breakage" point, but not "we should not be doing that".


--
Alexander E. Patrakov

Re: Re: usrmerge -- plan B?

[Alexander E. Patrakov]

Marco d'Itri wrote:


On Nov 21, Michael Stone  wrote:


How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage whatsoever

Actually I have quite a lot personally, with exactly zero problems.
On some of them I also enjoy advantages of merged-/usr, like having 
multiple containers share the same /usr.


I agree with the argument in this thread that usrmerge should be either 
Essential or not in Buster at all. I have no Debian systems in 
production, but the company that I have a contract with has some Ubuntu 
systems, and we are considering (as in: there is a successful PoC) the 
use of OSTree for deployment of some production systems. OSTree folks 
recommend merged /usr, so I do install usrmerge in the LXC container 
that is being used as a "golden" tree for deployment. And it works quite 
well for us with just a few extra moves and symlinks, and extlinux as 
the boot loader (to work around removal of GRUB2-related files from the 
Debian package).


So: my opinion is that I would prefer usrmerge in Essential in Buster, 
for the benefit of people who use OSTree for deployment, and to reduce 
divergence between Debian and Ubuntu plans.


--
Alexander E. Patrakov

Re: usrmerge -- plan B?

[Daniele Nicolodi]

Hello Adam,

On 23/11/2018 08:19, Adam Borowski wrote:
> On Fri, Nov 23, 2018 at 03:14:44PM +0100, Matthias Klumpp wrote:
>> Am Fr., 23. Nov. 2018 um 14:47 Uhr schrieb Stephan Seitz
>> :
>>>
>>> On Fr, Nov 23, 2018 at 02:04:05 +0100, Matthias Klumpp wrote:
 If there are actual issues encountered, we can always revert a change
>>>
>>> And how do you revert this change? As far as I have understand you can’t
>>> remove the usrmerge package and have your system in the old state again.
>>
>> You don't - it's unstable, for testing these things. If it breaks, you
>> file a bug and fix the system.
> 
> Fix -- not completely restore from backups.  This is not an option with
> usrmerge; we have severity:critical for bugs with such a fallout.

I've seen this repeated multiple times in this thread, where does your
assessment that the usrmerge package can leave the system in a non
working state comes from?

In my experience, usrmerge stops the merge if it encounters some
unforeseen difficulty and hints to how to resolve the problem. Once the
problem has been resolved, usrmerge can be run again and it will pick up
from where it left.

Is your experience different? Can you be more precise on the kind of
errors or system breakage you encountered?

Thank you.

Best,
Daniele

Re: usrmerge -- plan B?

[Guillem Jover]

Hi!

On Wed, 2018-11-21 at 10:23:46 +0100, Adam Borowski wrote:
> with less confidence:
>  • making usrmerge Essential (large amount of effort, known issues)

Oh dear, no, please!

First off, as I've said in the past, I have no problem whatsoever with
the concept, I think it brings both advantages (cleans up the FHS,
makes some things easier, etc) and disadvantages (makes Debian perhaps
a worse development environment for upstreams that want to target other
systems, easier to miss pathnames that should not be under /usr leaking
over, or a proper conversion making it not possible to keep unmerged
/usr for those that would prefer that, for whatever reason, etc).

As a proof-of-concept to make it possible to test the thing, to unearth
bugs, as a quick way to deploy on throwaway containers, or even as a
pragmatic way to switch ones system if there's a need for that no
matter what, right now, the usrmerge package is great.

But I do have a very big problem with us even considering using it as
a way to deploy this change as part of our standard installation or
upgrade. This is a hack, it's subverting the dpkg view of the world,
and while this is intended to be supported somehow from dpkg PoV, I
consider that to be on best-effort basis. For example I still have
not merged the branch [Q] to make dpkg-query cope with the symlink
stuff, as I was waiting for some feedback, and it still has some
conceptual problems. Making usrmerge Essential is the "we cannot be
bothered to do a proper transition and move things to their proper
place" kind of plan. :(

  [Q] 


Please, if we decide we want to do merged /usr, let's do it properly.
I'm still quite appalled that debootstrap has been change *again*
(after it got reverted for stretch) to default to use this method for
new installations…

Thanks,
Guillem

Re: usrmerge -- plan B?

[Adam Borowski]

On Fri, Nov 23, 2018 at 03:14:44PM +0100, Matthias Klumpp wrote:
> Am Fr., 23. Nov. 2018 um 14:47 Uhr schrieb Stephan Seitz
> :
> >
> > On Fr, Nov 23, 2018 at 02:04:05 +0100, Matthias Klumpp wrote:
> > >If there are actual issues encountered, we can always revert a change
> >
> > And how do you revert this change? As far as I have understand you can’t
> > remove the usrmerge package and have your system in the old state again.
> 
> You don't - it's unstable, for testing these things. If it breaks, you
> file a bug and fix the system.

Fix -- not completely restore from backups.  This is not an option with
usrmerge; we have severity:critical for bugs with such a fallout.

And yeah, bugs of this severity prevent a package from being a part of the
next stable release.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ I've read an article about how lively happy music boosts
⣾⠁⢰⠒⠀⣿⡁ productivity.  You can read it, too, you just need the
⢿⡄⠘⠷⠚⠋⠀ right music while doing so.  I recommend Skepticism
⠈⠳⣄ (funeral doom metal).

Re: usrmerge -- plan B?

[Stephan Seitz]

On Fr, Nov 23, 2018 at 03:14:44 +0100, Matthias Klumpp wrote:

There are always unforseen issues to be expected when upgrading. And


Of course, and since a dist-upgrade will bring newer software you may 
already have to fix configuration files.



at the moment, the only issues that are known when installing the
usrmerge package is when there are different binaries with the same
name in /bin and /usr/bin (or /sbin), and I really don't think that
this is actually a likely scenario.


I don’t know but I’ve seen enough strange installations.

It would be good if the usrmerge package would do a dry-run as part of 
the installation. If there are duplicate files the list will be printed 
(or mailed) and the installation will fail without breaking the whole 
upgrade process. The the admin can fix the problem later.


Shade and sweet water!

Stephan

--
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |


smime.p7s
Description: S/MIME cryptographic signature

Re: usrmerge -- plan B?

[Michael Stone]

On Fri, Nov 23, 2018 at 03:14:44PM +0100, Matthias Klumpp wrote:

For these cases though maybe the usrmerge script could ask the admin
on what to do to handle these particular binaries, instead of failing.


Maybe, as I suggested upthread, there could be a preview mode in which 
the admin could be told what would happen, alerted if the process is 
going to fail, and given some guidance as to what to do--*before* 
blindly pulling the trigger. We could even ask people to report on 
whether the preview mode predicts issues on their systems in order to 
gather data--which I think is far more likely than people letting us 
know whether the process broke (past tense) their systems (for the sake 
of science).

Re: usrmerge -- plan B?

[Stephan Seitz]

On Fr, Nov 23, 2018 at 03:02:00 +0100, Hans wrote:

Am Freitag, 23. November 2018, 14:47:28 CET schrieb Stephan Seitz:
And how do you revert this change? As far as I have understand you 
can’t remove the usrmerge package and have your system in the old 
state again.

Making an image of the whole hard drive is always a good idea.


While this is easy done if the system is a VMware VM I know my fellow 
admins well enough that they will simply cancel the update because they 
are not interested in debugging the upgrade process for a single package 
they don’t need.



So, that is, where "testing" is standing for. Testing before release


You know I don’t have doubts that you won’t have problems (or very few) 
if you do a new testing installation. I don’t like Marco’s attitude but 
he is doing a good job. If he has tested his package as he said this 
isn’t the problem.


But Debian/testing doesn’t mean testing on real servers which have 
„survived” different Debian releases, where different admins copied files 
from one location to another, or with some third-party software.


If the update of 2% of our systems won’t work because of failing 
usrmerge I would be asked if Debian is the right distribution.

Good point. However, this should be told to Microsoft admins or their
deciders.


;-) But while there are no alternative Windows distributions Debian isn’t 
the only one and can be changed.


Shade and sweet water!

Stephan

--
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |


smime.p7s
Description: S/MIME cryptographic signature

Re: usrmerge -- plan B?

[Matthias Klumpp]

Am Fr., 23. Nov. 2018 um 14:47 Uhr schrieb Stephan Seitz
:
>
> On Fr, Nov 23, 2018 at 02:04:05 +0100, Matthias Klumpp wrote:
> >If there are actual issues encountered, we can always revert a change
>
> And how do you revert this change? As far as I have understand you can’t
> remove the usrmerge package and have your system in the old state again.

You don't - it's unstable, for testing these things. If it breaks, you
file a bug and fix the system.

> As others in the thread have mentioned they don’t see the risk with new
> installations but with old systems with different admins and third-party
> software.
>
> >anyway. During distribution upgrades there is a lot that can be wrong
> >and a lot of stuff the administrator needs to care about (config file
>
> Right, and it means he has enough to do and doesn’t want to debug the
> usrmerge. I don’t want to have a usrmerge at a dist-upgrade. You don’t
> really know the sequence of the package updates. I think the risk is too
> big to have a partial upgraded system.

For the sequence of installations, the APT maintainers can shed light
on what the proper plan could be to ensure the usrmerge update happens
at the right time.

> >with information to the system administrator on what to do in case of
> >an error, and works for 98% of all users anyway, I see no reason not
>
> If the update of 2% of our systems won’t work because of failing usrmerge
> I would be asked if Debian is the right distribution.

There are always unforseen issues to be expected when upgrading. And
at the moment, the only issues that are known when installing the
usrmerge package is when there are different binaries with the same
name in /bin and /usr/bin (or /sbin), and I really don't think that
this is actually a likely scenario.
For these cases though maybe the usrmerge script could ask the admin
on what to do to handle these particular binaries, instead of failing.

I am not strongly advocating for going down this route and actually
migrating all systems on update, but I do not want us to dismiss that
option because we are scared that something might go wrong without
actually knowing that there are unfixable cases where the update might
inevitably break on older installations. Instead, I would rather want
to try out the migration on a bigger scale - potentially and
temporarily break unstable (with a warning, of course), instead of
discussing over and over again potential issues that might not
actually be real and delaying a useful change because of that.

(TBH, for the buster release not switching the buildds to usrmerge and
keep debootstrap/the installer to install in an usrmerged
configuration and then do the final switch during bullseye seems
sensible and I don't see any issue this would cause. Of course if the
reproducible-builds test turns out that we only need to fix a small
amount of packages to make the usrmerge happen on buildds as well,
switching them as well could make sense still)

Cheers,
Matthias

-- 
I welcome VSRE emails. See http://vsre.info/

Re: usrmerge -- plan B?

[Hans]

Am Freitag, 23. November 2018, 14:47:28 CET schrieb Stephan Seitz:
> On Fr, Nov 23, 2018 at 02:04:05 +0100, Matthias Klumpp wrote:

> And how do you revert this change? As far as I have understand you can’t
> remove the usrmerge package and have your system in the old state again.

Making an image of the whole hard drive is always a good idea.

> As others in the thread have mentioned they don’t see the risk with new
> installations but with old systems with different admins and third-party
> software.
> 

Changes are always coming with a risk. However, ones who fear, should try on a 
canary. This is, what I would do. Ok, this might be a little more work, but it 
is is worth. 

> Right, and it means he has enough to do and doesn’t want to debug the
> usrmerge. I don’t want to have a usrmerge at a dist-upgrade. You don’t
> really know the sequence of the package updates. I think the risk is too
> big to have a partial upgraded system.
> 

So, that is, where "testing" is standing for. Testing before release
 
> If the update of 2% of our systems won’t work because of failing usrmerge
> I would be asked if Debian is the right distribution.

Good point. However, this should be told to Microsoft admins or their 
deciders.

> 
> Shade and sweet water!
> 
>   Stephan

I think, we have now 2-3 moths for testing until freeze begins. Shouldn't this 
time not beeing well used, to eliminate all issues?

Best 

Hans

Re: usrmerge -- plan B?

[Stephan Seitz]

On Fr, Nov 23, 2018 at 02:04:05 +0100, Matthias Klumpp wrote:

If there are actual issues encountered, we can always revert a change


And how do you revert this change? As far as I have understand you can’t 
remove the usrmerge package and have your system in the old state again.


As others in the thread have mentioned they don’t see the risk with new 
installations but with old systems with different admins and third-party 
software.



anyway. During distribution upgrades there is a lot that can be wrong
and a lot of stuff the administrator needs to care about (config file


Right, and it means he has enough to do and doesn’t want to debug the 
usrmerge. I don’t want to have a usrmerge at a dist-upgrade. You don’t 
really know the sequence of the package updates. I think the risk is too 
big to have a partial upgraded system.



with information to the system administrator on what to do in case of
an error, and works for 98% of all users anyway, I see no reason not


If the update of 2% of our systems won’t work because of failing usrmerge 
I would be asked if Debian is the right distribution.


Shade and sweet water!

Stephan

--
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |


smime.p7s
Description: S/MIME cryptographic signature

Re: usrmerge -- plan B?

[Matthias Klumpp]

Am Fr., 23. Nov. 2018 um 13:45 Uhr schrieb Ian Jackson
:
> Russ Allbery writes ("Re: usrmerge -- plan B?"):
> > This is a much better summary of the thread, and I wish that you would
> > have said this instead of claiming incorrectly that those same people are
> > the ones advocating for a full merge of all systems.
>
> Marco d'Itri writes ("Re: usrmerge -- plan B?"):
> > If you are seriously concerned with the small issuses caused by the
> > transition to merged-/usr then I have a better proposal.
> > Plan C: just stop supporting non-merged-/usr systems since these
> > problems are caused by having to support both, and there is no real
> > benefit in doing that other than pleasing the few people who are scared
> > by changes.

For this I actually wonder: Why don't we just do that and enforce the
usrmerge on unstable?
I think we are caring too much about the stability of the unstable
suite, and absolutely miss the chance of just running experiments to
gather data on the feasibility of changes.
What we could do in this case is for example to just make the usrmerge
migration mandatory for users of the unstable suite and see what kind
of issues and how many of them users will actually encounter. The
suite is called unstable afterall :-)
If there are actual issues encountered, we can always revert a change
and not have it go into stable, but in any case we will get a lot
better testing on the migration and a lot more data on whether there
actually are any issues.

Since new installations will be usrmerged by default anyway and there
is no reason or easy way to swich back to a split-usr system, I think
the issues related to a split-usr-system will go away in the long run
anyway. During distribution upgrades there is a lot that can be wrong
and a lot of stuff the administrator needs to care about (config file
changes, different featuresets of tools, software being removed from
the archive, ...), so if the usrmerge package has a sensible fallback
with information to the system administrator on what to do in case of
an error, and works for 98% of all users anyway, I see no reason not
to try using it and save us from an eternal transition period or pain
of maintaining two configurations.

Cheers,
Matthias

-- 
I welcome VSRE emails. See http://vsre.info/

Re: usrmerge -- plan B?

[Ansgar Burchardt]

On Fri, 2018-11-23 at 12:45 +, Ian Jackson wrote:
> Russ Allbery writes ("Re: usrmerge -- plan B?"):
> > This is a much better summary of the thread, and I wish that you would
> > have said this instead of claiming incorrectly that those same people are
> > the ones advocating for a full merge of all systems.
> 
> Marco d'Itri writes ("Re: usrmerge -- plan B?"):
> > If you are seriously concerned with the small issuses caused by the 
> > transition to merged-/usr then I have a better proposal.
> > Plan C: just stop supporting non-merged-/usr systems since these 
> > problems are caused by having to support both, and there is no real 
> > benefit in doing that other than pleasing the few people who are scared 
> > by changes.

What is this quote supposed to tell us?

Ansgar

Re: individual packages moving binaries from /bin to /usr/bin (was: Re: usrmerge -- plan B?)

[Ian Jackson]

Simon McVittie writes ("Re: individual packages moving binaries from /bin to 
/usr/bin (was: Re: usrmerge -- plan B?)"):
> I'm not sure yet what the best plan for merged /usr is. I would definitely
> like to make sure it's at least possible to continue to use merged
> /usr for special-purpose systems (particularly containers and embedded
> systems), even if it comes with major caveats like "can't reliably build
> Debian packages suitable for other systems";

To be very clear: I have no problem with this at all.

> I personally think everyone
> should be using sbuild or equivalent, either on a buildd or locally,
> to build "release-quality" packages suitable for distribution to other
> systems *anyway*, but I know that view isn't necessarily universal.

"Suitable for distribution to other systems" is rather a moveable
feast.  I absolutely agree if you mean formal publication as part of
some kind of release.

But I'm sure all of us have on occasion done ad-hoc builds and then
copied the .deb somewhere else to install it.  Indeed my own
experience is that during development I rarely use a chroot.  I think
someone should be able to build some software on their own computer
and give the binaries to a friend, without having to set up a chroot.

I also think that setting up a chroot should be made easier and that
more people should use chroots.  I don't think these views conflict.

> For at least special-purpose systems, merged /usr seems to work fine with
> stretch, and I was able to get it working in an Ubuntu 12.04 derivative
> by backporting a single-digit number of changes, so that particular genie
> has been out of the bottle for quite some time anyway.

Would it be helpful to make some of this explicit in Debian policy ?

IMO binary packages shipped by Debian should certainly support
installation on both merged-usr and separate-usr systems.

And I wouldn't object to a rule that our source packages must build
`correctly' on both such systems, subject to the caveat that the
results from a merged-usr build are not of general applicability and
should be used only in a closed environment where all the target
systems are also merged-usr.

Does that make sense ?

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: usrmerge -- plan B?

[Ian Jackson]

Russ Allbery writes ("Re: usrmerge -- plan B?"):
> This is a much better summary of the thread, and I wish that you would
> have said this instead of claiming incorrectly that those same people are
> the ones advocating for a full merge of all systems.

Marco d'Itri writes ("Re: usrmerge -- plan B?"):
> If you are seriously concerned with the small issuses caused by the 
> transition to merged-/usr then I have a better proposal.
> Plan C: just stop supporting non-merged-/usr systems since these 
> problems are caused by having to support both, and there is no real 
> benefit in doing that other than pleasing the few people who are scared 
> by changes.

Ian.

Re: individual packages moving binaries from /bin to /usr/bin (was: Re: usrmerge -- plan B?)

[W. Martin Borgert]

Quoting Holger Levsen :

On Thu, Nov 22, 2018 at 10:11:24PM +0100, W. Martin Borgert wrote:

Reminds me of the long /usr/doc -> /usr/share/doc transition in potato
times. And we did not even have dh in those days. Sounds good to me!


ITYM s#potato#slink, potato, woody, sarge, etch and lenny#

(Started in 1999 and finally fully finished on Sep 25th 2008 with
closing #322762.)

So I don't think this transition is such a positive example :-D


Why not positive? Yes, it took some time, but it worked out
without breakage. As we say: Gut Ding will Weile haben! :—)
(for non-Germans: "it takes time to do a thing well")

Re: individual packages moving binaries from /bin to /usr/bin (was: Re: usrmerge -- plan B?)

[Marco d'Itri]

On Nov 22, Ansgar Burchardt  wrote:

> Well, the iptables case is different: those are compat symlinks created
> by the package's maintainer scripts, not the /bin -> /usr/bin symlinks
> that merged-/usr sets up.
Actually iptables is different because it mixes (incorrectly) 
compatibility symlinks and diversions.

Correctly creating just compatibility symlinks is trivial and has been 
documented for years on the wiki.

> In case a later Debian release would merged-/usr mandatory, the
> conversion process would be less problematic as no files would be left
> to move (just replace individual symlinks with a directory symlink).
No: it would not be any easier than it is now because every Debian 
system can already be converted automatically. Having usrmerge move 
files /installed by packages/ is well tested.

-- 
ciao,
Marco


signature.asc
Description: PGP signature