Re: Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi Russ,

Quoting Russ Allbery (2022-09-20 00:05:23)
> Johannes Schauer Marin Rodrigues  writes:
> > in 2016 we filed our first DPKG_ROOT patch #824594 against
> > base-files. The dpkg version at the time just had included support for
> > the DPKG_ROOT variable being set for maintainer scripts and we were
> > excited to try out this new feature for creating foreign architecture
> > chroots. At the time we thought that no discussion on d-devel was
> > necessary before filing the bug because we knew only 10 source packages
> > had to add DPKG_ROOT to their maintainer scripts and because doing so
> > would not affect any normal installation.
> 
> [...]
> 
> Thank you for this excellent write-up!
> 
> This is exactly the type of fairly obscure Debian lore that, although it
> only affects a small number of packages, is worth documenting because it
> can be very difficult to understand otherwise why it's present or to debug
> problems caused by accidentally breaking it.
> 
> I would therefore love to see this documented in Policy.  The
> documentation doesn't have to be long, but even though this only affects a
> small handful of packages used during early bootstrapping, I think we
> should write it down somewhere official so that we have a record of what
> we're doing and how it's supposed to work (and what packages need to care
> about it).
> 
> If possible, could you write up a brief description along those lines and
> open a bug against debian-policy with that description?  We can then
> figure out where to put it in the document.

sure thing!

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020323

Lets continue the discussion about how to best include the relevant information
into policy over there.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Presenting DPKG_ROOT

[Russ Allbery]

Johannes Schauer Marin Rodrigues  writes:

> in 2016 we filed our first DPKG_ROOT patch #824594 against
> base-files. The dpkg version at the time just had included support for
> the DPKG_ROOT variable being set for maintainer scripts and we were
> excited to try out this new feature for creating foreign architecture
> chroots. At the time we thought that no discussion on d-devel was
> necessary before filing the bug because we knew only 10 source packages
> had to add DPKG_ROOT to their maintainer scripts and because doing so
> would not affect any normal installation.

[...]

Thank you for this excellent write-up!

This is exactly the type of fairly obscure Debian lore that, although it
only affects a small number of packages, is worth documenting because it
can be very difficult to understand otherwise why it's present or to debug
problems caused by accidentally breaking it.

I would therefore love to see this documented in Policy.  The
documentation doesn't have to be long, but even though this only affects a
small handful of packages used during early bootstrapping, I think we
should write it down somewhere official so that we have a record of what
we're doing and how it's supposed to work (and what packages need to care
about it).

If possible, could you write up a brief description along those lines and
open a bug against debian-policy with that description?  We can then
figure out where to put it in the document.

Thanks!

-- 
Russ Allbery (r...@debian.org)  

Re: Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Josh Triplett (2022-09-11 15:26:52)
> Johannes Schauer Marin Rodrigues wrote:
> > What do you think? Is this the right solution to the problem? A few more 
> > bits
> > about DPKG_ROOT as well as alternative solution proposals (including 
> > rejected
> > ones) can be found on this wiki page:
> > 
> > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap#Proposal:_chrootless_maintscripts
> > 
> > So lets come back to our question of scope: Right now, our DPKG_ROOT patches
> > are limited to Essential:yes, build-essential, apt and systemd. We also
> > restrict the mechanism to initial installations only and upgrades are not
> > supported. We also currently require that the system (and its tools) on the
> > outside must be the same as the chroot that is being created with 
> > DPKG_ROOT. As
> > far as enabling very early architecture bootstrap goes, this solves the
> > problem.
> > 
> > So what do you think? Is this okay? Is there a better solution?
> 
> So, first of all, I'm thrilled to see work on improving bootstrapping
> and cross-bootstrapping. I'm also thrilled to see DPKG_ROOT being
> discussed more broadly.
> 
> Regarding the specific solution: the DPKG_ROOT approach has concerned me
> since it was introduced, because maintainer scripts run as root, so a
> bug in one package's DPKG_ROOT support (let alone the absence of
> DPKG_ROOT support) would result in modifying the host system rather than
> the target chroot.
> 
> I would love to see the DPKG_ROOT support augmented with `unshare`, a
> mount namespace, a UID namespace, and a chroot, such that:
> 
> - The host filesystem is bind-mounted read-only.
> - Host devices are not available (only a minimal /dev); this will also
>   help ensure bootstraps don't depend on any aspect of the host system.
> - The maintainer scripts run as container-root, but not as host-root, so
>   that they can't accidentally change any aspect of the host.
> 
> This could still make use of the existing DPKG_ROOT support; this would
> be completely transparent to the maintainer scripts and tools ported
> thus far.
> 
> This would also allow bootstrapping as non-root, on systems that allow
> creating UID namespaces as non-root.
> 
> As a bonus, testsuites could use an overlayfs instead of a read-only
> bind mount, and then check afterwards if *any* changes occurred in the
> overlay, which would be a test failure.
> 
> Does that seem like a reasonable addition to the DPKG_ROOT support?

re-mounting the host filesystem read-only is a good idea for the machinery that
in the end will create chroots using DPKG_ROOT (for example rebootstrap). Note,
that as the normal user, you probably will never need to create a chroot with
DPKG_ROOT because even if you want to create a foreign architecture chroot,
qemu support is available for all our release arches. But if you are involved
in early architecture bootstrapping and want to make sure that chroot creation
does not modify your outer system, then you can create your chroot using either
of the following techniques.

DISCLAIMER: two of our patches have not been applied yet so DO NOT run any of
the below commands unless you really know what you are doing.

1. as the normal user using fakeroot

fakeroot mmdebstrap --variant=essential --mode=chrootless unstable 
chroot.tar

2. inside mmdebstrap in unshare mode as a customize-hook

mmdebstrap --mode=unshare --skip=setup,update,cleanup --variant=custom \
--setup-hook='mount -o remount,bind,ro /' \
--customize-hook='mmdebstrap --variant=essential --mode=chrootless unstable 
"$1"' \
'' chroot.tar

3. using the (undocumented) mmdebstrap --unshare-helper

mmdebstrap --unshare-helper mmdebstrap \
--setup-hook='mount -o remount,bind,ro /' \
--variant=essential --mode=chrootless unstable chroot.tar

4. using lxc unshare tools

lxc-usernsexec -- lxc-unshare -s 'MOUNT|PID|UTSNAME|IPC' -- mmdebstrap \
--setup-hook='mount -o remount,bind,ro /' \
--variant=essential --mode=chrootless unstable chroot.tar

If you set SOURCE_DATE_EPOCH and add a repo with the last two remaining patches
applied to the packages in the Essential:yes set, then the tarballs created by
either of the commands above will be bit-by-bit identical to tarballs created
like this:

mmdebstrap --variant=essential unstable chroot.tar

Re-mounting / as read-only with an unshared mount namespace is unfortunately
nothing we can test in our CI system because those do not support any unshare
operation. But as a test, re-mounting / read-only is also not useful because
there are scripts which will not error-out if they cannot write something and
we would thus never detect these unintended changes. So instead, in our CI, we
let the maintainer systems write whatever they want, but compare the whole
system state before and after, to make sure that no file on the outside was
modified (even in its metadata).

signature.asc
Description: signature

Re: Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Josh Triplett (2022-09-11 15:26:52)
> Johannes Schauer Marin Rodrigues wrote:
> > What do you think? Is this the right solution to the problem? A few more 
> > bits
> > about DPKG_ROOT as well as alternative solution proposals (including 
> > rejected
> > ones) can be found on this wiki page:
> > 
> > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap#Proposal:_chrootless_maintscripts
> > 
> > So lets come back to our question of scope: Right now, our DPKG_ROOT patches
> > are limited to Essential:yes, build-essential, apt and systemd. We also
> > restrict the mechanism to initial installations only and upgrades are not
> > supported. We also currently require that the system (and its tools) on the
> > outside must be the same as the chroot that is being created with 
> > DPKG_ROOT. As
> > far as enabling very early architecture bootstrap goes, this solves the
> > problem.
> > 
> > So what do you think? Is this okay? Is there a better solution?
> 
> So, first of all, I'm thrilled to see work on improving bootstrapping
> and cross-bootstrapping. I'm also thrilled to see DPKG_ROOT being
> discussed more broadly.
> 
> Regarding the specific solution: the DPKG_ROOT approach has concerned me
> since it was introduced, because maintainer scripts run as root, so a
> bug in one package's DPKG_ROOT support (let alone the absence of
> DPKG_ROOT support) would result in modifying the host system rather than the
> target chroot.

That is correct.

> I would love to see the DPKG_ROOT support augmented with `unshare`, a mount
> namespace, a UID namespace, and a chroot, such that:
> 
> - The host filesystem is bind-mounted read-only.
> - Host devices are not available (only a minimal /dev); this will also
>   help ensure bootstraps don't depend on any aspect of the host system.
> - The maintainer scripts run as container-root, but not as host-root, so
>   that they can't accidentally change any aspect of the host.
> 
> This could still make use of the existing DPKG_ROOT support; this would
> be completely transparent to the maintainer scripts and tools ported
> thus far.
> 
> This would also allow bootstrapping as non-root, on systems that allow
> creating UID namespaces as non-root.
> 
> As a bonus, testsuites could use an overlayfs instead of a read-only
> bind mount, and then check afterwards if *any* changes occurred in the
> overlay, which would be a test failure.
> 
> Does that seem like a reasonable addition to the DPKG_ROOT support?

Bind-mounting the host system read-only is an interesting idea that we haven't
tried out yet. To avoid unintended modifications of the host system we are
currently testing two different approaches in our CI system:

 1. we run the whole chroot creation inside fakeroot. That way, the maintainer
scripts think they are run as root but they actually cannot modify anything
important.

 2. we run the chroot creation as root but inside mmdebstrap in unshare mode.
We create a tarball of the outer system before and after running the
DPKG_ROOT method and then compare these two tarballs to make sure that no
changes were done in the outer system.

I think making the outer system read-only via bind-mounts is another valuable
test that we should implement.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Presenting DPKG_ROOT

[Josh Triplett]

Johannes Schauer Marin Rodrigues wrote:
> What do you think? Is this the right solution to the problem? A few more bits
> about DPKG_ROOT as well as alternative solution proposals (including rejected
> ones) can be found on this wiki page:
> 
> https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap#Proposal:_chrootless_maintscripts
> 
> So lets come back to our question of scope: Right now, our DPKG_ROOT patches
> are limited to Essential:yes, build-essential, apt and systemd. We also
> restrict the mechanism to initial installations only and upgrades are not
> supported. We also currently require that the system (and its tools) on the
> outside must be the same as the chroot that is being created with DPKG_ROOT. 
> As
> far as enabling very early architecture bootstrap goes, this solves the
> problem.
> 
> So what do you think? Is this okay? Is there a better solution?

So, first of all, I'm thrilled to see work on improving bootstrapping
and cross-bootstrapping. I'm also thrilled to see DPKG_ROOT being
discussed more broadly.

Regarding the specific solution: the DPKG_ROOT approach has concerned me
since it was introduced, because maintainer scripts run as root, so a
bug in one package's DPKG_ROOT support (let alone the absence of
DPKG_ROOT support) would result in modifying the host system rather than
the target chroot.

I would love to see the DPKG_ROOT support augmented with `unshare`, a
mount namespace, a UID namespace, and a chroot, such that:

- The host filesystem is bind-mounted read-only.
- Host devices are not available (only a minimal /dev); this will also
  help ensure bootstraps don't depend on any aspect of the host system.
- The maintainer scripts run as container-root, but not as host-root, so
  that they can't accidentally change any aspect of the host.

This could still make use of the existing DPKG_ROOT support; this would
be completely transparent to the maintainer scripts and tools ported
thus far.

This would also allow bootstrapping as non-root, on systems that allow
creating UID namespaces as non-root.

As a bonus, testsuites could use an overlayfs instead of a read-only
bind mount, and then check afterwards if *any* changes occurred in the
overlay, which would be a test failure.

Does that seem like a reasonable addition to the DPKG_ROOT support?

- Josh Triplett

Presenting DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi,

in 2016 we filed our first DPKG_ROOT patch #824594 against base-files. The dpkg
version at the time just had included support for the DPKG_ROOT variable being
set for maintainer scripts and we were excited to try out this new feature for
creating foreign architecture chroots. At the time we thought that no
discussion on d-devel was necessary before filing the bug because we knew only
10 source packages had to add DPKG_ROOT to their maintainer scripts and because
doing so would not affect any normal installation.

In hindsight, this was a mistake. We should've brought it up on d-devel and
explained our idea first before starting to file our bugs with patches. Luckily
in this very first bug, Santiago pushed back and requested to first see that
our concept really works before applying anything. This was excellent advice
and resulted in a CI job on salsa that regularly checks that Debian unstable
with our patches produces bit-by-bit identical chroots created with DPKG_ROOT
compared to chroots created the normal way. This proof of our method working
exactly as intended convinced Santiago and probably many other maintainers that
applied our patches to their source packages.

Today, six years later, all but two of our patches have been applied. The
transition is nearly complete.

So why are we (Helmut and me) writing you now that things are already done?

Firstly to apologize for having misjudged the situation in the past years. We
should've communicated DPKG_ROOT to all of d-devel at the very beginning to
allow for project wide discussion but decided not to do so. For that mistake we
are sorry.

Secondly, while we of course are hoping for a blessing of our contributions in
hindsight, we wanted to ask whether we maybe missed anything that makes our
DPKG_ROOT approach inferior to other ways to solve this problem. We also wanted
to ask about the scope of DPKG_ROOT. So what exactly is the problem we are
trying to solve?

In the very early days of a new architecture, emulation support is either not
available at all or too buggy to be useful for any practical purposes.  After
having cross-built the initial package set, these packages need to be installed
to create a chroot that can then be used to continue building packages natively
on the new architecture, completing the early bootstrap process. But creating
that chroot requires package maintainer scripts to be run but we cannot emulate
the new architecture to run any of its binaries. So how was this done in the
past? By performing the tasks that are usually carried out by package
maintainer scripts manually. We wanted to find a way that would allow for an
automatic creation of a foreign architecture chroot without being able to run
any of the binaries in it.

To solve this problem, since dpkg 1.18.10 (old-old-stable) the variable
`DPKG_ROOT` is set to the empty string in all maintainer script invocations.
All, except when dpkg is run with `--force-script-chrootless` and `--root` set
to a chroot directory path which will be stored in the `DPKG_ROOT` environment
variable. This is a “force” option because if set, dpkg will not do a chroot
call into the new chroot before calling the maintainer script, thus causing
undesired changes in the outer system instead. By installing packages in a way
that avoids the chroot call, maintainer scripts will run the tools from outside
the chroot instead of the foreign architecture tools inside the chroot.  These
tools need to know where they need to operate on and they use the value of the
`DPKG_ROOT` variable to get this information.

As of today, tools like dpkg-divert, dpkg-maintscript-helper,
deb-systemd-helper, or update-shells understand the `DPKG_ROOT` variable and
will do the right thing.  Maintainer script snippets as they are generated by
debhelper also already respect `DPKG_ROOT`. Where we need package-specific
patches is when maintainer scripts call general purpose tools like mv, cp,
chown or chmod, where it doesn’t make sense to let them support `DPKG_ROOT`
because those tools are not limited to be used in maintainer scripts. Source
packages in the Essential:yes and build-essential set that require changes
involving the `DPKG_ROOT` variable in their maintainer scripts are:

base-files, base-passwd, coreutils, dash, debianutils, dpkg, gcc-defaults,
glibc, pam, shadow

Usually, patches look like this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=983565;filename=coreutils_8.32-4.1.debdiff;msg=5

So if before the maintainer script ran `rm /usr/bin/touch` then it now runs `rm
"$DPKG_ROOT/usr/bin/touch"`. Since the `DPKG_ROOT` variable is usually empty,
this change will be a NO-OP in normal installations and only affects setups
that explicitly called dpkg in the way described above. Another way to support
`DPKG_ROOT` is to remove maintainer scripts altogether and replace them by
declarative methods, which was done in case of the transition from add-shell
and remove-shell to update-shells: