Bug#1020323: debian-policy: document DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2022-10-05 20:08:19)
> I think this text is already quite good. I am yet wondering about the scope
> of support that we mention here.
> 
> 1. You write that we want essential + build-essential. In practice, we
>also want things such as apt or systemd. I am wondering whether we
>should rephrase this in a less specific way that leaves open some
>packages beyond the mentioned set. Vagueness can be avoided by
>explaining the purpose: We target packages that are relevant to
>setting up an initial build daemon.
> 
> 2. We should likely mention that package upgrade and removal paths can
>freely ignore DPKG_ROOT. Maintainer scripts can assume that when
>DPKG_ROOT is in effect, it will be an initial installation. Something
>along this would have helped Michael in determining whether his
>recent changes to init script handling would affect DPKG_ROOT.
> 
> Let me try to extend your text:
> 
> To enable creating a foreign architecture Debian chroot during the early
> bootstrap of a new Debian architecture, maintainer scripts and utilities
> called by maintainer scripts of packages relevant to setting up a
> build daemon, should support operating on a custom chroot directory.
> [... keep rest of the text unchanged ...]
> Support for `DPKG_ROOT` in code that handles package upgrades or package
> removal is not needed.

I would replace "relevant to setting up a build daemon" with "relevant to be
installed on a build daemon (for example essential and build-essential
packages)". We are not interested in the packages that need to set up the build
daemon (that could be understood to mean packages like debian-installer or
debootstrap) but we are interested in the packages that are installed on such a
build daemon. Giving the essential and build-essential package set as examples
might further help with understanding what is meant here.

I would also replace "is not needed" with "is not required".

But those two are just nitpicks. Either version is fine by me and I think it
rather comes down to what policy editors think makes sense.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1020323: debian-policy: document DPKG_ROOT

[Helmut Grohne]

Hi Joshannes,

On Wed, Oct 05, 2022 at 02:35:30PM +0200, Johannes Schauer Marin Rodrigues 
wrote:
>To enable creating a foreign architecture Debian chroot during the early
>bootstrap of a new Debian architecture, maintainer scripts and utilities
>called by maintainer scripts of packages in the essential and
>build-essential set, should support operating on a custom chroot directory.
>This is to avoid running any of the foreign architecture utilities from the
>chroot, because those cannot be executed during the early bootstrapping
>phase of a new architecture.  Instead, by avoiding the chroot() call,
>utilities from the outside should operate on the chroot path given via the
>`DPKG_ROOT` environment variable.  This environment variable is set but
>empty during normal package installations.  If the `DPKG_ROOT` environment
>variable is not empty, then this indicates to the maintainer scripts and 
> the
>tools it executes, that a chroot is being built as part of an early
>architecture bootstrap and all operations should be performed in the chroot
>path given by the contents of the `DPKG_ROOT` environment variable. In that
>case, the maintainer script should not modify anything outside the chroot
>directory.

Thank you for writing this.

> I refrained from using "must" because we promised maintainers that they would
> not need to do the work themselves but will get patches sent from us. We do 
> not
> want to force work on maintainers by making it an RC bug if they do not 
> support
> DPKG_ROOT.
> 
> Helmut, what do you think?

I think this text is already quite good. I am yet wondering about the
scope of support that we mention here.

1. You write that we want essential + build-essential. In practice, we
   also want things such as apt or systemd. I am wondering whether we
   should rephrase this in a less specific way that leaves open some
   packages beyond the mentioned set. Vagueness can be avoided by
   explaining the purpose: We target packages that are relevant to
   setting up an initial build daemon.

2. We should likely mention that package upgrade and removal paths can
   freely ignore DPKG_ROOT. Maintainer scripts can assume that when
   DPKG_ROOT is in effect, it will be an initial installation. Something
   along this would have helped Michael in determining whether his
   recent changes to init script handling would affect DPKG_ROOT.

Let me try to extend your text:

To enable creating a foreign architecture Debian chroot during the early
bootstrap of a new Debian architecture, maintainer scripts and utilities
called by maintainer scripts of packages relevant to setting up a
build daemon, should support operating on a custom chroot directory.
[... keep rest of the text unchanged ...]
Support for `DPKG_ROOT` in code that handles package upgrades or
package removal is not needed.

Helmut

Bug#1020323: debian-policy: document DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Hi Russ,

thank you for your explanations, things are quite a bit clearer now.

Quoting Russ Allbery (2022-09-20 05:47:45)
> The point of putting this in Policy is to provide guidance to the
> packagers, not to the bootstrappers.  Presumably you already have other
> documentation that you maintain about how to bootstrap Debian that spells
> out what to do in what order; that's outside of Policy's remit.  What
> Policy is trying to do is to define for packagers what interface they have
> to implement, and DPKG_ROOT is now part of that interface.
> 
> So in other words, you can just say something like:
> 
> Maintainer scripts in essential or build-essential packages must
> preface all paths they act on in maintainer scripts with an expansion
> of the DPKG_ROOT environment variable.  This will normally be empty
> (and thus normally will not change anything), but in some situations
> it may be set to a bootstrapping path to tell packages to act under
> that path instead of on the root file system.
> 
> That wording probably isn't quite right, but I think that's the general
> idea.

okay, if you think this should be a new section, then let me try to come up
with a text here.

   To enable creating a foreign architecture Debian chroot during the early
   bootstrap of a new Debian architecture, maintainer scripts and utilities
   called by maintainer scripts of packages in the essential and
   build-essential set, should support operating on a custom chroot directory.
   This is to avoid running any of the foreign architecture utilities from the
   chroot, because those cannot be executed during the early bootstrapping
   phase of a new architecture.  Instead, by avoiding the chroot() call,
   utilities from the outside should operate on the chroot path given via the
   `DPKG_ROOT` environment variable.  This environment variable is set but
   empty during normal package installations.  If the `DPKG_ROOT` environment
   variable is not empty, then this indicates to the maintainer scripts and the
   tools it executes, that a chroot is being built as part of an early
   architecture bootstrap and all operations should be performed in the chroot
   path given by the contents of the `DPKG_ROOT` environment variable. In that
   case, the maintainer script should not modify anything outside the chroot
   directory.

I refrained from using "must" because we promised maintainers that they would
not need to do the work themselves but will get patches sent from us. We do not
want to force work on maintainers by making it an RC bug if they do not support
DPKG_ROOT.

Helmut, what do you think?

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1020323: debian-policy: document DPKG_ROOT

[Russ Allbery]

Johannes Schauer Marin Rodrigues  writes:

>  * where to document this? Other variables set for maintainer scripts
>like DPKG_MAINTSCRIPT_PACKAGE or DPKG_MAINTSCRIPT_ARCH do not seem to
>be documented either even though (according to codesearch.d.n) they
>are used in hundreds of places

In general, Policy doesn't need to be a copy of dpkg's documentation, so I
don't think we need to list everything that dpkg makes available.

The important thing that Policy is adding isn't to catalog the facilities
available.  It's to tell you *when* to use those facilities.  In the case
of some of those variables, that's the obvious "whenever it's convenient
to get that information from an environment variable for whatever you're
trying to do," and Policy doesn't need to say anything about it.  But
here, some packages *have* to use this environment variable or they will
be buggy.  That's the sort of thing Policy should describe.

I would tend to add a section on bootstrapping support (since that's what
this is for) to chapter 6 at the end.  It could just talk about DPKG_ROOT
for now and we can add any other maintainer script considerations that
come up there in the future.

>  * what about scope? The Essential:yes and build-essential package set
>are certainly a *must*. We need these to start building natively on a
>new architecture. But do we need more than those? Having apt would be
>handy but not strictly necessary (dpkg -i can always be used
>instead). Having an init system would be handy but not strictly
>necessary (can always boot with init=/bin/bash).

I think this is up to you to define, but we need to be able to describe
it.  Starting with essential and build-essential packages sounds great,
and Policy already defines both of those terms, so that's easy.  If you
need to add more packages, we will need to figure out how to describe them
without special-case listings of package names.

>  * what can maintainer scripts supporting DPKG_ROOT expect from the
>system on the outside? Are its package versions the same as the
>system inside the chroot? Is it even Debian? Right now we do all our
>tests such that the system on the outside is identical to the one we
>want to create the chroot for except that it is of the native
>architecture. But should such a restriction be placed in policy?

It sounds like you're not imposing any restrictions on the *package* here,
so there's no need to say anything unless at some point you need to do so.

The point of putting this in Policy is to provide guidance to the
packagers, not to the bootstrappers.  Presumably you already have other
documentation that you maintain about how to bootstrap Debian that spells
out what to do in what order; that's outside of Policy's remit.  What
Policy is trying to do is to define for packagers what interface they have
to implement, and DPKG_ROOT is now part of that interface.

So in other words, you can just say something like:

Maintainer scripts in essential or build-essential packages must
preface all paths they act on in maintainer scripts with an expansion
of the DPKG_ROOT environment variable.  This will normally be empty
(and thus normally will not change anything), but in some situations
it may be set to a bootstrapping path to tell packages to act under
that path instead of on the root file system.

That wording probably isn't quite right, but I think that's the general
idea.

>  * what about upgrades? We only want to create a chroot and some of our
>patches are made much simpler because we ignore upgrades. If newer
>package versions are required, the bootstrapper can just create a new
>chroot without upgrading anything.

If maintainers don't have to worry about this for upgrades, you can just
say so.  I think it's up to you whether you think that is important or
not.  I would be inclined to say that it's *safe* to add DPKG_ROOT to
paths even on upgrade actions, but you only *must* do so for maintainer
script actions that run during initial installation.

Thank you very much for starting this discussion!

-- 
Russ Allbery (r...@debian.org)  

Bug#1020323: debian-policy: document DPKG_ROOT

[Johannes Schauer Marin Rodrigues]

Source: debian-policy
Version: 4.6.1.1
Severity: wishlist
User: debian-d...@lists.debian.org
Usertags: dpkg-root-support
X-Debbugs-Cc: jo...@debian.org, debian-cr...@lists.debian.org

Hi,

in [1] Russ asked us to submit a policy bug about DPKG_ROOT so here it
goes. :)

[1] https://lists.debian.org/874jx3yq8s@hope.eyrie.org

Here is one description of DPKG_ROOT from the dpkg man page:

DPKG_ROOT
Defined by dpkg on the maintainer script environment to indicate
which installation to act on (since dpkg 1.18.5).  The value is
intended to be prepended to any path maintainer scripts operate on.
During normal operation, this variable is empty.  When installing
packages into a different instdir, dpkg normally invokes maintainer
scripts using chroot(2) and leaves this variable empty, but if
--force-script-chrootless is specified then the chroot(2) call is
skipped and instdir is non-empty.

In practice this means that if (and only if) dpkg is called with
--root=X --force-script-chrootless then all maintainer scripts will be
called with the DPKG_ROOT variable set to X.

Why is this useful? In the very early days of a new architecture,
emulation support is either not available at all or too buggy to be
useful for any practical purposes.  After having cross-built the initial
package set, these packages need to be installed to create a chroot that
can then be used to continue building packages natively on the new
architecture, completing the early bootstrap process. But creating that
chroot requires package maintainer scripts to be run but we cannot
emulate the new architecture to run any of its binaries.

By installing packages with --force-script-chrootless, maintainer
scripts are called without being inside the chroot and thus they will
call the native binaries from the outside. To be able to know the chroot
directory they are supposed to operate on, the DPKG_ROOT variable is set
to the chroot directory. In practice, this means that if a maintainer
script ran this before:

chmod root:root "/path/to/file"

Then it now has to run

chmod root:root "$DPKG_ROOT/path/to/file"

More information about DPKG_ROOT can be found here:

 * 
https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap#Proposal:_chrootless_maintscripts
 * https://debconf22.debconf.org/talks/23-what-is-dpkg_root-and-what-is-it-not/
 * https://lists.debian.org/166289720850.2390.3729551131862514967@localhost

I think there is no problem with letting policy describe what the
DPKG_ROOT variable being set to a non-empty string means for a
maintainer script. There is also no problem in documenting what the
whole point of this exercise is.

But there are also a lot of open questions for which I do not have any
answers yet:

 * where to document this? Other variables set for maintainer scripts
   like DPKG_MAINTSCRIPT_PACKAGE or DPKG_MAINTSCRIPT_ARCH do not seem to be
   documented either even though (according to codesearch.d.n) they are used in
   hundreds of places
 * what about scope? The Essential:yes and build-essential package set are
   certainly a *must*. We need these to start building natively on a new
   architecture. But do we need more than those? Having apt would be handy
   but not strictly necessary (dpkg -i can always be used instead). Having an
   init system would be handy but not strictly necessary (can always boot with
   init=/bin/bash).
 * what can maintainer scripts supporting DPKG_ROOT expect from the system on
   the outside? Are its package versions the same as the system inside the
   chroot? Is it even Debian? Right now we do all our tests such that the
   system on the outside is identical to the one we want to create the
   chroot for except that it is of the native architecture. But should such a
   restriction be placed in policy?
 * what about upgrades? We only want to create a chroot and some of our patches
   are made much simpler because we ignore upgrades. If newer package versions
   are required, the bootstrapper can just create a new chroot without
   upgrading anything.

Thank you for your consideration!

cheers, josch