Bug#948115: Revise init script Policy based on GR result

[Russ Allbery]

Package: debian-policy
Version: 4.4.1.2
Severity: important

Per recent (non-BTS) discussion, this patch is a first draft at
reconciling Policy with the recent GR result.  Summary of changes:

* Change section headings and anchors to reflect the more general topic
* Add recommended naming convention for service units
* Encourage including an init script if there is no service unit
* Describe including an init script alongside a service unit as optional
* Recommend appropriate naming of an init script alongside a service unit
* Remove recommendation to include an init script
* Use init script rather than initscript consistently
* Move some things around that belong more naturally in other sections
* Be consistent about saying update-rc.d is a requirement
* Remove the section on alternate init systems, which is now not relevant

Policy itself has no links to the previous anchors.  This might
break external links; let me know if you feel like that's a larger
issue than I thought it was and we can look at keeping the old
(but pretty wildly inaccurate) anchors.

diff --git a/policy/ch-opersys.rst b/policy/ch-opersys.rst
index 4551196..47d9fe4 100644
--- a/policy/ch-opersys.rst
+++ b/policy/ch-opersys.rst
@@ -315,46 +315,53 @@ set to this value.
 The Debian autobuilders set HOME to ``/nonexistent`` so that packages
 which try to write to a home directory will fail to build.
 
-.. _s-sysvinit:
+.. _s-services:
 
-System run levels and ``init.d`` scripts
-
+Starting system services
+
 
-.. _s-etc-init.d:
+.. _s-services-intro:
 
 Introduction
 
 
-The ``/etc/init.d`` directory contains the scripts executed by ``init``
-at boot time and when the init state (or "runlevel") is changed (see
-:manpage:`init(8)`).
-
-``systemd`` uses dependency information contained within the init
-scripts and symlinks in ``/etc/rcn.d`` to decide which scripts to run
-and in which order. The ``sysv-rc`` runlevel system uses symlinks in
-``/etc/rcn.d`` to decide which scripts to run and in which order; see
-the ``README.runlevels`` file shipped with ``sysv-rc`` for
-implementation details. Other alternatives might exist.
-
-Maintainer scripts must use ``update-rc.d`` as described below to
-interact with the service manager for requests such as enabling or
-disabling services. They should use ``invoke-rc.d`` as described below
-to invoke initscripts for requests such as starting and stopping
-service.
+Packages that include system services should include ``systemd`` service
+units to start or stop those services.  See :manpage:`systemd.service(5)`
+for details on the syntax of a service unit file.  In the common case that
+a package includes a single system service, the service unit should have
+the same name as the package plus the ``.service`` extension.
+
+If the package does not include a service unit (if, for example, no one
+has yet written one), including an init script, as described below, to
+start the service is encouraged.
+
+Packages including a service unit may optionally include an init script to
+support other init systems.  In this case, the init script should have the
+same name as the ``systemd`` service unit so that ``systemd`` will ignore
+it and use the service unit instead.  Packages may also support other init
+systems by including configuration in the native format of those init
+systems.
+
+If a service unit is not present, ``systemd`` uses dependency information
+contained within the init scripts and symlinks in ``/etc/rcn.d`` to decide
+which scripts to run and in which order.  The ``sysv-rc`` runlevel system
+for ``sysvinit`` uses the same symlinks in ``/etc/rcn.d`` to decide which
+scripts to run and in which order at boot time and when the init state (or
+"runlevel") is changed.  See the ``README.runlevels`` file shipped with
+``sysv-rc`` for implementation details.  Other alternatives might exist.
+
+The sections below describe how to write those scripts and configure those
+symlinks.
 
 .. _s-writing-init:
 
 Writing the scripts
 ~~~
 
-Packages that include system services should include ``systemd`` service
-units to start or stop those services.  See :manpage:`systemd.service(5)`.
-
-To support other init systems, packages that include daemons for system
-services should place scripts in ``/etc/init.d`` to start or stop those
-services at boot time or during a change of runlevel. These scripts should
-be named ``/etc/init.d/package``, and they should accept one argument,
-saying what to do:
+Init scripts are placed in ``/etc/init.d``.  In the common case that a
+package starts a single service, they should be named
+``/etc/init.d/package``.  They should accept one argument, saying what to
+do:
 
 ``start``
 start the service,
@@ -381,10 +388,9 @@ saying what to do:
 ``status``
 report the current status of the service
 
-The ``start``, ``stop``, ``restart``, and ``force-reload`` options
-should be 

Re: Proposal for next steps for systemd-related policy

[Russ Allbery]

Sam Hartman  writes:

> I haven't been following the consensus around making service units more
> recommended.  Ignoring that discussion, but folding in the GR:

> Maintainers are recommended to install at least one of a service unit or
> init script.  Maintainers are encouraged to install an service unit and
> may install an init script.

> But if you've gotten to a point where service units are recommended all
> the time (no service unit but an init script is a bug) then: Maintainers
> are recommended to install a service unit.  If maintainers do not
> install a service unit, they are encouraged to install an init script;
> in other situations installing an init script is optional.

Thinking about this some more, and re-reading the text of the GR, this
sounds correct to me as well.  I'm not sure that Policy is currently
structured in such a way as to make saying this straightforward, but I'm
taking a look now and will create a bug with a patch for discussion and
review.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#944920: Revise terminology used to specify requirements

[Russ Allbery]

Sean Whitton  writes:

> Let's definitely reconsider those 'must' requirements in response to
> this work, but let's not commit ourselves to the idea that it's always a
> bug for the Release Team's conception of an RC bug, and Policy 'must'
> requirements, to disagree.

> The Release Team's conception of RC bugs, and the text of Policy, are
> generated and updated by different processes, for different purposes.  I
> think Debian benefits from that diversity of normative processes, and it
> would harm that to try too hard to keep the output of the two processes
> in perfect sync.

Yes, that's put better than I put it.  Thank you.  I was too focused on
the fact that I suspect we'll find some musts that are too aggressive, but
indeed, allowing these things to differ is part of why I'm proposing
explicit language to allow the Release Team to downgrade requirements to
recommendations.

-- 
Russ Allbery (r...@debian.org)  

Bug#944920: Revise terminology used to specify requirements

[Russ Allbery]

Russ Allbery  writes:

> I agree, but let's also fix existing incorrect wording.  I reviewed
> every instance of may and optional in Policy, and I think this larger
> diff will make wording (mostly) consistent.  I've tried not to change
> the sense of any of these Policy statements (even though a few are
> questionable and should probably be revisited).

Here is an updated version of this patch, including the upgrading
checklist entry.  I tried using a word diff, but I think the line diff is
more readable and easier to understand.  I at least found it easier to
understand the wording changes in that format (maybe this is just my long
familiarity with reviewing line diffs).

I think this is ready for seconds, assuming that one agrees with the three
places that I made semantic changes (/usr/lib64, init-system-helpers, and
deciding on encouraged for debian/missing-sources).

Most of this diff is changing normative "may not" phrasings to "must not"
or some other equivalent, and changing other uses of "may" to "could" or
other wordings.

This retains the statement about the release team's role in changing the
severity of Policy requirements, since I think that was the outcome of the
side conversation.  I'll work on an appendix accumulating all Policy
requirements in a separate patch.

diff --git a/policy/ch-archive.rst b/policy/ch-archive.rst
index b8ba081..e37ebee 100644
--- a/policy/ch-archive.rst
+++ b/policy/ch-archive.rst
@@ -329,8 +329,8 @@ management tools.
 the user doesn't select anything else. It doesn't include many large
 applications.
 
-No two packages that both have a priority of ``standard`` or higher
-may conflict with each other.
+Two packages that both have a priority of ``standard`` or higher must
+not conflict with each other.
 
 ``optional``
 This is the default priority for the majority of the archive. Unless
diff --git a/policy/ch-binary.rst b/policy/ch-binary.rst
index 74a1690..f1e518e 100644
--- a/policy/ch-binary.rst
+++ b/policy/ch-binary.rst
@@ -362,8 +362,8 @@ adding or removing diversions, package maintainer scripts 
must provide
 the ``--package`` flag to ``dpkg-divert`` and must not use ``--local``.
 
 All packages which supply an instance of a common command name (or, in
-general, filename) should generally use ``update-alternatives``, so that
-they may be installed together. If ``update-alternatives`` is not used,
+general, filename) should generally use ``update-alternatives`` so that
+they can be installed together. If ``update-alternatives`` is not used,
 then each package must use ``Conflicts`` to ensure that other packages
 are removed. (In this case, it may be appropriate to specify a conflict
 against earlier versions of something that previously did not use
diff --git a/policy/ch-controlfields.rst b/policy/ch-controlfields.rst
index 0d7a3e9..8d43130 100644
--- a/policy/ch-controlfields.rst
+++ b/policy/ch-controlfields.rst
@@ -72,7 +72,7 @@ Whitespace must not appear inside names (of packages, 
architectures,
 files or anything else) or version numbers, or between the characters of
 multi-character version relationships.
 
-The presence and purpose of a field, and the syntax of its value may
+The presence and purpose of a field, and the syntax of its value, may
 differ between types of control files.
 
 Field names are not case-sensitive, but it is usual to capitalize the
@@ -553,7 +553,7 @@ The three components here are:
 ``epoch``
 This is a single (generally small) unsigned integer. It may be
 omitted, in which case zero is assumed. If it is omitted then the
-``upstream_version`` may not contain any colons.
+``upstream_version`` must not contain any colons.
 
 Epochs can help when the upstream version numbering scheme
 changes, but they must be used with care.  You should not change
@@ -572,19 +572,19 @@ The three components here are:
 respect to the ``upstream_version`` is described below. The
 ``upstream_version`` portion of the version number is mandatory.
 
-The ``upstream_version`` may contain only alphanumerics [#]_ and
+The ``upstream_version`` must contain only alphanumerics [#]_ and
 the characters ``.`` ``+`` ``-`` ``~`` (full stop, plus, hyphen,
 tilde) and should start with a digit. If there is no
 ``debian_revision`` then hyphens are not allowed.
 
 ``debian_revision``
 This part of the version number specifies the version of the Debian
-package based on the upstream version. It may contain only
+package based on the upstream version. It must contain only
 alphanumerics and the characters ``+`` ``.`` ``~`` (plus, full stop,
 tilde) and is compared in the same way as the ``upstream_version`` is.
 
 It is optional; if it isn't present then the ``upstream_version``
-may not contain a hyphen. This format represents the case where a
+must not contain a hyphen. This format represents the case where a
 piece of software was written specifically 

Bug#944920: Revise terminology used to specify requirements

[Russ Allbery]

Sean Whitton  writes:
> On Sun 17 Nov 2019 at 05:48PM -08, Russ Allbery wrote:

>>  is being used.) You must not include the ``/etc/rcn.d`` directories
>> -themselves in the archive either. (Only the ``sysvinit`` package may do
>> -so.)
>> +themselves in the archive either. (Only the ``init-system-helpers``
>> +package may do so.)

> Likewise, isn't this a semantic change?

This is, but I think it's a bookkeeping change.  Those directories are in
init-system-helpers rather than sysvinit in the archive right now, and
that clearly shouldn't be a policy violation.

Ideally it should probably be in a separate commit, but in looking at this
again I also want to change "may do so" to "is permitted to do so."  I can
break it out if needed, but I'd rather commit it as part of this set of
changes (and will document it separately in debian/changelog; I don't
think it warrants adding to upgrading-checklist since it only affects one
set of maintainers who already know).

>> @@ -797,14 +798,13 @@ the upstream tarball.  In order to satisfy the DFSG 
>> for packages in
>>  2. include a copy of the sources in the ``debian/missing-sources``
>> directory.
>>
>> -There is an optional convention to organise the contents of
>> -``debian/missing-sources`` in the following way.  For a sourceless
>> -file ``foo`` in the subdirectory ``bar`` of the upstream tarball,
>> -where the source of ``foo`` has extension ``baz``, the source is to be
>> -located at ``debian/missing-sources/bar/foo.baz``.  For example,
>> -according to this convention, the C source code of an executable
>> -``checksum/util`` is to be located at
>> -``debian/missing-sources/checksum/util.c``.
>> +Package maintainers are encouraged to use the following convention to
>> +organize the contents of ``debian/missing-sources``: for a sourceless file
>> +``foo`` in the subdirectory ``bar`` of the upstream tarball, where the
>> +source of ``foo`` has extension ``baz``, the source is to be located at
>> +``debian/missing-sources/bar/foo.baz``. For example, according to this
>> +convention, the C source code of an executable ``checksum/util`` would be
>> +located at ``debian/missing-sources/checksum/util.c``.

> I don't think this should be strengthened to something Policy
> encourages, or if it should, we should discuss it in a separate bug.  So
> I'd like to suggest using none of the magic words in this paragraph,
> just starting it with "There is a convention to organise ..."

I think this is a change where the distinction between optional and
encouraged is valuable and this would have been written as encouraged if
we had that concept.  There's not much point in a convention unless we
advise maintainers to follow it, and that seems like an appropriate use of
Policy advice.

Does that make sense?  I can revise it if you don't like it after that
explanation, but it feels perfect for "encourage."

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal for next steps for systemd-related policy

[Sam Hartman]

> "Sean" == Sean Whitton  writes:

Sean> Hello Russ,
Sean> On Sun 29 Dec 2019 at 10:47am -08, Russ Allbery wrote:

>> This is a tentative proposal for next steps from a Policy standpoint 
given
>> the result of .  I thought it
>> might be helpful to lay out a possible way to sequence the work.

Sean> Thank you for writing this.

>> 1. Downgrade the requirement to include an init script in a package with 
a
>> systemd unit to "encouraged."  This is the direct outcome of the GR, so
>> I think we should make this change before the next normative upload,
>> since there's no point in Policy being inconsistent with the GR result.
>> 
>> I think there's some discussion to be had here about whether the
>> correct level is "encouraged" or "optional."  I'd also like to revise
>> and merge my change that adds "encouraged" first, although if we decide
>> "optional" is correct, that sequencing is, well, optional.

Sean> Under the new description of these words in #944920, I think we would
Sean> have to use 'encouraged'.  That new text says that 'optional' is meant
Sean> to be used purely for clarificatory purposes, but incorporating the
Sean> result of the GR into Policy would not be a matter of clarifying
Sean> something else said in Policy, so far as I can tell.

Sean> I think it's useful for 'optional' to be reserved for its 
clarificatory
Sean> role.

>> 3. Start a discussion on debian-devel to see if we can come up with some
>> idea for how to declare dependencies on availability of system
>> services.
>> 
>> My thought process here is that while the GR permits packages to start
>> using systemd facilities directly, doing that without somehow declaring
>> that requirement in package metadata seems likely to cause bugs and
>> upgrade issues, so we should try to provide some better facilities.  I
>> think there's an obvious gap here where we need a mechanism to declare
>> a dependency on a system facility (as distinct from a package that may

I haven't been following the consensus around making service units more
recommended.
Ignoring that discussion, but folding in the GR:

Maintainers are recommended to install at least one of a service unit or
init script.
Maintainers are encouraged to install an service unit and may install an
init script.

But if you've gotten to a point where service units are recommended all
the time (no service unit but an init script is a bug) then: Maintainers
are recommended to install a service unit.  If maintainers do not
install a service unit, they are encouraged to install an init script;
in other situations installing an init script is optional.

That at least is my take on what Proposal B implies in the new policy
terminology.
BTW, I like the new terms!  Great work

Re: Proposal for next steps for systemd-related policy

[Russ Allbery]

Simon McVittie  writes:

> Do you mean "systemd features", or do you mean system services more
> generally?

I'm hopeful that we can solve the more general problem, or at least make
forward progress on it, since as you mention we've had this problem for
years and have worked around it in various ways.  This is primarily
bringing an existing problem to the fore.

One of the open questions that I have, and that I think we'll need to talk
about, is what the UI should look like.  I think it should be possible to
install a package that depends on a system facility that isn't available,
but we need to somehow warn people that the thing they're trying to do
won't work as-is, and ideally figure out some way for reasonable things to
happen in chroots, containers, and other restricted environments.

Most of the ways that I can tentatively imagine we could approach this
problem would require changes to dpkg, apt, and aptitude; I don't think
package dependencies, even used creatively, will quite work here.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal for next steps for systemd-related policy

[Simon McVittie]

On Sun, 29 Dec 2019 at 10:47:44 -0800, Russ Allbery wrote:
> 3. Start a discussion on debian-devel to see if we can come up with some
>idea for how to declare dependencies on availability of system
>services.

Do you mean "systemd features", or do you mean system services more
generally?

If the latter, we've had this class of bugs for years, particularly in
chroots, even before systemd existed: if a package relies on a system
service (perhaps most frequently the D-Bus system bus, but also things
like avahi-daemon, BlueZ, firewalld and nslcd), package dependencies
can ensure that the system service is *installed*, but cannot ensure
that it is *running*.

For example, I've seen this recently with packages adding build-time tests
that fail when run in a chroot (or a Docker container) that does not have
a D-Bus system bus running, because "Depends: dbus" isn't sufficient to
make that facility available in a chroot. Such tests need to either run
a mock system bus or be skipped gracefully when run by Debian buildds,
and if skipped at build-time they should preferably run as autopkgtests
instead. This is not unique to systemd: the same situation would exist
on pre-systemd Debian versions, or in Devuan (assuming Devuan builds in
a minimal chroot like Debian does).

I agree that having both systemd and sysvinit does make this situation
more problematic, because expectations for a bootable system with a
non-default init system like sysvinit are higher than expectations for
a chroot or Docker-style container with no init system at all.

autopkgtest documents the isolation-container restriction as implying
that services *can* be started, although it doesn't actually document
that restriction as implying that installed services *will* be
started. Amusingly, this means a Docker virtualization backend for
autopkgtest will probably be unable to advertise isolation-container,
because Docker containers usually don't run an init system like sysvinit
or systemd.

smcv

Re: Proposal for next steps for systemd-related policy

[Sean Whitton]

Hello,

On Sun 29 Dec 2019 at 04:36pm -08, Russ Allbery wrote:

> Right, what I think is in scope for Policy is advising packagers on which
> readiness signaling mechanism to use if upstream supports several.  If one
> is relatively new to packaging daemons, this may not be something on one's
> radar to look at, but there are substantial, if subtle, advantages to
> using Type=notify if upstream already supports it.  (Type=dbus is fine, of
> course, and we probably shouldn't doument that as well, although my guess
> is that things that register with D-Bus are more likely to already know
> about these subtleties.)

Yes, let's be sure to get this in.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Bug#944920: Revise terminology used to specify requirements

[Sean Whitton]

Hello,

On Sun 29 Dec 2019 at 11:20am -08, Russ Allbery wrote:

> Paul Gevers  writes:
>> On 21-11-2019 13:59, Paul Gevers wrote:
>
>>> [Disclaimer: the words below are as a member of the release team, but
>>> not necessarily those of the team. We haven't discussed this yet.]
>
>> We have had a discussion, and there were no objections against my vision
>> below.
>
>>> I can envision that if Policy carries such a summary list, our policy
>>> would mention the version of Policy it was based on, to make sure that
>>> Policy doesn't suddenly change what we as the RT agreed on.
>
>> So, yes, we would welcome the Policy to maintain a summary list that we
>> could reference. We already acknowledge that there will be items in the
>> Policy text that we balance differently for RC-ness, so there will be
>> exceptions maintained by us.
>
> Thanks, Paul!  This is now on me to compose this list, and I'll let you
> know when that's done.  Once that's complete, we can do a reconciliation.
> I'm inclined to downgrade Policy musts that the release team does not
> consider likely to be release-critical in the future, for instance.

Let's definitely reconsider those 'must' requirements in response to
this work, but let's not commit ourselves to the idea that it's always a
bug for the Release Team's conception of an RC bug, and Policy 'must'
requirements, to disagree.

The Release Team's conception of RC bugs, and the text of Policy, are
generated and updated by different processes, for different purposes.  I
think Debian benefits from that diversity of normative processes, and it
would harm that to try too hard to keep the output of the two processes
in perfect sync.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Proposal for next steps for systemd-related policy

[Sean Whitton]

Hello Russ,

On Sun 29 Dec 2019 at 10:47am -08, Russ Allbery wrote:

> This is a tentative proposal for next steps from a Policy standpoint given
> the result of .  I thought it
> might be helpful to lay out a possible way to sequence the work.

Thank you for writing this.

> 1. Downgrade the requirement to include an init script in a package with a
>systemd unit to "encouraged."  This is the direct outcome of the GR, so
>I think we should make this change before the next normative upload,
>since there's no point in Policy being inconsistent with the GR result.
>
>I think there's some discussion to be had here about whether the
>correct level is "encouraged" or "optional."  I'd also like to revise
>and merge my change that adds "encouraged" first, although if we decide
>"optional" is correct, that sequencing is, well, optional.

Under the new description of these words in #944920, I think we would
have to use 'encouraged'.  That new text says that 'optional' is meant
to be used purely for clarificatory purposes, but incorporating the
result of the GR into Policy would not be a matter of clarifying
something else said in Policy, so far as I can tell.

I think it's useful for 'optional' to be reserved for its clarificatory
role.

> 3. Start a discussion on debian-devel to see if we can come up with some
>idea for how to declare dependencies on availability of system
>services.
>
>My thought process here is that while the GR permits packages to start
>using systemd facilities directly, doing that without somehow declaring
>that requirement in package metadata seems likely to cause bugs and
>upgrade issues, so we should try to provide some better facilities.  I
>think there's an obvious gap here where we need a mechanism to declare
>a dependency on a system facility (as distinct from a package that may
>be installed but not running) such as tmpfiles.d, sysusers.d, the
>ability to rely on DynamicUser without creating a user for that purpose
>in maintainer scripts, systemd D-Bus facilities, socket activation, or
>so forth.
>
>This would be a way to provide better UI when someone on a sysvinit
>system tries to install a package that requires a systemd facility (via
>an upgrade scenario, for example).  It also exposes in the archive what
>facilities are blocking use of packages on non-systemd systems, and
>thus provides a prioritized list of work for anyone who is pursuing
>exploration per paragraph two of the GR.
>
>This obviously is going to require some hashing out, and may well
>require support from the dpkg, apt, and aptitude teams.

Indeed, the GR discussions made it clear that this is something we need.

> 4. Parallel to point 3, start fleshing out Policy recommendations for best
>practices for systemd units.  Some initial candidates for security
>hardening:
>
>- DynamicUser (*without* removing useradd, see below)
>- ProtectSystem
>- ProtectHome
>- PrivateTmp
>- PrivateDevices
>- SystemCallFilter
>
>We probably also want to recommend Type=notify where possible and
>Type=exec where not, over Type=forking, when the daemon supports that.
>OOMScoreAdjust may also be worth setting some policy around.  We
>should, of course, have an open discussion of other things that people
>would like to see documented as best practices.
>
>These are chosen because they only affect the unit file and don't add
>any additional assumptions on the availability of other services.

This would be very cool to have in Policy.

> 5. As the outcome of point 3 concludes, start documenting use of other
>desirable services that would allow simplification of Debian packages
>and allow package maintainers to use those services rather than the
>ones that Policy currently requires.  My personal priority list looks
>like (roughly in order):
>
>- DynamicUser without useradd of a system user
>- tmpfiles.d
>- sysusers.d
>- Timer units
>- Socket activation
>
>but we should have an open discussion of what other facilities people
>think Debian would benefit from.  I think we should prioritize the
>cases where it would be relatively straightforward for other init
>systems to provide the same functionality; for example, tmpfiles.d and
>sysusers.d can be supported by simply arranging to run the relevant
>systemd binaries at appropriate times.  DynamicUser is a bit harder,
>but I think the minimum functionality to let the package keep working
>could be added to start-stop-daemon or some wrapper that it execs.
>
>As time goes on, we'll get a better feel for how much work folks will
>be doing going forward on supporting other init systems, and thus on
>how quickly we should move versus giving them time to determine how
>they want to support equivalent functionality.