Re: Proposal for next steps for systemd-related policy

[Russ Allbery]

Sam Hartman  writes:

> I haven't been following the consensus around making service units more
> recommended.  Ignoring that discussion, but folding in the GR:

> Maintainers are recommended to install at least one of a service unit or
> init script.  Maintainers are encouraged to install an service unit and
> may install an init script.

> But if you've gotten to a point where service units are recommended all
> the time (no service unit but an init script is a bug) then: Maintainers
> are recommended to install a service unit.  If maintainers do not
> install a service unit, they are encouraged to install an init script;
> in other situations installing an init script is optional.

Thinking about this some more, and re-reading the text of the GR, this
sounds correct to me as well.  I'm not sure that Policy is currently
structured in such a way as to make saying this straightforward, but I'm
taking a look now and will create a bug with a patch for discussion and
review.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal for next steps for systemd-related policy

[Sam Hartman]

> "Sean" == Sean Whitton  writes:

Sean> Hello Russ,
Sean> On Sun 29 Dec 2019 at 10:47am -08, Russ Allbery wrote:

>> This is a tentative proposal for next steps from a Policy standpoint 
given
>> the result of .  I thought it
>> might be helpful to lay out a possible way to sequence the work.

Sean> Thank you for writing this.

>> 1. Downgrade the requirement to include an init script in a package with 
a
>> systemd unit to "encouraged."  This is the direct outcome of the GR, so
>> I think we should make this change before the next normative upload,
>> since there's no point in Policy being inconsistent with the GR result.
>> 
>> I think there's some discussion to be had here about whether the
>> correct level is "encouraged" or "optional."  I'd also like to revise
>> and merge my change that adds "encouraged" first, although if we decide
>> "optional" is correct, that sequencing is, well, optional.

Sean> Under the new description of these words in #944920, I think we would
Sean> have to use 'encouraged'.  That new text says that 'optional' is meant
Sean> to be used purely for clarificatory purposes, but incorporating the
Sean> result of the GR into Policy would not be a matter of clarifying
Sean> something else said in Policy, so far as I can tell.

Sean> I think it's useful for 'optional' to be reserved for its 
clarificatory
Sean> role.

>> 3. Start a discussion on debian-devel to see if we can come up with some
>> idea for how to declare dependencies on availability of system
>> services.
>> 
>> My thought process here is that while the GR permits packages to start
>> using systemd facilities directly, doing that without somehow declaring
>> that requirement in package metadata seems likely to cause bugs and
>> upgrade issues, so we should try to provide some better facilities.  I
>> think there's an obvious gap here where we need a mechanism to declare
>> a dependency on a system facility (as distinct from a package that may

I haven't been following the consensus around making service units more
recommended.
Ignoring that discussion, but folding in the GR:

Maintainers are recommended to install at least one of a service unit or
init script.
Maintainers are encouraged to install an service unit and may install an
init script.

But if you've gotten to a point where service units are recommended all
the time (no service unit but an init script is a bug) then: Maintainers
are recommended to install a service unit.  If maintainers do not
install a service unit, they are encouraged to install an init script;
in other situations installing an init script is optional.

That at least is my take on what Proposal B implies in the new policy
terminology.
BTW, I like the new terms!  Great work

Re: Proposal for next steps for systemd-related policy

[Russ Allbery]

Simon McVittie  writes:

> Do you mean "systemd features", or do you mean system services more
> generally?

I'm hopeful that we can solve the more general problem, or at least make
forward progress on it, since as you mention we've had this problem for
years and have worked around it in various ways.  This is primarily
bringing an existing problem to the fore.

One of the open questions that I have, and that I think we'll need to talk
about, is what the UI should look like.  I think it should be possible to
install a package that depends on a system facility that isn't available,
but we need to somehow warn people that the thing they're trying to do
won't work as-is, and ideally figure out some way for reasonable things to
happen in chroots, containers, and other restricted environments.

Most of the ways that I can tentatively imagine we could approach this
problem would require changes to dpkg, apt, and aptitude; I don't think
package dependencies, even used creatively, will quite work here.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal for next steps for systemd-related policy

[Simon McVittie]

On Sun, 29 Dec 2019 at 10:47:44 -0800, Russ Allbery wrote:
> 3. Start a discussion on debian-devel to see if we can come up with some
>idea for how to declare dependencies on availability of system
>services.

Do you mean "systemd features", or do you mean system services more
generally?

If the latter, we've had this class of bugs for years, particularly in
chroots, even before systemd existed: if a package relies on a system
service (perhaps most frequently the D-Bus system bus, but also things
like avahi-daemon, BlueZ, firewalld and nslcd), package dependencies
can ensure that the system service is *installed*, but cannot ensure
that it is *running*.

For example, I've seen this recently with packages adding build-time tests
that fail when run in a chroot (or a Docker container) that does not have
a D-Bus system bus running, because "Depends: dbus" isn't sufficient to
make that facility available in a chroot. Such tests need to either run
a mock system bus or be skipped gracefully when run by Debian buildds,
and if skipped at build-time they should preferably run as autopkgtests
instead. This is not unique to systemd: the same situation would exist
on pre-systemd Debian versions, or in Devuan (assuming Devuan builds in
a minimal chroot like Debian does).

I agree that having both systemd and sysvinit does make this situation
more problematic, because expectations for a bootable system with a
non-default init system like sysvinit are higher than expectations for
a chroot or Docker-style container with no init system at all.

autopkgtest documents the isolation-container restriction as implying
that services *can* be started, although it doesn't actually document
that restriction as implying that installed services *will* be
started. Amusingly, this means a Docker virtualization backend for
autopkgtest will probably be unable to advertise isolation-container,
because Docker containers usually don't run an init system like sysvinit
or systemd.

smcv

Re: Proposal for next steps for systemd-related policy

[Sean Whitton]

Hello,

On Sun 29 Dec 2019 at 04:36pm -08, Russ Allbery wrote:

> Right, what I think is in scope for Policy is advising packagers on which
> readiness signaling mechanism to use if upstream supports several.  If one
> is relatively new to packaging daemons, this may not be something on one's
> radar to look at, but there are substantial, if subtle, advantages to
> using Type=notify if upstream already supports it.  (Type=dbus is fine, of
> course, and we probably shouldn't doument that as well, although my guess
> is that things that register with D-Bus are more likely to already know
> about these subtleties.)

Yes, let's be sure to get this in.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Proposal for next steps for systemd-related policy

[Sean Whitton]

Hello Russ,

On Sun 29 Dec 2019 at 10:47am -08, Russ Allbery wrote:

> This is a tentative proposal for next steps from a Policy standpoint given
> the result of .  I thought it
> might be helpful to lay out a possible way to sequence the work.

Thank you for writing this.

> 1. Downgrade the requirement to include an init script in a package with a
>systemd unit to "encouraged."  This is the direct outcome of the GR, so
>I think we should make this change before the next normative upload,
>since there's no point in Policy being inconsistent with the GR result.
>
>I think there's some discussion to be had here about whether the
>correct level is "encouraged" or "optional."  I'd also like to revise
>and merge my change that adds "encouraged" first, although if we decide
>"optional" is correct, that sequencing is, well, optional.

Under the new description of these words in #944920, I think we would
have to use 'encouraged'.  That new text says that 'optional' is meant
to be used purely for clarificatory purposes, but incorporating the
result of the GR into Policy would not be a matter of clarifying
something else said in Policy, so far as I can tell.

I think it's useful for 'optional' to be reserved for its clarificatory
role.

> 3. Start a discussion on debian-devel to see if we can come up with some
>idea for how to declare dependencies on availability of system
>services.
>
>My thought process here is that while the GR permits packages to start
>using systemd facilities directly, doing that without somehow declaring
>that requirement in package metadata seems likely to cause bugs and
>upgrade issues, so we should try to provide some better facilities.  I
>think there's an obvious gap here where we need a mechanism to declare
>a dependency on a system facility (as distinct from a package that may
>be installed but not running) such as tmpfiles.d, sysusers.d, the
>ability to rely on DynamicUser without creating a user for that purpose
>in maintainer scripts, systemd D-Bus facilities, socket activation, or
>so forth.
>
>This would be a way to provide better UI when someone on a sysvinit
>system tries to install a package that requires a systemd facility (via
>an upgrade scenario, for example).  It also exposes in the archive what
>facilities are blocking use of packages on non-systemd systems, and
>thus provides a prioritized list of work for anyone who is pursuing
>exploration per paragraph two of the GR.
>
>This obviously is going to require some hashing out, and may well
>require support from the dpkg, apt, and aptitude teams.

Indeed, the GR discussions made it clear that this is something we need.

> 4. Parallel to point 3, start fleshing out Policy recommendations for best
>practices for systemd units.  Some initial candidates for security
>hardening:
>
>- DynamicUser (*without* removing useradd, see below)
>- ProtectSystem
>- ProtectHome
>- PrivateTmp
>- PrivateDevices
>- SystemCallFilter
>
>We probably also want to recommend Type=notify where possible and
>Type=exec where not, over Type=forking, when the daemon supports that.
>OOMScoreAdjust may also be worth setting some policy around.  We
>should, of course, have an open discussion of other things that people
>would like to see documented as best practices.
>
>These are chosen because they only affect the unit file and don't add
>any additional assumptions on the availability of other services.

This would be very cool to have in Policy.

> 5. As the outcome of point 3 concludes, start documenting use of other
>desirable services that would allow simplification of Debian packages
>and allow package maintainers to use those services rather than the
>ones that Policy currently requires.  My personal priority list looks
>like (roughly in order):
>
>- DynamicUser without useradd of a system user
>- tmpfiles.d
>- sysusers.d
>- Timer units
>- Socket activation
>
>but we should have an open discussion of what other facilities people
>think Debian would benefit from.  I think we should prioritize the
>cases where it would be relatively straightforward for other init
>systems to provide the same functionality; for example, tmpfiles.d and
>sysusers.d can be supported by simply arranging to run the relevant
>systemd binaries at appropriate times.  DynamicUser is a bit harder,
>but I think the minimum functionality to let the package keep working
>could be added to start-stop-daemon or some wrapper that it execs.
>
>As time goes on, we'll get a better feel for how much work folks will
>be doing going forward on supporting other init systems, and thus on
>how quickly we should move versus giving them time to determine how
>they want to support equivalent functionality.

Re: Proposal for next steps for systemd-related policy

[Matthew Vernon]

Hi,

I broadly agree with what you said.

Russ Allbery  writes:

> 2. Do nothing further before January 6th.  It's still the holidays, and
>subsequent steps are going to require some discussion, and I think it's
>worth taking a breath.

This, particularly.

>My thought process here is that while the GR permits packages to start
>using systemd facilities directly, doing that without somehow declaring
>that requirement in package metadata seems likely to cause bugs and
>upgrade issues, so we should try to provide some better facilities.  I
>think there's an obvious gap here where we need a mechanism to declare
>a dependency on a system facility (as distinct from a package that may
>be installed but not running) such as tmpfiles.d, sysusers.d, the
>ability to rely on DynamicUser without creating a user for that purpose
>in maintainer scripts, systemd D-Bus facilities, socket activation, or
>so forth.

This is a sound point; systemd provides an increasing range of
facilities, and it'd be good to keep track of which of them are being
used where and how.

>As time goes on, we'll get a better feel for how much work folks will
>be doing going forward on supporting other init systems, and thus on
>how quickly we should move versus giving them time to determine how
>they want to support equivalent functionality.

I suspect such effort is going to be at a low ebb in the immediate
aftermath of this GR, so this will need some care.

Regards,

Matthew

-- 
"At least you know where you are with Microsoft."
"True. I just wish I'd brought a paddle."
http://www.debian.org

Re: Proposal for next steps for systemd-related policy

[Russ Allbery]

Russ Allbery  writes:

> Right, what I think is in scope for Policy is advising packagers on
> which readiness signaling mechanism to use if upstream supports several.
> If one is relatively new to packaging daemons, this may not be something
> on one's radar to look at, but there are substantial, if subtle,
> advantages to using Type=notify if upstream already supports it.
> (Type=dbus is fine, of course, and we probably shouldn't doument that as
> well, although my guess is that things that register with D-Bus are more
> likely to already know about these subtleties.)

That was supposed to be "we probably should document that as well."  I
think I edited that sentence too much.  Sorry about the confusion.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal for next steps for systemd-related policy

[Russ Allbery]

Simon McVittie  writes:
> On Sun, 29 Dec 2019 at 10:47:44 -0800, Russ Allbery wrote:

>>We probably also want to recommend Type=notify where possible and
>>Type=exec where not, over Type=forking, when the daemon supports
>>that.

> I'm not sure what you mean by "where possible" - it'll usually be
> possible to implement Type=notify (even if a libsystemd dependency is
> unacceptable or not feasible in the implementation language, the
> protocol is designed to be reimplementable) but it won't always be
> useful.

I mean something relatively modest.  If upstream already supports
systemd's notification protocol, I think we should recommend that as the
best choice.  (I'm dubious about asking Debian packagers to add support.)

> Type=exec doesn't provide a way for the daemon to signal that it has
> opened listening sockets or done whatever else is necessary for units
> that depend on it to be able to start without error, so daemons that
> might be depended on by other units will normally need to use either
> Type=notify, Type=dbus or Type=forking, then make sure they do basic
> setup (listen on sockets, etc.) before they notify, request their D-Bus
> names or do the traditional double-fork daemonization trick (whichever
> is appropriate for their Type).

Ah, yes, it's a good point that Type=forking when properly implemented is
somewhat better than Type=exec on getting dependency ordering correct.
But this is the sort of nuance that I think we should document somewhere.

Getting Type=forking right is surprisingly complicated.  The best
documentation for how to do that I've seen is, ironically, in the systemd
documentation, in daemon(7).  If the daemon double-forks and exits and
then binds to network sockets and so forth, which is common, the
advantages over Type=exec are basically lost.

> However, I don't think this is necessarily in-scope for Policy?
> Implementing readiness-signalling if required seems like it might be the
> sort of "don't be buggy" requirement that shouldn't necessarily need to
> be said explicitly.

Right, what I think is in scope for Policy is advising packagers on which
readiness signaling mechanism to use if upstream supports several.  If one
is relatively new to packaging daemons, this may not be something on one's
radar to look at, but there are substantial, if subtle, advantages to
using Type=notify if upstream already supports it.  (Type=dbus is fine, of
course, and we probably shouldn't doument that as well, although my guess
is that things that register with D-Bus are more likely to already know
about these subtleties.)

> I say "when run from a systemd service" so that services like
> dbus-daemon meet this recommendation - by default it double-forks for
> compatibility with LSB-style init scripts and its own historical
> behaviour, but the systemd unit dbus.service runs "dbus-daemon --nofork"
> which disables the double-fork code path, and I think that's a perfectly
> good implementation of what we want.

Yes, completely agreed.

>>- DynamicUser without useradd of a system user

> One warning for this one is that users referred to in dbus-daemon
> system.d/*.conf snippets, typically ,
> currently need to be pre-created by sysusers.d, useradd or equivalent
> (because dbus-daemon resolves usernames to uids during its own
> startup/reload, to populate internal data structures that are all
> designed in terms of uids). Other references to users in non-D-Bus
> configuration might well behave similarly.

Right, there are a lot of subtleties for when DynamicUser can and can't be
used that we'll need to hash out and document.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposal for next steps for systemd-related policy

[Simon McVittie]

On Sun, 29 Dec 2019 at 10:47:44 -0800, Russ Allbery wrote:
>We probably also want to recommend Type=notify where possible and
>Type=exec where not, over Type=forking, when the daemon supports that.

I'm not sure what you mean by "where possible" - it'll usually be
possible to implement Type=notify (even if a libsystemd dependency is
unacceptable or not feasible in the implementation language, the protocol
is designed to be reimplementable) but it won't always be useful.

Type=exec doesn't provide a way for the daemon to signal that it has
opened listening sockets or done whatever else is necessary for units that
depend on it to be able to start without error, so daemons that might be
depended on by other units will normally need to use either Type=notify,
Type=dbus or Type=forking, then make sure they do basic setup (listen on
sockets, etc.) before they notify, request their D-Bus names or do the
traditional double-fork daemonization trick (whichever is appropriate
for their Type). However, I don't think this is necessarily in-scope
for Policy? Implementing readiness-signalling if required seems like
it might be the sort of "don't be buggy" requirement that shouldn't
necessarily need to be said explicitly.

I think perhaps a better recommendation might be that it's preferred for
systemd services that run daemons to run them as a foreground process
(without forking or "daemonizing" during startup), with a footnote or
parenthetical note saying that this recommendation is incompatible with
Type=forking, which should therefore only be used if readiness-signalling
is required and alternatives like the preferred Type=notify or Type=dbus
are unavailable?

I say "when run from a systemd service" so that services like dbus-daemon
meet this recommendation - by default it double-forks for compatibility
with LSB-style init scripts and its own historical behaviour, but the
systemd unit dbus.service runs "dbus-daemon --nofork" which disables the
double-fork code path, and I think that's a perfectly good implementation
of what we want.

>- DynamicUser without useradd of a system user

One warning for this one is that users referred to in dbus-daemon
system.d/*.conf snippets, typically ,
currently need to be pre-created by sysusers.d, useradd or equivalent
(because dbus-daemon resolves usernames to uids during its own
startup/reload, to populate internal data structures that are all designed
in terms of uids). Other references to users in non-D-Bus configuration
might well behave similarly.

smcv

Proposal for next steps for systemd-related policy

[Russ Allbery]

Hi all,

This is a tentative proposal for next steps from a Policy standpoint given
the result of .  I thought it
might be helpful to lay out a possible way to sequence the work.

1. Downgrade the requirement to include an init script in a package with a
   systemd unit to "encouraged."  This is the direct outcome of the GR, so
   I think we should make this change before the next normative upload,
   since there's no point in Policy being inconsistent with the GR result.

   I think there's some discussion to be had here about whether the
   correct level is "encouraged" or "optional."  I'd also like to revise
   and merge my change that adds "encouraged" first, although if we decide
   "optional" is correct, that sequencing is, well, optional.

2. Do nothing further before January 6th.  It's still the holidays, and
   subsequent steps are going to require some discussion, and I think it's
   worth taking a breath.

3. Start a discussion on debian-devel to see if we can come up with some
   idea for how to declare dependencies on availability of system
   services.

   My thought process here is that while the GR permits packages to start
   using systemd facilities directly, doing that without somehow declaring
   that requirement in package metadata seems likely to cause bugs and
   upgrade issues, so we should try to provide some better facilities.  I
   think there's an obvious gap here where we need a mechanism to declare
   a dependency on a system facility (as distinct from a package that may
   be installed but not running) such as tmpfiles.d, sysusers.d, the
   ability to rely on DynamicUser without creating a user for that purpose
   in maintainer scripts, systemd D-Bus facilities, socket activation, or
   so forth.

   This would be a way to provide better UI when someone on a sysvinit
   system tries to install a package that requires a systemd facility (via
   an upgrade scenario, for example).  It also exposes in the archive what
   facilities are blocking use of packages on non-systemd systems, and
   thus provides a prioritized list of work for anyone who is pursuing
   exploration per paragraph two of the GR.

   This obviously is going to require some hashing out, and may well
   require support from the dpkg, apt, and aptitude teams.

4. Parallel to point 3, start fleshing out Policy recommendations for best
   practices for systemd units.  Some initial candidates for security
   hardening:

   - DynamicUser (*without* removing useradd, see below)
   - ProtectSystem
   - ProtectHome
   - PrivateTmp
   - PrivateDevices
   - SystemCallFilter

   We probably also want to recommend Type=notify where possible and
   Type=exec where not, over Type=forking, when the daemon supports that.
   OOMScoreAdjust may also be worth setting some policy around.  We
   should, of course, have an open discussion of other things that people
   would like to see documented as best practices.

   These are chosen because they only affect the unit file and don't add
   any additional assumptions on the availability of other services.

5. As the outcome of point 3 concludes, start documenting use of other
   desirable services that would allow simplification of Debian packages
   and allow package maintainers to use those services rather than the
   ones that Policy currently requires.  My personal priority list looks
   like (roughly in order):

   - DynamicUser without useradd of a system user
   - tmpfiles.d
   - sysusers.d
   - Timer units
   - Socket activation

   but we should have an open discussion of what other facilities people
   think Debian would benefit from.  I think we should prioritize the
   cases where it would be relatively straightforward for other init
   systems to provide the same functionality; for example, tmpfiles.d and
   sysusers.d can be supported by simply arranging to run the relevant
   systemd binaries at appropriate times.  DynamicUser is a bit harder,
   but I think the minimum functionality to let the package keep working
   could be added to start-stop-daemon or some wrapper that it execs.

   As time goes on, we'll get a better feel for how much work folks will
   be doing going forward on supporting other init systems, and thus on
   how quickly we should move versus giving them time to determine how
   they want to support equivalent functionality.

-- 
Russ Allbery (r...@debian.org)