Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 18 Sep 2022 20:52:23 -0700 Russ Allbery  wrote:
> Ansgar  writes:
> 
> > I would like to recommend packages to use tmpfiles.d(5) to manage
> > creating directories in locations such as /var or /etc instead of
> > maintainer scripts.
> 
> > It could also be used to create trivial files below /var that
should be
> > recreated if deleted by accident (such as atd's sequence number
which is
> > currently created by the maintainer script).
> 
> Hi all,
> 
> Here's where I think we currently are with this proposal:
> 
> * There should no longer be any blocker to adding a dependency on
>   systemd-tmpfiles since systemd-standalone-tmpfiles exists.
> 
> * So far as I can tell, dh_installtmpfiles adds the correct stanza to
the
>   postinst script for a service to run systemd-tmpfiles after the
package
>   has installed its tmpfiles.d fragment.  I believe that addresses
>   creating these files on installation on systems without any init
system?
>   (Obviously if tmpfs file systems are subsequently reset on such a
>   system, there is nothing in place to recreate these tmp files, but
I
>   think that's expected and not something we can address?)
> 
> * Guillem plans to add support to dpkg to register these files
properly as
>   package-associated files and also handle creation of the files. 
This is
>   great, and I am 100% in favor of it.  I don't think that blocks
this
>   change; to the contrary, I think this change is a good incremental
step
>   towards that world, since it moves temporary file creation out of
>   maintainer scripts into a declarative syntax.  dpkg can then either
>   consume the same syntax or packages can later convert that syntax
to
>   whatever dpkg uses, depending on how the dpkg implementation works,
>   which will be a simple and easy-to-detect migration that Lintian
can
>   diagnose.
> 
> Therefore, I don't see anything blocking adopting this as a policy
now,
> and it seems like an obviously good idea to me.
> 
> Am I missing something?  If not, I think the next step is for someone
to
> propose wording.

I've done an initial attempt to define the wording, although I'm sure
it will need quite a few changes. Attached as a patch, and also
available on Salsa:

https://salsa.debian.org/bluca/policy/-/commits/tmpfiles

Happy to move/reword/change/enhance as required.

> In order to handle installations that have no init system, I think we
may
> have to require that the dependency be listed as:
> 
> systemd-standalone-tmpfiles | systemd-tmpfiles
> 
> to avoid trying to install systemd into an init-free chroot.  Maybe I
have
> that wrong?  Currently, dh_installtmpfiles doesn't add a dependency
at
> all, probably because it assumes that the package will use a fallback
to
> create the files in cases without an init system?  Either way, we
should
> address this in the Policy wording, and then encode that in
> dh_installtmpfiles if needed.

For now I've kept only a mention of the 'systemd-tmpfiles' virtual
package. As maintainers we would really prefer if the 'main'
implementation is pulled in whenever possible. When a minimal
installation is desired (ie, a minbase), it is possible to manually
specify the -standalone variant.

This was a controversial point last year, see:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017441

We could even decide that no dependency is added at all by dh, and
instead the build tool needs to decide if it's building an image where
tmpfiles snippets need to be ran, and if so pull in the preferred
alternative.

-- 
Kind regards,
Luca Boccassi
From  Mon Sep 17 00:00:00 2001
From: Luca Boccassi 
Date: Tue, 9 May 2023 01:38:13 +0100
Subject: [PATCH] Define tmpfiles.d interface and usage

---
 locales/ja/LC_MESSAGES/ch-files.po|  97 --
 .../ja/LC_MESSAGES/ch-maintainerscripts.po| 293 ++
 policy/ch-files.rst   |  39 +++
 policy/ch-maintainerscripts.rst   |   6 +
 4 files changed, 288 insertions(+), 147 deletions(-)

diff --git a/locales/ja/LC_MESSAGES/ch-files.po b/locales/ja/LC_MESSAGES/ch-files.po
index c241418..ec9d329 100644
--- a/locales/ja/LC_MESSAGES/ch-files.po
+++ b/locales/ja/LC_MESSAGES/ch-files.po
@@ -10,7 +10,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: Debian Policy Manual 4.1.6.0\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2021-08-17 20:06-0700\n"
+"POT-Creation-Date: 2023-05-09 01:31+0100\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME \n"
 "Language-Team: LANGUAGE \n"
@@ -963,7 +963,60 @@ msgid ""
 " to ASCII when it is possible to do so."
 msgstr ""
 
-#: ../../ch-files.rst:726
+#: ../../ch-files.rst:728
+msgid "tmpfiles.d"
+msgstr ""
+
+#: ../../ch-files.rst:730
+msgid ""
+"Packages might need additional files or directories to implement their "
+"functionality. Directories that are located under ``/var/`` or ``/etc/``,"
+" and files that are located under ``/var/``, should not be created "

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 08 May 2023 14:14:30 -0700 Russ Allbery  wrote:
> Guillem Jover  writes:
> > On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:
> 
> >> […] I suspect Policy should say something stronger and more
general,
> >> namely that no package in Debian should divert a file from another
> >> package unless this is arranged cooperatively between the packages
to
> >> solve some specific (unusual) problem.
> 
> > ,--- §3.9 ---
> > You should not use "dpkg-divert" on a file belonging to another
> > package without consulting the maintainer of that package first.
When
> > adding or removing diversions, package maintainer scripts must
provide
> > the "--package" flag to "dpkg-divert" and must not use "--local".
> > `---
> 
> Oh, thank you!  I had completely forgotten that we said something
about
> this under maintainer scripts.
> 
> That doesn't entirely cover this case (because systemd and udev may
not be
> "that package" in this sense), but it covers much of the general
case.

Would you like me to reword/move the new snippet?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Guillem Jover  writes:
> On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:

>> […] I suspect Policy should say something stronger and more general,
>> namely that no package in Debian should divert a file from another
>> package unless this is arranged cooperatively between the packages to
>> solve some specific (unusual) problem.

> ,--- §3.9 ---
> You should not use "dpkg-divert" on a file belonging to another
> package without consulting the maintainer of that package first. When
> adding or removing diversions, package maintainer scripts must provide
> the "--package" flag to "dpkg-divert" and must not use "--local".
> `---

Oh, thank you!  I had completely forgotten that we said something about
this under maintainer scripts.

That doesn't entirely cover this case (because systemd and udev may not be
"that package" in this sense), but it covers much of the general case.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Guillem Jover]

On Mon, 2023-05-08 at 08:48:49 -0700, Russ Allbery wrote:
> […] I suspect Policy should say something stronger and more general,
> namely that no package in Debian should divert a file from another package
> unless this is arranged cooperatively between the packages to solve some
> specific (unusual) problem.

,--- §3.9 ---
You should not use "dpkg-divert" on a file belonging to another
package without consulting the maintainer of that package first. When
adding or removing diversions, package maintainer scripts must provide
the "--package" flag to "dpkg-divert" and must not use "--local".
`---

> To me, this feels similar to the case of one
> package modifying the configuration files of another package, where we
> explicitly prohibit one package modifying the configuration of another
> package except through an interface provided by the package whose
> configuration is being modified.

,--- §10.7.4 ---
If the configuration file cannot be shared as described above, the
packages must be marked as conflicting with each other. Two packages
that specify the same file as a "conffile" must conflict. This is an
instance of the general rule about not sharing files. Neither
alternatives nor diversions are likely to be appropriate in this case;
in particular, "dpkg" does not handle diverted "conffile"s well.
`---

Regards,
Guillem

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Sean Whitton  writes:
> On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:

>> In other words, dpkg-divert is primarily for local administrators,
>> non-Policy-compliant local packages that are doing unusual things, and
>> the occasional rare problem that requires special coordination between
>> packages, not something that Debian packages should be doing to other
>> packages without explicit coordination.

>> The rule about systemd and udev files doesn't entirely fall out of that
>> statement,

> Hmm, could you explain why?

It didn't fall out of the above statement because the systemd unit file
may not be shipped with the systemd package, but by some other random
package, so you could have an explicit coordination with the package that
provides the unit file but still be doing something that the systemd
maintainers don't want to support.

I think it does fall out of the somewhat squishier statement that you
shouldn't use diversions when there's some other available mechanism to
accomplish the same goal.

> I don't mean to dismiss the significant impact on the systemd
> maintainers that's being claimed, but specifically calling out udev and
> systemd configuration files seems strangely specific, for Policy, to me.

I think they're a special case of the general rule that if there's some
mechanism other than diversions to do what you want, you should use it
instead, but it's such a common special case that we should call it out
explicitly, particularly since a lot of people right now don't seem to
know about masking or drop-ins.  So in other words, I think I basically
agree with this, but I think it's worth spending some words on systemd and
udev, more as a communication strategy than anything else.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 20:00, Sean Whitton  wrote:
>
> Hello,
>
> On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:
>
> > In other words, dpkg-divert is primarily for local administrators,
> > non-Policy-compliant local packages that are doing unusual things, and the
> > occasional rare problem that requires special coordination between
> > packages, not something that Debian packages should be doing to other
> > packages without explicit coordination.
> >
> > The rule about systemd and udev files doesn't entirely fall out of that
> > statement,
>
> Hmm, could you explain why?
>
> > so we can still include a specific statement about them, noting that
> > drop-ins and masking make dpkg-divert unnecessary (and those
> > facilities produce better tool behavior) and therefore it should not
> > be used.
> >
> > So, ideally, the way I'd prefer to move forward is for us to add a new
> > section to the main Policy manual on diversions (probably 10.11), document
> > that this is primarily a tool for local administrators and local packages
> > to override the behavior of Debian, and that its use between Debian
> > packages should be rare, should involve coordination between the packages,
> > and should only be used to solve problems that cannot be handled through
> > other facilities such as alternatives or package-specific tools like
> > systemd's support for drop-ins and masking.  And then explicitly call out
> > systemd and udev configuration as cases where dpkg-divert should not be
> > used, alongside conffiles and critical system files.
>
> I don't mean to dismiss the significant impact on the systemd
> maintainers that's being claimed, but specifically calling out udev and
> systemd configuration files seems strangely specific, for Policy, to me.
>
> The general prohibition seems justified, and then I would be inclined to
> use the systemd and udev configuration files case as an example, which
> explains and justifies the existence of the rule.  So instead of
>
> [don't use dpkg-divert on other packages's files]  In particular,
> packages should not divert systemd and udev configuration files.
> Doing so leads to confusing command output.
>
> it seems more natural to me to do something like
>
> [don't use dpkg-divert on other packages's files]  For example,
> diverting another package's systemd units or udev configuration
> files can result in confusing command output.
>
> If this was a mistake that maintainers seemed to habitually make, the
> former would make sense, but so far I think we have just one or a few
> concrete examples, and so the correct thing to do seems to me to be to
> add normative language for only the general case.

Hi,

The specific difference, for which I think an explicit call out is
needed, is because these config files are shipped by some packages but
are not used _by_ them, they are consumed by systemd (or udev, or
kmod, etc). Specifically, if package A ships a.service, and package B
overrides it, even if the maintainers of A and B agree, that's still
not good enough for me, as they are really affecting systemd, which is
the consumer and the provider of the interface they are using, and
ultimately the first port of call for bug reports. This is especially
true for udev.

So in my latest revision of the patch, the general rule is as
requested by Russ and as you mention it, but there is an explicit,
stricter rule to cover this case, which is important to me. Policy
calls out core component software in many places, such as dpkg, and
systemd is already mentioned in other parts of the policy, so it did
not seem too far-fetched to me.

I am of course open to re-wording, adjustments, etc as deemed necessary.

Changeset at: https://salsa.debian.org/bluca/policy/-/tree/systemd_overrides

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sean Whitton]

Hello,

On Mon 08 May 2023 at 08:48AM -07, Russ Allbery wrote:

> In other words, dpkg-divert is primarily for local administrators,
> non-Policy-compliant local packages that are doing unusual things, and the
> occasional rare problem that requires special coordination between
> packages, not something that Debian packages should be doing to other
> packages without explicit coordination.
>
> The rule about systemd and udev files doesn't entirely fall out of that
> statement,

Hmm, could you explain why?

> so we can still include a specific statement about them, noting that
> drop-ins and masking make dpkg-divert unnecessary (and those
> facilities produce better tool behavior) and therefore it should not
> be used.
>
> So, ideally, the way I'd prefer to move forward is for us to add a new
> section to the main Policy manual on diversions (probably 10.11), document
> that this is primarily a tool for local administrators and local packages
> to override the behavior of Debian, and that its use between Debian
> packages should be rare, should involve coordination between the packages,
> and should only be used to solve problems that cannot be handled through
> other facilities such as alternatives or package-specific tools like
> systemd's support for drop-ins and masking.  And then explicitly call out
> systemd and udev configuration as cases where dpkg-divert should not be
> used, alongside conffiles and critical system files.

I don't mean to dismiss the significant impact on the systemd
maintainers that's being claimed, but specifically calling out udev and
systemd configuration files seems strangely specific, for Policy, to me.

The general prohibition seems justified, and then I would be inclined to
use the systemd and udev configuration files case as an example, which
explains and justifies the existence of the rule.  So instead of

[don't use dpkg-divert on other packages's files]  In particular,
packages should not divert systemd and udev configuration files.
Doing so leads to confusing command output.

it seems more natural to me to do something like

[don't use dpkg-divert on other packages's files]  For example,
diverting another package's systemd units or udev configuration
files can result in confusing command output.

If this was a mistake that maintainers seemed to habitually make, the
former would make sense, but so far I think we have just one or a few
concrete examples, and so the correct thing to do seems to me to be to
add normative language for only the general case.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 18:31, Russ Allbery  wrote:
>
> Sam Hartman  writes:
>
> > In cases where the change being made is permitted by policy, I think you
> > have made a compelling case to vastly prefer the native systemd and udev
> > mechanisms to dpkg-divert.  I don't think that your case is strong
> > enough to forbid dpkg-divert.
>
> > As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> > It just gives surprising results to someone coming from the systemd
> > universe.
>
> I think (and please correct me, Luca, if I'm wrong) that Luca is trying to
> declare, on behalf of the systemd maintainers, that this method of
> disabling a systemd configuration file is unsupported and may not work.
> To me that does warrant a Policy "should not" even if in specific
> situations it works currently, because it implies that this approach is
> fragile and may well stop working or cause bugs in the future with no
> further notification (since that's essentially the definition of
> unsupported).
>
> Also, even apart from that, I personally would support a Policy "should
> not" for using diversions in any case where another mechanism is available
> that solves the same problem.  Diversions are a low-level tool with a lot
> of sharp edges and should be used very carefully and avoided when there is
> some other approach available.

Yes, that is precisely what I meant.

I have applied your suggestions and added a 10.10 chapter that has a
generic 'should' rule, and a more strict 'must' rule for systemd
files. I am pushing to the same branch, if you prefer me to attach
directly to the mail too let me know and I can do that, otherwise the
branch is on Salsa:

https://salsa.debian.org/bluca/policy/-/tree/systemd_overrides

I kept the wording for the dpkg-divert appendix too, because people
are bound to find it when googling, so as long as it's there I think
it should mention this too. Once it gets removed/reworked it can go.

On request from Marco, the kmod maintainer, I've also added the same
constraint for modprobe.d/ files, for exactly the same reason, as kmod
supports overrides, drop-ins and so on. I've kept it as a separate
commit on top of the other changes, given I am not involved with kmod
directly.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

Sam Hartman  writes:

> In cases where the change being made is permitted by policy, I think you
> have made a compelling case to vastly prefer the native systemd and udev
> mechanisms to dpkg-divert.  I don't think that your case is strong
> enough to forbid dpkg-divert.

> As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> It just gives surprising results to someone coming from the systemd
> universe.

I think (and please correct me, Luca, if I'm wrong) that Luca is trying to
declare, on behalf of the systemd maintainers, that this method of
disabling a systemd configuration file is unsupported and may not work.
To me that does warrant a Policy "should not" even if in specific
situations it works currently, because it implies that this approach is
fragile and may well stop working or cause bugs in the future with no
further notification (since that's essentially the definition of
unsupported).

Also, even apart from that, I personally would support a Policy "should
not" for using diversions in any case where another mechanism is available
that solves the same problem.  Diversions are a low-level tool with a lot
of sharp edges and should be used very carefully and avoided when there is
some other approach available.

> But consider a package that diverts several resources, several of them
> systemd and several of them not systemd.  The maintainer might
> legitimately want to use the same mechanism for all the
> overriding/masking so that systemd resources and non-systemd resources
> were handled the same.

I'm not really convinced by this the way that I would be if we were
talking about alternatives.  With alternatives, the slave links mean that
managing a group of similar changes together is important, but dpkg-divert
has no equivalent and every diversion already has to be maintained
separately.  Given that, I think the burden of asking people to use
masking instead of diversion for systemd configuration files is a fairly
minor request, so I weigh the problems on the systemd side higher than
what feels like a modest convenience.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 16:48, Russ Allbery  wrote:
>
> I think your X-Debbugs-Cc was syntactically invalid and thus didn't work.
> I manually added in the other addresses in this reply.

Thanks - email is hard!

> Luca Boccassi  writes:
>
> > It has come to my attention that there is one package in Debian using
> > dpkg-divert to mask a systemd configuration file (an udev rule).
> > Speaking as one of the maintainers, both upstream and downstream, I find
> > this greatly undesirable for several reasons that I will outline
> > later. Hence I would like to propose explicitly mentioning that
> > dpkg-divert must not be used for systemd configuration files (units,
> > rules, etc), and instead the supported workflow (drop-ins, masking, etc)
> > must be used, both by packages and administrators. This is already
> > standard practice, and again there is only one instance that needs
> > correcting as far as I understand, and I have already provided a bug and
> > a MR for that [1][2]. So the impact of this policy change should be
> > minimal, and it's mostly to ensure more such instances are accidentally
> > added in the future.
>
> > I have a draft policy update, that adds a paragraph to the dpkg-divert
> > section of the policy. It is attached here, and also available on Salsa
> > on my fork [3].
>
> The part of Policy that you edited with this patch is basically
> unmaintained and should ideally be removed in favor of actual Policy.  (I
> had started looking at that a long time ago and then never finished.)  All
> of those appendices from the old packaging manual predate better
> documentation maintained elsewhere (such as in the dpkg package) and are
> ambiguous with regards to whether they set requirements for Debian
> packages, document things for local administrators, or something else.
> The Policy manual warns that they may not be normative, and people often
> don't think to read them (for good reason).
>
> In the case of diversions, while I certainly agree with your proposed
> rule, I suspect Policy should say something stronger and more general,
> namely that no package in Debian should divert a file from another package
> unless this is arranged cooperatively between the packages to solve some
> specific (unusual) problem.  To me, this feels similar to the case of one
> package modifying the configuration files of another package, where we
> explicitly prohibit one package modifying the configuration of another
> package except through an interface provided by the package whose
> configuration is being modified.

I'd like this to go a step further - rules, units and so on can (and
must!) be shipped by other packages, not just from src:systemd.
But as I mentioned in the other reply, bugs come to the systemd bug
tracker most often, and make our lives more difficult, even if it's a
third package that ships the configuration.
So, I'd very much want to make the rule stronger for this use case,
and forbid it even if the respective maintainers agree between
themselves that package A's unit should be diverted by package B's,
because ultimately both A and B are feeding configuration into
systemd/udev/etc, and this is just not a supported mechanism to apply
such changes.

> In other words, dpkg-divert is primarily for local administrators,
> non-Policy-compliant local packages that are doing unusual things, and the
> occasional rare problem that requires special coordination between
> packages, not something that Debian packages should be doing to other
> packages without explicit coordination.
>
> The rule about systemd and udev files doesn't entirely fall out of that
> statement, so we can still include a specific statement about them, noting
> that drop-ins and masking make dpkg-divert unnecessary (and those
> facilities produce better tool behavior) and therefore it should not be
> used.
>
> So, ideally, the way I'd prefer to move forward is for us to add a new
> section to the main Policy manual on diversions (probably 10.11), document
> that this is primarily a tool for local administrators and local packages
> to override the behavior of Debian, and that its use between Debian
> packages should be rare, should involve coordination between the packages,
> and should only be used to solve problems that cannot be handled through
> other facilities such as alternatives or package-specific tools like
> systemd's support for drop-ins and masking.  And then explicitly call out
> systemd and udev configuration as cases where dpkg-divert should not be
> used, alongside conffiles and critical system files.

Ok, I can look at adding 10.11 - I very naively searched for existing
paragraphs mentioning diverts and found the one I extended, I did not
realize it was not proper part of Policy, thanks for the pointer.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

On Mon, 8 May 2023 at 16:39, Sam Hartman  wrote:
>
> > "Luca" == Luca Boccassi  writes:
>
> Luca> It has come to my attention that there is one package in
> Luca> Debian using dpkg-divert to mask a systemd configuration file
> Luca> (an udev rule).  Speaking as one of the maintainers, both
> Luca> upstream and downstream, I find this greatly undesirable for
> Luca> several reasons that I will outline later. Hence I would like
> Luca> to propose explicitly mentioning that dpkg- divert must not be
> Luca> used for systemd configuration files (units, rules, etc), and
> Luca> instead the supported workflow (drop-ins, masking, etc) must
> Luca> be used, both by packages and administrators.
>
> First, I thought there was already a prohibition about one package
> mucking with another package's configuration.
> In many cases it sounds like it's already against policy for one package
> to change the default systemd or udev configuration--at least for
> packages in the archive.
> (I am skepticle that amazon-ec2-utils complies with policy in general).
>
>
> In cases where the change being made is permitted by policy, I think you
> have made a compelling case to vastly prefer the native systemd and udev
> mechanisms to dpkg-divert.
> I don't think that your case is strong enough to forbid dpkg-divert.
>
> As far as I can tell reading your reasoning, dpkg-divert *works fine*.
> It just gives surprising results to someone coming from the systemd
> universe.
>
> But consider a package that diverts several resources, several of them
> systemd and several of them not systemd.
> The maintainer might legitimately want to use the same mechanism for all
> the overriding/masking  so that systemd resources and non-systemd
> resources were handled the same.
>
> There's a real trade off there, and we generally leave those to
> maintainers.
>
> So, I'd support policy advice (ENCOURAGED) rather than policy
> requirements (MUST) in this case.
>
> I do think that if a maintainer violates this advice without a good
> reason, important is a more appropriate bug severity than wishlist, but
> unfortunately we don't have a good way to specify that in policy
> language.
>
> I would not support policy recommendation language (RECOMMENDED/SHOULD)
> because that has a connotation that failing to follow the recommendation
> is always a bug, and I don't think that's true here.

The problem is that when there are udev/systemd bugs they get directed
to the systemd team, not to the package shipping a divert, because
diverts are essentially invisible to normal users. It is already very
difficult and very time consuming to support these packages,
especially udev because we are essentially dealing with the
intersection of an infinite combination of hardware and software, and
anything that makes our lives harder is detrimental to the project at
large. If there was a significant, or even any benefit at all, then it
could be discussed, but I fail to see any. I do not find the
theoretical point about multiple diversions very compelling - we use
different mechanisms for different things all the time, but more
importantly such a case simply has never surfaced in the 10 past years
or so since we ship systemd by default, and longer since we ship udev.

Moreover, as upstream developer, I can guarantee you that masking via
diversion like this is NOT supported, and there could be more bugs
lurking, and we categorically do not intend to support or help with
such cases. I have no intention to spend any time investigating
whether such bugs exist and document them. Therefore, given there is
only one case in the whole distro so impact is minimal, the best
option to me seems to flat out forbid it, so that we never get into
that bad situation in the first place.

Kind regards,
Luca Boccassi

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Russ Allbery]

I think your X-Debbugs-Cc was syntactically invalid and thus didn't work.
I manually added in the other addresses in this reply.

Luca Boccassi  writes:

> It has come to my attention that there is one package in Debian using
> dpkg-divert to mask a systemd configuration file (an udev rule).
> Speaking as one of the maintainers, both upstream and downstream, I find
> this greatly undesirable for several reasons that I will outline
> later. Hence I would like to propose explicitly mentioning that
> dpkg-divert must not be used for systemd configuration files (units,
> rules, etc), and instead the supported workflow (drop-ins, masking, etc)
> must be used, both by packages and administrators. This is already
> standard practice, and again there is only one instance that needs
> correcting as far as I understand, and I have already provided a bug and
> a MR for that [1][2]. So the impact of this policy change should be
> minimal, and it's mostly to ensure more such instances are accidentally
> added in the future.

> I have a draft policy update, that adds a paragraph to the dpkg-divert
> section of the policy. It is attached here, and also available on Salsa
> on my fork [3].

The part of Policy that you edited with this patch is basically
unmaintained and should ideally be removed in favor of actual Policy.  (I
had started looking at that a long time ago and then never finished.)  All
of those appendices from the old packaging manual predate better
documentation maintained elsewhere (such as in the dpkg package) and are
ambiguous with regards to whether they set requirements for Debian
packages, document things for local administrators, or something else.
The Policy manual warns that they may not be normative, and people often
don't think to read them (for good reason).

In the case of diversions, while I certainly agree with your proposed
rule, I suspect Policy should say something stronger and more general,
namely that no package in Debian should divert a file from another package
unless this is arranged cooperatively between the packages to solve some
specific (unusual) problem.  To me, this feels similar to the case of one
package modifying the configuration files of another package, where we
explicitly prohibit one package modifying the configuration of another
package except through an interface provided by the package whose
configuration is being modified.

In other words, dpkg-divert is primarily for local administrators,
non-Policy-compliant local packages that are doing unusual things, and the
occasional rare problem that requires special coordination between
packages, not something that Debian packages should be doing to other
packages without explicit coordination.

The rule about systemd and udev files doesn't entirely fall out of that
statement, so we can still include a specific statement about them, noting
that drop-ins and masking make dpkg-divert unnecessary (and those
facilities produce better tool behavior) and therefore it should not be
used.

So, ideally, the way I'd prefer to move forward is for us to add a new
section to the main Policy manual on diversions (probably 10.11), document
that this is primarily a tool for local administrators and local packages
to override the behavior of Debian, and that its use between Debian
packages should be rare, should involve coordination between the packages,
and should only be used to solve problems that cannot be handled through
other facilities such as alternatives or package-specific tools like
systemd's support for drop-ins and masking.  And then explicitly call out
systemd and udev configuration as cases where dpkg-divert should not be
used, alongside conffiles and critical system files.

-- 
Russ Allbery (r...@debian.org)  

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> It has come to my attention that there is one package in
Luca> Debian using dpkg-divert to mask a systemd configuration file
Luca> (an udev rule).  Speaking as one of the maintainers, both
Luca> upstream and downstream, I find this greatly undesirable for
Luca> several reasons that I will outline later. Hence I would like
Luca> to propose explicitly mentioning that dpkg- divert must not be
Luca> used for systemd configuration files (units, rules, etc), and
Luca> instead the supported workflow (drop-ins, masking, etc) must
Luca> be used, both by packages and administrators.

First, I thought there was already a prohibition about one package
mucking with another package's configuration.
In many cases it sounds like it's already against policy for one package
to change the default systemd or udev configuration--at least for
packages in the archive.
(I am skepticle that amazon-ec2-utils complies with policy in general).


In cases where the change being made is permitted by policy, I think you
have made a compelling case to vastly prefer the native systemd and udev
mechanisms to dpkg-divert.
I don't think that your case is strong enough to forbid dpkg-divert.

As far as I can tell reading your reasoning, dpkg-divert *works fine*.
It just gives surprising results to someone coming from the systemd
universe.

But consider a package that diverts several resources, several of them
systemd and several of them not systemd.
The maintainer might legitimately want to use the same mechanism for all
the overriding/masking  so that systemd resources and non-systemd
resources were handled the same.

There's a real trade off there, and we generally leave those to
maintainers.

So, I'd support policy advice (ENCOURAGED) rather than policy
requirements (MUST) in this case.

I do think that if a maintainer violates this advice without a good
reason, important is a more appropriate bug severity than wishlist, but
unfortunately we don't have a good way to specify that in policy
language.

I would not support policy recommendation language (RECOMMENDED/SHOULD)
because that has a connotation that failing to follow the recommendation
is always a bug, and I don't think that's true here.

--Sam

Bug#1035733: debian -policy: packages must not use dpkg-divert to override default systemd configuraton files

[Luca Boccassi]

Package: debian-policy
X-Debbugs-CC: bi...@debian.org pkg-systemd-maintain...@lists.alioth.debian.org

It has come to my attention that there is one package in Debian using
dpkg-divert to mask a systemd configuration file (an udev rule).
Speaking as one of the maintainers, both upstream and downstream, I
find this greatly undesirable for several reasons that I will outline
later. Hence I would like to propose explicitly mentioning that dpkg-
divert must not be used for systemd configuration files (units, rules,
etc), and instead the supported workflow (drop-ins, masking, etc) must
be used, both by packages and administrators. This is already standard
practice, and again there is only one instance that needs correcting as
far as I understand, and I have already provided a bug and a MR for
that [1][2]. So the impact of this policy change should be minimal, and
it's mostly to ensure more such instances are accidentally added in the
future.

I have a draft policy update, that adds a paragraph to the dpkg-divert
section of the policy. It is attached here, and also available on Salsa
on my fork [3].

The full reasoning below is what I provided on the MR for the one
existing instance, and I'll copy it mostly unchanged as I hope it's
exhaustive enough. It uses the one existing instance I found as
concrete example. This is not intended to single out the maintainer or
assign blame, but merely to illustrate the point with a concrete and
real use case. Quoting from the MR:

One of the main goals behind the systemd (and its udev component)
project is to unify how the low-level userspace components of a Linux
distro work, so that the exact same mechanisms, patterns, behaviours
and interfaces apply to a multitude of use cases, implementations and
tools. A core part of this is the configuration system. The
configuration system supports a complex schema of main contents,
overrides, drop-ins, masking and aliasing. This system is used and
understood by all systemd components, across all Linux distributions,
with the same interface, look and feel, so that users can feel at home
and know how to work the system regardless of the vendor, and so that
programs can rely on a stable and common interface that doesn't have to
be endlessly customized depending on which vendor or distribution it is
used by. The concept of 'masking' a configuration is well understood,
ubiquitous and fully supported by all the tooling, including in input
_and_ output, and logging, and so on. By using the supported mechanism
for masking we ensure that there are no surprises for users, coders and
vendors. When an unsupported masking mechanism is used, as in this
case, the fact that the original item is masked is completely hidden to
the systemd components, and thus to the interface provided to the user.
This causes at best confusion and misunderstanding, and at worst bugs
that will inevitably fall on the systemd maintainers, causing increased
workload for an already over stretched team.

A simple and obvious example of what I am referring to is already
included in the git commit for this change. Consider what happens when
udev parses the vanilla configuration (ie: without amazon-ec2-utils
installed):

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

The user/system/log is clearly notified that the file is parsed, there
are no errors, and thus is used.

Now consider what happens when the current version of amazon-ec2-utils
is installed:

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Reading rules file: /lib/udev/rules.d/60-cdrom_id.rules

It looks exactly the same. But something extremely different is
actually happening, in fact the opposite! The file is empty, so the
vanilla rule is effectively masked, but nothing and nobody is notified
of this very important fact when udev is running. One would have to
query dpkg-divert to figure this out, but this is something that even
someone like me, who can reasonably consider myself a proficient Debian
user, would not think of looking at.

Now finally consider what happens if amazon-ec2-utils with my proposed
change is installed:

$ udevadm test /dev/cdrom 2>&1 | grep 60-cdrom_id
Skipping overridden file '/lib/udev/rules.d/60-cdrom_id.rules'.
Skipping empty file: /etc/udev/rules.d/60-cdrom_id.rules

There is an extremely clear, obvious and expected indication that the
original rule is ignored because it is overridden, _and_ that the
override is empty, as it's a masking operation. Now udev is a bit older
than the other components, so "masking" is not named explicitly as it
would be for example by systemctl, but it is the same operation and the
same result and the same interface. Something I should probably improve
a bit upstream, I will take a note of this.

Here's another example with systemctl and units. I have masked systemd-
homed.service via `systemctl mask` and this is what the interface shows
me when I check the status:

$ systemctl status