Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Josh Triplett]

Sean Whitton wrote:
> On Sun 17 Sep 2023 at 10:52am -07, Russ Allbery wrote:
> > So far as I can tell, the only important part is that the directory
> > be registered in tmpfiles.d (or a service unit) so that it can be
> > recreated when needed.
>
> Something which I don't think has been mentioned yet is that we
> explicitly permit the local administrator to nuke /usr/share/doc, even
> though that is a similar sort of inconsistency between the dpkg db and
> the filesystem as the ones that are bothering Luca.

One key difference between the two is the existence of Policy 12.3:
> Packages must not require the existence of any files in
> "/usr/share/doc/" in order to function.  [6] Any files that are used
> or read by programs but are also useful as stand alone documentation
> should be installed elsewhere, such as under "/usr/share/package/",
> and then included via symbolic links in "/usr/share/doc/package".

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 17 Sept 2023 at 17:35, Luca Boccassi  wrote:
>
> To do the tmpfiles purge/reset I have two WIP PRs, one against
> sd-tmpfiles, and one against debhelper. I need to pick them up again
> and finish that, and I am aiming to do so within the next couple of
> months.

Sorry for bumping the thread, but wanted to follow-up that tmpfiles
--purge mentioned above is now merged upstream and will be part of
v256 which should be in rc1 in the next couple of weeks. Once rc1 is
there I'll work again on the debhelper integration.

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sean Whitton]

Hello,

On Sun 17 Sep 2023 at 10:52am -07, Russ Allbery wrote:

> I don't see how shipping empty directories in a deb package affects this
> in any way.  You're going to have to be considerably more specific about
> what invariant is being violated or what error you're expecting.
>
> One of the key things that I'm stuck on, and which you haven't addressed
> that I've seen, is that you're talking here about a system managed with a
> normal package manager, and therefore running all maintainer scripts.  The
> maintainer scripts under this tmpfiles.d proposal will create directories
> and files in /var, because the whole point is that systemd-tmpfiles is run
> from postinst.  But you are saying that creating directories and files in
> /var with the package manager will break this configuration.  I'm missing
> something here.  There's nothing special about dpkg creating the directory
> versus postinst creating the directory.  So far as I can tell, the only
> important part is that the directory be registered in tmpfiles.d (or a
> service unit) so that it can be recreated when needed.

Something which I don't think has been mentioned yet is that we
explicitly permit the local administrator to nuke /usr/share/doc, even
though that is a similar sort of inconsistency between the dpkg db and
the filesystem as the ones that are bothering Luca.

> I understand that you are trying to do something new, and you are worried
> that this will interfere with it, but so far you have not been able to
> explain any way in which this *would* cause you problems.  You just don't
> like the vibes.  And I hear that, and sometimes that sort of bad feeling
> is a valuable architectural clue, but in this case you're really going to
> have to be more specific because I'm not seeing it.
>
> Also, we clearly cannot ship a system without /var *now* because oodles of
> other packages put things there, so a lot of work is left to do before
> this is a technical vision that Debian can implement, if we do indeed
> decide to implement it.  Policy can always change later; perhaps by the
> time that we're ready to implement this vision, dpkg will have a way of
> registering files managed by tmpfiles.d and there's no reason to ship the
> empty directories in the package.  So theoretical future problems are not
> very persuasive to me at this stage, particularly if you can't be specific
> about what those problems would be.
>
>> [...]
>
> I understand that you would like to finish this, but Debian so far has not
> even decided to *start* this, so while I'm willing to take this into
> account when writing Policy, it's not a controlling design principle.  If
> you get an agreement in Debian that this is something we're going to work
> towards, then it will become a significant factor in evaluating Policy
> changes.
>
> I'm really trying to meet you halfway here by adopting and fleshing out
> proposals that you're putting forward primarily for a project that Debian
> isn't currently doing, but which have clear other benefits regardless of
> whether we do the factory reset project.  I don't want to argue over end
> goals when we can agree on intermediate steps regardless of the end goal.
> But I really want to emphasize here that we are not *currently* writing
> Policy to support factory reset because there is no decision in Debian yet
> that we are supporting factory reset.
>
> This is not directly relevant to this bug, but for the record, if you want
> Debian to support factory reset, one good way to make that more likely is
> to write a DEP with the details of precisely what that would mean, roughly
> what sorts of things would need to change in packaging, and a list of what
> the benefits would be.  I personally think a lot of the benefits are
> rather compelling, but no one has yet made a proper case for it in Debian.
> You and Marco and a few other people just write email messages on other
> topics that treat the desirability of factory reset as a foregone
> conclusion, or mention a few benefits in passing and without specifics,
> which of course is fine for casual discussion but which is not a real
> attempt at persuasion and doesn't get us closer to making a real decision.
>
> In other words, this is advice that I constantly give myself because I am
> very bad at this and have to be reminded repeatedly: stop arguing with
> people about specifics and use that energy to write down a design with a
> justification.  It will make the arguments much less annoying and
> repetitive, because a lot of the repetition is because everyone else is
> sadly incapable of reading your mind and doesn't understand what you are
> assuming is well-known.

I very much agree.  It also seems worth mentioning that if we are able
to support factory-reset, we will also want to minimise how much it gets
in the way of people who have no interest in that sort of feature --
most of our classic Debian desktop power users, I suspect (which
includes most DDs).

Russ, I think 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 17 Sept 2023 at 18:53, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > Let me clarify, here I meant something much simpler - the image
> > installed is a 'normal' one, with r/w root and managed by apt as usual
> > (ie: not immutable image-based) but with a repart.d snippet that causes
> > a new /var to be created on-the-fly on first boot if missing, with
> > tpm-bound encryption (and similar treatment for /home, although
> > unrelated here). This is a very low-hanging fruit of a pattern that
> > would allow to achieve decent local data protection on an otherwise
> > pretty much vanilla setup. But if you need to ship /var from packages,
> > then it goes out of the window.
>
> I don't see how shipping empty directories in a deb package affects this
> in any way.  You're going to have to be considerably more specific about
> what invariant is being violated or what error you're expecting.

How could it _not_ affect it? You'd have two separate components
stepping on each other's toes and claiming ownership of the same
resources, with no hope of having any real synchronization apart from
manually crafting it and hoping for the best, assuming it's even
possible - how can one even express things like aging, acls, xattrs,
and all the other things that tmpfiles.d can do, via dpkg? On top of
that, the point is to be able to keep a consistent system at all times
- but the package tree is going to be immediately inconsistent. If I
wanted to work against the system, I would just do that. The purpose
was to keep things coherent instead.

> > I don't think it is particularly useful, and mixing package content
> > with variable data smells like trouble to me.
>
> I understand that you don't like the idea.  You've made that very clear.
> However, we are *currently* doing this, so obviously nothing that we
> currently support is going to break from continuing to do this.

It doesn't seem to me that we really are though: on my bookworm laptop
I have 5357 packages installed, which ship in total 196 unique
directories under /var, so 1 pkg out of 27 actually does it. I have
4487 unique directories under /var, 20 times as many as they are
shipped by packages.

> > We are working on that. This means that tmpfiles.d would be able to both
> > create the files/dirs when needed, and remove them when unneeded, ie on
> > purge - as far as I can tell, that would be the only useful thing that a
> > dpkg integration would provide.
>
> dpkg -S is the most useful feature this supports for me personally (and
> some related things, such as cruft-finding).

We could add a systemd-tmpfiles --inventory or so, that outputs the
same, with a json mode too, if it could be useful. Give it paths, it
gives out a list of tmpfiles.d that touch them, which then you can
trace back to their packages.

> > On the contrary I suspect it would break things or at the very least get
> > in the way, as explained above. Of course I haven't tested this as we
> > aren't there yet, so can't be 100% sure. But from what I've seen from
> > some experiments, expectations around /var being fixed and
> > package-managed is already creating some headaches, and requiring
> > workarounds.
>
> Specifics!  Specifics!  My kingdom for specifics!  :)  Bug numbers for
> these headaches would be helpful, or detailed descriptions, or
> *something*.  You're giving me nothing to work with here, which means that
> I'm likely to go forward with requiring some of these empty directories be
> registered with dpkg because that's the less invasive change and avoids a
> regression.

I'm afraid that would require significantly more time that I have
available, so I just can't give you a working prototype anytime soon.

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Alexandre Detiste]

Le dim. 17 sept. 2023 à 21:26, Russ Allbery  a écrit :
> Based on previous discussion with Guillem, I think the direction Guillem
> is headed is something like adding a new member (or field in another
> member) to the deb format that holds a list of volatile files that the
> package considers itself responsible for.  I think I agree with Guillem's
> position (at least as I understand it) that it doesn't make sense for dpkg
> to parse other files to populate that list.  That can very easily be
> handled outside of dpkg.

Yes, it's better to keep dpkg clean & generic and
handle the fast moving parts elsewhere.
(like not tainting dpkg forever with the oneshot UsrMerge final migration)

I understand why redhat would bolt tmpfiles directly into rpm,
but it's a more integrated ecosystem.

> So the idea would be that the package would install tmpfiles.d files,
> service units, and similar files as normal, and then debhelper would parse
> those files, extract the list of paths that they manage, and use that to
> write a control file or the like that dpkg consumes to register those
> files.

Yes.

And debputy too, which is a rewrite of debhelper, but in Python,
with different assumptions (everything in one machine-editable
yaml instead of so many tiny files in debian/ )

> those files.
and not only files but glob patterns: /var/cache/[package]/**


> If I'm correct about that design, an intermediate step in that direction
> from where cruft is today would be to add that logic to debhelper and then
> have debhelper ship that database in the package in
> /usr/share/cruft/ or some similar directory, and then cruft could
> just consume that database of registered paths to get attribution
> information until such time as that can move into dpkg.

Yes, that's what dh_cruft does. I use it extensively for my proprietary
Debian-derivative, but did not try to push it down Debian packagers
(because it's in Perl and I cannot really maintain it,
so the tmpfiles.d parsing is not there yet).

Having today machine readable data (instead of "if $1 == purge... rm -rf ...")
make it easier to later turn it into another packaging format.

> This design is just off the top of my head, and I'm probably missing some
> problems and some details.

You've got a good grasp of the important parts.

So this tool can be used to recover trashed system
- I receive 10 years old filesystem images that
have seen much "sudo make install" abuse -
but also be used to make the distribution better as a whole.

So I file bugs regularly against random _other_ packages
for the dynamic things piuparts won't notice by design;
because it doesn't run the tools in packages it tests.
(most of the times it's a missing "rm .." in postrm);


Maybe having cruft run in last step of all autopkgtests
would ensure all dynamic files are registered.
 (maybe only on Salsa first)

The first step would be that autopkgtest itself provides
this authoritative list of allowed file names.

(see from :  unexplained: /  )
https://ci.debian.net/data/autopkgtest/testing/amd64/c/cruft-ng/37993513/log.gz

Greetings

---

I uploaded a new "cruft-ng" with a new "cruft --package PACKAGE" option
that dumps this one package volatile files ruleset to ease import in
another tool.

And "cruft FILE" now works closer to  [ "dpkg -S" + volatile files ].

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Alexandre Detiste  writes:

> The ugly magic behind the curtain:

> ls /usr/libexec/cruft/ -1
>   LOGROTATE -> that parses these for path
>   SERVICES -> added today reading this discussion, it reads
>  CacheDirectory= & StateDirectory= from *.service
>   TMPFILES  -> that parses these for path

> This whole thing, while being already usefull & used,
> is more like a mockup of what could be a "perfect" dpkg.

> These tiny shell scripts could be converted into something else
> that plugs into dpkg ... for example tiny .so plugins that answer
> which package own which dynamic file ?

> (for runtime evaluation, other possibility is debhelper magic at
> compile-time that generate a list of possible files)

Based on previous discussion with Guillem, I think the direction Guillem
is headed is something like adding a new member (or field in another
member) to the deb format that holds a list of volatile files that the
package considers itself responsible for.  I think I agree with Guillem's
position (at least as I understand it) that it doesn't make sense for dpkg
to parse other files to populate that list.  That can very easily be
handled outside of dpkg.

So the idea would be that the package would install tmpfiles.d files,
service units, and similar files as normal, and then debhelper would parse
those files, extract the list of paths that they manage, and use that to
write a control file or the like that dpkg consumes to register those
files.

If I'm correct about that design, an intermediate step in that direction
from where cruft is today would be to add that logic to debhelper and then
have debhelper ship that database in the package in
/usr/share/cruft/ or some similar directory, and then cruft could
just consume that database of registered paths to get attribution
information until such time as that can move into dpkg.

This design is just off the top of my head, and I'm probably missing some
problems and some details.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Bill Allombert  writes:

> As I said, filling the caches in /var/cache. For that they need to exist
> with correct ownership and permissions.

Sorry, I think I saw that and then edited my message more and lost it
again.

That use case makes sense to me, and without the directory already
present, you have to know what directory to create and you have to get the
ownership and permissions correct.  But there's a couple of reasons why I
don't think that's a problem:

1. Installing the package creates the directories since it invokes
   systemd-tmpfiles via postinst, so the directory will normally be there
   with correct ownership and permissions.  The only case where it
   wouldn't be is in cases where the packages were installed without
   running postinst, which feels like an unusual use case.

2. Presumably you would be copying these caches from another system, which
   will normally have the directory with correct ownership and
   permissions.  This isn't necessarily true if you're mixing versions of
   Debian, of course, but in that case it's not clear the cache format
   will be correct either.  Also, you need to get the ownership and
   permissions of the files right, which the directory structure doesn't
   necessarily help you with, and if you're copying that over already, the
   same mechanism can handle the ownership and permissions of the parent
   directory.

So, by definition any directory that's shipped in the deb cannot have
dynamic ownership, which also limits the range of permissions it could
have.

> even populate /var/www with your website, etc.

/var/www is a whole separate problem that I agree has not yet been
addressed and would need to be.  We've known that /var/www is weird for a
while (we have a special exception in the FHS for it because it's breaking
the FHS file system layout rules), and there have been a few attempts to
handle it some other way, but none of them so far have been successful.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Bill Allombert]

On Sun, Sep 17, 2023 at 08:28:56AM -0700, Russ Allbery wrote:
> Bill Allombert  writes:
> > On Sun, Sep 17, 2023 at 10:41:55AM +0200, Marco d'Itri wrote:
> >> On Sep 17, Russ Allbery  wrote:
> 
> >>> (I am a little confused by this wording, but I think what you're
> >>> saying is that /usr is encrypted and read-only, and /var is recreated
> >>> on each boot.  That at least is my understanding of the pattern that
> >>> you're trying to enable.)
> 
> >> The general idea is to be able to create /var on the first boot.
> 
> > Does not that would break users expectation that the system image
> > contains /var before the first boot ?
> 
> > A lot of things in /var are caches that are mostly instance-independent
> > and can be prefilled, but for that, users expect a minimal directory
> > hierarchy to be present before first boot.
> 
> Not that I think we're particularly close to achieving this design
> currently (and to be clear we haven't decided we're working towards this
> yet), but while I understand why a user would have that expectation today,
> I'm not sure why it would practically matter.  If all of that directory
> structure appears on first boot, and no static data is stored in /var,
> what use case requires the directory structure already exist in /var
> before the first boot?

As I said, filling the caches in /var/cache. For that they need to
exist with correct ownership and permissions.

Most of the cache in /var/cache/ (some are in /var/lib actually) 
do not depend on the host configuration, and can be regenerated/redownloaded as
needed, but not for free.

For example you might want to populate
/var/cache/apt/archives/ with the debs you need install later on (for example
for a pbuilder-like system), populate /var/lib/texmf/fonts/ with your fonts, or
even populate /var/www with your website, etc.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Alexandre Detiste]

So there are 3 distinct interlinked goals:

- tmpfiles.d itself
  - recovering from missing /var (+ later /etc)
  - volatile file tracking

Just finishing the first step without going to far in either
other tracks would be so great already.


Le dim. 17 sept. 2023 à 19:57, Russ Allbery  a écrit :
> Luca Boccassi  writes:
> > We are working on that. This means that tmpfiles.d would be able to both
> > create the files/dirs when needed, and remove them when unneeded, ie on
> > purge - as far as I can tell, that would be the only useful thing that a
> > dpkg integration would provide.
>
> dpkg -S is the most useful feature this supports for me personally (and
> some related things, such as cruft-finding).

I agree.

And when "dpkg -S file" fail,
just try "cruft /path/file".

It is slow, the implementation *is* bad - more like a unit test -
but should know about most files, and it still faster
than starting googling for the filename and getting distracted:

$ LANG=C dpkg -S  /var/log/dpkg.log
dpkg-query: no path found matching pattern /var/log/dpkg.log
$ time cruft /var/log/dpkg.log
dpkg
real0m7,129s

The ugly magic behind the curtain:

ls /usr/libexec/cruft/ -1
  LOGROTATE -> that parses these for path
  SERVICES -> added today reading this discussion, it reads
 CacheDirectory= & StateDirectory= from *.service
  TMPFILES  -> that parses these for path


This whole thing, while being already usefull & used,
is more like a mockup of what could be a "perfect" dpkg.

These tiny shell scripts could be converted into something else
that plugs into dpkg ... for example tiny .so plugins that answer
which package own which dynamic file ?

(for runtime evaluation, other possibility is debhelper magic at compile-time
that generate a list of possible files)

> More generally, dropping directories that are currently registered with
> dpkg from dpkg's database is a regression.

I agree.


I've mounted /var/cache/apt/archives everywhere on a tmpfs since forever,
apt will now how/when to recreate /partial subdir,
but yet it's nice to have it in dpkg register.

> Specifics!  Specifics!  My kingdom for specifics!  :)

There's indeed not so much remaining in the way for /var,
some files could be replaced by tmpfiles.d generated symlinks

$ cat /var/lib/dpkg/info/*.list | grep ^/var | sort -u | while read f;
do test -f $f && echo $f; done | wc -l
256

$ dpkg -S $(cat /var/lib/dpkg/info/*.list | grep ^/var | sort -u |
while read f; do test -f $f && echo $f; done) | cut -d: -f1 | sort -u
aspell-en
dictionaries-common
ifrench: /var/lib/dictionaries-common/ispell/ifrench -> generated metadata
plocate:   CACHEDIR.TAG
poppler-data:only transitional symlinks to /usr/share/poppler ?
ttf-mscorefonts-installer: only one file that could be symlinked from /usr
xserver-common:   /var/lib/xkb/README.compiled, could be a
symlink from /usr/share/doc

Maybe /etc in a later track.

Greetings,

(I find this thread greatly inspirational)

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> Let me clarify, here I meant something much simpler - the image
> installed is a 'normal' one, with r/w root and managed by apt as usual
> (ie: not immutable image-based) but with a repart.d snippet that causes
> a new /var to be created on-the-fly on first boot if missing, with
> tpm-bound encryption (and similar treatment for /home, although
> unrelated here). This is a very low-hanging fruit of a pattern that
> would allow to achieve decent local data protection on an otherwise
> pretty much vanilla setup. But if you need to ship /var from packages,
> then it goes out of the window.

I don't see how shipping empty directories in a deb package affects this
in any way.  You're going to have to be considerably more specific about
what invariant is being violated or what error you're expecting.

One of the key things that I'm stuck on, and which you haven't addressed
that I've seen, is that you're talking here about a system managed with a
normal package manager, and therefore running all maintainer scripts.  The
maintainer scripts under this tmpfiles.d proposal will create directories
and files in /var, because the whole point is that systemd-tmpfiles is run
from postinst.  But you are saying that creating directories and files in
/var with the package manager will break this configuration.  I'm missing
something here.  There's nothing special about dpkg creating the directory
versus postinst creating the directory.  So far as I can tell, the only
important part is that the directory be registered in tmpfiles.d (or a
service unit) so that it can be recreated when needed.

> I don't think it is particularly useful, and mixing package content
> with variable data smells like trouble to me.

I understand that you don't like the idea.  You've made that very clear.
However, we are *currently* doing this, so obviously nothing that we
currently support is going to break from continuing to do this.

I understand that you are trying to do something new, and you are worried
that this will interfere with it, but so far you have not been able to
explain any way in which this *would* cause you problems.  You just don't
like the vibes.  And I hear that, and sometimes that sort of bad feeling
is a valuable architectural clue, but in this case you're really going to
have to be more specific because I'm not seeing it.

Also, we clearly cannot ship a system without /var *now* because oodles of
other packages put things there, so a lot of work is left to do before
this is a technical vision that Debian can implement, if we do indeed
decide to implement it.  Policy can always change later; perhaps by the
time that we're ready to implement this vision, dpkg will have a way of
registering files managed by tmpfiles.d and there's no reason to ship the
empty directories in the package.  So theoretical future problems are not
very persuasive to me at this stage, particularly if you can't be specific
about what those problems would be.

> I'd much rather finish the factory reset workstream for lifetime
> management, so that tmpfiles can handle tmpfiles purges, without needing
> to involve dpkg.

I understand that you would like to finish this, but Debian so far has not
even decided to *start* this, so while I'm willing to take this into
account when writing Policy, it's not a controlling design principle.  If
you get an agreement in Debian that this is something we're going to work
towards, then it will become a significant factor in evaluating Policy
changes.

I'm really trying to meet you halfway here by adopting and fleshing out
proposals that you're putting forward primarily for a project that Debian
isn't currently doing, but which have clear other benefits regardless of
whether we do the factory reset project.  I don't want to argue over end
goals when we can agree on intermediate steps regardless of the end goal.
But I really want to emphasize here that we are not *currently* writing
Policy to support factory reset because there is no decision in Debian yet
that we are supporting factory reset.

This is not directly relevant to this bug, but for the record, if you want
Debian to support factory reset, one good way to make that more likely is
to write a DEP with the details of precisely what that would mean, roughly
what sorts of things would need to change in packaging, and a list of what
the benefits would be.  I personally think a lot of the benefits are
rather compelling, but no one has yet made a proper case for it in Debian.
You and Marco and a few other people just write email messages on other
topics that treat the desirability of factory reset as a foregone
conclusion, or mention a few benefits in passing and without specifics,
which of course is fine for casual discussion but which is not a real
attempt at persuasion and doesn't get us closer to making a real decision.

In other words, this is advice that I constantly give myself because I am
very bad at this and have to be reminded 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 17 Sept 2023 at 00:12, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > Aside from more practical considerations, shipping /var content in
> > packages is problematic because it's supposed to be local variable data,
> > that can be removed without breaking a system.
>
> Unless I'm missing something, including the directory in the deb won't
> make any difference here.  dpkg won't break if a directory that was
> included in the package is deleted.  It would show as an inconsistency if
> someone checked the file system against the dpkg database, but as soon as
> systemd-tmpfiles runs, it will create the directory again and fix the
> inconsistency, so I don't see what problems that would create.

That was a remark about the concept itself, independently of tmpfiles
- variable data is variable data, and static data is static data.
Having variable data among static package content sounds conceptually
wrong to me, simply from a high-level design perspective.

> > More practically, one of the purposes of the hermetic-usr pattern is to
> > allow several modernizations. The easiest one to achieve is to create
> > /var/ on firstboot, and encrypt it against the tpm, so that it can be
> > enabled by default, always, so we can't have packages ship and expect
> > content in /var from their packages.
>
> (I am a little confused by this wording, but I think what you're saying is
> that /usr is encrypted and read-only, and /var is recreated on each boot.
> That at least is my understanding of the pattern that you're trying to
> enable.)

Let me clarify, here I meant something much simpler - the image
installed is a 'normal' one, with r/w root and managed by apt as usual
(ie: not immutable image-based) but with a repart.d snippet that
causes a new /var to be created on-the-fly on first boot if missing,
with tpm-bound encryption (and similar treatment for /home, although
unrelated here). This is a very low-hanging fruit of a pattern that
would allow to achieve decent local data protection on an otherwise
pretty much vanilla setup. But if you need to ship /var from packages,
then it goes out of the window.

In a sense, for the immutable /usr case it gets easier - you just
don't ship a package manager at all, and then you can do all the
mangling you want when building the image, as consistency no longer
matters, and you don't install/upgrade by definition. But I strongly
believe we need to do some serious steps forward in the
security-by-default aspects of all flavours, including the 'vanilla'
ones, where this consistency matters a lot, for obvious reasons - you
do upgrades/installs on those.

And yes, I would hope to have a concrete proposal for encrypting by
default to submit to the project for at least something like this for
Trixie, once more pieces have fallen into place. All mainstream
distributions are looking into some variation of this. It is way past
time Linux caught up with OSX and Windows on these aspects, and it
would be great if for once Debian wasn't left behind as usual.

> > On top of that, as you mentioned already things will inevitably get out
> > of sync, and one will have to duplicate everything.
>
> One would need to duplicate empty directories in /var (that don't have
> dynamic ownership).  I'm dubious that's a significant burden (it's two or
> three lines in debian/rules), but if it is, one could even automate this
> in debhelper by parsing tmpfiles.d if one really wanted to.  The main
> thing that could get out of sync is the ownership, which is indeed not
> ideal, but I'm also not sure it's going to cause major problems even if
> people do get it wrong.  I was trying to remember if dpkg changes
> directory (as opposed to file) ownership if it sees a directory owned by
> an unexpected user.  I kind of think it doesn't, but I'm not sure about
> that.
>
> The benefit we gain from this is attribution of the directories in the
> dpkg database, which is useful (although I understand that one can argue
> about how useful).

I don't think it is particularly useful, and mixing package content
with variable data smells like trouble to me. I'd much rather finish
the factory reset workstream for lifetime management, so that tmpfiles
can handle tmpfiles purges, without needing to involve dpkg. We are
working on that. This means that tmpfiles.d would be able to both
create the files/dirs when needed, and remove them when unneeded, ie
on purge - as far as I can tell, that would be the only useful thing
that a dpkg integration would provide. I am pretty sure running
checksums on local variable data would be a pretty useless exercise
given, well, it's variable. Or was there anything else aside from
these two aspects?

To do the tmpfiles purge/reset I have two WIP PRs, one against
sd-tmpfiles, and one against debhelper. I need to pick them up again
and finish that, and I am aiming to do so within the next couple of
months.

> So... I think the answer to my question of whether this will interfere
> with 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Marco d'Itri]

On Sep 17, Bill Allombert  wrote:

> Does not that would break users expectation that the system image contains 
> /var
> before the first boot ?
I am not aware of such expectations.

> A lot of things in /var are caches that are mostly instance-independent and 
> can
> be prefilled, but for that, users expect a minimal directory hierarchy to be
> present before first boot.
Can you show some examples of how this would work in practice?

> It seems your scheme favors some usecase over some others.
There are always tradeoffs, but my use case does not forbid the other 
one: worst case it requires one more mkdir while copying that data.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Bill Allombert  writes:
> On Sun, Sep 17, 2023 at 10:41:55AM +0200, Marco d'Itri wrote:
>> On Sep 17, Russ Allbery  wrote:

>>> (I am a little confused by this wording, but I think what you're
>>> saying is that /usr is encrypted and read-only, and /var is recreated
>>> on each boot.  That at least is my understanding of the pattern that
>>> you're trying to enable.)

>> The general idea is to be able to create /var on the first boot.

> Does not that would break users expectation that the system image
> contains /var before the first boot ?

> A lot of things in /var are caches that are mostly instance-independent
> and can be prefilled, but for that, users expect a minimal directory
> hierarchy to be present before first boot.

Not that I think we're particularly close to achieving this design
currently (and to be clear we haven't decided we're working towards this
yet), but while I understand why a user would have that expectation today,
I'm not sure why it would practically matter.  If all of that directory
structure appears on first boot, and no static data is stored in /var,
what use case requires the directory structure already exist in /var
before the first boot?

I think you're thinking of cases where the user puts data into /var and
expects it to be used by the system after boot, but configuration data
would go into /etc, so I'm not sure what data that would be.

Also, I think that scenario would still work.  My understanding of the
design is that /var isn't tmpfs; while there's no precreated directory
structure, the user could still make one if they wanted.  There wouldn't
be the guide of existing empty directories, but this is a fairly
sophisticated use case, IMO.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Bill Allombert]

On Sun, Sep 17, 2023 at 10:41:55AM +0200, Marco d'Itri wrote:
> On Sep 17, Russ Allbery  wrote:
> 
> > (I am a little confused by this wording, but I think what you're saying is
> > that /usr is encrypted and read-only, and /var is recreated on each boot.
> > That at least is my understanding of the pattern that you're trying to
> > enable.)
> The general idea is to be able to create /var on the first boot.

Does not that would break users expectation that the system image contains /var
before the first boot ?

A lot of things in /var are caches that are mostly instance-independent and can
be prefilled, but for that, users expect a minimal directory hierarchy to be
present before first boot.

It seems your scheme favors some usecase over some others.

Cheers
-- 
Bill. 

Imagine a large red swirl here. 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Marco d'Itri]

On Sep 17, Russ Allbery  wrote:

> (I am a little confused by this wording, but I think what you're saying is
> that /usr is encrypted and read-only, and /var is recreated on each boot.
> That at least is my understanding of the pattern that you're trying to
> enable.)
The general idea is to be able to create /var on the first boot.
If /var can be populated programmatically then a system can be trivially 
replicated by sharing (or copying) /usr and by copying /etc.

BTW, I do not expect that tmpfiles.d(5) will be the standard method used 
to create most directories below /var.
Usually the CacheDirectory, LogsDirectory and StateDirectory directives 
are more convenient and flexible.

> The benefit we gain from this is attribution of the directories in the
> dpkg database, which is useful (although I understand that one can argue
> about how useful).
Not enough to justify having multiple sources of truth is my opinion.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> Aside from more practical considerations, shipping /var
Luca> content in packages is problematic because it's supposed to be
Luca> local variable data,

I agree with the above.

Luca> that can be removed without breaking a
Luca> system.

Says who?
Where?
Do we have any agreement within Debian that is true for Debian systems?
If so, where?  This is the first I am hearing about the idea I should be
able to delete local variable data and have the system  still work.

If you're talking about *empty directories in /var* or *cache
directories in /var*,
I support that as a goal.  I think it is a new goal though, and I'm
uncomfortable stating it as a reality.
(I think tmpfiles.d helps us achieve that and that's one of the
compelling reasons for tmpfiles.d).

But WRT other data in /var, I don't think we've agreed that being able
to delete it is a goal.

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Alexandre Detiste]

Le dim. 17 sept. 2023 à 01:15, Russ Allbery  a écrit :
> Luca Boccassi  writes:
>
> One would need to duplicate empty directories in /var (that don't have
> dynamic ownership).  I'm dubious that's a significant burden (it's two or
> three lines in debian/rules), but if it is,
> one could even automate this
> in debhelper by parsing tmpfiles.d if one really wanted to.

That would be awesome. The less (open) code in d/rules, the better.

There's already this draft RM that this feature could build upon.
https://salsa.debian.org/debian/debhelper/-/merge_requests/102/

Greets

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> Aside from more practical considerations, shipping /var content in
> packages is problematic because it's supposed to be local variable data,
> that can be removed without breaking a system.

Unless I'm missing something, including the directory in the deb won't
make any difference here.  dpkg won't break if a directory that was
included in the package is deleted.  It would show as an inconsistency if
someone checked the file system against the dpkg database, but as soon as
systemd-tmpfiles runs, it will create the directory again and fix the
inconsistency, so I don't see what problems that would create.

> More practically, one of the purposes of the hermetic-usr pattern is to
> allow several modernizations. The easiest one to achieve is to create
> /var/ on firstboot, and encrypt it against the tpm, so that it can be
> enabled by default, always, so we can't have packages ship and expect
> content in /var from their packages.

(I am a little confused by this wording, but I think what you're saying is
that /usr is encrypted and read-only, and /var is recreated on each boot.
That at least is my understanding of the pattern that you're trying to
enable.)

Here too, I don't see how including an empty directory in /var in the deb
will make any difference here.  When you create such a system, you would
delete /var, so it wouldn't matter if packages created files in there (and
in fact, under every proposal in this bug, installing packages *would*
create files there, since systemd-tmpfiles would be invoked by the
maintainer scripts anyway).

> On top of that, as you mentioned already things will inevitably get out
> of sync, and one will have to duplicate everything.

One would need to duplicate empty directories in /var (that don't have
dynamic ownership).  I'm dubious that's a significant burden (it's two or
three lines in debian/rules), but if it is, one could even automate this
in debhelper by parsing tmpfiles.d if one really wanted to.  The main
thing that could get out of sync is the ownership, which is indeed not
ideal, but I'm also not sure it's going to cause major problems even if
people do get it wrong.  I was trying to remember if dpkg changes
directory (as opposed to file) ownership if it sees a directory owned by
an unexpected user.  I kind of think it doesn't, but I'm not sure about
that.

The benefit we gain from this is attribution of the directories in the
dpkg database, which is useful (although I understand that one can argue
about how useful).

So... I think the answer to my question of whether this will interfere
with your use case is no?  I understand that you don't want to do it, and
expected that, and that opinion is important for the discussion, but I'm
also trying to figure out if it will *break* anything.

> And if dpkg gets the ability to read tmpfiles.d - then that's great,
> and even more reasons not to change policy for something that would
> only be a temporary stop-gap.

I'm not going to assume that this is going to happen on any particular
time scale.  dpkg has to gain a mechanism for registering transient files
first, which in my understanding depends on other significant dpkg
architectural work.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Simon McVittie  writes:

> The key piece of information that was missing from your previous
> proposal was that systemd-tmpfiles interface versions match upstream
> systemd version numbers. As a concrete example, if someone wants to
> upload an implementation other than the one from systemd, it cannot have

> Provides: systemd-tmpfiles (= 254)

> until it has at least a basic implementation of the new "X", "C+" and
> "--graceful" features introduced in systemd 254.

Yeah, I had missed that, thank you.  I think that's now captured.

> If the package benefits from running tmpfiles.d but does not strictly
> require it (for example dbus-daemon, where /run/dbus/containers is
> needed for some optional functionality), would a Recommends or Suggests
> be allowed by this wording, or are we intending for this to be a
> mandatory hard dependency?

I'm not sure it's going to make a lot of difference in practice, since I
think it will be hard to end up with a system that doesn't have a
systemd-tmpfiles implementation installed, but I agree that in theory this
is too strong.  I'll try to come up with a rewording that allows for the
possibility of Recommends or Suggests.  Maybe just a parenthetical that
says or Recommends or Suggests if this more accurately fits the nature of
the dependency?

I think apart from this and resolving whether to add empty directories
into the deb, the remaining issue before we can merge this is to make sure
that the sysvinit maintainers are okay with adding the requirement that
the init system invoke a systemd-tmpfiles implementation periodically.  As
I would expect, the systemd-standalone-tmpfiles package only provides the
binary, not any init system integration.  Does anyone know if that
integration has already been done to invoke systemd-tmpfiles during boot
on systems using sysvinit?

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Fri, 15 Sept 2023 at 22:18, Russ Allbery  wrote:
>
> Guillem Jover  writes:
>
> > Not shipping these empty directories in the .deb seems like a regression
> > or a disservice to me. Even for things that might get deleted because
> > things like our policy or the FHS allows for it (say stuff under
> > /var/cache), as «dpkg --verify» can be useful. Because of course, these
> > in addition, can be defined via tmpfiles.d, so that they can possibly be
> > recreated if needed (until dpkg provides its own interfaces to do that).
>
> Luca, are there any drawbacks for your purposes in both shipping the
> directories in the deb *and* defining them with tmpfiles.d for those cases
> where it is possible to ship them in the deb (no dynamic owner or group,
> for instance)?  At first glance, it feels like this should be fine, since
> tmpfiles.d will recreate the directories and dpkg will then be happy with
> them.
>
> It does potentially create problems if dpkg and tmpfiles.d have different
> ideas about what the ownership or permissions of the directories should
> be, but at present I don't think such conflicts would create a lot of
> practical problems (tmpfiles.d should essentialy always win), so I think
> it's a fairly minor point.
>
> It's a bit more complicated to specify in Policy because it's not possible
> to include the directory in the deb file in cases where it needs to have
> ownership set based on users or groups created dynamically by the
> maintainer scripts, but hopefully not overly complicated.

Aside from more practical considerations, shipping /var content in
packages is problematic because it's supposed to be local variable
data, that can be removed without breaking a system. This is by
definition not the case if the system's state becomes inconsistent
because packages, that have fixed content, ship files that can then be
removed locally. This is one of the many reasons why recently rpm
moved its database into /usr, as that's really what it is tied to and
where packages ship files into. AFAIK a few people have already
started some time ago to fix this on a package-by-package basis -
fortunately it's not that many, iirc.

More practically, one of the purposes of the hermetic-usr pattern is
to allow several modernizations. The easiest one to achieve is to
create /var/ on firstboot, and encrypt it against the tpm, so that it
can be enabled by default, always, so we can't have packages ship and
expect content in /var from their packages. This is a concrete and
achievable step forward to catch up to other OSes, as Linux is
embarrassingly behind Windows and OSX on the security aspect, and we
have a lot of work to do.

On top of that, as you mentioned already things will inevitably get
out of sync, and one will have to duplicate everything. Also
inevitably it will end up being wrong in cases where different
metadata has to be specified, with conflicts. This seems just busywork
that we can spare ourselves.

And if dpkg gets the ability to read tmpfiles.d - then that's great,
and even more reasons not to change policy for something that would
only be a temporary stop-gap. rpm recently gained native support for
sysusers.d and I believe they are looking into tmpfiles.d next, so
it's the right thing to do regardless.

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Guillem Jover  writes:

> Not shipping these empty directories in the .deb seems like a regression
> or a disservice to me. Even for things that might get deleted because
> things like our policy or the FHS allows for it (say stuff under
> /var/cache), as «dpkg --verify» can be useful. Because of course, these
> in addition, can be defined via tmpfiles.d, so that they can possibly be
> recreated if needed (until dpkg provides its own interfaces to do that).

Luca, are there any drawbacks for your purposes in both shipping the
directories in the deb *and* defining them with tmpfiles.d for those cases
where it is possible to ship them in the deb (no dynamic owner or group,
for instance)?  At first glance, it feels like this should be fine, since
tmpfiles.d will recreate the directories and dpkg will then be happy with
them.

It does potentially create problems if dpkg and tmpfiles.d have different
ideas about what the ownership or permissions of the directories should
be, but at present I don't think such conflicts would create a lot of
practical problems (tmpfiles.d should essentialy always win), so I think
it's a fairly minor point.

It's a bit more complicated to specify in Policy because it's not possible
to include the directory in the deb file in cases where it needs to have
ownership set based on users or groups created dynamically by the
maintainer scripts, but hopefully not overly complicated.

This has the valuable benefit, as Guillem points out, of retaining dpkg
database awareness of the association between that directory and a package
until such time as dpkg is aware of files defined in tmpfiles.d (directly
or indirectly via debhelper magic to register the tmpfiles.d targets with
a new dpkg dynamic file database; the latter is my guess about where we're
headed based on previous discussions).

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Guillem Jover]

Hi!

On Tue, 2023-09-12 at 22:17:44 -0700, Russ Allbery wrote:
> Russ Allbery  writes:
> > Russ Allbery  writes:
> >> Maybe the right way to do this is just have two examples, one as the
> >> default and another if you're using tmpfiles.d functionality added in a
> >> specific version of systemd that's newer than the version shipped with
> >> the stable version of Debian prior to the one you're targeting.
> 
> > Here's an updated version with that change plus some other minor fixes.
> 
> Er, right, helps to rebase first.  Here's the actual patch.

> diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> index b34c183..fa3e5be 100644
> --- a/policy/ch-files.rst
> +++ b/policy/ch-files.rst
> @@ -722,6 +722,70 @@ The name of the files and directories installed by 
> binary packages
>  outside the system PATH must be encoded in UTF-8 and should be
>  restricted to ASCII when it is possible to do so.
>  
> +.. _s-tmpfiles.d:
> +
> +Volatile and temporary files (``tmpfiles.d``)
> +-
> +
> +Some packages require empty directories in ``/var`` or ``/etc``, or

Not shipping these empty directories in the .deb seems like a
regression or a disservice to me. Even for things that might get
deleted because things like our policy or the FHS allows for it (say
stuff under /var/cache), as «dpkg --verify» can be useful. Because of
course, these in addition, can be defined via tmpfiles.d, so that they
can possibly be recreated if needed (until dpkg provides its own
interfaces to do that).

> +symlinks or files with trivial content in ``/var``, to implement their
> +functionality.  Examples include directories under ``/var/cache`` that are
> +writable by the package as cache areas, an initially-empty directory in
> +``/etc`` intended for local overrides added by the local system
> +administrator, or a file in ``/var`` that should default to a symlink
> +elsewhere on the system but may be changed later.
> +
> +Rather than include these symlinks, files, or directories in the binary
> +package or create them in package maintainer scripts, packages should use
> +the ``tmpfiles.d`` mechanism to specify the files and directories that
> +should be created.  This allows associating these files and directories
> +with specific packages (not currently possible when creating them in
> +maintainer scripts),

Well, this association would then only be indirect, instead of being able
to get at them via say «dpkg-query --search» or «dpkg-query --listfiles».

>   and allows local administrators to delete the
> +contents of directories such as ``/var/cache`` with the assurance that
> +``tmpfiles.d`` can recreate the necessary file structure without
> +reinstalling packages or re-running maintainer scripts.

Thanks,
Guillem

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Wed, 13 Sept 2023 at 15:37, Sam Hartman  wrote:
>
> > "Russ" == Russ Allbery  writes:
>
> I don't know if this needs seconds, but I reviewed all the text and it
> looks good.
> If seconds are required, I second.

Same, in case ownership passes to Russ, seconded/approved/you have my
sword/etc etc

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sam Hartman]

> "Russ" == Russ Allbery  writes:

I don't know if this needs seconds, but I reviewed all the text and it
looks good.
If seconds are required, I second.


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Tue, 12 Sep 2023 at 22:17:44 -0700, Russ Allbery wrote:
> >> Maybe the right way to do this is just have two examples, one as the
> >> default and another if you're using tmpfiles.d functionality added in a
> >> specific version of systemd that's newer than the version shipped with
> >> the stable version of Debian prior to the one you're targeting.

Yes, I think that reads better than what I initially suggested. Thanks!

The key piece of information that was missing from your previous proposal
was that systemd-tmpfiles interface versions match upstream systemd
version numbers. As a concrete example, if someone wants to upload an
implementation other than the one from systemd, it cannot have

Provides: systemd-tmpfiles (= 254)

until it has at least a basic implementation of the new "X", "C+" and
"--graceful" features introduced in systemd 254.

> +All packages that ship ``tmpfiles.d`` configuration should declare a
> +dependency on::
> +
> +default-systemd-tmpfiles | systemd-tmpfiles

If the package benefits from running tmpfiles.d but does not strictly
require it (for example dbus-daemon, where /run/dbus/containers is needed
for some optional functionality), would a Recommends or Suggests be
allowed by this wording, or are we intending for this to be a mandatory
hard dependency?

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Russ Allbery  writes:
> Russ Allbery  writes:

>> Maybe the right way to do this is just have two examples, one as the
>> default and another if you're using tmpfiles.d functionality added in a
>> specific version of systemd that's newer than the version shipped with
>> the stable version of Debian prior to the one you're targeting.

> Here's an updated version with that change plus some other minor fixes.

Er, right, helps to rebase first.  Here's the actual patch.

-- 
Russ Allbery (r...@debian.org)  

diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..fa3e5be 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -722,6 +722,70 @@ The name of the files and directories installed by binary packages
 outside the system PATH must be encoded in UTF-8 and should be
 restricted to ASCII when it is possible to do so.
 
+.. _s-tmpfiles.d:
+
+Volatile and temporary files (``tmpfiles.d``)
+-
+
+Some packages require empty directories in ``/var`` or ``/etc``, or
+symlinks or files with trivial content in ``/var``, to implement their
+functionality.  Examples include directories under ``/var/cache`` that are
+writable by the package as cache areas, an initially-empty directory in
+``/etc`` intended for local overrides added by the local system
+administrator, or a file in ``/var`` that should default to a symlink
+elsewhere on the system but may be changed later.
+
+Rather than include these symlinks, files, or directories in the binary
+package or create them in package maintainer scripts, packages should use
+the ``tmpfiles.d`` mechanism to specify the files and directories that
+should be created.  This allows associating these files and directories
+with specific packages (not currently possible when creating them in
+maintainer scripts), and allows local administrators to delete the
+contents of directories such as ``/var/cache`` with the assurance that
+``tmpfiles.d`` can recreate the necessary file structure without
+reinstalling packages or re-running maintainer scripts.
+
+For information on how to specify files and directories that should be
+managed by the ``tmpfiles.d`` mechanism, see :manpage:`tmpfiles.d(5)`.
+
+If the files or directories are only needed by a system service or
+otherwise should only be created when that service is running, packages
+should define those files and directories in the ``systemd`` unit for the
+service (and, for alternative init systems, in the configuration for that
+init system) instead of using the ``tmpfiles.d`` mechanism.  See
+:ref:`s-services-dirs` for more details.
+
+The ``tmpfiles.d`` mechanism may also be used to create and manage files
+and directories under ephemeral file systems such as ``/tmp`` and
+``/run``, although these are more likely to be associated with a running
+service and in those cases should be defined in the ``systemd`` unit for
+the service.
+
+All packages that ship ``tmpfiles.d`` configuration should declare a
+dependency on::
+
+default-systemd-tmpfiles | systemd-tmpfiles
+
+If the package uses ``tmpfiles.d`` features that are not supported by all
+implementations of the ``systemd-tmpfiles`` virtual package in the stable
+release prior to the release being targeted, instead use::
+
+default-systemd-tmpfiles (>= v) | systemd-tmpfiles (>= v)
+
+where ``v`` is the version of ``systemd`` in which the features were
+introduced.
+
+All packages that ship ``tmpfiles.d`` configuration should arrange for
+that configuration to be processed during package installation.  This
+should be handled by the packaging helper framework; for example, packages
+using ``debhelper`` should use ``dh_installtmpfiles``, which will add the
+appropriate commands to the package maintainer scripts.
+
+The init system must ensure that ``tmpfiles.d`` configuration is applied
+during boot and that ``tmpfiles.d`` cleanup rules are invoked
+periodically.  See :manpage:`systemd-tmpfiles(8)` for more information on
+how to do this.
+
 .. [#]
If you are using GCC, ``-fPIC`` produces code with relocatable
position independent code, which is required for most architectures
diff --git a/policy/ch-maintainerscripts.rst b/policy/ch-maintainerscripts.rst
index 724074c..e43340f 100644
--- a/policy/ch-maintainerscripts.rst
+++ b/policy/ch-maintainerscripts.rst
@@ -50,6 +50,11 @@ absolute pathname. Maintainer scripts should also not reset the
 appending package-specific directories. These considerations really
 apply to all shell scripts.
 
+Maintainer scripts should not be used to create empty directories in
+``/var`` or ``/etc``, or symlinks or files with trivial content in
+``/var``.  Instead, use the ``tmpfiles.d`` mechanism to manage those
+directories and files.  See :ref:`s-tmpfiles.d` for more information.
+
 .. _s-idempotency:
 
 Maintainer scripts idempotency
diff --git a/policy/ch-opersys.rst b/policy/ch-opersys.rst
index 64c0ff6..e80e9b4 100644

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Russ Allbery  writes:

> Maybe the right way to do this is just have two examples, one as the
> default and another if you're using tmpfiles.d functionality added in a
> specific version of systemd that's newer than the version shipped with
> the stable version of Debian prior to the one you're targeting.

Here's an updated version with that change plus some other minor fixes.

-- 
Russ Allbery (r...@debian.org)  

diff --git a/debian/changelog b/debian/changelog
index 4cead28..44a3710 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -24,17 +24,6 @@ debian-policy (4.6.3.0) UNRELEASED; urgency=medium
 Seconded: Helmut Grohne 
 Seconded: Guillem Jover 
 Closes: #970234
-  * Policy: Binary and Description fields may be absent in .changes
-Wording: Russ Allbery 
-Seconded: Sam Hartman 
-Seconded: Guillem Jover 
-Closes: #963524
-  * Policy: systemd units are required to start and stop system services
-Wording: Luca Boccassi 
-Wording: Russ Allbery 
-Seconded: Luca Boccassi 
-Seconded: Sam Hartman 
-Closes: #1039102
 
  -- Sean Whitton   Wed, 14 Jun 2023 16:58:40 +0100
 
diff --git a/policy/ch-controlfields.rst b/policy/ch-controlfields.rst
index d5c9d68..4bab7df 100644
--- a/policy/ch-controlfields.rst
+++ b/policy/ch-controlfields.rst
@@ -278,7 +278,7 @@ The fields in this file are:
 
 -  :ref:`Source ` (mandatory)
 
--  :ref:`Binary ` (mandatory in some cases)
+-  :ref:`Binary ` (mandatory)
 
 -  :ref:`Architecture ` (mandatory)
 
@@ -292,7 +292,7 @@ The fields in this file are:
 
 -  :ref:`Changed-By `
 
--  :ref:`Description ` (mandatory in some cases)
+-  :ref:`Description ` (mandatory)
 
 -  :ref:`Closes `
 
@@ -812,16 +812,12 @@ See :ref:`s-descriptions` for further information on
 this.
 
 In a ``.changes`` file, the ``Description`` field contains a summary of
-the descriptions of the binary packages being uploaded. If no binary
-packages are being uploaded, this field will not be present.
-
-When used inside a ``.changes`` file, the ``Description`` field has a
-different format than in source or binary control files. It is a multiline
-field with one line per binary package. The first line of the field value
-(the part on the same line as ``Description:``) is always empty. Each
-subsequent line is indented by one space and contains the name of a binary
-package, a space, a hyphen (``-``), a space, and the short description
-line from that package.
+the descriptions for the packages being uploaded. For this case, the
+first line of the field value (the part on the same line as
+``Description:``) is always empty. It is a multiline field, with one
+line per package. Each line is indented by one space and contains the
+name of a binary package, a space, a hyphen (``-``), a space, and the
+short description line from that package.
 
 .. _s-f-Distribution:
 
@@ -931,8 +927,7 @@ every architecture. The source control file doesn't contain details of
 which architectures are appropriate for which of the binary packages.
 
 When it appears in a ``.changes`` file, it lists the names of the binary
-packages being uploaded, separated by whitespace (not commas). If no
-binary packages are being uploaded, this field will not be present.
+packages being uploaded, separated by whitespace (not commas).
 
 .. _s-f-Installed-Size:
 
diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..fa3e5be 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -722,6 +722,70 @@ The name of the files and directories installed by binary packages
 outside the system PATH must be encoded in UTF-8 and should be
 restricted to ASCII when it is possible to do so.
 
+.. _s-tmpfiles.d:
+
+Volatile and temporary files (``tmpfiles.d``)
+-
+
+Some packages require empty directories in ``/var`` or ``/etc``, or
+symlinks or files with trivial content in ``/var``, to implement their
+functionality.  Examples include directories under ``/var/cache`` that are
+writable by the package as cache areas, an initially-empty directory in
+``/etc`` intended for local overrides added by the local system
+administrator, or a file in ``/var`` that should default to a symlink
+elsewhere on the system but may be changed later.
+
+Rather than include these symlinks, files, or directories in the binary
+package or create them in package maintainer scripts, packages should use
+the ``tmpfiles.d`` mechanism to specify the files and directories that
+should be created.  This allows associating these files and directories
+with specific packages (not currently possible when creating them in
+maintainer scripts), and allows local administrators to delete the
+contents of directories such as ``/var/cache`` with the assurance that
+``tmpfiles.d`` can recreate the necessary file structure without
+reinstalling packages or re-running maintainer scripts.
+
+For information on how to specify files and directories 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Mon, 11 Sep 2023, 17:58 Russ Allbery,  wrote:

> Luca Boccassi  writes:
>
> > Two more things went missing: Simon's suggestion on the versioned
> > dependencies on the virtual packages,
>
> Ah, yes, I'm sorry, I talked myself out of that and then completely forgot
> the previous discussion so didn't say anything.
>
> My concern is that it felt like we were providing a detailed description
> of an entirely normal dependency management situation (you always have to
> depend on the version of a package you use that provides the interface
> you're using unless it's old enough that it doesn't matter), and I wasn't
> sure what was special about this one that warranted spelling that out
> other than the need to add the version constraint to both stanzas.  So I
> kept that part but omitted the rest.
>
> The phrasing Simon proposed I think would be appropriate if we thought
> most packages would need a version constraint, but I didn't think the
> functionality was changing that quickly.  Am I wrong about that?  It felt
> awkward to include the version constraints and then tell people to remove
> them if they're going to be able to remove them 95% of the time, but I
> don't know if that's the case.
>
> Maybe the right way to do this is just have two examples, one as the
> default and another if you're using tmpfiles.d functionality added in a
> specific version of systemd that's newer than the version shipped with the
> stable version of Debian prior to the one you're targeting.
>

Ok, I have no opinion either way and am fine with the solution you and
Simon agree on

> and the link from the tmpfiles section to the service directory section
> > (given it was moved).
>
> It's there (last sentence):
>
> +If the files or directories are only needed by a system service or
> +otherwise should only be created when that service is running, packages
> +should define those files and directories in the ``systemd`` unit for the
> +service (and, for alternative init systems, in the configuration for that
> +init system) instead of using the ``tmpfiles.d`` mechanism.  See
> +:ref:`s-services-dirs` for more details.
>
> You don't need to spell out the section title; Sphinx defaults to adding
> that for you based on the heading that you're linking to.  (I think we are
> excessively explicit in a bunch of places in Policy currently due to a
> conversion artifact from DebianDoc-XML.)
>

Got it, thanks

>

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> Two more things went missing: Simon's suggestion on the versioned
> dependencies on the virtual packages,

Ah, yes, I'm sorry, I talked myself out of that and then completely forgot
the previous discussion so didn't say anything.

My concern is that it felt like we were providing a detailed description
of an entirely normal dependency management situation (you always have to
depend on the version of a package you use that provides the interface
you're using unless it's old enough that it doesn't matter), and I wasn't
sure what was special about this one that warranted spelling that out
other than the need to add the version constraint to both stanzas.  So I
kept that part but omitted the rest.

The phrasing Simon proposed I think would be appropriate if we thought
most packages would need a version constraint, but I didn't think the
functionality was changing that quickly.  Am I wrong about that?  It felt
awkward to include the version constraints and then tell people to remove
them if they're going to be able to remove them 95% of the time, but I
don't know if that's the case.

Maybe the right way to do this is just have two examples, one as the
default and another if you're using tmpfiles.d functionality added in a
specific version of systemd that's newer than the version shipped with the
stable version of Debian prior to the one you're targeting.

> and the link from the tmpfiles section to the service directory section
> (given it was moved).

It's there (last sentence):

+If the files or directories are only needed by a system service or
+otherwise should only be created when that service is running, packages
+should define those files and directories in the ``systemd`` unit for the
+service (and, for alternative init systems, in the configuration for that
+init system) instead of using the ``tmpfiles.d`` mechanism.  See
+:ref:`s-services-dirs` for more details.

You don't need to spell out the section title; Sphinx defaults to adding
that for you based on the heading that you're linking to.  (I think we are
excessively explicit in a bunch of places in Policy currently due to a
conversion artifact from DebianDoc-XML.)

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Mon, 11 Sept 2023 at 07:01, Russ Allbery  wrote:
>
> As usual, the things I notice only after I post text, even though I'd
> already read it several times.
>
> Russ Allbery  writes:
>
> > +Volatile and temporary files (``tmpfiles.d``)
> > +-
> > +
> > +Some packages require empty directories, files with trivial content (such
> > +as short fixed strings), or symlinks in ``/var`` or ``/etc`` to implement
> > +their functionality.
>
> Luca carefully worded this to avoid talking about files in /etc, and then
> I lost that distinction.  I now have:
>
> Some packages require empty directories in ``/var`` or ``/etc``, or
> symlinks or files with trivial content in ``/var``, to implement their
> functionality.

Two more things went missing: Simon's suggestion on the versioned
dependencies on the virtual packages, and the link from the tmpfiles
section to the service directory section (given it was moved).

Apart from that, looks good to me, thanks.

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

As usual, the things I notice only after I post text, even though I'd
already read it several times.

Russ Allbery  writes:

> +Volatile and temporary files (``tmpfiles.d``)
> +-
> +
> +Some packages require empty directories, files with trivial content (such
> +as short fixed strings), or symlinks in ``/var`` or ``/etc`` to implement
> +their functionality.

Luca carefully worded this to avoid talking about files in /etc, and then
I lost that distinction.  I now have:

Some packages require empty directories in ``/var`` or ``/etc``, or
symlinks or files with trivial content in ``/var``, to implement their
functionality.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> Moved as suggested. Also incorporated your suggestion on the versioned
> virtual package dependency verbatim.

Okay, I felt like doing editing this evening, apparently, so even though
only you, Sam, and Simon had a chance to respond, I went ahead and did the
editing.  I'm guessing we still have some discussion to get through, but
attached is a revised diff that I think captures the content of your diff
but adds some additional explanation and justification that I was kind of
craving.  Please let me know if I messed up any of the meaning here.

Note that this adds a must (held over from Luca's "required") for init
systems.  I don't want to introduce that into Policy until the sysvinit
maintainers have had a chance to weigh in or someone can confirm that
sysvinit already handles running systemd-tmpfiles appropriately when it is
installed.

I should note that I dropped the admonition in the maintainer scripts
section to use upstream's tmpfiles.d files because admonitions of this
type (from Lintian and elsewhere) annoy me.  The maintainer is in the best
possible position to balance the advantages of using upstream
configuration that is shared across distributions, bugs in the upstream
version that aren't fixed, upstream's ability to maintain those files
directly, whether upstream accepts contributions promptly, and whether
there are Debian-specific integration concerns that need to be addressed.

Less personally and more specific to Policy, making appropriate decisions
about when to use upstream files and when to use Debian-specific files is
a maintainer experience and expertise issue, not a Policy issue.  Policy
defines how the packages should work and is agnostic about where the
pieces of it come from.  If we want to give maintainers advice on how to
integrate upstream packages, I think that should go into Developers
Reference instead.

There was some earlier discussion in this bug about the possibility of
using tmpfiles.d to manage things like /run directories that, under
sysvinit, are currently managed in a somewhat ad hoc and untrackable way,
such as via mkdir in the init script.  I still think there's something
there, but I don't see a good way to describe it without creating possible
problems, so I left it out.

> We don't have to handle it with this change/bug and as mentioned I've
> already reworded it as suggested, but to clarify my thinking there, the
> place I was coming from was the factory reset and first boot angle. When
> doing a first boot with only the OS vendor tree under /usr and nothing
> else, you want to get to a working system, and if there are complex
> files created under /var by maintainer scripts, that's kinda of a
> problem.

Should Debian decide to adopt the OS vendor tree concept, I certainly
understand how what gnubg does would interfere with that.  This seemed
like the best of a set of bad options at the time.  I may adopt Simon's
idea of just putting the generated file in /usr, which would also allow me
to drop a Debian-specific patch; I didn't do that because putting files in
/usr that dpkg doesn't know about felt icky, but Simon is correct that
there are a lot of other precedents.

> Perhaps those complex binaries should be created by oneshot services
> that run at boot with a ConditionPathExists=!/var/some/complex/binary
> other than maintainer scripts? That way if /var is blown away, you still
> get a working system on next boot.

Yes, I could also do something like that.  Of course, the point may be
moot if upstream never ports GNU Backgammon to anything newer than Gtk+ 2,
and the chances of that port currently aren't looking great.

> But again, happy to shelve this for now, as it's a more complex topic.

Agreed, we don't have to cross this bridge today.

-- 
Russ Allbery (r...@debian.org)  

diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..cc685fe 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -722,6 +722,65 @@ The name of the files and directories installed by binary packages
 outside the system PATH must be encoded in UTF-8 and should be
 restricted to ASCII when it is possible to do so.
 
+.. _s-tmpfiles.d:
+
+Volatile and temporary files (``tmpfiles.d``)
+-
+
+Some packages require empty directories, files with trivial content (such
+as short fixed strings), or symlinks in ``/var`` or ``/etc`` to implement
+their functionality.  Examples include directories under ``/var/cache``
+that are writable by the package as cache areas, an initially-empty
+directory in ``/etc`` intended for local overrides added by the local
+system administrator, or a file in ``/var`` that should default to a
+symlink elsewhere on the system but may be changed later.
+
+Rather than include these files and directories in the binary package or
+create them in package maintainer scripts, packages should use the
+``tmpfiles.d`` mechanism to specify 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Edward Little]

Please remove the following email address:  e.little...@gmail.com

On Sat, Sep 9, 2023 at 10:54 PM Russ Allbery  wrote:

> Luca Boccassi  writes:
>
> > Sure, updated as suggested.
>
> I have a bunch of minor wording fixes that I'd want to make at this before
> merging, but that should be straightforward to do.  Before I invest the
> time in that, I want to check the opinions of everyone else following
> along and see if the semantics of Luca's change have general approval.
>
> Could folks take a look at this patch and see if the basic gist of it is
> something that they would second (or, for that matter, is something they
> would object to)?  I think I would second it (with wording adjustments),
> with one caveat mentioned below.  The whole thing is at:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#295
>
> Luca, am I right that service directories are specific to, well, services?
> If so, what would you think of moving them to Policy 9.3 alongside the
> other discussion of systemd units?  They feel out of place here, since
> packages that do not use services cannot use this functionality, and
> there's already a statement in the tmpfiles.d section pointing to them as
> more appropriate for services.
>
> > +Packages might need additional files or directories to implement their
> > +functionality. Directories that are located under ``/var/`` or
> > +``/etc/``, and files that are located under ``/var/``, should not be
> > +created manually via maintainer scripts, but instead be declaratively
> > +defined via the `tmpfiles.d
> > +`_
> > +interface.  Files and directories under ephemeral filesystems such as
> > +``/tmp/`` may also be created and managed via ``tmpfiles.d`` snippets.
>
> I understand the empty directory part, but I don't believe "files that are
> located under /var" is correct unless you specifically mean *empty* files
> (and even then, I'm not clear on precisely when this would be needed).
> For example, /var/lib/gnubg/gnubg_ts0.bd is created by the gnubg package
> maintainer script, and I can see no possible way that action could (or
> should) be handled by the tmpfiles.d mechanism.
>
> What am I missing?
>
> --
> Russ Allbery (r...@debian.org)  
>
>

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> On Sun, 10 Sept 2023 at 11:31, Simon McVittie  wrote:
>> 
>> On Sat, 09 Sep 2023 at 19:51:50 -0700, Russ Allbery wrote:
>> > Luca, am I right that service directories are specific to,
>> well, services?  > If so, what would you think of moving them to
>> Policy 9.3 alongside the > other discussion of systemd units?
>> They feel out of place here, since > packages that do not use
>> services cannot use this functionality
>> 
>> I'm not Luca, but I think you're correct here.

Luca> Moved as suggested. Also incorporated your suggestion on the
Luca> versioned virtual package dependency verbatim.

>> > Luca Boccassi  writes: > > +Packages might
>> need additional files or directories to implement their > >
>> +functionality. Directories that are located under ``/var/`` or >
>> > +``/etc/``, and files that are located under ``/var/``, should
>> not be > > +created manually via maintainer scripts, but instead
>> be declaratively > > +defined via the `tmpfiles.d > >
>> +`_
>> > > +interface.  Files and directories under ephemeral
>> filesystems such as > > +``/tmp/`` may also be created and
>> managed via ``tmpfiles.d`` snippets.
>> >
>> > I understand the empty directory part, but I don't believe
>> "files that are > located under /var" is correct unless you
>> specifically mean *empty* files > (and even then, I'm not clear
>> on precisely when this would be needed).  > For example,
>> /var/lib/gnubg/gnubg_ts0.bd is created by the gnubg package >
>> maintainer script, and I can see no possible way that action
>> could (or > should) be handled by the tmpfiles.d mechanism.
>> 
>> In general tmpfiles.d handles files that exist only as metadata:
>> symbolic links (for which the target is the only interesting
>> fact), empty files (for which the existence and
>> ownership/permissions are the only interesting facts),
>> directories (ditto) and so on.
>> 
>> It can also handle files that have static initial contents that
>> do not vary between systems, but can change in a system-specific
>> way later, with initial contents either hard-coded in the
>> tmpfiles.d snippet (for short text strings) or copied from
>> somewhere below /usr (canonically /usr/share/factory).
>> 
>> Files generated by non-trivial imperative code, like
>> machine-specific initial contents (/var/lib/dbus/machine-id) or
>> some sort of compiler (/var/lib/gnubg/gnubg_ts0.bd, as far as I
>> can tell), are out of scope for tmpfiles.d, so I think you're
>> right to be concerned that Luca's wording as written is asking
>> gnubg to do something that is unimplementable.
>> ch-maintainerscripts.rst has the same issue.
>> 
>> Perhaps "files with trivial contents that are located under /var"
>> would be a good wording that is not overly specific about
>> implementation details, covers the 90% case, and leaves room for
>> exceptions by declaring packages like dbus and gnubg to be
>> non-trivial?

Luca> I have reworded as suggested, citing symlinks or short fixed
Luca> strings as examples.

I second this patch, and do not need to additionally review Russ's
minor rewordings

--Sam


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 10 Sept 2023 at 11:31, Simon McVittie  wrote:
>
> On Sat, 09 Sep 2023 at 19:51:50 -0700, Russ Allbery wrote:
> > Luca, am I right that service directories are specific to, well, services?
> > If so, what would you think of moving them to Policy 9.3 alongside the
> > other discussion of systemd units?  They feel out of place here, since
> > packages that do not use services cannot use this functionality
>
> I'm not Luca, but I think you're correct here.

Moved as suggested. Also incorporated your suggestion on the versioned
virtual package dependency verbatim.

> > Luca Boccassi  writes:
> > > +Packages might need additional files or directories to implement their
> > > +functionality. Directories that are located under ``/var/`` or
> > > +``/etc/``, and files that are located under ``/var/``, should not be
> > > +created manually via maintainer scripts, but instead be declaratively
> > > +defined via the `tmpfiles.d
> > > +`_
> > > +interface.  Files and directories under ephemeral filesystems such as
> > > +``/tmp/`` may also be created and managed via ``tmpfiles.d`` snippets.
> >
> > I understand the empty directory part, but I don't believe "files that are
> > located under /var" is correct unless you specifically mean *empty* files
> > (and even then, I'm not clear on precisely when this would be needed).
> > For example, /var/lib/gnubg/gnubg_ts0.bd is created by the gnubg package
> > maintainer script, and I can see no possible way that action could (or
> > should) be handled by the tmpfiles.d mechanism.
>
> In general tmpfiles.d handles files that exist only as metadata: symbolic
> links (for which the target is the only interesting fact), empty files
> (for which the existence and ownership/permissions are the only interesting
> facts), directories (ditto) and so on.
>
> It can also handle files that have static initial contents that do not
> vary between systems, but can change in a system-specific way later,
> with initial contents either hard-coded in the tmpfiles.d snippet (for
> short text strings) or copied from somewhere below /usr (canonically
> /usr/share/factory).
>
> Files generated by non-trivial imperative code, like machine-specific
> initial contents (/var/lib/dbus/machine-id) or some sort of compiler
> (/var/lib/gnubg/gnubg_ts0.bd, as far as I can tell), are out of scope for
> tmpfiles.d, so I think you're right to be concerned that Luca's wording
> as written is asking gnubg to do something that is unimplementable.
> ch-maintainerscripts.rst has the same issue.
>
> Perhaps "files with trivial contents that are located under /var" would be
> a good wording that is not overly specific about implementation details,
> covers the 90% case, and leaves room for exceptions by declaring packages
> like dbus and gnubg to be non-trivial?

I have reworded as suggested, citing symlinks or short fixed strings
as examples.

> If /var/lib/gnubg/gnubg_ts0.bd is deterministically compiled from
> files shipped in the .deb as a time/space trade-off, is only written
> during package management operations, and is otherwise read-only, then
> perhaps it should live in /usr, but that's orthogonal to wanting to use
> tmpfiles.d where feasible. (Prior art for similar situations includes
> Python bytecode, glibc locales, GLib gschemas.compiled and giomodule.cache,
> and so on.)

We don't have to handle it with this change/bug and as mentioned I've
already reworded it as suggested, but to clarify my thinking there,
the place I was coming from was the factory reset and first boot
angle. When doing a first boot with only the OS vendor tree under /usr
and nothing else, you want to get to a working system, and if there
are complex files created under /var by maintainer scripts, that's
kinda of a problem. This is where tmpfiles.d plus Credentials
(https://systemd.io/CREDENTIALS/ see examples toward the end
especially) can come in and help, even with non-trivial files, such as
users, passwords and ssh keys.
Perhaps those complex binaries should be created by oneshot services
that run at boot with a ConditionPathExists=!/var/some/complex/binary
other than maintainer scripts? That way if /var is blown away, you
still get a working system on next boot.

But again, happy to shelve this for now, as it's a more complex topic.
See attached file for new revision, also pushed to Salsa.
From  Mon Sep 17 00:00:00 2001
From: Luca Boccassi 
Date: Tue, 9 May 2023 01:38:13 +0100
Subject: [PATCH] Define service directories and tmpfiles.d interfaces and
 usage

---
 policy/ch-files.rst | 49 +
 policy/ch-maintainerscripts.rst |  7 +
 policy/ch-opersys.rst   | 21 ++
 3 files changed, 77 insertions(+)

diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..7d0837c 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -722,6 +722,55 @@ 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

In general I support this direction.

On Sun, 25 Jun 2023 at 16:55:44 +0100, Luca Boccassi wrote:
> Packages shipping ``tmpfiles.d`` snippets should
> +depend on the appropriate virtual packages in the following order:
> +``default-systemd-tmpfiles | systemd-tmpfiles``.

I think it's worth saying explicitly that if the package relies on
functionality added in systemd version n, where n is newer than some
reasonable cutoff, it should depend on
default-systemd-tmpfiles (>= n) | systemd-tmpfiles (>= n)
instead.

If a package is targeting testing/unstable, then it can drop the version
constraint in the very common case where its tmpfiles.d snippet is processed
correctly by stable's systemd-tmpfiles. Similarly if it's targeting stable,
it can drop the version constraint if oldstable's systemd would have been
enough.

Perhaps this?

Packages shipping ``tmpfiles.d`` snippets should
depend on the appropriate virtual packages in the following order:
``default-systemd-tmpfiles (>= v) | systemd-tmpfiles (>= v)``,
where *v* is a version of systemd that provides all ``tmpfiles.d``
features that are required. The version constraint may be
omitted if it is satisfied by all implementations of the
``systemd-tmpfiles`` virtual package supported in the previous stable
release.

(If debhelper generates the dependency, in practice it's probably enough
for it to generate the unversioned dependency, and in the rare case
where a tmpfiles.d snippet needs new features, maintainers can add
the versioned dependency themselves.)

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Sat, 09 Sep 2023 at 19:51:50 -0700, Russ Allbery wrote:
> Luca, am I right that service directories are specific to, well, services?
> If so, what would you think of moving them to Policy 9.3 alongside the
> other discussion of systemd units?  They feel out of place here, since
> packages that do not use services cannot use this functionality

I'm not Luca, but I think you're correct here.

> Luca Boccassi  writes:
> > +Packages might need additional files or directories to implement their
> > +functionality. Directories that are located under ``/var/`` or
> > +``/etc/``, and files that are located under ``/var/``, should not be
> > +created manually via maintainer scripts, but instead be declaratively
> > +defined via the `tmpfiles.d
> > +`_
> > +interface.  Files and directories under ephemeral filesystems such as
> > +``/tmp/`` may also be created and managed via ``tmpfiles.d`` snippets.
> 
> I understand the empty directory part, but I don't believe "files that are
> located under /var" is correct unless you specifically mean *empty* files
> (and even then, I'm not clear on precisely when this would be needed).
> For example, /var/lib/gnubg/gnubg_ts0.bd is created by the gnubg package
> maintainer script, and I can see no possible way that action could (or
> should) be handled by the tmpfiles.d mechanism.

In general tmpfiles.d handles files that exist only as metadata: symbolic
links (for which the target is the only interesting fact), empty files
(for which the existence and ownership/permissions are the only interesting
facts), directories (ditto) and so on.

It can also handle files that have static initial contents that do not
vary between systems, but can change in a system-specific way later,
with initial contents either hard-coded in the tmpfiles.d snippet (for
short text strings) or copied from somewhere below /usr (canonically
/usr/share/factory).

Files generated by non-trivial imperative code, like machine-specific
initial contents (/var/lib/dbus/machine-id) or some sort of compiler
(/var/lib/gnubg/gnubg_ts0.bd, as far as I can tell), are out of scope for
tmpfiles.d, so I think you're right to be concerned that Luca's wording
as written is asking gnubg to do something that is unimplementable.
ch-maintainerscripts.rst has the same issue.

Perhaps "files with trivial contents that are located under /var" would be
a good wording that is not overly specific about implementation details,
covers the 90% case, and leaves room for exceptions by declaring packages
like dbus and gnubg to be non-trivial?

If /var/lib/gnubg/gnubg_ts0.bd is deterministically compiled from
files shipped in the .deb as a time/space trade-off, is only written
during package management operations, and is otherwise read-only, then
perhaps it should live in /usr, but that's orthogonal to wanting to use
tmpfiles.d where feasible. (Prior art for similar situations includes
Python bytecode, glibc locales, GLib gschemas.compiled and giomodule.cache,
and so on.)

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> Sure, updated as suggested.

I have a bunch of minor wording fixes that I'd want to make at this before
merging, but that should be straightforward to do.  Before I invest the
time in that, I want to check the opinions of everyone else following
along and see if the semantics of Luca's change have general approval.

Could folks take a look at this patch and see if the basic gist of it is
something that they would second (or, for that matter, is something they
would object to)?  I think I would second it (with wording adjustments),
with one caveat mentioned below.  The whole thing is at:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#295

Luca, am I right that service directories are specific to, well, services?
If so, what would you think of moving them to Policy 9.3 alongside the
other discussion of systemd units?  They feel out of place here, since
packages that do not use services cannot use this functionality, and
there's already a statement in the tmpfiles.d section pointing to them as
more appropriate for services.

> +Packages might need additional files or directories to implement their
> +functionality. Directories that are located under ``/var/`` or
> +``/etc/``, and files that are located under ``/var/``, should not be
> +created manually via maintainer scripts, but instead be declaratively
> +defined via the `tmpfiles.d
> +`_
> +interface.  Files and directories under ephemeral filesystems such as
> +``/tmp/`` may also be created and managed via ``tmpfiles.d`` snippets.

I understand the empty directory part, but I don't believe "files that are
located under /var" is correct unless you specifically mean *empty* files
(and even then, I'm not clear on precisely when this would be needed).
For example, /var/lib/gnubg/gnubg_ts0.bd is created by the gnubg package
maintainer script, and I can see no possible way that action could (or
should) be handled by the tmpfiles.d mechanism.

What am I missing?

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 23 Jul 2023 12:17:05 +0100 Luca Boccassi 
wrote:
> On Tue, 20 Jun 2023 22:53:24 +0100 Luca Boccassi 
> wrote:
> > On Sun, 18 Jun 2023 13:38:12 +0100 Luca Boccassi 
> > wrote:
> > > On Sun, 18 Jun 2023 at 13:03, Sean Whitton
> 
> > wrote:
> > > >
> > > > Hello,
> > > >
> > > > On Fri 16 Jun 2023 at 05:57PM +01, Luca Boccassi wrote:
> > > >
> > > > > Is there anything needed from me to make progress on this?
Any
> > changes
> > > > > required to the last revision posted?
> > > >
> > > > Yes, Russ posted some comments on your most recent revision, I
> > believe.
> > > 
> > > Last one I can find with specifics is:
> > > 
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#140
> > > 
> > > Which I think I have addressed in a follow-up. Did I miss
> something?
> > 
> > Russ, anything I've missed that you want me to change from the most
> > recent revision at
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#160 ?
> 
> Monthly ping. Anything I can do to unblock this?

Another monthly ping. Would be great to be able to make progress on
this.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Tue, 20 Jun 2023 22:53:24 +0100 Luca Boccassi 
wrote:
> On Sun, 18 Jun 2023 13:38:12 +0100 Luca Boccassi 
> wrote:
> > On Sun, 18 Jun 2023 at 13:03, Sean Whitton

> wrote:
> > >
> > > Hello,
> > >
> > > On Fri 16 Jun 2023 at 05:57PM +01, Luca Boccassi wrote:
> > >
> > > > Is there anything needed from me to make progress on this? Any
> changes
> > > > required to the last revision posted?
> > >
> > > Yes, Russ posted some comments on your most recent revision, I
> believe.
> > 
> > Last one I can find with specifics is:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#140
> > 
> > Which I think I have addressed in a follow-up. Did I miss
something?
> 
> Russ, anything I've missed that you want me to change from the most
> recent revision at
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#160 ?

Monthly ping. Anything I can do to unblock this?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Ansgar]

Hi Luca,

On Tue, 2023-06-20 at 22:53 +0100, Luca Boccassi wrote:
> Russ, anything I've missed that you want me to change from the most
> recent revision at
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#160 ?

I suggest a minor change:

In the paragraph

+---
| Init systems other than ``systemd`` should allow providing the same
| functionality as appropriate for each system, for example managing the
| directories from the init script shipped by the package.
+---

change "init script shipped by the package" to "appropriate service
startup configuration".

This should address concerns raise on d-devel@ that some packages might
not ship an init script. It also better covers alternative init systems
that do something more interesting than just starting the same sysvinit
scripts of old (not sure if any do).

Ansgar

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 25 Jun 2023 at 16:52, Ansgar  wrote:
>
> Hi Luca,
>
> On Tue, 2023-06-20 at 22:53 +0100, Luca Boccassi wrote:
> > Russ, anything I've missed that you want me to change from the most
> > recent revision at
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#160 ?
>
> I suggest a minor change:
>
> In the paragraph
>
> +---
> | Init systems other than ``systemd`` should allow providing the same
> | functionality as appropriate for each system, for example managing the
> | directories from the init script shipped by the package.
> +---
>
> change "init script shipped by the package" to "appropriate service
> startup configuration".
>
> This should address concerns raise on d-devel@ that some packages might
> not ship an init script. It also better covers alternative init systems
> that do something more interesting than just starting the same sysvinit
> scripts of old (not sure if any do).

Sure, updated as suggested.

Kind regards,
Luca Boccassi
From 8538f41dc235e9d9be4101595d469d389db693ef Mon Sep 17 00:00:00 2001
From: Luca Boccassi 
Date: Tue, 9 May 2023 01:38:13 +0100
Subject: [PATCH] Define service directories and tmpfiles.d interfaces and
 usage

---
 policy/ch-files.rst | 64 +
 policy/ch-maintainerscripts.rst |  7 
 2 files changed, 71 insertions(+)

diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..830f69c 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -722,6 +722,70 @@ The name of the files and directories installed by binary packages
 outside the system PATH must be encoded in UTF-8 and should be
 restricted to ASCII when it is possible to do so.
 
+.. _s-service-directories:
+
+Service Directories
+---
+
+Services might need auxiliary directories under ``/var/``, ``/run/`` or
+``/etc``. Instead of shipping empty directories in packages, or creating them
+with custom code in maintainer scripts, services should use ``systemd``'s native
+settings to ensure the required directories are created regardless of the
+privilege level under which the services are running. The relevant settings are
+`RuntimeDirectory=, StateDirectory=, CacheDirectory=, LogsDirectory= and
+ConfigurationDirectory=
+`_,
+covering respectively ``/run/``, ``/var/lib/``, ``/var/cache/``, ``/var/log/``
+and ``/etc/`` for system services, and the equivalent XDG-defined location for
+user services.
+
+Init systems other than ``systemd`` should allow providing the same
+functionality as appropriate for each system, for example managing the
+directories from the appropriate service startup configuration.
+
+.. _s-tmpfiles.d:
+
+tmpfiles.d
+--
+
+Packages might need additional files or directories to implement their
+functionality. Directories that are located under ``/var/`` or ``/etc/``, and
+files that are located under ``/var/``, should not be created manually via
+maintainer scripts, but instead be declaratively defined via the `tmpfiles.d
+`_ interface.
+Files and directories under ephemeral filesystems such as ``/tmp/`` may also be
+created and managed via ``tmpfiles.d`` snippets.
+
+When ownership of a directory can be clearly tied to a specific service,
+``Service Directories`` should be preferred to ``tmpfiles.d`` snippets.
+
+The ``tmpfiles.d`` file format is defined by the ``systemd`` project, and is
+guaranteed to be stable. Details about the syntax and installation paths are
+defined by its `reference implementation's documentation,
+`_ and will
+not be redefined here.
+
+``tmpfiles.d`` snippets should be usable on systems that do not boot (such as a
+very minimal chroot image), and also on systems booting with init systems other
+than ``systemd``.
+
+``tmpfiles.d`` snippets should be integrated in packages using automated shared
+tooling rather than by manually writing package-specific code in maintainers
+scripts. For example, packages built using ``debhelper`` should make use of the
+``dh_installtmpfiles`` addon. Packages shipping ``tmpfiles.d`` snippets should
+depend on the appropriate virtual packages in the following order:
+``default-systemd-tmpfiles | systemd-tmpfiles``.
+
+Init systems are required to integrate with ``tmpfiles.d`` and run the service
+that applies them on boot, and regularly for cleanup purposes, depending on the
+appropriate package providing the appropriate implementation that best
+integrates with each system. For example, ``systemd`` will make the reference
+implementation available when its main package is installed. The documentation
+for the reference implementation, `systemd-tmpfiles,
+`_
+explains how to call the program so that the appropriate ``tmpfiles.d`` snippets
+are applied at 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 18 Jun 2023 13:38:12 +0100 Luca Boccassi 
wrote:
> On Sun, 18 Jun 2023 at 13:03, Sean Whitton 
wrote:
> >
> > Hello,
> >
> > On Fri 16 Jun 2023 at 05:57PM +01, Luca Boccassi wrote:
> >
> > > Is there anything needed from me to make progress on this? Any
changes
> > > required to the last revision posted?
> >
> > Yes, Russ posted some comments on your most recent revision, I
believe.
> 
> Last one I can find with specifics is:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#140
> 
> Which I think I have addressed in a follow-up. Did I miss something?

Russ, anything I've missed that you want me to change from the most
recent revision at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#160 ?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 18 Jun 2023 at 13:03, Sean Whitton  wrote:
>
> Hello,
>
> On Fri 16 Jun 2023 at 05:57PM +01, Luca Boccassi wrote:
>
> > Is there anything needed from me to make progress on this? Any changes
> > required to the last revision posted?
>
> Yes, Russ posted some comments on your most recent revision, I believe.

Last one I can find with specifics is:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945269#140

Which I think I have addressed in a follow-up. Did I miss something?

Kind regards,
Luca Boccassi

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sean Whitton]

Hello,

On Fri 16 Jun 2023 at 05:57PM +01, Luca Boccassi wrote:

> Is there anything needed from me to make progress on this? Any changes
> required to the last revision posted?

Yes, Russ posted some comments on your most recent revision, I believe.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Fri, 16 Jun 2023 10:51:17 +0100 Sean Whitton
 wrote:
> Hello,
> 
> On Tue 13 Jun 2023 at 05:58PM +01, Mark Hindley wrote:
> 
> > There is a new upstream version of elogind[1] that is already
packaged in
> > Devuan[2] although that uncovered up an upstream issue that I am
waiting to be
> > resolved[3]. So, maybe by the end of this month?
> >
> > However, that is only considering whether the packaging and
dependencies can be
> > made to work (like Simon McVittie, I think they probably can).
> >
> > I remain much less convinced that there is a consensus for
requiring packages to
> > use tmpfiles.d(5) for /var, /tmp and maybe /etc. The recent thread
on
> > debian-devel demonstrated a range of opinion. Thorsten and Bill
have just raised
> > valid points about chroots.
> >
> > So, whilst I am happy to test the dependency changes in elogind,
enshrining this
> > as a 'should' in the Policy now seems, at least, premature.
> 
> Cool, thank you.  This will simplify resolving this bug.

Is there anything needed from me to make progress on this? Any changes
required to the last revision posted?

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sean Whitton]

Hello,

On Tue 13 Jun 2023 at 05:58PM +01, Mark Hindley wrote:

> There is a new upstream version of elogind[1] that is already packaged in
> Devuan[2] although that uncovered up an upstream issue that I am waiting to be
> resolved[3]. So, maybe by the end of this month?
>
> However, that is only considering whether the packaging and dependencies can 
> be
> made to work (like Simon McVittie, I think they probably can).
>
> I remain much less convinced that there is a consensus for requiring packages 
> to
> use tmpfiles.d(5) for /var, /tmp and maybe /etc. The recent thread on
> debian-devel demonstrated a range of opinion. Thorsten and Bill have just 
> raised
> valid points about chroots.
>
> So, whilst I am happy to test the dependency changes in elogind, enshrining 
> this
> as a 'should' in the Policy now seems, at least, premature.

Cool, thank you.  This will simplify resolving this bug.

> Reading the proposed text as somebody who is particularly interested
> in non-systemd systems, I am struck by the inconsistency between
>
>   Init systems other than ``systemd`` should allow providing the same
>   functionality as appropriate for each system, for example managing the
>   directories from the init script shipped by the package.
>
> and the fact that we no longer expect packages to include init scripts 
> alongside
> their systemd units and even accept their removal, even if other interested
> people offer to maintain them and provide tested patches.

I'm sympathetic, though, this in itself is not a Policy issue.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Thorsten Glaser]

On Tue, 13 Jun 2023, Russ Allbery wrote:

>Ah, I think I understand what you're getting at.  You're talking about
>using the init script of a daemon as this sort of wrapper script for

Not really. I was talking about normal programs, not dæmons.
I have the expectation that these, when invoked, create their
necessary temporary files/directories when they are placed on
volatile storage, i.e. /tmp and /run and possibly /var/tmp.

>nothing we're talking about here changes that behavior, because nothing
>needs to be pre-created.

Ah, good then.

For dæmons, running them in chroots is usually more tricky
anyway. Ideally, just /etc/init.d/foo {stop|start} would
work, but there’s situations where that doesn’t suffice.
If maintainers can get the former working, fine.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg


/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against  Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Ansgar]

On Tue, 2023-06-13 at 12:51 -0700, Russ Allbery wrote:
> I still am curious if it's safe to configure the same files in both
> tmpfiles.d and in the unit file, because it would make it much easier for
> those who want to support other init systems to do so.

Having this configured in two places would at least be confusing for
users: where would users need to change settings? In the unit file? In
the tmpfiles files? Both?

Ansgar

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Tue, 13 Jun 2023 at 20:51, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > That paragraph is in the context of StateDirectory= and
> > RuntimeDirectory=. These are unit files options, so it's up to
> > alternative init systems to provide alternative and integrate them in
> > the (eventual) init script, just as they are defined in the systemd
> > unit. I've mentioned those explicitly as you indicated earlier in this
> > thread:
>
> Ah!  Sorry, I did inded not understand the context correctly, and should
> have just not replied until I had a chance to read the context.  That's my
> fault.
>
> I agree with this statement with respect to systemd unit features.  This
> is a consequences of the fact that units are the preferred daemon
> configuration and package maintainers are allowed to use its features (GR
> 2019-002).  We should be encouraging them to use those features correctly
> and documenting how to do so, but that necessarily means that init scripts
> for use with other init systems will need to provide a version of the same
> feature in some other way.  We have had several GRs on this topic, and
> Policy does need to follow the outcome of those GRs unless someone
> overturns them with another GR.
>
> That being said:
>
> > Again, the rationale is: when there is a strong ownership model tied to
> > an individual service those are best as the lifecycle and permissions
> > are handled, when there is no owner or no specific owner or particular
> > metadata requirements then tmpfiles.d are best.
>
> I still am curious if it's safe to configure the same files in both
> tmpfiles.d and in the unit file, because it would make it much easier for
> those who want to support other init systems to do so.

As long as they both do the exact same thing, the one that runs last
is a no-op, I don't believe there would be any issue. But they have to
be kept in sync, not only w.r.t. directory name, but metadata too (ie:
ownership). See it this way: if the features provided by tmpfiles.d
and unit files (w.r.t. directories) were on a venn diagram, there
would be an intersection, yes, but there would not be a complete
overlap. There are things tmpfiles.d can do, that StateDirectory= and
friends cannot, and vice-versa.
In other words, for your stated goal of helping those who want to
support other init systems, it would only cover a subset of cases -
and these are real world cases, there are many services out there
making use of these features. We cannot be restricted, in terms of
unit files options, to what tmpfiles.d can provide, as that is not
realistic.

This is why I added that paragraph, indicating that other init systems
should provide equivalent functionality (and bear in mind I am not a
policy guru, so if 'should' is the wrong word I can change as needed,
it's perfectly fine as far as I'm concerned if other init systems
simply say, run everything as root and forget about security. It's
their choice what features they provide and I was not intending to
force them to do anything.).

Kind regards,
Luca Boccassi

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> That paragraph is in the context of StateDirectory= and
> RuntimeDirectory=. These are unit files options, so it's up to
> alternative init systems to provide alternative and integrate them in
> the (eventual) init script, just as they are defined in the systemd
> unit. I've mentioned those explicitly as you indicated earlier in this
> thread:

Ah!  Sorry, I did inded not understand the context correctly, and should
have just not replied until I had a chance to read the context.  That's my
fault.

I agree with this statement with respect to systemd unit features.  This
is a consequences of the fact that units are the preferred daemon
configuration and package maintainers are allowed to use its features (GR
2019-002).  We should be encouraging them to use those features correctly
and documenting how to do so, but that necessarily means that init scripts
for use with other init systems will need to provide a version of the same
feature in some other way.  We have had several GRs on this topic, and
Policy does need to follow the outcome of those GRs unless someone
overturns them with another GR.

That being said:

> Again, the rationale is: when there is a strong ownership model tied to
> an individual service those are best as the lifecycle and permissions
> are handled, when there is no owner or no specific owner or particular
> metadata requirements then tmpfiles.d are best.

I still am curious if it's safe to configure the same files in both
tmpfiles.d and in the unit file, because it would make it much easier for
those who want to support other init systems to do so.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Tue, 13 Jun 2023 10:55:38 -0700 Russ Allbery  wrote:

> >   Init systems other than ``systemd`` should allow providing the same
> >   functionality as appropriate for each system, for example managing the
> >   directories from the init script shipped by the package.
>
> This sort of requirement is exactly what we should be getting rid of
> by
> using systemd-tmpfiles uniformly instead.  We should be trying to
> minimize
> the extra work required to support non-systemd init systems in every
> package if we want non-systemd init systems to remain viable, because
> we
> know that work largely won't happen.
> 
> So, in other words, I haven't read the latest version of the patch
> yet,
> but if that wording is in there and I'm understanding the context
> correctly, I think that's the opposite of what we should be saying
> and we
> should take it out in favor of saying everything should just invoke
> systemd-tmpfiles or some equivalent implementation that uses the same
> configuration.
> 
> If other init systems arrange for systemd-tmpfiles to be run when
> appropriate (at boot, mostly), then there is no need to provide
> fallbacks
> via, for instance, init scripts with different functionality than the
> systemd units.  This is the whole reason why we did the work to
> package a
> standalone systemd-tmpfiles package that can be used regardless of
> the
> init system.

That paragraph is in the context of StateDirectory= and
RuntimeDirectory=. These are unit files options, so it's up to
alternative init systems to provide alternative and integrate them in
the (eventual) init script, just as they are defined in the systemd
unit. I've mentioned those explicitly as you indicated earlier in this
thread:

> However, reading that documentation, it sounds like most of the cases for
> other directories are handled by other systemd unit configuration
> directives.  We should say that explicitly here and reproduce the list of
> other directories that should be handled directly by the unit file if
> that's what we want people to do.

Again, the rationale is: when there is a strong ownership model tied to
an individual service those are best as the lifecycle and permissions
are handled, when there is no owner or no specific owner or particular
metadata requirements then tmpfiles.d are best.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Thorsten Glaser  writes:
> On Tue, 13 Jun 2023, Russ Allbery wrote:

>> namely if you're running anything in a chroot that needs directories
>> created in /tmp and /run, the chroot either needs to have a persistent
>> /tmp and /run or you have to arrange for it to run at least some init
>> scripts during boot.

> I very much disagree here. Both /tmp and /run are volatile, and for /tmp
> there usually even are cronjobs that delete old files, so programs that
> need anything in there must create it themselves at startup (via wrapper
> scripts if needed) if absent.

Ah, I think I understand what you're getting at.  You're talking about
using the init script of a daemon as this sort of wrapper script for
running it in a chroot, by invoking the init script outside of an init
system as root, but inside the chroot.

This works in some situations when the init script has no other
dependencies, but is going to start bit-rotting because init scripts are
less frequently tested and people are going to forget to add separate code
to handle cases that are already handled by tmpfiles.d or the systemd
unit.  The replacement is to first run systemd-tmpfiles --create in the
chroot and then manually run the init script as before.  That does add an
additional step to the (somewhat rare) case of running daemons in chroots,
but it has the advantage of not having to add runtime code to every
package to create directories (often incorrectly, without handling edge
cases), and it's not that different from what you'd need to do to start
daemons in chroots that have dependencies on other daemons.  (It would
also be easy to add to a generic wrapper script for starting a daemon in a
Debian chroot that does this for you.)

If you're just talking about programs that need temporary directories in
/tmp (not /run, which is owned by root) owned by the same user that the
program is running as, or programs that only run as root creating PID
files in /run, then that is unrelated to this bug so far as I can tell;
nothing we're talking about here changes that behavior, because nothing
needs to be pre-created.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Thorsten Glaser]

On Tue, 13 Jun 2023, Russ Allbery wrote:

>namely if you're running anything
>in a chroot that needs directories created in /tmp and /run, the chroot
>either needs to have a persistent /tmp and /run or you have to arrange for
>it to run at least some init scripts during boot.

I very much disagree here. Both /tmp and /run are volatile, and for /tmp
there usually even are cronjobs that delete old files, so programs that
need anything in there must create it themselves at startup (via wrapper
scripts if needed) if absent.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg


/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against  Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Thorsten Glaser  writes:

> what you described of course does not work for /tmp and /run.
> It is viable for /var/tmp etc.

Well, it does work for /tmp and /run as well as anything else can possibly
work for /tmp and /run inside a chroot, namely if you're running anything
in a chroot that needs directories created in /tmp and /run, the chroot
either needs to have a persistent /tmp and /run or you have to arrange for
it to run at least some init scripts during boot.

My experience is that mostly the sorts of stuff that needs specific
directories in /tmp and /run isn't run in a chroot, and when it is, often
they just use persistent /tmp and /run.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Thorsten Glaser]

On Tue, 13 Jun 2023, Russ Allbery wrote:

>Thorsten Glaser  writes:
>
>> Therefore I belive that Policy ought to *not* recommend any solution
>> that depends on starting dæmons or init scripts to create temporary
>> files/directories that are necessary for programs to work.
>
>This is handled by this proposal, no?  That's the point of requiring
>integration with maintainer scripts (via triggers or direct invocation).
>My understanding is that this is exactly what dh_installtmpfiles already
>does, via generating an explicit call to systemd-tmpfiles --create.

Hmm, that sounds okay-ish, except…

>Or am I missing something?

what you described of course does not work for /tmp and /run.
It is viable for /var/tmp etc.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg


/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against  Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Ansgar  writes:
> On Tue, 2023-06-13 at 18:36 +0200, Marco d'Itri wrote:
>> On Jun 13, Bill Allombert  wrote:

>>> Conversely, sometimes I need to use chroots to test init scripts.
>>> start-stop-daemon should not refuse to run in a chroot if policy-
>>> rc.d allows
>>> it.

>> I suggest that you try systemd-nspawn for this purpose.

> Or podman or docker or various other things.

> Plain chroots and an unclean environment which violates various
> assumptions system startup scripts make are not a great way to test
> stuff.

Unless I'm very mistaken about how dh_installtmpfiles and systemd-tmpfiles
works (possible, I guess), I don't think we need to have this argument in
this bug.  If I am right in assuming that nothing about this proposal will
change how chroots work, arguing with people about why they use chroots is
just going to add heat without any light.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Mark Hindley  writes:

> I remain much less convinced that there is a consensus for requiring
> packages to use tmpfiles.d(5) for /var, /tmp and maybe /etc.

Well, for /tmp, /var/tmp, and /run, I think this is the right approach,
unless there is some other systemd unit configuration that is even better.
This is also already what we're doing across the archive, which is a
pretty strong indication of consensus.  The alternative for /tmp,
/var/tmp, and /run is to add manual mkdirs to systemd units or init
scripts (that normally then have to be run as root, even when the daemon
doesn't), and that's clearly inferior and a poor technical approach.

Given that, if we want to keep everything working with non-systemd init
systems, they're pretty much going to have to invoke systemd-tmpfiles
during boot or things are going to start breaking just because they're not
tested (or some equivalent; there was some discussion of adding support
for the tmpfiles.d format directly to dpkg so that dpkg is also aware of
what package owns these files).

I am considerably more dubious about /var and /etc in general.  I don't
think there's a consensus for managing files or directories there with
systemd-tmpfiles, and most packages just ship the directories in the
package.  I understand that this is part of the overall goal of making
Linux distributions only ship files in /usr, but the project as a whole
has not signed on to that goal yet.

However, what's being proposed here is something much narrower: we should
not be managing directories in those paths *with mkdir in maintainer
scripts*, and instead should use systemd-tmpfiles in that specific case.
For those rare cases where packages are manually creating directories in
shell instead of shipping them in the package for whatever reason,
switching to systemd-tmpfiles feels like an obviously correct improvement
as long as the maintainer script arranges for systemd-tmpfiles to be
invoked so that it will create the directories.  It allows us to move
something from imperative maintainer scripts that are very hard to analyze
to declarative files that have well-understood semantics and handle all
the edge cases that human-written maintainer scripts tend not to manage.
This change should also be invisible to other init systems since the files
and directories are still being created by the maintainer scripts as
always, just using a different program.

However, the really compelling use of systemd-tmpfiles is for /tmp, /run,
and /var/tmp, replacing all the various hacks and workarounds we have had
for decades to create those files during boot, at least in the cases where
the functionality isn't handled directly by a systemd unit.  It's that
behavior that I think deserves a should.

For the persistent directory case in /var and /etc, I think "encouraged"
(Policy advice) is more appropriate at this point, since creating the
directories in maintainer scripts is not *broken*, and we should not be
making those packages instantly buggy.

> The recent thread on debian-devel demonstrated a range of
> opinion. Thorsten and Bill have just raised valid points about chroots.

I don't understand the point about chroots.  It seemed to be based on a
fundmental misunderstanding of how systemd-tmpfiles works?  Thorsten
seemed to think it was a daemon; it's not.  It's essentially a fancy
version of mkdir + chmod + chown plus some other things that supports a
declarative syntax for specifying what directories should exist.  A good
analogy for a different type of operation would be start-stop-daemon
(except systemd-tmpfiles supports declarative configuration, which is even
better).

> Reading the proposed text as somebody who is particularly interested in
> non-systemd systems, I am struck by the inconsistency between

>   Init systems other than ``systemd`` should allow providing the same
>   functionality as appropriate for each system, for example managing the
>   directories from the init script shipped by the package.

This sort of requirement is exactly what we should be getting rid of by
using systemd-tmpfiles uniformly instead.  We should be trying to minimize
the extra work required to support non-systemd init systems in every
package if we want non-systemd init systems to remain viable, because we
know that work largely won't happen.

So, in other words, I haven't read the latest version of the patch yet,
but if that wording is in there and I'm understanding the context
correctly, I think that's the opposite of what we should be saying and we
should take it out in favor of saying everything should just invoke
systemd-tmpfiles or some equivalent implementation that uses the same
configuration.

If other init systems arrange for systemd-tmpfiles to be run when
appropriate (at boot, mostly), then there is no need to provide fallbacks
via, for instance, init scripts with different functionality than the
systemd units.  This is the whole reason why we did the work to package a
standalone 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Ansgar]

On Tue, 2023-06-13 at 18:36 +0200, Marco d'Itri wrote:
> On Jun 13, Bill Allombert  wrote:
> 
> > Conversely, sometimes I need to use chroots to test init scripts.
> > start-stop-daemon should not refuse to run in a chroot if policy-
> > rc.d allows
> > it.
> I suggest that you try systemd-nspawn for this purpose.
> 

Or podman or docker or various other things.

Plain chroots and an unclean environment which violates various
assumptions system startup scripts make are not a great way to test
stuff.

Ansgar

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Mark Hindley]

Sean,

On Tue, Jun 13, 2023 at 05:15:15PM +0100, Sean Whitton wrote:
> > In principle and just looking at the dependencies this seems a viable
> > solution.  It is very similar to the way we handle the logind and
> > default-logind virtual packages.
> 
> Thank you for reviewing.  Do you have a rough idea of how long it would
> be until you could confirm that this is viable, and implement it in sid?

There is a new upstream version of elogind[1] that is already packaged in
Devuan[2] although that uncovered up an upstream issue that I am waiting to be
resolved[3]. So, maybe by the end of this month?

However, that is only considering whether the packaging and dependencies can be
made to work (like Simon McVittie, I think they probably can).

I remain much less convinced that there is a consensus for requiring packages to
use tmpfiles.d(5) for /var, /tmp and maybe /etc. The recent thread on
debian-devel demonstrated a range of opinion. Thorsten and Bill have just raised
valid points about chroots.

So, whilst I am happy to test the dependency changes in elogind, enshrining this
as a 'should' in the Policy now seems, at least, premature.

Reading the proposed text as somebody who is particularly interested in
non-systemd systems, I am struck by the inconsistency between

  Init systems other than ``systemd`` should allow providing the same
  functionality as appropriate for each system, for example managing the
  directories from the init script shipped by the package.

and the fact that we no longer expect packages to include init scripts alongside
their systemd units and even accept their removal, even if other interested
people offer to maintain them and provide tested patches.

With best wishes

Mark

[1]  https://qa.debian.org/cgi-bin/watch?pkg=elogind

[2]  
https://pkginfo.devuan.org/cgi-bin/package-query.html?c=package=elogind=252.9-1~rc1

[3]  https://github.com/elogind/elogind/issues/258

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Marco d'Itri]

On Jun 13, Bill Allombert  wrote:

> Conversely, sometimes I need to use chroots to test init scripts.
> start-stop-daemon should not refuse to run in a chroot if policy-rc.d allows
> it.
I suggest that you try systemd-nspawn for this purpose.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Thorsten Glaser  writes:

> Therefore I belive that Policy ought to *not* recommend any solution
> that depends on starting dæmons or init scripts to create temporary
> files/directories that are necessary for programs to work.

This is handled by this proposal, no?  That's the point of requiring
integration with maintainer scripts (via triggers or direct invocation).
My understanding is that this is exactly what dh_installtmpfiles already
does, via generating an explicit call to systemd-tmpfiles --create.

Or am I missing something?

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sean Whitton]

Hello Mark,

On Tue 13 Jun 2023 at 01:51PM +01, Mark Hindley wrote:

> On Tue, Jun 06, 2023 at 11:58:07AM +0100, Simon McVittie wrote:
>> Exactly. My hope is that if we had:
>>
>> Package: systemd
>> Architecture: linux-any
>> Provides: default-systemd-tmpfiles, systemd-tmpfiles
>> Conflicts: systemd-tmpfiles
>> Replaces: systemd-tmpfiles
>>
>> Package: systemd-standalone-tmpfiles
>> Architecture: linux-any
>> Provides: systemd-tmpfiles
>> Conflicts: systemd-tmpfiles
>> Replaces: systemd-tmpfiles
>>
>> Package: elogind
>> Depends: systemd-standalone-tmpfiles# or Recommends?
>
> In principle and just looking at the dependencies this seems a viable
> solution.  It is very similar to the way we handle the logind and
> default-logind virtual packages.

Thank you for reviewing.  Do you have a rough idea of how long it would
be until you could confirm that this is viable, and implement it in sid?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Thorsten Glaser]

On Tue, 13 Jun 2023, Bill Allombert wrote:

>I agree, chroots are important to consider, and the system should not
>make assumptions how and why there are used.

Thanks!

>Conversely, sometimes I need to use chroots to test init scripts.
>start-stop-daemon should not refuse to run in a chroot if policy-rc.d
>allows it.

TTBOMK this works-ish. It certainly starts and stops things, but if
you have the same thing running outside of the chroot, interference
may happen. You’ll probably want a separate pid namespace (I think)
at least, and make sure that, when leaving the chroot, everything
started in it is in fact terminated; sometimes, things like to keep
hanging around. This is easier to manage with VMs or (probably; I
don’t like to use them myself) container-ish thingies.

In my schroot setup I used to start a vncserver in a persistent
chroot back when my main system was x32 and vncserver didn’t like
that nor was coïnstallable (hence the i386 chroot).

My “enter a Debian chroot” script, to use e.g. with a Grml live ISO
to fix the bootloader (or to work under qemu-user with an RPi µSD
image before moving it into the embedded machine), certainly tries
hard to create a policy-rc.d to disable dæmon starting should the
user need to install packages, so it generally will work.
https://evolvis.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=shellsnippets/shellsnippets.git;a=blob;f=posix/sysadmin/debchroot.sh;hb=HEAD
in case someone’s interested, it’s more complete than grml-chroot.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg


/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against  Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Bill Allombert]

On Tue, Jun 13, 2023 at 05:43:17PM +0200, Thorsten Glaser wrote:
> On Mon, 5 Jun 2023, Simon McVittie wrote:
> 
> >No init system at all, (C.), can only happen when starting with a
> >minbase debootstrap or equivalent (because a default debootstrap
> >includes the init metapackage due to its Priority: required). In
> >this scenario it *usually* doesn't really matter whether we
> >install systemd or systemd-tmpfiles-standalone. systemd is somewhat
> 
> This is not quite true; there is one really important use case:
> chroots. I have multiple chroots (sid, stretch, buster) on one
> of my bullseye systems which I use with schroot, but that could
> just as well be any other chroot, to run individual software in
> it. They are, as is proper, configured to not run any services
> (via policy-rc.d).

I agree, chroots are important to consider, and the system should not make
assumptions how and why there are used.

Conversely, sometimes I need to use chroots to test init scripts.
start-stop-daemon should not refuse to run in a chroot if policy-rc.d allows
it.

Cheers,
Bill.

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Thorsten Glaser]

On Mon, 5 Jun 2023, Simon McVittie wrote:

>No init system at all, (C.), can only happen when starting with a
>minbase debootstrap or equivalent (because a default debootstrap
>includes the init metapackage due to its Priority: required). In
>this scenario it *usually* doesn't really matter whether we
>install systemd or systemd-tmpfiles-standalone. systemd is somewhat

This is not quite true; there is one really important use case:
chroots. I have multiple chroots (sid, stretch, buster) on one
of my bullseye systems which I use with schroot, but that could
just as well be any other chroot, to run individual software in
it. They are, as is proper, configured to not run any services
(via policy-rc.d).

>I also think that Policy shouldn't be recommending this interface without

Therefore I belive that Policy ought to *not* recommend any
solution that depends on starting dæmons or init scripts to
create temporary files/directories that are necessary for
programs to work.

>- if we get a useful non-Linux port (admittedly this looks increasingly
>  unlikely) which cannot compile src:systemd, then their reimplementation

hurd-amd64 is just shy of being uploaded; work on Debian GNU/Hurd
is active and things seem to look good on that front. (The pools
for hurd-amd64 have just been created a week or two ago.)

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg


/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against  Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Mark Hindley]

Simon,

Thanks for your care and insight with this and apologies for the delay in
replying (mails to elog...@packages.debian.org have been held up on a
mailserver).

On Tue, Jun 06, 2023 at 11:58:07AM +0100, Simon McVittie wrote:
> Exactly. My hope is that if we had:
> 
> Package: systemd
> Architecture: linux-any
> Provides: default-systemd-tmpfiles, systemd-tmpfiles
> Conflicts: systemd-tmpfiles
> Replaces: systemd-tmpfiles
> 
> Package: systemd-standalone-tmpfiles
> Architecture: linux-any
> Provides: systemd-tmpfiles
> Conflicts: systemd-tmpfiles
> Replaces: systemd-tmpfiles
> 
> Package: elogind
> Depends: systemd-standalone-tmpfiles# or Recommends?

In principle and just looking at the dependencies this seems a viable solution.
It is very similar to the way we handle the logind and default-logind virtual
packages.

Mark

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Wed, 7 Jun 2023 at 11:46, Luca Boccassi  wrote:
>
> On Wed, 7 Jun 2023 at 11:29, Simon McVittie  wrote:
> >
> > On Tue, 06 Jun 2023 at 20:40:52 -0700, Russ Allbery wrote:
> > > Luca Boccassi  writes:
> > > > +Packages might need additional files or directories to implement their
> > > > +functionality. Directories that are located under ``/var/`` or
> > > > +``/etc/``, and files that are located under ``/var/``, must not be
> > > > +created manually via maintainer scripts, but instead be declaratively
> > > > +defined via the `tmpfiles.d
> > > > +`_
> > > > +interface.
> > >
> > > This is an oddly specific list of directories and not at all the
> > > directories that I would have expected to be handled by tmpfiles.d.
> >
> > Sorry, in my previous reading of this bug I had been concentrating on
> > the mechanics of how to make tmpfiles.d(5) something that maintainers can
> > rely on if it's convenient/helpful, and I'd missed that Luca is asking
> > for its use to be mandatory in some cases.
> >
> > I would personally be inclined to concentrate on making tmpfiles.d(5)
> > something that we can rely on and encourage the use of where appropriate,
> > even on non-systemd systems, so that (upstream and downstream) maintainers
> > can move towards it of their own accord because it's more convenient
> > than other options, and put aside the question of making it generally a
> > "should" or "must" for the moment.
> >
> > I believe (please correct me if I'm wrong) that Luca's intention here
> > is that this is drawing a line between:
> >
> > - the static files of the OS: /usr and the /usr-like top-level directories
> >   (the ones that get merged by the /usr merge), which should be statically
> >   shipped in packages and managed by dpkg (or on "immutable" systems
> >   that use image-based/tree-based upgrades, maybe by ostree or casync
> >   or similar, from an tree originally constructed from dpkg packages)
> >
> > - this specific system's persistent state: /var and parts of /etc
> >
> > with the intention of eventually enabling functionality like being
> > able to do a "factory reset" to the equivalent of a freshly installed
> > system by deleting (most of) /etc and /var, rebooting, and letting the
> > OS re-create them from a template below /usr; or doing the equivalent
> > for individual packages by deleting only their part of /etc and /var.
> >
> > /etc is somewhere between static files and state, because traditionally
> > it has been a mixture of files that the sysadmin or installer must provide
> > (like /etc/passwd); configuration files that are shipped by a package and
> > can be edited by the sysadmin (like /etc/systemd/logind.conf); and
> > integration glue that links up one package with another, can in principle
> > be edited by the sysadmin, but in practice is rarely edited
> > (like /etc/profile.d/flatpak.sh).
> >
> > Various upstream projects including systemd have been trying to reduce
> > the extent to which /etc and /var are included in the data.tar.* of a .deb
> > or other packaging systems' equivalents, by moving the integration glue
> > to a /usr-like directory (/lib/udev/rules.d, /usr/share/dbus-1/system.d),
> > reserving the corresponding /etc directory for sysadmin configuration
> > (/etc/udev/rules.d, /etc/dbus-1/system.d), and providing a way for the
> > sysadmin to "mask" any integration files they want the system to ignore.
> >
> > If we disregard conffiles and configuration files in /etc for the
> > moment, there are basically three ways for a package to get a file onto
> > the running system:
> >
> > - ship it in the data.tar.* of a .deb
> > - create it from a maintainer script and also during boot
> > - maybe via tmpfiles.d(5)
> > - or maybe open-coded
> > - have the package create it at runtime, on-demand
> > - this clearly doesn't work if the package's code runs unprivileged
> >   and relies on root having created a directory for it already
> >
> > For /usr and the /usr-like directories, shipping files in the .deb is by
> > far the most common, although a few packages need to create files here
> > via maintainer scripts or triggers (for example
> > /usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache which is a summary
> > of files created by multiple packages, and is updated whenever those
> > packages are added, removed or changed).
> >
> > For /run, /tmp and /var/tmp, I think there's consensus that shipping files
> > in those directories in the .deb is a bug, because at the next reboot,
> > the file will be deleted, leaving the files that dpkg thinks it's managing
> > out of sync with the files that actually exist. At the moment, these are
> > variously created by maintainer scripts, systemd units/init scripts, or the
> > daemons themselves, with some duplication, and no good way to get an
> > overview of which packages "own" which locations: dpkg doesn't know anything
> > about them, and 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Wed, 7 Jun 2023 at 11:29, Simon McVittie  wrote:
>
> On Tue, 06 Jun 2023 at 20:40:52 -0700, Russ Allbery wrote:
> > Luca Boccassi  writes:
> > > +Packages might need additional files or directories to implement their
> > > +functionality. Directories that are located under ``/var/`` or
> > > +``/etc/``, and files that are located under ``/var/``, must not be
> > > +created manually via maintainer scripts, but instead be declaratively
> > > +defined via the `tmpfiles.d
> > > +`_
> > > +interface.
> >
> > This is an oddly specific list of directories and not at all the
> > directories that I would have expected to be handled by tmpfiles.d.
>
> Sorry, in my previous reading of this bug I had been concentrating on
> the mechanics of how to make tmpfiles.d(5) something that maintainers can
> rely on if it's convenient/helpful, and I'd missed that Luca is asking
> for its use to be mandatory in some cases.
>
> I would personally be inclined to concentrate on making tmpfiles.d(5)
> something that we can rely on and encourage the use of where appropriate,
> even on non-systemd systems, so that (upstream and downstream) maintainers
> can move towards it of their own accord because it's more convenient
> than other options, and put aside the question of making it generally a
> "should" or "must" for the moment.
>
> I believe (please correct me if I'm wrong) that Luca's intention here
> is that this is drawing a line between:
>
> - the static files of the OS: /usr and the /usr-like top-level directories
>   (the ones that get merged by the /usr merge), which should be statically
>   shipped in packages and managed by dpkg (or on "immutable" systems
>   that use image-based/tree-based upgrades, maybe by ostree or casync
>   or similar, from an tree originally constructed from dpkg packages)
>
> - this specific system's persistent state: /var and parts of /etc
>
> with the intention of eventually enabling functionality like being
> able to do a "factory reset" to the equivalent of a freshly installed
> system by deleting (most of) /etc and /var, rebooting, and letting the
> OS re-create them from a template below /usr; or doing the equivalent
> for individual packages by deleting only their part of /etc and /var.
>
> /etc is somewhere between static files and state, because traditionally
> it has been a mixture of files that the sysadmin or installer must provide
> (like /etc/passwd); configuration files that are shipped by a package and
> can be edited by the sysadmin (like /etc/systemd/logind.conf); and
> integration glue that links up one package with another, can in principle
> be edited by the sysadmin, but in practice is rarely edited
> (like /etc/profile.d/flatpak.sh).
>
> Various upstream projects including systemd have been trying to reduce
> the extent to which /etc and /var are included in the data.tar.* of a .deb
> or other packaging systems' equivalents, by moving the integration glue
> to a /usr-like directory (/lib/udev/rules.d, /usr/share/dbus-1/system.d),
> reserving the corresponding /etc directory for sysadmin configuration
> (/etc/udev/rules.d, /etc/dbus-1/system.d), and providing a way for the
> sysadmin to "mask" any integration files they want the system to ignore.
>
> If we disregard conffiles and configuration files in /etc for the
> moment, there are basically three ways for a package to get a file onto
> the running system:
>
> - ship it in the data.tar.* of a .deb
> - create it from a maintainer script and also during boot
> - maybe via tmpfiles.d(5)
> - or maybe open-coded
> - have the package create it at runtime, on-demand
> - this clearly doesn't work if the package's code runs unprivileged
>   and relies on root having created a directory for it already
>
> For /usr and the /usr-like directories, shipping files in the .deb is by
> far the most common, although a few packages need to create files here
> via maintainer scripts or triggers (for example
> /usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache which is a summary
> of files created by multiple packages, and is updated whenever those
> packages are added, removed or changed).
>
> For /run, /tmp and /var/tmp, I think there's consensus that shipping files
> in those directories in the .deb is a bug, because at the next reboot,
> the file will be deleted, leaving the files that dpkg thinks it's managing
> out of sync with the files that actually exist. At the moment, these are
> variously created by maintainer scripts, systemd units/init scripts, or the
> daemons themselves, with some duplication, and no good way to get an
> overview of which packages "own" which locations: dpkg doesn't know anything
> about them, and systemd knows about some but not all of them.
>
> tmpfiles.d seems like a good way to keep track of who "owns" those
> transient files and directories. I think a Policy "must" is probably too
> strong here, but a "should" might be 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Wed, 7 Jun 2023 at 04:40, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> > index b34c183..30ce013 100644
> > --- a/policy/ch-files.rst
> > +++ b/policy/ch-files.rst
> > @@ -722,6 +722,43 @@ The name of the files and directories installed by 
> > binary packages
> >  outside the system PATH must be encoded in UTF-8 and should be
> >  restricted to ASCII when it is possible to do so.
> >
> > +.. _s-tmpfiles.d:
> > +
> > +tmpfiles.d
> > +--
> > +
> > +Packages might need additional files or directories to implement their
> > +functionality. Directories that are located under ``/var/`` or
> > +``/etc/``, and files that are located under ``/var/``, must not be
> > +created manually via maintainer scripts, but instead be declaratively
> > +defined via the `tmpfiles.d
> > +`_
> > +interface.
>
> This is an oddly specific list of directories and not at all the
> directories that I would have expected to be handled by tmpfiles.d.  My
> naive expectation would be that the most common path handled by tmpfiles.d
> would be /run, /tmp, and /var/tmp.  I read through the other messages in
> this bug but I didn't see the explanation for why those directories in
> particular.
>
> Given that neither /var (apart from /var/tmp) nor /etc are transient, I
> would have expected packages to simply ship the directories under those
> paths that they need in the package.  Why is that not the right approach?
> Could you explain when tmpfiles.d should be used instead of shipping the
> directory in the package?

It's not only for transient hierarchies as in memory based, but also
for hierarchies that can be blown away and the system is supposed to
recover from. /var is state, and if random things under it are lost,
it should be possible to recover without having to throw away the
whole installation and start over. Of course there will be some
limits, but for the really trivial stuff (ie: "I need to store some
cached data somewhere so I need a directory under /var") it should be
possible to cleanly re-set up in an automated fashion after a reboot.
Not only this, but tmpfiles.d help also enforce that the right
metadata (dac/mac/acl) are set, again without having to reinstall
packages. Also more specifically, for image-based systems we want to
be able to have the vendor tree (ie: content shipped by packages)
restricted to /usr and optionally /etc. One of the goals we have for
Trixie is to no longer ship anything under /var (and /boot) hard-coded
in packages.

So the list of directories there is provided as an example, I can make
it more or less specific as needed.

> > +The ``tmpfiles.d`` file format is defined by the ``systemd`` project, and 
> > is
> > +guaranteed to be stable. Ideally, such definitions should be defined 
> > upstream
> > +where applicable, and shipped as they are by Debian packages.
> > +
> > +Details about the syntax and installation paths for ``tmpfiles.d`` are 
> > defined
> > +by its `reference implementation's documentation,
> > +`_ and 
> > will
> > +not be redefined here.
>
> I'm clearly missing something, but I was naively expecting this section to
> start with something more like this:
>
> Packages that need to create files or directories in file systems that
> may be deleted on each reboot (for example, ``/run``, ``/tmp``, and
> ``/var/tmp``) should do so via configuration files in the
> ``/usr/lib/tmpfiles.d`` directory. The syntax of those files is
> defined by the `systemd tmpfiles.d documentation
> `__.
>
> However, reading that documentation, it sounds like most of the cases for
> other directories are handled by other systemd unit configuration
> directives.  We should say that explicitly here and reproduce the list of
> other directories that should be handled directly by the unit file if
> that's what we want people to do.

I'd be more than delighted to recommend using
StateDirectory=/RuntimeDirectory= and friends as the first choice, and
can certainly do so. However, note that those work best for cases
where there is a service which is the clear owner (they can be shared,
with some caveats that we are trying to solve upstream if ephemeral
users are used, but still needs an owner), and this is important for
lifetime management (ie: when to delete). There are also cases where
there is no clear service that is the owner and to whose lifecycle the
directory should be tied to, and for that tmpfiles.d is the solution.

> What's the plan for creating directories in /run on non-systemd systems?
> I had assumed that tmpfiles.d would be used for those directories so that
> we can use the same mechanism for both systemd and non-systemd systems,
> but it looks like that's not ideal for systemd systems.  Is it safe to 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Tue, 06 Jun 2023 at 20:40:52 -0700, Russ Allbery wrote:
> Luca Boccassi  writes:
> > +Packages might need additional files or directories to implement their
> > +functionality. Directories that are located under ``/var/`` or
> > +``/etc/``, and files that are located under ``/var/``, must not be
> > +created manually via maintainer scripts, but instead be declaratively
> > +defined via the `tmpfiles.d
> > +`_
> > +interface.
> 
> This is an oddly specific list of directories and not at all the
> directories that I would have expected to be handled by tmpfiles.d.

Sorry, in my previous reading of this bug I had been concentrating on
the mechanics of how to make tmpfiles.d(5) something that maintainers can
rely on if it's convenient/helpful, and I'd missed that Luca is asking
for its use to be mandatory in some cases.

I would personally be inclined to concentrate on making tmpfiles.d(5)
something that we can rely on and encourage the use of where appropriate,
even on non-systemd systems, so that (upstream and downstream) maintainers
can move towards it of their own accord because it's more convenient
than other options, and put aside the question of making it generally a
"should" or "must" for the moment.

I believe (please correct me if I'm wrong) that Luca's intention here
is that this is drawing a line between:

- the static files of the OS: /usr and the /usr-like top-level directories
  (the ones that get merged by the /usr merge), which should be statically
  shipped in packages and managed by dpkg (or on "immutable" systems
  that use image-based/tree-based upgrades, maybe by ostree or casync
  or similar, from an tree originally constructed from dpkg packages)

- this specific system's persistent state: /var and parts of /etc

with the intention of eventually enabling functionality like being
able to do a "factory reset" to the equivalent of a freshly installed
system by deleting (most of) /etc and /var, rebooting, and letting the
OS re-create them from a template below /usr; or doing the equivalent
for individual packages by deleting only their part of /etc and /var.

/etc is somewhere between static files and state, because traditionally
it has been a mixture of files that the sysadmin or installer must provide
(like /etc/passwd); configuration files that are shipped by a package and
can be edited by the sysadmin (like /etc/systemd/logind.conf); and
integration glue that links up one package with another, can in principle
be edited by the sysadmin, but in practice is rarely edited
(like /etc/profile.d/flatpak.sh).

Various upstream projects including systemd have been trying to reduce
the extent to which /etc and /var are included in the data.tar.* of a .deb
or other packaging systems' equivalents, by moving the integration glue
to a /usr-like directory (/lib/udev/rules.d, /usr/share/dbus-1/system.d),
reserving the corresponding /etc directory for sysadmin configuration
(/etc/udev/rules.d, /etc/dbus-1/system.d), and providing a way for the
sysadmin to "mask" any integration files they want the system to ignore.

If we disregard conffiles and configuration files in /etc for the
moment, there are basically three ways for a package to get a file onto
the running system:

- ship it in the data.tar.* of a .deb
- create it from a maintainer script and also during boot
- maybe via tmpfiles.d(5)
- or maybe open-coded
- have the package create it at runtime, on-demand
- this clearly doesn't work if the package's code runs unprivileged
  and relies on root having created a directory for it already

For /usr and the /usr-like directories, shipping files in the .deb is by
far the most common, although a few packages need to create files here
via maintainer scripts or triggers (for example
/usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache which is a summary
of files created by multiple packages, and is updated whenever those
packages are added, removed or changed).

For /run, /tmp and /var/tmp, I think there's consensus that shipping files
in those directories in the .deb is a bug, because at the next reboot,
the file will be deleted, leaving the files that dpkg thinks it's managing
out of sync with the files that actually exist. At the moment, these are
variously created by maintainer scripts, systemd units/init scripts, or the
daemons themselves, with some duplication, and no good way to get an
overview of which packages "own" which locations: dpkg doesn't know anything
about them, and systemd knows about some but not all of them.

tmpfiles.d seems like a good way to keep track of who "owns" those
transient files and directories. I think a Policy "must" is probably too
strong here, but a "should" might be reasonable?

For the persistent parts of /var, several packages ship regular files
(as opposed to directories) in the .deb. I think there might be rough
consensus that doing so is at least a "code smell", but quite a 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> diff --git a/policy/ch-files.rst b/policy/ch-files.rst
> index b34c183..30ce013 100644
> --- a/policy/ch-files.rst
> +++ b/policy/ch-files.rst
> @@ -722,6 +722,43 @@ The name of the files and directories installed by 
> binary packages
>  outside the system PATH must be encoded in UTF-8 and should be
>  restricted to ASCII when it is possible to do so.
>  
> +.. _s-tmpfiles.d:
> +
> +tmpfiles.d
> +--
> +
> +Packages might need additional files or directories to implement their
> +functionality. Directories that are located under ``/var/`` or
> +``/etc/``, and files that are located under ``/var/``, must not be
> +created manually via maintainer scripts, but instead be declaratively
> +defined via the `tmpfiles.d
> +`_
> +interface.

This is an oddly specific list of directories and not at all the
directories that I would have expected to be handled by tmpfiles.d.  My
naive expectation would be that the most common path handled by tmpfiles.d
would be /run, /tmp, and /var/tmp.  I read through the other messages in
this bug but I didn't see the explanation for why those directories in
particular.

Given that neither /var (apart from /var/tmp) nor /etc are transient, I
would have expected packages to simply ship the directories under those
paths that they need in the package.  Why is that not the right approach?
Could you explain when tmpfiles.d should be used instead of shipping the
directory in the package?

> +The ``tmpfiles.d`` file format is defined by the ``systemd`` project, and is
> +guaranteed to be stable. Ideally, such definitions should be defined upstream
> +where applicable, and shipped as they are by Debian packages.
> +
> +Details about the syntax and installation paths for ``tmpfiles.d`` are 
> defined
> +by its `reference implementation's documentation,
> +`_ and will
> +not be redefined here.

I'm clearly missing something, but I was naively expecting this section to
start with something more like this:

Packages that need to create files or directories in file systems that
may be deleted on each reboot (for example, ``/run``, ``/tmp``, and
``/var/tmp``) should do so via configuration files in the
``/usr/lib/tmpfiles.d`` directory. The syntax of those files is
defined by the `systemd tmpfiles.d documentation
`__.

However, reading that documentation, it sounds like most of the cases for
other directories are handled by other systemd unit configuration
directives.  We should say that explicitly here and reproduce the list of
other directories that should be handled directly by the unit file if
that's what we want people to do.

What's the plan for creating directories in /run on non-systemd systems?
I had assumed that tmpfiles.d would be used for those directories so that
we can use the same mechanism for both systemd and non-systemd systems,
but it looks like that's not ideal for systemd systems.  Is it safe to use
*both* (e.g.) RuntimeDirectory= *and* tmpfiles.d for the same directory so
that tmpfiles.d is used for non-systemd systems?

> +``tmpfiles.d`` snippets should be detected at package build time by
> +tools such as ``debhelper``, packaged, and the appropriate snippet to
> +call them on installation, upgrade, removal, purge and other steps as
> +required, should be automatically added by helpers such as
> +``dh_installtmpfiles``.

Policy should not say things like this.  It's the job of Policy to define
what those snippets are and be specific.  It's Policy-compliant to not use
debhelper at all (we've had some discussions of changing that, but we
haven't yet).  debhelper is one implementation of Policy (and
additional non-Policy stuff), so we should be defining (at a high level)
what it needs to do and what any non-debhelper parallel implementation
should do.

Also, isn't the most obvious way to implement this to use triggers?  That
runs into the problem that Policy still doesn't have any documentation of
triggers, but (unlike debhelper) we can just require that implementations
of the tmpfiles.d mechanism pick up new files via triggers and do the
right thing, which feels appealing.

> Packages shipping ``tmpfiles.d`` snippets should depend on the
> +appropriate virtual packages in the following order:
> +``default-systemd-tmpfiles | systemd-tmpfiles``.

I read the discussion of this and agree this is the best approach.

> +Init systems are required to integrate with ``tmpfiles.d`` and run the
> +service that applies them on boot, and regularly for cleanup
> +purposes. The documentation for the reference implementation,
> +`systemd-tmpfiles,
> +`_
> +explains how to call the program so that the appropriate ``tmpfiles.d``
> +snippets are applied at the appropriate 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Tue, 06 Jun 2023 13:07:37 +0100 Luca Boccassi 
wrote:
> Sounds like a good plan to me.

Updated as suggested.

-- 
Kind regards,
Luca Boccassi
From 20a655663c17914699e72e48a74daca03fd42a22 Mon Sep 17 00:00:00 2001
From: Luca Boccassi 
Date: Tue, 9 May 2023 01:38:13 +0100
Subject: [PATCH] Define tmpfiles.d interface and usage

---
 policy/ch-files.rst | 37 +
 policy/ch-maintainerscripts.rst |  6 ++
 2 files changed, 43 insertions(+)

diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..30ce013 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -722,6 +722,43 @@ The name of the files and directories installed by binary packages
 outside the system PATH must be encoded in UTF-8 and should be
 restricted to ASCII when it is possible to do so.
 
+.. _s-tmpfiles.d:
+
+tmpfiles.d
+--
+
+Packages might need additional files or directories to implement their
+functionality. Directories that are located under ``/var/`` or ``/etc/``, and
+files that are located under ``/var/``, must not be created manually via
+maintainer scripts, but instead be declaratively defined via the `tmpfiles.d
+`_ interface.
+The ``tmpfiles.d`` file format is defined by the ``systemd`` project, and is
+guaranteed to be stable. Ideally, such definitions should be defined upstream
+where applicable, and shipped as they are by Debian packages.
+
+Details about the syntax and installation paths for ``tmpfiles.d`` are defined
+by its `reference implementation's documentation,
+`_ and will
+not be redefined here.
+
+``tmpfiles.d`` snippets should be usable on systems that do not boot (such as a
+very minimal chroot image), and also on systems booting with init systems other
+than ``systemd``.
+
+``tmpfiles.d`` snippets should be detected at package build time by tools such
+as ``debhelper``, packaged, and the appropriate snippet to call them on
+installation, upgrade, removal, purge and other steps as required, should be
+automatically added by helpers such as ``dh_installtmpfiles``. Packages shipping
+``tmpfiles.d`` snippets should depend on the appropriate virtual packages in the
+following order: ``default-systemd-tmpfiles | systemd-tmpfiles``.
+
+Init systems are required to integrate with ``tmpfiles.d`` and run the service
+that applies them on boot, and regularly for cleanup purposes. The documentation
+for the reference implementation, `systemd-tmpfiles,
+`_
+explains how to call the program so that the appropriate ``tmpfiles.d`` snippets
+are applied at the appropriate time.
+
 .. [#]
If you are using GCC, ``-fPIC`` produces code with relocatable
position independent code, which is required for most architectures
diff --git a/policy/ch-maintainerscripts.rst b/policy/ch-maintainerscripts.rst
index 724074c..320949d 100644
--- a/policy/ch-maintainerscripts.rst
+++ b/policy/ch-maintainerscripts.rst
@@ -50,6 +50,12 @@ absolute pathname. Maintainer scripts should also not reset the
 appending package-specific directories. These considerations really
 apply to all shell scripts.
 
+Maintainer scripts should not be used to create or remove auxiliary files and/or
+directories that packages may need, such as those in ``/var/`` or ``/etc/``.
+Instead, :ref:`s-tmpfiles.d` snippets should be shipped, being ideally provided by
+the upstream sources, if any. For more details about the ``tmpfiles.d``
+interface, see :ref:`s-tmpfiles.d`.
+
 .. _s-idempotency:
 
 Maintainer scripts idempotency
-- 
2.39.2



signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Tue, 6 Jun 2023 11:58:07 +0100 Simon McVittie 
wrote:
> On Tue, 06 Jun 2023 at 11:37:51 +0100, Sean Whitton wrote:
> > On Sun 04 Jun 2023 at 02:56PM +01, Simon McVittie wrote:
> > > Another possible mitigation which I haven't previously seen
proposed
> > > is giving *elogind* a Depends or Recommends on systemd-*-
standalone.
> > > I think that would work to mitigate the failure mode with (1.)
and (B.),
> > > and the installed-size argument seems less interesting here
because the
> > > sort of systems that require elogind are already much larger
anyway.
> > > Would the elogind maintainers be willing to consider this? Does
anyone
> > > see a reason why it wouldn't work?
> > 
> > So to confirm, you think that if the elogind maintainers did this,
then
> > default-systemd-tmpfiles could point at systemd rather than
> > systemd-standalone-tmpfiles, which the systemd maintainers prefer,
but
> > in addition, there aren't any scenarios in which people's systems
are
> > likely to be re-arranged when they don't want them to be?
> 
> Exactly. My hope is that if we had:
> 
> Package: systemd
> Architecture: linux-any
> Provides: default-systemd-tmpfiles, systemd-tmpfiles
> Conflicts: systemd-tmpfiles
> Replaces: systemd-tmpfiles
> 
> Package: systemd-standalone-tmpfiles
> Architecture: linux-any
> Provides: systemd-tmpfiles
> Conflicts: systemd-tmpfiles
> Replaces: systemd-tmpfiles
> 
> Package: elogind
> Depends: systemd-standalone-tmpfiles    # or Recommends?
> 
> Package: foo-service # any package that requires
tmpfiles.d(5)
> Depends: default-systemd-tmpfiles | systemd-tmpfiles
> 
> # optionally, if someone does the work
> Package: openrc-tmpfiles    # any other
implementation
> Architecture: hurd-any kfreebsd-any
> Provides: default-systemd-tmpfiles, systemd-tmpfiles
> Conflicts: systemd-tmpfiles
> Replaces: systemd-tmpfiles
> 
> then the right thing (or at least *a* right thing) would happen in
> all cases:
> 
> * install foo-service on a systemd-booted system:
>   systemd is already installed and the dependency is satisfied
> 
> * install foo-service on a sysvinit-booted desktop system with
elogind:
>   elogind is already installed, therefore systemd-standalone-tmpfiles
is
>   already installed and the dependency is satisfied, avoiding
#1016006 etc.
> 
> * install foo-service on a sysvinit-booted headless system with no
elogind:
>   systemd gets installed as a dependency by default, which is what
the
>   systemd maintainers would prefer to happen when there are no
compelling
>   space constraints; but the user can specifically ask for
>   systemd-standalone-tmpfiles if that's what they'd prefer
> 
> * install foo-service in a container with no init system at all:

Sounds like a good plan to me.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Tue, 06 Jun 2023 at 11:37:51 +0100, Sean Whitton wrote:
> On Sun 04 Jun 2023 at 02:56PM +01, Simon McVittie wrote:
> > Another possible mitigation which I haven't previously seen proposed
> > is giving *elogind* a Depends or Recommends on systemd-*-standalone.
> > I think that would work to mitigate the failure mode with (1.) and (B.),
> > and the installed-size argument seems less interesting here because the
> > sort of systems that require elogind are already much larger anyway.
> > Would the elogind maintainers be willing to consider this? Does anyone
> > see a reason why it wouldn't work?
> 
> So to confirm, you think that if the elogind maintainers did this, then
> default-systemd-tmpfiles could point at systemd rather than
> systemd-standalone-tmpfiles, which the systemd maintainers prefer, but
> in addition, there aren't any scenarios in which people's systems are
> likely to be re-arranged when they don't want them to be?

Exactly. My hope is that if we had:

Package: systemd
Architecture: linux-any
Provides: default-systemd-tmpfiles, systemd-tmpfiles
Conflicts: systemd-tmpfiles
Replaces: systemd-tmpfiles

Package: systemd-standalone-tmpfiles
Architecture: linux-any
Provides: systemd-tmpfiles
Conflicts: systemd-tmpfiles
Replaces: systemd-tmpfiles

Package: elogind
Depends: systemd-standalone-tmpfiles# or Recommends?

Package: foo-service # any package that requires tmpfiles.d(5)
Depends: default-systemd-tmpfiles | systemd-tmpfiles

# optionally, if someone does the work
Package: openrc-tmpfiles# any other implementation
Architecture: hurd-any kfreebsd-any
Provides: default-systemd-tmpfiles, systemd-tmpfiles
Conflicts: systemd-tmpfiles
Replaces: systemd-tmpfiles

then the right thing (or at least *a* right thing) would happen in
all cases:

* install foo-service on a systemd-booted system:
  systemd is already installed and the dependency is satisfied

* install foo-service on a sysvinit-booted desktop system with elogind:
  elogind is already installed, therefore systemd-standalone-tmpfiles is
  already installed and the dependency is satisfied, avoiding #1016006 etc.

* install foo-service on a sysvinit-booted headless system with no elogind:
  systemd gets installed as a dependency by default, which is what the
  systemd maintainers would prefer to happen when there are no compelling
  space constraints; but the user can specifically ask for
  systemd-standalone-tmpfiles if that's what they'd prefer

* install foo-service in a container with no init system at all:
  systemd gets installed as a dependency by default, which is what the
  systemd maintainers would prefer to happen when there are no compelling
  space constraints; but the user can specifically ask for
  systemd-standalone-tmpfiles if that's what they'd prefer

* (optionally) install foo-service on hurd-i386:
  openrc-tmpfiles (or whatever) gets installed as a dependency

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sean Whitton]

Hello,

On Sun 04 Jun 2023 at 02:56PM +01, Simon McVittie wrote:

> So I think the only realistic options for packages that hard-require
> this functionality (not all do) are:
>
> 1. Depends: systemd | systemd-tmpfiles
> 2. Depends: systemd-tmpfiles-standalone | systemd-tmpfiles
> 3. Depends: default-systemd-tmpfiles | systemd-tmpfiles
>
> (where the third one is equivalent to one of the first two, depending on
> how default-systemd-tmpfiles was implemented), possibly with some more
> less-preferred options between the first "|" and the virtual-package
> fallback.

As you wrote in another message, it's very cheap future-proofing to use
a virtual package for whichever actual solution we're implementing, so I
think we should do that.

> Another possible mitigation which I haven't previously seen proposed
> is giving *elogind* a Depends or Recommends on systemd-*-standalone.
> I think that would work to mitigate the failure mode with (1.) and (B.),
> and the installed-size argument seems less interesting here because the
> sort of systems that require elogind are already much larger anyway.
> Would the elogind maintainers be willing to consider this? Does anyone
> see a reason why it wouldn't work?

So to confirm, you think that if the elogind maintainers did this, then
default-systemd-tmpfiles could point at systemd rather than
systemd-standalone-tmpfiles, which the systemd maintainers prefer, but
in addition, there aren't any scenarios in which people's systems are
likely to be re-arranged when they don't want them to be?

> As a maintainer of system services, I would not be at all happy with
> expecting the maintainer of every system service that requires tmpfiles
> to have this conversation again and again. Obviously as a technical
> committee member and occasional Policy contributor, I've chosen to take
> on some of the burden of decisions like this one; but when I'm working
> on dbus or polkitd or even openarena-server, I should be able to follow
> a project decision, and be reasonably confident that I won't get angry
> RC bug reports about it (and if I do, I should be able to refer the bug
> reporter to a project decision instead of having to re-litigate it).
> Putting this sort of thing on individual package maintainers seems like
> a recipe for making it no longer fun to maintain a package.

Yes, let's figure out a standard solution and write it in Policy.
The reasoning may well be applicable to similar things in the future.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Helmut Grohne]

On Sun, Jun 04, 2023 at 02:56:59PM +0100, Simon McVittie wrote:
> I think one way or another, if anyone is going to set a package-level
> dependency on systemd-tmpfiles, the first (preferred) dependency needs to
> be on either a concrete provider (systemd or systemd-tmpfiles-standalone
> in this case), or a default-systemd-tmpfiles virtual package
> that only has one provider per architecture (which is the way
> {default-,}dbus-{system,session}-bus are handled). Otherwise, you
> can get a non-deterministic choice of default implementation, which
> seems strictly worse than either depending on systemd or depending on
> systemd-tmpfiles-standalone - if you're unlucky, it can have all the
> disadvantages of either one of those.

Thank you for the elaborate writeup. There is little to add to what you
write except for one minor aspect.

>   - actual result: apt's heuristic might have difficulty realising that
> it needs to do that

I think we should be able to guide apt here. I recently had to look into
Replaces and in that process I also had to re-read policy section 7.6.2.
It details the "other" use of Replaces to guide a package manager (e.g.
apt) for changing implementations of an interface - which is exactly
what we are talking about here. In essence, it says that we should do:

Provides: systemd-tmpfiles
Conflicts: systemd-tmpfiles
Replaces: systemd-tmpfiles

And systemd-standalone-tmpfiles does that. :) But systemd does not. :(
systemd misses out on Conflicts and Replaces. I guess (but have not
verified) that once these are added, apt would be happier to "upgrade"
systemd-standalone-tmpfiles to systemd when needed.

I've also experimented with a minimal chroot, installed the standalone
tools and the asked apt to install libbiometric0 (which happens to have
a dependency on systemd) and apt was quite happy with removing the
standalone variants. This is still missing the consumers of the provided
facilities though, so it might not be representative.

Is there any concrete evidence of apt having difficulties in a real
situation? Or maybe a constructed example demonstrating this? Thanks for
being cautious, but I'd also like to understand whether this is
hypothetical or real.

Helmut

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Mon, 5 Jun 2023 09:53:39 +0100 Simon McVittie 
wrote:
> On Mon, 05 Jun 2023 at 01:36:25 +0100, Luca Boccassi wrote:
> > If it is useful, adding a "default-tmpfiles" or so virtual package
> > would be fine by me - but with the kfreebsd port being retired
soon,
> > and i386 (for hurd) going the way of the dodo, I'm not sure it
would
> > be very useful? I don't think it would be a problem to add it, if
it
> > turns out to be of use though.
> 
> What this would look like, in src:systemd:
> 
> Package: systemd
> Provides: default-systemd-tmpfiles, systemd-tmpfiles
> 
> Package: systemd-standalone-tmpfiles
> Provides: systemd-tmpfiles
> 
> (or maybe the other way round, depending what conclusion we come to
on
> the choice between (1.) and (2.)), and then in dependent packages:
> 
> Package: foo-service
> Depends: default-systemd-tmpfiles | systemd-tmpfiles
> 
> The benefits of that over having foo-service depend directly on
> "systemd | systemd-tmpfiles" or
> "systemd-standalone-tmpfiles | systemd-tmpfiles" are fairly minor,
but the
> cost is also fairly minor; and if we want this, we should set it up
> *before* lots of packages start adding a dependency on the -tmpfiles
or
> -sysusers interfaces. The benefits I see are:
> 
> - if we find out that the dependency we first added is a practical
problem
>   for whatever reason, we can swap the default with a src:systemd
upload,
>   without needing multiple maintainers to touch all the dependent
packages;
> 
> - if we get a useful non-Linux port (admittedly this looks
increasingly
>   unlikely) which cannot compile src:systemd, then their
reimplementation
>   of these interfaces can have Provides: default-systemd-tmpfiles on
that
>   architecture, without affecting Linux architectures
>   (the same way dbus-x11 Provides default-dbus-session-bus on non-
Linux
>   ports, even though it's dbus-user-session that Provides the virtual
>   package on Linux)

Sure, that sounds reasonable and simple enough to do, no objection from
me.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Mon, 5 Jun 2023 10:11:46 +0100 Simon McVittie 
wrote:
> On Mon, 05 Jun 2023 at 01:36:25 +0100, Luca Boccassi wrote:
> > Our time is worth more than 80K or whatever it is of disk space in
a
> > throw-away container.
> 
> I agree that the systemd maintainers' time is a limited resource that
we
> should not waste, but that size estimate is off by a couple of orders
of
> magnitude. On amd64, aptitude tells me systemd-standalone-tmpfiles
and
> -sysusers are about 700K of Uncompressed Size between them, while the
> full systemd and libsystemd-shared packages add up to about 15M. For
> genuinely throwaway containers, yes, it's not worth optimizing this,
> but for containers that will be archived in a registry and/or kept
> running longer-term, that seems like enough that maintainers of
Docker
> containers, etc. will want to use the standalone binaries if they are
> sufficient for the container's needs.
> 
> (This is ignoring any extra library dependencies that might be
required by
> systemd and libsystemd-shared but unnecessary for the standalone
binaries;
> if there are any, then obviously the effective size delta increases.)

Sure, but again, those special cases that really care about that
particular angle can simply adjust their dependencies accordingly,
there's nothing stopping them from doing so. It does not need to be the
default.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Mon, 05 Jun 2023 at 01:36:25 +0100, Luca Boccassi wrote:
> On Sun, 4 Jun 2023 at 14:56, Simon McVittie  wrote:
> > So I think the only realistic options for packages that hard-require
> > this functionality (not all do) are:
> >
> > 1. Depends: systemd | systemd-tmpfiles
> > 2. Depends: systemd-tmpfiles-standalone | systemd-tmpfiles
> > 3. Depends: default-systemd-tmpfiles | systemd-tmpfiles

In case it's not obvious, (2.) should say
systemd-standalone-tmpfiles | systemd-tmpfiles (and the same everywhere
else that I mentioned systemd-tmpfiles-standalone).

> Our time is worth more than 80K or whatever it is of disk space in a
> throw-away container.

I agree that the systemd maintainers' time is a limited resource that we
should not waste, but that size estimate is off by a couple of orders of
magnitude. On amd64, aptitude tells me systemd-standalone-tmpfiles and
-sysusers are about 700K of Uncompressed Size between them, while the
full systemd and libsystemd-shared packages add up to about 15M. For
genuinely throwaway containers, yes, it's not worth optimizing this,
but for containers that will be archived in a registry and/or kept
running longer-term, that seems like enough that maintainers of Docker
containers, etc. will want to use the standalone binaries if they are
sufficient for the container's needs.

(This is ignoring any extra library dependencies that might be required by
systemd and libsystemd-shared but unnecessary for the standalone binaries;
if there are any, then obviously the effective size delta increases.)

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Mon, 05 Jun 2023 at 01:36:25 +0100, Luca Boccassi wrote:
> If it is useful, adding a "default-tmpfiles" or so virtual package
> would be fine by me - but with the kfreebsd port being retired soon,
> and i386 (for hurd) going the way of the dodo, I'm not sure it would
> be very useful? I don't think it would be a problem to add it, if it
> turns out to be of use though.

What this would look like, in src:systemd:

Package: systemd
Provides: default-systemd-tmpfiles, systemd-tmpfiles

Package: systemd-standalone-tmpfiles
Provides: systemd-tmpfiles

(or maybe the other way round, depending what conclusion we come to on
the choice between (1.) and (2.)), and then in dependent packages:

Package: foo-service
Depends: default-systemd-tmpfiles | systemd-tmpfiles

The benefits of that over having foo-service depend directly on
"systemd | systemd-tmpfiles" or
"systemd-standalone-tmpfiles | systemd-tmpfiles" are fairly minor, but the
cost is also fairly minor; and if we want this, we should set it up
*before* lots of packages start adding a dependency on the -tmpfiles or
-sysusers interfaces. The benefits I see are:

- if we find out that the dependency we first added is a practical problem
  for whatever reason, we can swap the default with a src:systemd upload,
  without needing multiple maintainers to touch all the dependent packages;

- if we get a useful non-Linux port (admittedly this looks increasingly
  unlikely) which cannot compile src:systemd, then their reimplementation
  of these interfaces can have Provides: default-systemd-tmpfiles on that
  architecture, without affecting Linux architectures
  (the same way dbus-x11 Provides default-dbus-session-bus on non-Linux
  ports, even though it's dbus-user-session that Provides the virtual
  package on Linux)

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 4 Jun 2023 at 14:56, Simon McVittie  wrote:
>
> (Newly cc'd elogind maintainers: Please see #945269 for context)
>
> On Sun, 04 Jun 2023 at 12:15:41 +0100, Luca Boccassi wrote:
> > On Sun, 4 Jun 2023 at 12:02, Sean Whitton  wrote:
> > > On Tue 09 May 2023 at 01:44AM +01, Luca Boccassi wrote:
> > > > For now I've kept only a mention of the 'systemd-tmpfiles' virtual
> > > > package. As maintainers we would really prefer if the 'main'
> > > > implementation is pulled in whenever possible. When a minimal
> > > > installation is desired (ie, a minbase), it is possible to manually
> > > > specify the -standalone variant.
> > > >
> > > > This was a controversial point last year, see:
> > > >
> > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017441
> > >
> > > Hmm.  I don't have personal experience with this sort of thing, but
> > > based on some of the examples in that bug, it seems like doing this
> > > could cause apt to change people's systems around in ways they strongly
> > > disprefer.  What you propose seems like it could cause unpleasant
> > > surprises.
> >
> > Only due to bugs in said other packages, or if the wrong commands are
> > passed to apt/aptitude/etc.
>
> I think one way or another, if anyone is going to set a package-level
> dependency on systemd-tmpfiles, the first (preferred) dependency needs to
> be on either a concrete provider (systemd or systemd-tmpfiles-standalone
> in this case), or a default-systemd-tmpfiles virtual package
> that only has one provider per architecture (which is the way
> {default-,}dbus-{system,session}-bus are handled). Otherwise, you
> can get a non-deterministic choice of default implementation, which
> seems strictly worse than either depending on systemd or depending on
> systemd-tmpfiles-standalone - if you're unlucky, it can have all the
> disadvantages of either one of those.
>
> So I think the only realistic options for packages that hard-require
> this functionality (not all do) are:
>
> 1. Depends: systemd | systemd-tmpfiles
> 2. Depends: systemd-tmpfiles-standalone | systemd-tmpfiles
> 3. Depends: default-systemd-tmpfiles | systemd-tmpfiles
>
> (where the third one is equivalent to one of the first two, depending on
> how default-systemd-tmpfiles was implemented), possibly with some more
> less-preferred options between the first "|" and the virtual-package
> fallback.
>
> I also think that Policy shouldn't be recommending this interface without
> also being able to give guidance on what an appropriate dependency would
> look like, because if some packages choose (1.) and some choose (2.),
> again we'll get a non-deterministic result, depending on which packages
> you happen to install first and what their maintainers think; and I don't
> want individual services' maintainers to be required to constantly argue
> whether (1.) or (2.) is better.
>
> In #1017441 the debhelper maintainers declined to generate such a
> dependency, but even if they had wanted to generate it centrally, we'd
> have the same distro-integration considerations about saying what the
> right dependency would look like.
>
> Before describing pros and cons of those options in different scenarios,
> I want to reiterate that installing the systemd package (which contains
> the default/recommended implementation of the systemd-tmpfiles interface)
> does not imply using systemd as an init system, and everywhere in this
> message that I have said "systemd", I specifically mean the systemd
> package and not systemd-as-pid-1.
>
> >From the point of view of init systems, I think the interesting scenarios
> are:
>
> A. systemd package installed (typically via systemd-sysv)
> B. non-systemd init system (typically sysvinit) and no systemd package
> C. no init system at all (typically in Docker or a chroot)
>
> For (A.), there is no ambiguity: systemd is installed and provides the
> tmpfiles interface, and everything is fine. Any dependency sequence ((1.),
> (2.) or (3.) above) should give us the result we want. The only problem
> scenario I can think of for (A.) with (2.) is:
>
> - a package (let's say foo-service) has chosen (2.) and so depends on
>   systemd-tmpfiles-standalone | systemd-tmpfiles
> - we install a base system with neither foo-service nor systemd
>   - this must be a minbase debootstrap or similar, because a default
> debootstrap already includes systemd-sysv
> - we install foo-service first, without systemd
>   - systemd-tmpfiles-standalone is pulled in
> - afterwards, we install systemd (via the init metapackage and
>   systemd-sysv, or directly)
>   - desired result: systemd-tmpfiles-standalone is removed and replaced by
> systemd
>   - actual result: apt's heuristic might have difficulty realising that
> it needs to do that
>
> I would have expected that anyone wanting an environment with an init
> system is more likely to include it in the bootstrap or in the first
> batch of post-bootstrap additions than to install it later, but maybe
> that's 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

(Newly cc'd elogind maintainers: Please see #945269 for context)

On Sun, 04 Jun 2023 at 12:15:41 +0100, Luca Boccassi wrote:
> On Sun, 4 Jun 2023 at 12:02, Sean Whitton  wrote:
> > On Tue 09 May 2023 at 01:44AM +01, Luca Boccassi wrote:
> > > For now I've kept only a mention of the 'systemd-tmpfiles' virtual
> > > package. As maintainers we would really prefer if the 'main'
> > > implementation is pulled in whenever possible. When a minimal
> > > installation is desired (ie, a minbase), it is possible to manually
> > > specify the -standalone variant.
> > >
> > > This was a controversial point last year, see:
> > >
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017441
> >
> > Hmm.  I don't have personal experience with this sort of thing, but
> > based on some of the examples in that bug, it seems like doing this
> > could cause apt to change people's systems around in ways they strongly
> > disprefer.  What you propose seems like it could cause unpleasant
> > surprises.
> 
> Only due to bugs in said other packages, or if the wrong commands are
> passed to apt/aptitude/etc.

I think one way or another, if anyone is going to set a package-level
dependency on systemd-tmpfiles, the first (preferred) dependency needs to
be on either a concrete provider (systemd or systemd-tmpfiles-standalone
in this case), or a default-systemd-tmpfiles virtual package
that only has one provider per architecture (which is the way
{default-,}dbus-{system,session}-bus are handled). Otherwise, you
can get a non-deterministic choice of default implementation, which
seems strictly worse than either depending on systemd or depending on
systemd-tmpfiles-standalone - if you're unlucky, it can have all the
disadvantages of either one of those.

So I think the only realistic options for packages that hard-require
this functionality (not all do) are:

1. Depends: systemd | systemd-tmpfiles
2. Depends: systemd-tmpfiles-standalone | systemd-tmpfiles
3. Depends: default-systemd-tmpfiles | systemd-tmpfiles

(where the third one is equivalent to one of the first two, depending on
how default-systemd-tmpfiles was implemented), possibly with some more
less-preferred options between the first "|" and the virtual-package
fallback.

I also think that Policy shouldn't be recommending this interface without
also being able to give guidance on what an appropriate dependency would
look like, because if some packages choose (1.) and some choose (2.),
again we'll get a non-deterministic result, depending on which packages
you happen to install first and what their maintainers think; and I don't
want individual services' maintainers to be required to constantly argue
whether (1.) or (2.) is better.

In #1017441 the debhelper maintainers declined to generate such a
dependency, but even if they had wanted to generate it centrally, we'd
have the same distro-integration considerations about saying what the
right dependency would look like.

Before describing pros and cons of those options in different scenarios,
I want to reiterate that installing the systemd package (which contains
the default/recommended implementation of the systemd-tmpfiles interface)
does not imply using systemd as an init system, and everywhere in this
message that I have said "systemd", I specifically mean the systemd
package and not systemd-as-pid-1.

>From the point of view of init systems, I think the interesting scenarios
are:

A. systemd package installed (typically via systemd-sysv)
B. non-systemd init system (typically sysvinit) and no systemd package
C. no init system at all (typically in Docker or a chroot)

For (A.), there is no ambiguity: systemd is installed and provides the
tmpfiles interface, and everything is fine. Any dependency sequence ((1.),
(2.) or (3.) above) should give us the result we want. The only problem
scenario I can think of for (A.) with (2.) is:

- a package (let's say foo-service) has chosen (2.) and so depends on
  systemd-tmpfiles-standalone | systemd-tmpfiles
- we install a base system with neither foo-service nor systemd
  - this must be a minbase debootstrap or similar, because a default
debootstrap already includes systemd-sysv
- we install foo-service first, without systemd
  - systemd-tmpfiles-standalone is pulled in
- afterwards, we install systemd (via the init metapackage and
  systemd-sysv, or directly)
  - desired result: systemd-tmpfiles-standalone is removed and replaced by
systemd
  - actual result: apt's heuristic might have difficulty realising that
it needs to do that

I would have expected that anyone wanting an environment with an init
system is more likely to include it in the bootstrap or in the first
batch of post-bootstrap additions than to install it later, but maybe
that's wrong? Or are there other problem scenarios here?

I'll come back to (B.), non-default init, in a moment.

No init system at all, (C.), can only happen when starting with a
minbase debootstrap or equivalent (because a default 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 4 Jun 2023 at 12:02, Sean Whitton  wrote:
>
> Hello,
>
> On Tue 09 May 2023 at 01:44AM +01, Luca Boccassi wrote:
>
> > I've done an initial attempt to define the wording, although I'm sure
> > it will need quite a few changes. Attached as a patch, and also
> > available on Salsa:
> >
> > https://salsa.debian.org/bluca/policy/-/commits/tmpfiles
> >
> > Happy to move/reword/change/enhance as required.
>
> Thanks.
>
> > For now I've kept only a mention of the 'systemd-tmpfiles' virtual
> > package. As maintainers we would really prefer if the 'main'
> > implementation is pulled in whenever possible. When a minimal
> > installation is desired (ie, a minbase), it is possible to manually
> > specify the -standalone variant.
> >
> > This was a controversial point last year, see:
> >
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017441
>
> Hmm.  I don't have personal experience with this sort of thing, but
> based on some of the examples in that bug, it seems like doing this
> could cause apt to change people's systems around in ways they strongly
> disprefer.  What you propose seems like it could cause unpleasant
> surprises.

Only due to bugs in said other packages, or if the wrong commands are
passed to apt/aptitude/etc.

> > We could even decide that no dependency is added at all by dh, and
> > instead the build tool needs to decide if it's building an image where
> > tmpfiles snippets need to be ran, and if so pull in the preferred
> > alternative.
>
> This is a highly inspecific response, but: aren't things expressed by
> dependencies generally less work for everyone than more special cases to
> be handled by each build tool?

Well, sure, but at some point it's either one or the other: set up the
dependencies so that the default case (which in debian is systemd)
works out of the box and the right thing happens magically, and people
who want to use non-default init systems (which are supported for the
purpose of "experimenting and exploring alternatives") will need to
fix their packages and provide the right instructions to apt, or no
dependency is set at all and bootstrapping/image building/etc tools
need to adapt and manually pull the right tools at the right time.

I'm in favour of the first one, to be clear, but I am open to the
second one for policy if that's what you prefer, and say nothing, in
the interest of moving forward with the change, and at least codify
how things should look like, leaving the low-level arrangements out of
it.

Kind regards,
Luca Boccassi

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Sean Whitton]

Hello,

On Tue 09 May 2023 at 01:44AM +01, Luca Boccassi wrote:

> I've done an initial attempt to define the wording, although I'm sure
> it will need quite a few changes. Attached as a patch, and also
> available on Salsa:
>
> https://salsa.debian.org/bluca/policy/-/commits/tmpfiles
>
> Happy to move/reword/change/enhance as required.

Thanks.

> For now I've kept only a mention of the 'systemd-tmpfiles' virtual
> package. As maintainers we would really prefer if the 'main'
> implementation is pulled in whenever possible. When a minimal
> installation is desired (ie, a minbase), it is possible to manually
> specify the -standalone variant.
>
> This was a controversial point last year, see:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017441

Hmm.  I don't have personal experience with this sort of thing, but
based on some of the examples in that bug, it seems like doing this
could cause apt to change people's systems around in ways they strongly
disprefer.  What you propose seems like it could cause unpleasant
surprises.

> We could even decide that no dependency is added at all by dh, and
> instead the build tool needs to decide if it's building an image where
> tmpfiles snippets need to be ran, and if so pull in the preferred
> alternative.

This is a highly inspecific response, but: aren't things expressed by
dependencies generally less work for everyone than more special cases to
be handled by each build tool?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 18 Sep 2022 20:52:23 -0700 Russ Allbery  wrote:
> Ansgar  writes:
> 
> > I would like to recommend packages to use tmpfiles.d(5) to manage
> > creating directories in locations such as /var or /etc instead of
> > maintainer scripts.
> 
> > It could also be used to create trivial files below /var that
should be
> > recreated if deleted by accident (such as atd's sequence number
which is
> > currently created by the maintainer script).
> 
> Hi all,
> 
> Here's where I think we currently are with this proposal:
> 
> * There should no longer be any blocker to adding a dependency on
>   systemd-tmpfiles since systemd-standalone-tmpfiles exists.
> 
> * So far as I can tell, dh_installtmpfiles adds the correct stanza to
the
>   postinst script for a service to run systemd-tmpfiles after the
package
>   has installed its tmpfiles.d fragment.  I believe that addresses
>   creating these files on installation on systems without any init
system?
>   (Obviously if tmpfs file systems are subsequently reset on such a
>   system, there is nothing in place to recreate these tmp files, but
I
>   think that's expected and not something we can address?)
> 
> * Guillem plans to add support to dpkg to register these files
properly as
>   package-associated files and also handle creation of the files. 
This is
>   great, and I am 100% in favor of it.  I don't think that blocks
this
>   change; to the contrary, I think this change is a good incremental
step
>   towards that world, since it moves temporary file creation out of
>   maintainer scripts into a declarative syntax.  dpkg can then either
>   consume the same syntax or packages can later convert that syntax
to
>   whatever dpkg uses, depending on how the dpkg implementation works,
>   which will be a simple and easy-to-detect migration that Lintian
can
>   diagnose.
> 
> Therefore, I don't see anything blocking adopting this as a policy
now,
> and it seems like an obviously good idea to me.
> 
> Am I missing something?  If not, I think the next step is for someone
to
> propose wording.

I've done an initial attempt to define the wording, although I'm sure
it will need quite a few changes. Attached as a patch, and also
available on Salsa:

https://salsa.debian.org/bluca/policy/-/commits/tmpfiles

Happy to move/reword/change/enhance as required.

> In order to handle installations that have no init system, I think we
may
> have to require that the dependency be listed as:
> 
> systemd-standalone-tmpfiles | systemd-tmpfiles
> 
> to avoid trying to install systemd into an init-free chroot.  Maybe I
have
> that wrong?  Currently, dh_installtmpfiles doesn't add a dependency
at
> all, probably because it assumes that the package will use a fallback
to
> create the files in cases without an init system?  Either way, we
should
> address this in the Policy wording, and then encode that in
> dh_installtmpfiles if needed.

For now I've kept only a mention of the 'systemd-tmpfiles' virtual
package. As maintainers we would really prefer if the 'main'
implementation is pulled in whenever possible. When a minimal
installation is desired (ie, a minbase), it is possible to manually
specify the -standalone variant.

This was a controversial point last year, see:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017441

We could even decide that no dependency is added at all by dh, and
instead the build tool needs to decide if it's building an image where
tmpfiles snippets need to be ran, and if so pull in the preferred
alternative.

-- 
Kind regards,
Luca Boccassi
From  Mon Sep 17 00:00:00 2001
From: Luca Boccassi 
Date: Tue, 9 May 2023 01:38:13 +0100
Subject: [PATCH] Define tmpfiles.d interface and usage

---
 locales/ja/LC_MESSAGES/ch-files.po|  97 --
 .../ja/LC_MESSAGES/ch-maintainerscripts.po| 293 ++
 policy/ch-files.rst   |  39 +++
 policy/ch-maintainerscripts.rst   |   6 +
 4 files changed, 288 insertions(+), 147 deletions(-)

diff --git a/locales/ja/LC_MESSAGES/ch-files.po b/locales/ja/LC_MESSAGES/ch-files.po
index c241418..ec9d329 100644
--- a/locales/ja/LC_MESSAGES/ch-files.po
+++ b/locales/ja/LC_MESSAGES/ch-files.po
@@ -10,7 +10,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: Debian Policy Manual 4.1.6.0\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2021-08-17 20:06-0700\n"
+"POT-Creation-Date: 2023-05-09 01:31+0100\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME \n"
 "Language-Team: LANGUAGE \n"
@@ -963,7 +963,60 @@ msgid ""
 " to ASCII when it is possible to do so."
 msgstr ""
 
-#: ../../ch-files.rst:726
+#: ../../ch-files.rst:728
+msgid "tmpfiles.d"
+msgstr ""
+
+#: ../../ch-files.rst:730
+msgid ""
+"Packages might need additional files or directories to implement their "
+"functionality. Directories that are located under ``/var/`` or ``/etc/``,"
+" and files that are located under ``/var/``, should not be created "

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Ansgar  writes:

> I would like to recommend packages to use tmpfiles.d(5) to manage
> creating directories in locations such as /var or /etc instead of
> maintainer scripts.

> It could also be used to create trivial files below /var that should be
> recreated if deleted by accident (such as atd's sequence number which is
> currently created by the maintainer script).

Hi all,

Here's where I think we currently are with this proposal:

* There should no longer be any blocker to adding a dependency on
  systemd-tmpfiles since systemd-standalone-tmpfiles exists.

* So far as I can tell, dh_installtmpfiles adds the correct stanza to the
  postinst script for a service to run systemd-tmpfiles after the package
  has installed its tmpfiles.d fragment.  I believe that addresses
  creating these files on installation on systems without any init system?
  (Obviously if tmpfs file systems are subsequently reset on such a
  system, there is nothing in place to recreate these tmp files, but I
  think that's expected and not something we can address?)

* Guillem plans to add support to dpkg to register these files properly as
  package-associated files and also handle creation of the files.  This is
  great, and I am 100% in favor of it.  I don't think that blocks this
  change; to the contrary, I think this change is a good incremental step
  towards that world, since it moves temporary file creation out of
  maintainer scripts into a declarative syntax.  dpkg can then either
  consume the same syntax or packages can later convert that syntax to
  whatever dpkg uses, depending on how the dpkg implementation works,
  which will be a simple and easy-to-detect migration that Lintian can
  diagnose.

Therefore, I don't see anything blocking adopting this as a policy now,
and it seems like an obviously good idea to me.

Am I missing something?  If not, I think the next step is for someone to
propose wording.

In order to handle installations that have no init system, I think we may
have to require that the dependency be listed as:

systemd-standalone-tmpfiles | systemd-tmpfiles

to avoid trying to install systemd into an init-free chroot.  Maybe I have
that wrong?  Currently, dh_installtmpfiles doesn't add a dependency at
all, probably because it assumes that the package will use a fallback to
create the files in cases without an init system?  Either way, we should
address this in the Policy wording, and then encode that in
dh_installtmpfiles if needed.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Sat, 30 Nov 2019 at 10:58:23 -0800, Russ Allbery wrote:
> Guillem Jover  writes:
> > I don't mind much how this could end up being hooked into various init
> > systems and chroot/container managers. I can see how it could be done by
> > the respective imeplementations themselves or provided by dpkg itself,
> > I'm open to any of these TBH.
> 
> One possibly interesting option would be for dpkg to take responsibility
> for writing the tmpfiles.d configuration file for a package based on its
> own metadata

I think the other way round could also make sense: having dpkg create
variable/transient directories, and track them as owned by the package,
using its tmpfiles.d fragment(s) as input (either directly, or via a
debhelper step that takes tmpfiles.d as input and registers it with dpkg
as output).

That way, in the hopefully-common case where the upstream developer
of a system service installs a tmpfiles.d fragment for the files that
their package owns (most likely generated at build-time with appropriate
Autoconf/etc. @substitutions@), the packaged version will do the right
thing, without the packager having to take action specifically.

Prior art/naming: I think RPM "ghost files" are files tracked as owned
but not actually shipped in the RPM?

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Guillem Jover  writes:

> The way I see it, any pathname "owned" by a package is within the realm
> of dpkg, which is in charge of tracking and managing the filesystem
> contents for system software. I've, for example, always found it a big
> defficiency that when querying dpkg about system pathnames it cannot
> give a proper answer in many cases. The same with verification of system
> pathnames. But this goes beyond temporary files, this includes things
> like log or cache files, or other volatile pathnames, for example.

I completely agree with all of this.  I was curious what you anticipated
using as an implementation strategy.

> I don't mind much how this could end up being hooked into various init
> systems and chroot/container managers. I can see how it could be done by
> the respective imeplementations themselves or provided by dpkg itself,
> I'm open to any of these TBH.

One possibly interesting option would be for dpkg to take responsibility
for writing the tmpfiles.d configuration file for a package based on its
own metadata (and registering that file as part of the package), which
accomplishes a lot of the same goals (for instance, local administrators
used to other systemd-based distributions will find configuration in the
expected place and working in the expected way) but leaves open support
from other init systems.

Obviously it would be a lot easier to do that if the package format dpkg
consumed used a compatible format to systemd-tmpfiles (and there would be
some, probably minor, learning-curve benefits if it used the same format).

> To me that seems like an extremely unsatisfying and restrictive way to
> characterize cross-distribution. This implies GNU/Linux-only ones, not
> even things like musl-based, for example. I consider portability of
> paramount importance, and I'll not stop caring about it, even if the
> Debian project decided otherwise, less so as an upstream. (See my GR
> amendment. :)

I understand, and I don't want you to stop caring about that.  I think
there are approaches that might satisfy both goals at the same time, such
as dpkg using systemd's native services and configuration files to
integrate with systemd-using systems.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Guillem Jover]

Hi!

On Fri, 2019-11-29 at 09:13:47 -0800, Russ Allbery wrote:
> Guillem Jover  writes:
> > As I mentioned on debian-devel, I think major parts of this and of the
> > sysuser stuff fall under dpkg realm. And my plan is to implement this
> > kind of functionality natively in dpkg, regardless of whatever the
> > outcome of that GR is.
> 
> > Of course some parts of the functionality needed to manage/track
> > temporary pathnames requires integration or hooking into an init system
> > or chroot/container manager, but that's true regardless of the
> > implementation.
> 
> Could you say more about how you think dpkg would be able to handle
> systemd-tmpfiles?  As you mention, this is something that has to be done
> at every system boot, which seems pretty far afield from dpkg's domain.
> Do you want to maintain custom dpkg plugins for every init system that a
> dpkg system may support?

The way I see it, any pathname "owned" by a package is within the
realm of dpkg, which is in charge of tracking and managing the
filesystem contents for system software. I've, for example, always
found it a big defficiency that when querying dpkg about system
pathnames it cannot give a proper answer in many cases. The same with
verification of system pathnames. But this goes beyond temporary files,
this includes things like log or cache files, or other volatile
pathnames, for example.

These kinds of pathnames are also potential package cruft, that should
be cleaned up on package remove/purge (ideally in a configurable way,
as in “I want logs to be left behind after purge”), and w/o requiring
maintainer scripts.

  

I don't mind much how this could end up being hooked into various init
systems and chroot/container managers. I can see how it could be done by
the respective imeplementations themselves or provided by dpkg itself,
I'm open to any of these TBH.

> (My recollection is that dpkg supports other OSes like Solaris.)

  
  

> (systemd-sysusers is another matter and a bit more obviously doable with
> dpkg, although if Sam's option three wins, I'd ask everyone in the
> project, including you as dpkg maintainer, to consider the possible
> benefits of using cross-distribution mechanisms instead of Debian-specific
> mechanisms.)

  

To me that seems like an extremely unsatisfying and restrictive way
to characterize cross-distribution. This implies GNU/Linux-only ones,
not even things like musl-based, for example. I consider portability
of paramount importance, and I'll not stop caring about it, even if
the Debian project decided otherwise, less so as an upstream. (See my
GR amendment. :)

I find integration within Debian and with the wider dpkg ecosystem to
be way more interesting, and interfaces in dpkg that can work anywhere
where it would be available. That's why I'll work on this no matter
what, even if Debian decided to not use these features (which I'd find
rather unfortunate), I'd spend time on this at the expense of other
stuff.

Thanks,
Guillem

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Guillem Jover  writes:

> As I mentioned on debian-devel, I think major parts of this and of the
> sysuser stuff fall under dpkg realm. And my plan is to implement this
> kind of functionality natively in dpkg, regardless of whatever the
> outcome of that GR is.

> Of course some parts of the functionality needed to manage/track
> temporary pathnames requires integration or hooking into an init system
> or chroot/container manager, but that's true regardless of the
> implementation.

Could you say more about how you think dpkg would be able to handle
systemd-tmpfiles?  As you mention, this is something that has to be done
at every system boot, which seems pretty far afield from dpkg's domain.
Do you want to maintain custom dpkg plugins for every init system that a
dpkg system may support?  (My recollection is that dpkg supports other
OSes like Solaris.)

(systemd-sysusers is another matter and a bit more obviously doable with
dpkg, although if Sam's option three wins, I'd ask everyone in the
project, including you as dpkg maintainer, to consider the possible
benefits of using cross-distribution mechanisms instead of Debian-specific
mechanisms.)

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Guillem Jover]

On Fri, 2019-11-22 at 10:12:06 -0800, Russ Allbery wrote:
> Ansgar  writes:
> > I think no option says we shouldn't use services that don't rely on
> > systemd as pid-1 (which also includes widely used things like udev).
> 
> I agree, but if, say, Sam's option 3 wins, we can directly incorporate
> this, whereas if, say, Ian's option wins, then we should have a
> conversation with the sysvinit maintainers to see if they believe they can
> adopt systemd-tmpfiles directly, and we should give them at least six
> months (obviously shortened if it happens sooner) to update the packaging
> and set up a mechanism to run systemd-tmpfiles (or a replacement of their
> choice) at boot.

As I mentioned on debian-devel, I think major parts of this and of
the sysuser stuff fall under dpkg realm. And my plan is to implement
this kind of functionality natively in dpkg, regardless of whatever
the outcome of that GR is.

Of course some parts of the functionality needed to manage/track
temporary pathnames requires integration or hooking into an init system
or chroot/container manager, but that's true regardless of the
implementation.

Thanks,
Guillem

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Ansgar  writes:

> I think no option says we shouldn't use services that don't rely on
> systemd as pid-1 (which also includes widely used things like udev).

I agree, but if, say, Sam's option 3 wins, we can directly incorporate
this, whereas if, say, Ian's option wins, then we should have a
conversation with the sysvinit maintainers to see if they believe they can
adopt systemd-tmpfiles directly, and we should give them at least six
months (obviously shortened if it happens sooner) to update the packaging
and set up a mechanism to run systemd-tmpfiles (or a replacement of their
choice) at boot.

If you want to start those conversations now, then we could get the
proposal to a place where it would be fine for all possible GR options,
but I suspect the people that you'd need to talk to are focused on the GR
at the moment.

Meanwhile, it's going to take at least a week to hash out wording (just
based on past experience), and there's no reason not to start that now,
and by that point we may be pretty near a vote anyway.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Ansgar]

On Fri, 2019-11-22 at 09:05 -0800, Russ Allbery wrote:
> Simon McVittie  writes:
> > I had also thought that, in the current state of non-systemd-init-world,
> > having a dependency on a package containing systemd-tmpfiles was likely
> > to be practically problematic because it would indirectly conflict
> > with elogind, via libsystemd0 and its elogind reimplementation - but
> > looking more closely, systemd-tmpfiles and systemd-sysusers depend on
> > /lib/systemd/libsystemd-shared-243.so (and not on libsystemd0) so that
> > shouldn't actually be a problem.

As I said I'm interested in supporting environments with no init
installed and believe alternative init implementations happen to have
the same or similar needs here.  For environments with no init, it
would be useful to split out the -tmpfiles and -sysuser programs into a
separate package as people using container images are always concerned
about image sizes.

I tried linking -tmpfiles and -sysusers against a static version of
libsystemd-shared which increased binary size a bit: -tmpfiles was 212
kB (up from current 84 kB), -sysusers 137 kB (up from current 51 kB). 
But this should allow to safely split out the programs into a smaller
binary package.

> There's also the problem that while systemd-tmpfiles doesn't currently
> depend on systemd as PID 1, I'm dubious upstream is willing to make any
> guarantees that this will continue to be the case.

I would be surprised if upstream would break installing packages in
containers which often don't have systemd as pid-1 (as they have no
init at all).  There also seems no reason for them to use facilities
provided by pid-1.

> It's also not clear
> whether the project would be comfortable with us adopting new Policy
> guidance that inherently breaks the non-Linux ports (since I believe the
> systemd package does not build there).

We can make the guidance apply to only Linux and use other means on
non-Linux platform, just as we would for other Linux-specific parts
(any systemd .service files for example).  Altenratively, non-Linux
ports could choose to use an alternative implementation (which
reportedly already exists outside Debian).

> Let's hold off on this until the GR.  The GR won't be too far into the
> future, and will provide a very clear path forward for proposals like
> this.
> 
> Thank you very much for starting to work on it, though, and I'd be very
> happy to see specific proposed language in this bug that we can consider
> once the GR makes it clear how the project wants us to proceed with
> facilities like this.

I think no option says we shouldn't use services that don't rely on
systemd as pid-1 (which also includes widely used things like udev).

Ansgar

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Simon McVittie  writes:
> On Fri, 22 Nov 2019 at 14:05:50 +0100, Ansgar wrote:

>> I'm fairly sure that systemd-tmpfiles doesn't require systemd as pid-1

> It doesn't, but its dh_installsystemd integration currently does, so
> maintainer scripts relying on it would currently be buggy. I think it
> would be premature to recommend systemd-tmpfiles in Policy before it can
> be used in a non-buggy way.

> I had also thought that, in the current state of non-systemd-init-world,
> having a dependency on a package containing systemd-tmpfiles was likely
> to be practically problematic because it would indirectly conflict
> with elogind, via libsystemd0 and its elogind reimplementation - but
> looking more closely, systemd-tmpfiles and systemd-sysusers depend on
> /lib/systemd/libsystemd-shared-243.so (and not on libsystemd0) so that
> shouldn't actually be a problem.

There's also the problem that while systemd-tmpfiles doesn't currently
depend on systemd as PID 1, I'm dubious upstream is willing to make any
guarantees that this will continue to be the case.  It's also not clear
whether the project would be comfortable with us adopting new Policy
guidance that inherently breaks the non-Linux ports (since I believe the
systemd package does not build there).

Let's hold off on this until the GR.  The GR won't be too far into the
future, and will provide a very clear path forward for proposals like
this.

Thank you very much for starting to work on it, though, and I'd be very
happy to see specific proposed language in this bug that we can consider
once the GR makes it clear how the project wants us to proceed with
facilities like this.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Fri, 22 Nov 2019 at 14:05:50 +0100, Ansgar wrote:
> I'm fairly sure that systemd-tmpfiles doesn't require systemd as pid-1

It doesn't, but its dh_installsystemd integration currently does, so
maintainer scripts relying on it would currently be buggy. I think it
would be premature to recommend systemd-tmpfiles in Policy before it can
be used in a non-buggy way.

I had also thought that, in the current state of non-systemd-init-world,
having a dependency on a package containing systemd-tmpfiles was likely
to be practically problematic because it would indirectly conflict
with elogind, via libsystemd0 and its elogind reimplementation - but
looking more closely, systemd-tmpfiles and systemd-sysusers depend on
/lib/systemd/libsystemd-shared-243.so (and not on libsystemd0) so that
shouldn't actually be a problem.

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Ansgar]

On Fri, 2019-11-22 at 12:07 +, Simon McVittie wrote:
> On Fri, 22 Nov 2019 at 09:03:29 +0100, Ansgar wrote:
> > I would like to recommend packages to use tmpfiles.d(5) to manage
> > creating directories in locations such as /var or /etc instead of
> > maintainer scripts.
> 
> Using tmpfiles.d(5) seems like a good thing to encourage, but using
> them *instead of* maintainer scripts is less simple. My understanding
> is that in current Debian policies and status, if a package relies
> on tmpfiles.d(5) for important functionality and does not depend on a
> tmpfiles implementation, that would usually be a grave bug.

Sure, I don't see a problem with that in principle though?

> The only tmpfiles implementation in Debian right now (that I know of) is
> the combination of systemd-tmpfiles(8), systemd-tmpfiles-setup.service
> (for creation of tmpfiles during boot), the maintainer script
> fragments generated by dh_installsystemd (for creation of tmpfiles
> during installation), and systemd as pid 1.

I'm fairly sure that systemd-tmpfiles doesn't require systemd as pid-1
(though one GR option seems to imply systemd-tmpfiles and -sysusers are
somehow facilities that need alternative implementations for other
inits, but I don't think that claim was verified).

Both -tmpfiles and -sysusers seem to work for example in Docker
containers which don't run systemd-init.

> I'm not entirely sure why
> systemd doesn't have a trigger on tmpfiles.d, but perhaps it's because
> the sequencing would be wrong: a package's tmpfiles generally need to be
> created before its postinst reaches the point where services are started,
> and I don't think triggers would guarantee that.

I think the maintainer scripts would currently need to call systemd-
tmpfiles once, yes.  (The same would be true for -sysusers; see also
the --replace option in systemd-sysusers(8).)

> ( is an alternative implementation,
> but is not in Debian and would require adjustments to dh_installsystemd.)

I know it exists, but I don't think we need an alternative
implementation?  It would be useful for non-Linux, but if people don't
want to package it then we could also just recommend using
tmpfiles.d(5) on Linux only.

On Linux I see no advantage of having multiple implementations. People
are of course free to replace them locally if they want though, just
like they can for coreutils.

> Note that I am deliberately saying "without systemd as pid 1" rather than
> "with a non-systemd init", because there is a subtle difference. If you
> install (for example) dbus into a chroot or container that does not run an
> init system at all, then the current dh_installsystemd fragment will see
> that /run/systemd/system doesn't exist, so /usr/lib/tmpfiles.d/dbus.conf
> will not be processed and /var/lib/dbus will not be created.

We don't have to use dh_installsystemd and I think it was proposed to
use a different helper for tmpfiles anyway (for other reasons).

> (Reference:
> look in /var/lib/dpkg/info/dbus.postinst for "Automatically added
> by dh_installsystemd".) This doesn't matter for the D-Bus system
> service, because the D-Bus system service will not be started in those
> chroots/containers (there is no init to start it), but it does matter
> for dbus-x11 (which also wants to see /var/lib/dbus/machine-id for
> the autolaunching protocol) and anything else that is using the D-Bus
> machine ID.
> 
> This means that even if GR 2019-002 resolves that we will stop supporting
> non-systemd init systems, there will still be two cases to consider:
> systemd vs. no init at all.

Sure, I'm quite interested in supporting the no-init case (I think the
request to make the `init` package non-essential and optional came from
me ;-)).  I wouldn't be surprised if other distributions also care
about that, even when they only support systemd as init.

(It might even be useful to move helpers like -tmpfiles and -sysusers
to a separate packages so container images won't pull in the full
systemd package.)

"Other inits" are probably not much different han "no init", except
that they should call systemd-tmpfiles at some appropriate time during
boot.

> Another potential difficulty is that some tmpfiles.d(5) fragments might
> assume that systemd is installed: certain guarantees/constraints,
> such as /etc/machine-id behaving as documented in machine-id(5),
> are known to be true on systems with systemd as pid 1, but not on
> Debian systems in general. dbus' tmpfiles fragment creates a symlink
> /var/lib/dbus/machine-id -> /etc/machine-id based on the assumption that
> machine-id(5) is as documented, which is not necessarily true without
> systemd (again, this can mean either sysvinit as pid 1, or no init
> at all).

I think it is fine to blacklist specific cases such as using the
machine-id (at least without an explicit dependency).  I don't expect
many packages to use these anyway.  Same for the options related to
(btrfs) subvolumes and some 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Simon McVittie]

On Fri, 22 Nov 2019 at 09:03:29 +0100, Ansgar wrote:
> I would like to recommend packages to use tmpfiles.d(5) to manage
> creating directories in locations such as /var or /etc instead of
> maintainer scripts.

Using tmpfiles.d(5) seems like a good thing to encourage, but using
them *instead of* maintainer scripts is less simple. My understanding
is that in current Debian policies and status, if a package relies
on tmpfiles.d(5) for important functionality and does not depend on a
tmpfiles implementation, that would usually be a grave bug.

The only tmpfiles implementation in Debian right now (that I know of) is
the combination of systemd-tmpfiles(8), systemd-tmpfiles-setup.service
(for creation of tmpfiles during boot), the maintainer script
fragments generated by dh_installsystemd (for creation of tmpfiles
during installation), and systemd as pid 1. I'm not entirely sure why
systemd doesn't have a trigger on tmpfiles.d, but perhaps it's because
the sequencing would be wrong: a package's tmpfiles generally need to be
created before its postinst reaches the point where services are started,
and I don't think triggers would guarantee that.

( is an alternative implementation,
but is not in Debian and would require adjustments to dh_installsystemd.)

It is of course fine to ship tmpfiles.d(5) fragments (like dbus does)
as long as the package either depends on systemd as pid 1, or has some
redundant way to create enough tmpfiles that it can work without systemd
as pid 1; and I think it might be good for Policy to recommend that
required directories in /run, /var and /etc are listed in tmpfiles.d(5)
fragments, particularly /run and /var/cache.

Ways to avoid solely relying on tmpfiles.d(5) include creating
directories from maintainer scripts (possibly redundantly), shipping
directories (other than in /run!) in the .deb (possibly redundantly),
and creating directories on a just-in-time basis in init-scripts or
init-script-equivalent constructs like systemd services (possibly
redundantly).

Note that I am deliberately saying "without systemd as pid 1" rather than
"with a non-systemd init", because there is a subtle difference. If you
install (for example) dbus into a chroot or container that does not run an
init system at all, then the current dh_installsystemd fragment will see
that /run/systemd/system doesn't exist, so /usr/lib/tmpfiles.d/dbus.conf
will not be processed and /var/lib/dbus will not be created. (Reference:
look in /var/lib/dpkg/info/dbus.postinst for "Automatically added
by dh_installsystemd".) This doesn't matter for the D-Bus system
service, because the D-Bus system service will not be started in those
chroots/containers (there is no init to start it), but it does matter
for dbus-x11 (which also wants to see /var/lib/dbus/machine-id for
the autolaunching protocol) and anything else that is using the D-Bus
machine ID.

This means that even if GR 2019-002 resolves that we will stop supporting
non-systemd init systems, there will still be two cases to consider:
systemd vs. no init at all.

Another potential difficulty is that some tmpfiles.d(5) fragments might
assume that systemd is installed: certain guarantees/constraints,
such as /etc/machine-id behaving as documented in machine-id(5),
are known to be true on systems with systemd as pid 1, but not on
Debian systems in general. dbus' tmpfiles fragment creates a symlink
/var/lib/dbus/machine-id -> /etc/machine-id based on the assumption that
machine-id(5) is as documented, which is not necessarily true without
systemd (again, this can mean either sysvinit as pid 1, or no init
at all).

For some packages these issues could be solved by a dependency on
systemd (in this case not systemd-sysv the init system, just systemd
the collection of executables) and suitable adjustments to the
dh_installsystemd maintainer script fragment; or by a dependency on
systemd | opentmpfiles and suitable adjustments to dh_installsystemd,
if someone packages .

> This has advantages over maintainer scripts

I agree, and I would like to seem more use of tmpfiles.d(5), but I think
it needs to be thought through some more.

> As far as I understand the current GR is unrelated to adopting things
> like systemd-tmpfiles.

I don't think that's true: as long as the only working implementation
of tmpfiles.d(5) in Debian requires systemd as pid 1, and Debian aims to
support pid 1 other than systemd, we can't rely on tmpfiles.d(5). Some of
the possible results of the current GR would change those constraints.
Someone packaging opentmpfiles could also change those constraints.

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Ansgar]

Package: debian-policy

I would like to recommend packages to use tmpfiles.d(5) to manage
creating directories in locations such as /var or /etc instead of
maintainer scripts.

It could also be used to create trivial files below /var that should be
recreated if deleted by accident (such as atd's sequence number which is
currently created by the maintainer script).

This has advantages over maintainer scripts:

 - It's declarative instead of imperative.

 - Directories/files get recreated automatically if deleted;
   previously packages would sometimes just fail if one deleted, for
   example, a subdirectory of /var/cache.

 - Manipulating ownership of files is prone to race conditions or fun
   with hard- and symlinks; many maintainer scripts don't do this fully
   correct.

 - It brings stateless systems (which start with empty /etc, /var) a bit
   closer.

I'm also fine with alternatives such as using `CacheDirectory=`,
`StateDirectory=`, `ConfigurationDirectory=` or `LogsDirectory=` in
.service files where appropriate.

I have no opinion on implementation details so far.

As far as I understand the current GR is unrelated to adopting things
like systemd-tmpfiles.

Ansgar