Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Marco d'Itri]

On Sep 17, Russ Allbery  wrote:

> (I am a little confused by this wording, but I think what you're saying is
> that /usr is encrypted and read-only, and /var is recreated on each boot.
> That at least is my understanding of the pattern that you're trying to
> enable.)
The general idea is to be able to create /var on the first boot.
If /var can be populated programmatically then a system can be trivially 
replicated by sharing (or copying) /usr and by copying /etc.

BTW, I do not expect that tmpfiles.d(5) will be the standard method used 
to create most directories below /var.
Usually the CacheDirectory, LogsDirectory and StateDirectory directives 
are more convenient and flexible.

> The benefit we gain from this is attribution of the directories in the
> dpkg database, which is useful (although I understand that one can argue
> about how useful).
Not enough to justify having multiple sources of truth is my opinion.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Does iproute2 moving config files to /usr/lib violate section 10.7.2?

[Bill Allombert]

On Sun, Sep 17, 2023 at 12:12:18AM +0200, Daniel Gröber wrote:
> Sam, Russ, Bill,
> 
> Thanks for your input. To be quite frank I still don't see how the
> interpretation of allowing configuration files outside of /etc can be
> supported based on the policy text.
> 
> Ultimately I'm just concerned about the UX aspects of admins suddenly
> having to go hunting for config files all over their system when packages
> start implementing this config-in-/usr business en mass.

It seems your issue is more with the documentation and transition to these
new schemes, which is a valid concern I mentionned in my answer.

There are two common schemes:
1/ the default file is read and then the config file is read to 
override/supplement it.
In that case the package should ship with dummy config files that document 
where is the file
they override

2/ the default file is only read when the config file does not exist.
This is more annoying since this precludes shipping with dummy config files.
In that case I recommend including a README file listing the path to the
default files that can be overrided.
Especially if this is a new scheme that users cannot yet be aware of.

Cheers,
Bill.

Bug#1051371: Post-/usr-merge paths for script interpreters

[Bill Allombert]

On Sun, Sep 17, 2023 at 08:52:17AM +0200, Ansgar wrote:
> > Control: unblock 1051371 by 1050001
> > 
> > Ansgar  writes:
> > 
> > > However, there is a proposal by Jackson for an alternative filesystem
> > > layout based on symlink farms in consideration by the technical
> > > committee.  This advocates removing compat symlinks in /bin, /sbin over
> > > time[1], thus requiring (c).
> > 
> > This is not a correct summary of Ian's proposal.  In the message that you
> > linked, Ian says:
> > 
> >     /bin and /lib etc. remain directories (so there is no aliasing).  All
> >     actual files are shipped in /usr.  / contains compatibility symlinks
> >     pointing into /usr, for those files/APIs/programs where this is needed
> >     (which is far from all of them).  Eventualloy, over time, the set of
> >     compatibility links is reduced to a mere handful.
> > 
> > I am absolutely certain that Ian would consider /bin/sh to be one of the
> > programs for which a compatibility symlink is needed, and one of the
> > remaining handful of links that would exist indefinitely into the future.
> > Indeed, he mentions /bin/sh explicitly later in that message.
> 
> But the subject of this issue talks about "script interpreters", not
> just `/bin/sh` (which I guess is safe to assume would be one of the
> "handful").
> 
> It is unclear what files the Jackson symlink farm proposal would leave
> in /bin.  Would /bin/python3 stay?  Or would it stay first, but dropped
> at some later point?  What about /bin/perl, /bin/zsh, /bin/env, ...?

/bin/perl, /bin/env, /bin/python3 did not exist in the old scheme, so there is
no point in creating them now.

/bin/zsh and /bin/sed existed, though.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#1051582: Policy 9.3 (Starting system services) is largely obsolete

[Niels Thykier]

Russ Allbery:

Bill Allombert  writes:

[...]  Quite a lot of Policy is of the general format "here's a bunch of
complex things you need to do, wait, never mind, just use debhelper, go
see its documentation for the configuration files you should use instead"
and some of the rest of Policy is "here's a bunch of complex things you
need to do but if you follow these instructions instead of using debhelper
your package will be buggy."  This is not ideal!

I think there's a lot of appeal of having a thorough specification for
what debhelper is supposed to be doing.  It would enable future
competition around better packaging helpers (and I do think debhelper will
not be the last word).  But I also want to be realistic about whether
we're really capable of maintaining that specification.

--
Russ Allbery (r...@debian.org)  



Indeed. I have had the same challenge with the Policy.

I had a look at the introduction section of Policy to check who the 
target audience is.  I cannot find an explicit mention of the target 
audience. I suspect there is a conflict here on the content because we 
have different audiences in mind for the Policy and the Policy does not 
seem to be explicit here.


If the target audience is package maintainers, then 100% of all debian 
contributors should read it. Then we need "simple and easy to 
understand" language and examples, because we want *everybody* to 
understand this.  I agree with Russ that long winded sections of "here 
is how you do this manually but really just use debhelper" is directly 
counter productive to this audience.


If it is cross-package integration, are we targeting the ones 
integrating or the ones providing the integration points?  If it is 
both, then for more complex topics, it would make sense to split the 
topic into two. One for consumer and one for the provider of the 
integration, because for the consumer it will probably boil down to 
"install path at $X in your deb and run tool $Y" and then the consumer 
can skip the provider section as "not relevant".


  There is a separate here in whether the Policy editors have the 
capacity to maintain the documentation for the provider side of the 
integration points for all the integrations.  I think this is where Russ 
is arguing that we do not that capacity.  If so, it would also make 
sense to explicitly cut *out* that side of from policy in the Scope 
section. Maybe along the lines of "The Policy does not document the 
provider side of integration points directly. Instead, we provide links 
to the external documentation where available.".



If the target audience is tool-chain maintainers, then only 5-10 people 
need to read the policy and the entire document + related process seems 
completely over-engineered.  But then, for the same reason, I suspect 
this is an unlikely target audience for the Policy.



Either way, I think the root problem is that we are not all agreeing on 
scope and audience for the Policy. Until resolved, we can argue forever 
about whether X belongs in Policy.


Best regards,
Niels


PS: I also have other things to say about the concrete topic, but I will 
leave that for a later mail. I think the point above is so important 
that it should not be diluted with other topics.

Bug#1051371: Post-/usr-merge paths for script interpreters

[Ansgar]

On Sun, 2023-09-17 at 11:54 +0200, Bill Allombert wrote:
> /bin/perl, /bin/env, /bin/python3 did not exist in the old scheme, so
> there is no point in creating them now.

No, in Debian's current filesystem layout (Debian stable and later;
partly Debian oldoldstable and later) all of these exist.  They are
also used, both by software shipping in Debian and outside.

Dropping them would break existing software.  I'm assuming the Jackson
symlink proposal would retain them for this reason instead of breaking
software for no good reason.

Ansgar

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Bill Allombert]

On Sun, Sep 17, 2023 at 10:41:55AM +0200, Marco d'Itri wrote:
> On Sep 17, Russ Allbery  wrote:
> 
> > (I am a little confused by this wording, but I think what you're saying is
> > that /usr is encrypted and read-only, and /var is recreated on each boot.
> > That at least is my understanding of the pattern that you're trying to
> > enable.)
> The general idea is to be able to create /var on the first boot.

Does not that would break users expectation that the system image contains /var
before the first boot ?

A lot of things in /var are caches that are mostly instance-independent and can
be prefilled, but for that, users expect a minimal directory hierarchy to be
present before first boot.

It seems your scheme favors some usecase over some others.

Cheers
-- 
Bill. 

Imagine a large red swirl here. 

Re: Does iproute2 moving config files to /usr/lib violate section 10.7.2?

[Simon McVittie]

On Sun, 17 Sep 2023 at 02:03:52 +0200, Santiago Vila wrote:
> If the files are intended to be modified "in place" by the
> system admin, we call them configuration files (and we try hard
> to put them in /etc). If they are not intended to be modified by
> the system admin, we don't call them configuration files (and
> we try hard not to put them in /etc).

Perhaps an interesting example of this is that in packages originally
developed on an OS with less elaborate conffile handling than dpkg's
(like dbus, originally developed on/by Red Hat), there used to be a
common pattern of having:

1. a file in /etc that sysadmins are specifically asked *not* to edit,
   for which non-dpkg package managers would be asked to default to
   the equivalent of dpkg's "use the maintainer's version" (for example
   %config in RPM), like /etc/dbus-1/system.conf

2. a separate file for local changes, which might or might not exist with
   empty or template contents by default, for which non-dpkg package
   managers would be asked to default to the equivalent of dpkg's "keep
   the locally modified version" (for example %config(noreplace) in RPM),
   like /etc/dbus-1/system-local.conf

/etc/dbus-1/system.conf was traditionally a dpkg conffile because
of its location, but it wasn't really a configuration file by
the definition Santiago gave here (and it has now been moved to
/usr/share/dbus-1/system.conf as a result): it's more like part of the
implementation of dbus-daemon --system that, as an implementation detail,
happens to have been written in the dbus-daemon XML configuration language
rather than in C.

/etc/dbus-1/system-local.conf isn't created by default, but sysadmins
can create it if they want to, and it's a "true" configuration file
(although in practice I'd recommend "drop-ins" in /etc/dbus-1/system.d
as a better version of /etc/dbus-1/system-local.conf).

smcv

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Bill Allombert  writes:
> On Sun, Sep 17, 2023 at 10:41:55AM +0200, Marco d'Itri wrote:
>> On Sep 17, Russ Allbery  wrote:

>>> (I am a little confused by this wording, but I think what you're
>>> saying is that /usr is encrypted and read-only, and /var is recreated
>>> on each boot.  That at least is my understanding of the pattern that
>>> you're trying to enable.)

>> The general idea is to be able to create /var on the first boot.

> Does not that would break users expectation that the system image
> contains /var before the first boot ?

> A lot of things in /var are caches that are mostly instance-independent
> and can be prefilled, but for that, users expect a minimal directory
> hierarchy to be present before first boot.

Not that I think we're particularly close to achieving this design
currently (and to be clear we haven't decided we're working towards this
yet), but while I understand why a user would have that expectation today,
I'm not sure why it would practically matter.  If all of that directory
structure appears on first boot, and no static data is stored in /var,
what use case requires the directory structure already exist in /var
before the first boot?

I think you're thinking of cases where the user puts data into /var and
expects it to be used by the system after boot, but configuration data
would go into /etc, so I'm not sure what data that would be.

Also, I think that scenario would still work.  My understanding of the
design is that /var isn't tmpfs; while there's no precreated directory
structure, the user could still make one if they wanted.  There wouldn't
be the guide of existing empty directories, but this is a fairly
sophisticated use case, IMO.

-- 
Russ Allbery (r...@debian.org)  

Bug#1051371: Post-/usr-merge paths for script interpreters

[Russ Allbery]

Ansgar  writes:

> But the subject of this issue talks about "script interpreters", not
> just `/bin/sh` (which I guess is safe to assume would be one of the
> "handful").

> It is unclear what files the Jackson symlink farm proposal would leave
> in /bin.  Would /bin/python3 stay?  Or would it stay first, but dropped
> at some later point?  What about /bin/perl, /bin/zsh, /bin/env, ...?

Oh, okay, I see what you're syaing now.  This is a bit beyond the scope of
the areas of Policy touched by Luca's proposal, but I can see why it would
indeed arise under Ian's proposal.  We've introduced new /bin interfaces
for every binary in /usr/bin, and if we went down that path we'd remove
most of those interfaces again.  That is indeed an argument for (c) for
*most* things, just not the areas touched by this diff (with the possible
exception of /bin/csh; I'm not sure if that would qualify for an exception
or not, these days).

So yes, I agree that the resolution of this bug would significantly affect
what we want to say in Policy about /usr-merge in general, even if not
what we say about /bin/sh.

I still don't feel like we need to wait for the TC bug to be resolved,
since there is a standing TC decision to make /bin a symlink to /usr/bin
and we can always change our wording later if that decision changes, but
we need to wait for the buildd /usr-merge anyway, so either way I don't
think we're ready to merge patches for this bug right now.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Marco d'Itri]

On Sep 17, Bill Allombert  wrote:

> Does not that would break users expectation that the system image contains 
> /var
> before the first boot ?
I am not aware of such expectations.

> A lot of things in /var are caches that are mostly instance-independent and 
> can
> be prefilled, but for that, users expect a minimal directory hierarchy to be
> present before first boot.
Can you show some examples of how this would work in practice?

> It seems your scheme favors some usecase over some others.
There are always tradeoffs, but my use case does not forbid the other 
one: worst case it requires one more mkdir while copying that data.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Luca Boccassi]

On Sun, 17 Sept 2023 at 00:12, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > Aside from more practical considerations, shipping /var content in
> > packages is problematic because it's supposed to be local variable data,
> > that can be removed without breaking a system.
>
> Unless I'm missing something, including the directory in the deb won't
> make any difference here.  dpkg won't break if a directory that was
> included in the package is deleted.  It would show as an inconsistency if
> someone checked the file system against the dpkg database, but as soon as
> systemd-tmpfiles runs, it will create the directory again and fix the
> inconsistency, so I don't see what problems that would create.

That was a remark about the concept itself, independently of tmpfiles
- variable data is variable data, and static data is static data.
Having variable data among static package content sounds conceptually
wrong to me, simply from a high-level design perspective.

> > More practically, one of the purposes of the hermetic-usr pattern is to
> > allow several modernizations. The easiest one to achieve is to create
> > /var/ on firstboot, and encrypt it against the tpm, so that it can be
> > enabled by default, always, so we can't have packages ship and expect
> > content in /var from their packages.
>
> (I am a little confused by this wording, but I think what you're saying is
> that /usr is encrypted and read-only, and /var is recreated on each boot.
> That at least is my understanding of the pattern that you're trying to
> enable.)

Let me clarify, here I meant something much simpler - the image
installed is a 'normal' one, with r/w root and managed by apt as usual
(ie: not immutable image-based) but with a repart.d snippet that
causes a new /var to be created on-the-fly on first boot if missing,
with tpm-bound encryption (and similar treatment for /home, although
unrelated here). This is a very low-hanging fruit of a pattern that
would allow to achieve decent local data protection on an otherwise
pretty much vanilla setup. But if you need to ship /var from packages,
then it goes out of the window.

In a sense, for the immutable /usr case it gets easier - you just
don't ship a package manager at all, and then you can do all the
mangling you want when building the image, as consistency no longer
matters, and you don't install/upgrade by definition. But I strongly
believe we need to do some serious steps forward in the
security-by-default aspects of all flavours, including the 'vanilla'
ones, where this consistency matters a lot, for obvious reasons - you
do upgrades/installs on those.

And yes, I would hope to have a concrete proposal for encrypting by
default to submit to the project for at least something like this for
Trixie, once more pieces have fallen into place. All mainstream
distributions are looking into some variation of this. It is way past
time Linux caught up with OSX and Windows on these aspects, and it
would be great if for once Debian wasn't left behind as usual.

> > On top of that, as you mentioned already things will inevitably get out
> > of sync, and one will have to duplicate everything.
>
> One would need to duplicate empty directories in /var (that don't have
> dynamic ownership).  I'm dubious that's a significant burden (it's two or
> three lines in debian/rules), but if it is, one could even automate this
> in debhelper by parsing tmpfiles.d if one really wanted to.  The main
> thing that could get out of sync is the ownership, which is indeed not
> ideal, but I'm also not sure it's going to cause major problems even if
> people do get it wrong.  I was trying to remember if dpkg changes
> directory (as opposed to file) ownership if it sees a directory owned by
> an unexpected user.  I kind of think it doesn't, but I'm not sure about
> that.
>
> The benefit we gain from this is attribution of the directories in the
> dpkg database, which is useful (although I understand that one can argue
> about how useful).

I don't think it is particularly useful, and mixing package content
with variable data smells like trouble to me. I'd much rather finish
the factory reset workstream for lifetime management, so that tmpfiles
can handle tmpfiles purges, without needing to involve dpkg. We are
working on that. This means that tmpfiles.d would be able to both
create the files/dirs when needed, and remove them when unneeded, ie
on purge - as far as I can tell, that would be the only useful thing
that a dpkg integration would provide. I am pretty sure running
checksums on local variable data would be a pretty useless exercise
given, well, it's variable. Or was there anything else aside from
these two aspects?

To do the tmpfiles purge/reset I have two WIP PRs, one against
sd-tmpfiles, and one against debhelper. I need to pick them up again
and finish that, and I am aiming to do so within the next couple of
months.

> So... I think the answer to my question of whether this will interfere
> with you

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Luca Boccassi  writes:

> Let me clarify, here I meant something much simpler - the image
> installed is a 'normal' one, with r/w root and managed by apt as usual
> (ie: not immutable image-based) but with a repart.d snippet that causes
> a new /var to be created on-the-fly on first boot if missing, with
> tpm-bound encryption (and similar treatment for /home, although
> unrelated here). This is a very low-hanging fruit of a pattern that
> would allow to achieve decent local data protection on an otherwise
> pretty much vanilla setup. But if you need to ship /var from packages,
> then it goes out of the window.

I don't see how shipping empty directories in a deb package affects this
in any way.  You're going to have to be considerably more specific about
what invariant is being violated or what error you're expecting.

One of the key things that I'm stuck on, and which you haven't addressed
that I've seen, is that you're talking here about a system managed with a
normal package manager, and therefore running all maintainer scripts.  The
maintainer scripts under this tmpfiles.d proposal will create directories
and files in /var, because the whole point is that systemd-tmpfiles is run
from postinst.  But you are saying that creating directories and files in
/var with the package manager will break this configuration.  I'm missing
something here.  There's nothing special about dpkg creating the directory
versus postinst creating the directory.  So far as I can tell, the only
important part is that the directory be registered in tmpfiles.d (or a
service unit) so that it can be recreated when needed.

> I don't think it is particularly useful, and mixing package content
> with variable data smells like trouble to me.

I understand that you don't like the idea.  You've made that very clear.
However, we are *currently* doing this, so obviously nothing that we
currently support is going to break from continuing to do this.

I understand that you are trying to do something new, and you are worried
that this will interfere with it, but so far you have not been able to
explain any way in which this *would* cause you problems.  You just don't
like the vibes.  And I hear that, and sometimes that sort of bad feeling
is a valuable architectural clue, but in this case you're really going to
have to be more specific because I'm not seeing it.

Also, we clearly cannot ship a system without /var *now* because oodles of
other packages put things there, so a lot of work is left to do before
this is a technical vision that Debian can implement, if we do indeed
decide to implement it.  Policy can always change later; perhaps by the
time that we're ready to implement this vision, dpkg will have a way of
registering files managed by tmpfiles.d and there's no reason to ship the
empty directories in the package.  So theoretical future problems are not
very persuasive to me at this stage, particularly if you can't be specific
about what those problems would be.

> I'd much rather finish the factory reset workstream for lifetime
> management, so that tmpfiles can handle tmpfiles purges, without needing
> to involve dpkg.

I understand that you would like to finish this, but Debian so far has not
even decided to *start* this, so while I'm willing to take this into
account when writing Policy, it's not a controlling design principle.  If
you get an agreement in Debian that this is something we're going to work
towards, then it will become a significant factor in evaluating Policy
changes.

I'm really trying to meet you halfway here by adopting and fleshing out
proposals that you're putting forward primarily for a project that Debian
isn't currently doing, but which have clear other benefits regardless of
whether we do the factory reset project.  I don't want to argue over end
goals when we can agree on intermediate steps regardless of the end goal.
But I really want to emphasize here that we are not *currently* writing
Policy to support factory reset because there is no decision in Debian yet
that we are supporting factory reset.

This is not directly relevant to this bug, but for the record, if you want
Debian to support factory reset, one good way to make that more likely is
to write a DEP with the details of precisely what that would mean, roughly
what sorts of things would need to change in packaging, and a list of what
the benefits would be.  I personally think a lot of the benefits are
rather compelling, but no one has yet made a proper case for it in Debian.
You and Marco and a few other people just write email messages on other
topics that treat the desirability of factory reset as a foregone
conclusion, or mention a few benefits in passing and without specifics,
which of course is fine for casual discussion but which is not a real
attempt at persuasion and doesn't get us closer to making a real decision.

In other words, this is advice that I constantly give myself because I am
very bad at this and have to be reminded repeated

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Alexandre Detiste]

So there are 3 distinct interlinked goals:

- tmpfiles.d itself
  - recovering from missing /var (+ later /etc)
  - volatile file tracking

Just finishing the first step without going to far in either
other tracks would be so great already.


Le dim. 17 sept. 2023 à 19:57, Russ Allbery  a écrit :
> Luca Boccassi  writes:
> > We are working on that. This means that tmpfiles.d would be able to both
> > create the files/dirs when needed, and remove them when unneeded, ie on
> > purge - as far as I can tell, that would be the only useful thing that a
> > dpkg integration would provide.
>
> dpkg -S is the most useful feature this supports for me personally (and
> some related things, such as cruft-finding).

I agree.

And when "dpkg -S file" fail,
just try "cruft /path/file".

It is slow, the implementation *is* bad - more like a unit test -
but should know about most files, and it still faster
than starting googling for the filename and getting distracted:

$ LANG=C dpkg -S  /var/log/dpkg.log
dpkg-query: no path found matching pattern /var/log/dpkg.log
$ time cruft /var/log/dpkg.log
dpkg
real0m7,129s

The ugly magic behind the curtain:

ls /usr/libexec/cruft/ -1
  LOGROTATE -> that parses these for path
  SERVICES -> added today reading this discussion, it reads
 CacheDirectory= & StateDirectory= from *.service
  TMPFILES  -> that parses these for path


This whole thing, while being already usefull & used,
is more like a mockup of what could be a "perfect" dpkg.

These tiny shell scripts could be converted into something else
that plugs into dpkg ... for example tiny .so plugins that answer
which package own which dynamic file ?

(for runtime evaluation, other possibility is debhelper magic at compile-time
that generate a list of possible files)

> More generally, dropping directories that are currently registered with
> dpkg from dpkg's database is a regression.

I agree.


I've mounted /var/cache/apt/archives everywhere on a tmpfs since forever,
apt will now how/when to recreate /partial subdir,
but yet it's nice to have it in dpkg register.

> Specifics!  Specifics!  My kingdom for specifics!  :)

There's indeed not so much remaining in the way for /var,
some files could be replaced by tmpfiles.d generated symlinks

$ cat /var/lib/dpkg/info/*.list | grep ^/var | sort -u | while read f;
do test -f $f && echo $f; done | wc -l
256

$ dpkg -S $(cat /var/lib/dpkg/info/*.list | grep ^/var | sort -u |
while read f; do test -f $f && echo $f; done) | cut -d: -f1 | sort -u
aspell-en
dictionaries-common
ifrench: /var/lib/dictionaries-common/ispell/ifrench -> generated metadata
plocate:   CACHEDIR.TAG
poppler-data:only transitional symlinks to /usr/share/poppler ?
ttf-mscorefonts-installer: only one file that could be symlinked from /usr
xserver-common:   /var/lib/xkb/README.compiled, could be a
symlink from /usr/share/doc

Maybe /etc in a later track.

Greetings,

(I find this thread greatly inspirational)

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Bill Allombert]

On Sun, Sep 17, 2023 at 08:28:56AM -0700, Russ Allbery wrote:
> Bill Allombert  writes:
> > On Sun, Sep 17, 2023 at 10:41:55AM +0200, Marco d'Itri wrote:
> >> On Sep 17, Russ Allbery  wrote:
> 
> >>> (I am a little confused by this wording, but I think what you're
> >>> saying is that /usr is encrypted and read-only, and /var is recreated
> >>> on each boot.  That at least is my understanding of the pattern that
> >>> you're trying to enable.)
> 
> >> The general idea is to be able to create /var on the first boot.
> 
> > Does not that would break users expectation that the system image
> > contains /var before the first boot ?
> 
> > A lot of things in /var are caches that are mostly instance-independent
> > and can be prefilled, but for that, users expect a minimal directory
> > hierarchy to be present before first boot.
> 
> Not that I think we're particularly close to achieving this design
> currently (and to be clear we haven't decided we're working towards this
> yet), but while I understand why a user would have that expectation today,
> I'm not sure why it would practically matter.  If all of that directory
> structure appears on first boot, and no static data is stored in /var,
> what use case requires the directory structure already exist in /var
> before the first boot?

As I said, filling the caches in /var/cache. For that they need to
exist with correct ownership and permissions.

Most of the cache in /var/cache/ (some are in /var/lib actually) 
do not depend on the host configuration, and can be regenerated/redownloaded as
needed, but not for free.

For example you might want to populate
/var/cache/apt/archives/ with the debs you need install later on (for example
for a pbuilder-like system), populate /var/lib/texmf/fonts/ with your fonts, or
even populate /var/www with your website, etc.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Bill Allombert  writes:

> As I said, filling the caches in /var/cache. For that they need to exist
> with correct ownership and permissions.

Sorry, I think I saw that and then edited my message more and lost it
again.

That use case makes sense to me, and without the directory already
present, you have to know what directory to create and you have to get the
ownership and permissions correct.  But there's a couple of reasons why I
don't think that's a problem:

1. Installing the package creates the directories since it invokes
   systemd-tmpfiles via postinst, so the directory will normally be there
   with correct ownership and permissions.  The only case where it
   wouldn't be is in cases where the packages were installed without
   running postinst, which feels like an unusual use case.

2. Presumably you would be copying these caches from another system, which
   will normally have the directory with correct ownership and
   permissions.  This isn't necessarily true if you're mixing versions of
   Debian, of course, but in that case it's not clear the cache format
   will be correct either.  Also, you need to get the ownership and
   permissions of the files right, which the directory structure doesn't
   necessarily help you with, and if you're copying that over already, the
   same mechanism can handle the ownership and permissions of the parent
   directory.

So, by definition any directory that's shipped in the deb cannot have
dynamic ownership, which also limits the range of permissions it could
have.

> even populate /var/www with your website, etc.

/var/www is a whole separate problem that I agree has not yet been
addressed and would need to be.  We've known that /var/www is weird for a
while (we have a special exception in the FHS for it because it's breaking
the FHS file system layout rules), and there have been a few attempts to
handle it some other way, but none of them so far have been successful.

-- 
Russ Allbery (r...@debian.org)  

Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var

[Russ Allbery]

Alexandre Detiste  writes:

> The ugly magic behind the curtain:

> ls /usr/libexec/cruft/ -1
>   LOGROTATE -> that parses these for path
>   SERVICES -> added today reading this discussion, it reads
>  CacheDirectory= & StateDirectory= from *.service
>   TMPFILES  -> that parses these for path

> This whole thing, while being already usefull & used,
> is more like a mockup of what could be a "perfect" dpkg.

> These tiny shell scripts could be converted into something else
> that plugs into dpkg ... for example tiny .so plugins that answer
> which package own which dynamic file ?

> (for runtime evaluation, other possibility is debhelper magic at
> compile-time that generate a list of possible files)

Based on previous discussion with Guillem, I think the direction Guillem
is headed is something like adding a new member (or field in another
member) to the deb format that holds a list of volatile files that the
package considers itself responsible for.  I think I agree with Guillem's
position (at least as I understand it) that it doesn't make sense for dpkg
to parse other files to populate that list.  That can very easily be
handled outside of dpkg.

So the idea would be that the package would install tmpfiles.d files,
service units, and similar files as normal, and then debhelper would parse
those files, extract the list of paths that they manage, and use that to
write a control file or the like that dpkg consumes to register those
files.

If I'm correct about that design, an intermediate step in that direction
from where cruft is today would be to add that logic to debhelper and then
have debhelper ship that database in the package in
/usr/share/cruft/ or some similar directory, and then cruft could
just consume that database of registered paths to get attribution
information until such time as that can move into dpkg.

This design is just off the top of my head, and I'm probably missing some
problems and some details.

-- 
Russ Allbery (r...@debian.org)  

Bug#1051582: Policy 9.3 (Starting system services) is largely obsolete

[Russ Allbery]

Niels Thykier  writes:

> I had a look at the introduction section of Policy to check who the
> target audience is.  I cannot find an explicit mention of the target
> audience. I suspect there is a conflict here on the content because we
> have different audiences in mind for the Policy and the Policy does not
> seem to be explicit here.

> If the target audience is package maintainers, then 100% of all debian
> contributors should read it. Then we need "simple and easy to
> understand" language and examples, because we want *everybody* to
> understand this.  I agree with Russ that long winded sections of "here
> is how you do this manually but really just use debhelper" is directly
> counter productive to this audience.

> If it is cross-package integration, are we targeting the ones
> integrating or the ones providing the integration points?  If it is
> both, then for more complex topics, it would make sense to split the
> topic into two. One for consumer and one for the provider of the
> integration, because for the consumer it will probably boil down to
> "install path at $X in your deb and run tool $Y" and then the consumer
> can skip the provider section as "not relevant".

> If the target audience is tool-chain maintainers, then only 5-10 people
> need to read the policy and the entire document + related process seems
> completely over-engineered.  But then, for the same reason, I suspect
> this is an unlikely target audience for the Policy.

Ooo, this is a great framing of the problem.  I have a lot of thoughts
about this.  Unfortunately, I'm not sure if they're actionable thoughts
since my grand vision requires someone to sit down and do some serious
Policy restructuring and produce a radically different document.  But
maybe if I write them all down and enough people feel similarly, it would
be worth doing.  I would love to work on this, I am just afraid that it is
the sort of project that I would start and never finish and thus never
accomplish anything of use.

I think that ideally Policy targets all three of your audiences, but not
at the same time, and not with the same priority.

I have a lot of problems with the current structure of Policy, to the
extent that it sometimes interferes with my motivation for working on it,
and one of the big ones is that it doesn't follow any structured design
pattern for documentation, such as Divio [1].  It may have at one point,
but iteratively over the years, and due to some decisions like trying to
retain section numbering, we've diverged from that.  Even if it were
trying to be only one type of documentation, it doesn't consistently stay
in that lane.

[1] https://documentation.divio.com/

I think my ideal structure of Policy would have three major parts.

First, there would be Policy for Debian packagers.  This would focus on
the things someone packaging for Debian needs to know, and would be
organized roughly around task.  Example sections here would be:

* Choosing an archive area
* Files on the file system (FHS, ownership, permissions, etc.)
* Writing a debian/control file
* Writing a changelog
* Writing a copyright file
* Packaging a shared library
* Packaging a system service
* Using maintainer scripts

In other words, this section would consist primarily of Divio how-to
guides, with some intermixed Divio explanations.  (Tutorials I think are
outside the scope of Policy and are better handled elsewhere, such as in
debhelper documentation.)

(This is very rough, this is not the right order, etc.  This is just to
provide a feel of the nature of this section.)

This section would assume that you're using a packaging helper and not
tell you how to do things that the packaging helper is going to do for
you.  There's an awkward line here between describing what to do and being
debhelper documentation, but the basic idea is that this would tell you
what you are aiming for, and then your packaging helper documentation will
tell you where to put configuration files to achieve that.

Second section would be Policy the reference manual for interfaces in the
distribution.  Sections here would be:

* Complete list of control fields and their meanings.
* Specifications for the .dsc and .changes files (which packagers mostly
  never need to care about, but tool maintainers do).
* The detailed reference documentation on all the ways maintainer scripts
  can be called.
* Specification for the symbols and shlibs files.

This is all Divio reference stuff.  Whenever we have a comprehensive
explanation of the details of something, it goes here, and then we
liberally link to the reference section from the packaging-focused how-to
section for more details.

Finally, there is the reference manual for toolchain maintainers.  My
position on this is that it's best-effort documentation and should
probably be a non-normative appendix where toolchain maintainers are
relatively free to just make changes without going through a very formal
process as long as those changes reflect