Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Egmont Koblinger]

Hi guys,

On Wed, Feb 26, 2020 at 10:54 PM Bollinger, John C
 wrote:

> > Is it OK to clean $XDG_CACHE_HOME after a fixed time period by default?
>
> No, it is not, as a matter of policy and of the nature of $XDG_CACHE_HOME.  
> It is not safe or reasonable to assume that all of the contents are similar 
> to thumbnail or browser caches, that go stale over time and that can easily 
> and cheaply be repopulated.  What's more, those particular items are already 
> accounted for.  The default should be to protect users' data, including cache 
> data, about which the system has no specific knowledge or instructions.

I'm absolutely on John's side here.

Let's take ccache for example. It's a weird example because they don't
use XDG_CACHE_HOME, but they could :). I might compile a project, play
with it, get some result out of it, and think that I won't ever need
to work on it again, thus remove the git checkout tree. Then months
later something pops up and I need to check the project again. Having
the compilation cache still available might very easily save me in the
order of minutes or tens of minutes.

_My time_ is the most precious bottleneck, not the disk space or CPU
time or whatever technical. Automatic purging (based on no more
information than the timestamp of files) has much more chance to harm
than to help me.

> Overall, the assertion that the cache directory should be subject to cleaning 
> by default seems to be a solution without a problem.

I don't think it's even a solution to that non-problem. Because if I'm
running short on disk space, I need to take immediate action purging
some files, I cannot wait for the automatic cleaning to kick in who
knows when, in days or weeks, to hopefully free up enough space (which
it most likely won't, by the way).


cheers,
egmont
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

RE: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bollinger, John C]

On Wednesday, February 26, 2020 1:41 PM, Simon McVittie wrote:

> The only thing that the XDG_CACHE_HOME specification needs to agree on is the 
> contents and meaning of whatever file format gets used, not a concrete 
> implementation of the interpreter for that file format (the same way 
> freedesktop.org defines what .desktop files contain and mean, but it's up to 
> projects like GNOME to actually implement them in a compatible way).


Note that XDG_CACHE_HOME is covered by the XDG Base Directory spec.  It doesn't 
have its own, as far as I am aware.  I am uncertain whether the cleanup 
functionality, if accepted for specification, would belong in that spec or 
somewhere else.  In principle, it wouldn't need to be specific to 
$XDG_CACHE_HOME.

To the main point, I agree that any specifications that are adopted only need 
to (or should) address the format, but it is not clear whether everyone is on 
the same page about that.  This was the point of my question.  It matters 
because it may make a difference to some that there is not, in fact, an 
existing application that would do the wanted job on all systems of interest.


> You're assuming that systemd-tmpfiles is running in system mode


Yes, I was indeed making that assumption.


> However, `systemd-tmpfiles --user` can be invoked as an ordinary user, 
> unprivileged, to look at various paths including 
> ~/.config/user-tmpfiles.d/*.conf (which is the highest-precedence) and 
> create/clean per-user directories. I think the suggestion is that desktop 
> environments like GNOME should periodically run `systemd-tmpfiles --user`, or 
> a compatible reimplementation, as a way to clean per-user cached files.


Something along those lines would indeed be possible.  I did not intend to 
suggest otherwise, but I did want to establish that all parties agree that 
something along those lines -- a user process running periodically in the 
context of a login session -- was what everyone was considering.  This has 
implications that might be important to some people, such as on the timing and 
scope of each cleanup.


>> Why is that a strong argument? Perhaps this is part of the issue.
>> I don't see any particular imperative for cleaning up cache files.
>...
>> users should be responsible for their own usage.
>> If they are using too much, then that's what storage quotas are for.
>
> Some cache files (such as thumbnails and web browser caches) are created by 
> "the system" without the user being particularly aware of it happening. If 
> these have built up to the extent that  they are a substantial proportion of 
> the available storage (storage quota on a multi-user system or disk size on a 
> single-user system), then it isn't at all obvious to a non-knowledgeable user 
> why they have run out of space or what they should do about it.


I'd be inclined to say that cache files created by the desktop and by user 
applications, without explicit user request, are probably the vast majority of 
the contents of most users' cache directories.  I do not object to the 
proposition that there should be a way for users to clean these up, either on 
demand or on a schedule, without being cognizant of the details.  I am not 
overly concerned with browser or thumbprint caches specifically, as there are 
already mechanisms in place to address those, but sure, it would be swell if 
such management could be generalized and unified, so that each application 
didn't have to roll its own.


> That's why the "housekeeping" module of gnome-settings-daemon (which I think 
> is named for its original purpose, hosting Xsettings, but in practice is more 
> like a collection of miscellaneous background services used in GNOME) has 
> traditionally been responsible for cleaning up files like 
> non-recently-accessed thumbnails.


As I consider the matter further, I find that I also do not object more 
generally to cleanup policy being established on behalf of users on a 
per-application basis, especially if the application provides a good means for 
the user to adjust that policy.  This is the space where common browsers and 
GSD reside now.  I account cleanup mediated or directed by the application as 
part of the contract between user and application.  I acknowledge that this is 
a more nuanced position than probably was conveyed by my previous comments.  
Bringing it back around to the original questions, then:


[Benjamin:]
> Is it reasonable to standardise on the systemd tmpfiles.d format?

As a configuration format for describing cleanup properties, yes, it is 
reasonable.

However, it would be wise to evaluate the details of that format to determine 
whether it would benefit from a bit of customization for the more specialized 
role proposed for it.

> Is it OK to clean $XDG_CACHE_HOME after a fixed time period by default?

No, it is not, as a matter of policy and of the nature of $XDG_CACHE_HOME.  It 
is not safe or reasonable to assume that all of the contents are similar to 
t

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Simon McVittie]

On Wed, 26 Feb 2020 at 19:19:29 +, Bollinger, John C wrote:
> Let's do get some clarity on this point, as various comments have seemed
> to say different things.  Are you suggesting that the *configuration
> format* consumed by systemd-tempfiles be reused by some desktop
> component, or are you suggesting that cleanup tasks be delegated to
> systemd-tempfiles itself?

(Its name is systemd-tmpfiles.)

The only thing that the XDG_CACHE_HOME specification needs to agree on
is the contents and meaning of whatever file format gets used, not a
concrete implementation of the interpreter for that file format (the
same way freedesktop.org defines what .desktop files contain and mean,
but it's up to projects like GNOME to actually implement them in a
compatible way).

> Even on those systems that have systemd-tempfiles and have it enabled,
> it cannot be relied upon to clean up anything in user directories,
> because such directories may not be accessible to it.  On the systems I
> manage, for example, regular users' home directories reside on a network
> filesystem, and local-machine administrative accounts do not have any
> privileges there.

You're assuming that systemd-tmpfiles is running in system mode, with
privileges, looking at /etc/tmpfiles.d/*.conf, /run/tmpfiles.d/*.conf and
/usr/lib/tmpfiles.d/*.conf only. I agree that that's a non-starter.

However, `systemd-tmpfiles --user` can be invoked as an
ordinary user, unprivileged, to look at various paths including
~/.config/user-tmpfiles.d/*.conf (which is the highest-precedence) and
create/clean per-user directories. I think the suggestion is that desktop
environments like GNOME should periodically run `systemd-tmpfiles --user`,
or a compatible reimplementation, as a way to clean per-user cached files.

See tmpfiles.d(5) (file format) and systemd-tmpfiles(8) (reference
implementation) for more details.

> Why is that a strong argument? Perhaps this is part of the issue.
> I don't see any particular imperative for cleaning up cache files.
...
> users should be responsible for their own usage.
> If they are using too much, then that's what storage quotas are for.

Some cache files (such as thumbnails and web browser caches) are
created by "the system" without the user being particularly aware of
it happening. If these have built up to the extent that they are a
substantial proportion of the available storage (storage quota on a
multi-user system or disk size on a single-user system), then it isn't
at all obvious to a non-knowledgeable user why they have run out of
space or what they should do about it.

That's why the "housekeeping" module of gnome-settings-daemon (which
I think is named for its original purpose, hosting Xsettings, but in
practice is more like a collection of miscellaneous background services
used in GNOME) has traditionally been responsible for cleaning up files
like non-recently-accessed thumbnails.

smcv
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

RE: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bollinger, John C]

On Wednesday, February 26, 2020 11:19 AM, Benjamin Berg wrote:

> So, lets try to clarify a few points.

> I do think that it is a good idea to use tmpfiles.d and for applications to 
> ship appropriate configurations. There seems to be an overall agreement that 
> tmpfiles.d is an appropriate way of doing this and that we could standardise 
> on it.

Let's do get some clarity on this point, as various comments have seemed to say 
different things.  Are you suggesting that the *configuration format* consumed 
by systemd-tempfiles be reused by some desktop component, or are you suggesting 
that cleanup tasks be delegated to systemd-tempfiles itself?  Because the 
latter is a complete non-starter.  Even on those systems that have 
systemd-tempfiles and have it enabled, it cannot be relied upon to clean up 
anything in user directories, because such directories may not be accessible to 
it.  On the systems I manage, for example, regular users' home directories 
reside on a network filesystem, and local-machine administrative accounts do 
not have any privileges there.  Moreover, regular users do not have authority 
to modify the configuration of systemd-tempfiles, yet they must be afforded 
ultimate control over any cleanup policy applying to the space afforded to them.

> The strong argument in favour is that we have an external service that 
> guarantees cleanup even if the application is not running regularly.
> For example, I have multiple .cache/* directories with relevant amounts of 
> junk data for applications that I have not used in months to years.

Why is that a strong argument? Perhaps this is part of the issue.  I don't see 
any particular imperative for cleaning up cache files.  As far as I'm 
concerned, it is a "nice-to-have" feature, not a "must-have", and nice only if 
it does not interfere with my work.  I agree that it would be particularly nice 
to have a common component that is responsible for such cleanup so that one 
does not need to rely on individual applications to manage their own cache 
usage directly.  That's a feature that could even be sold to application 
developers, but ultimately, and perhaps this is my key point, users should be 
responsible for their own usage.  If they are using too much, then that's what 
storage quotas are for.  If they are not using too much, then what's the big 
deal about how they are using it?

> But, I would like to go one step further and make this an opt-out.
> Which is what appears to be triggering the opposition here. The argument in 
> favour is a bit more complicated, but I think it boils down
to:
>
>  * The behaviour becomes explicit rather than an implicit "keep
>   forever" policy. I do feel that this is really good in principle.

What do you think is good about that, exactly?  What actual problem does it 
solve, that cannot be addressed as well or better via some other mechanism?  
"Alice is using a lot of disk space" is not a problem in itself, nor is "Bob 
has a lot of files that he hasn't accessed in months".

> * In principle, I don't think it is sane to keep caches forever.

Perhaps the problem is with your conception of what caches, in the sense 
$XDG_CACHE_HOME, are or are for.  I get the impression that you have a pretty 
narrow view of that, which may be coloring your opinion here.

But even if all uses of $XDG_CACHE_HOME were analogous to browser caches and 
thumbnail caches (which is not the case), what is insane about leaving the 
cache contents lying around indefinitely, if you don't need the space for 
anything else?

> So I
>   do believe it makes sense to define the expectation that
>   $XDG_CACHE_HOME is cleaned eventually even if the application is not
>   run regularly.

I am doubtful that you'll attract a majority to that position, at least in its 
full generality, and I am confident that you will not develop a consensus 
around it.

> * If a user just stops using an application and removes the package,
>   then we should clean the cache. This works very nicely with an opt-
>   out solution, as the tmpfiles.d config is removed and the default
>   configuration kicks in and cleans things up.

That the package has been removed by a sysadmin is not a clear indication that 
it is reasonable to remove users' associated cache files.  Even if we interpret 
it that way, though, there's an important distinction between "can be removed" 
and "should be removed".  As a matter of general policy, the system should be 
exceedingly careful about mucking with user files.

As an additional practical matter, if users can install custom cache-management 
configuration -- as indeed they should be allowed to do -- then an opt-out 
solution does not ensure that removing the package would cause cleanup to kick 
in for all users.

> And yes, I do agree that it may well be a bit painful at the point when the 
> switch is flipped. I wouldn't expect that to happen for another 1-2 years 
> after the specification changed though; and I expect that

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

Hi,

So, lets try to clarify a few points.


I do think that it is a good idea to use tmpfiles.d and for
applications to ship appropriate configurations. There seems to be an
overall agreement that tmpfiles.d is an appropriate way of doing this
and that we could standardise on it.

The strong argument in favour is that we have an external service that
guarantees cleanup even if the application is not running regularly.
For example, I have multiple .cache/* directories with relevant amounts
of junk data for applications that I have not used in months to years.


But, I would like to go one step further and make this an opt-out.
Which is what appears to be triggering the opposition here. The
argument in favour is a bit more complicated, but I think it boils down
to:

 * The behaviour becomes explicit rather than an implicit "keep
   forever" policy. I do feel that this is really good in principle.
 * In principle, I don't think it is sane to keep caches forever. So I
   do believe it makes sense to define the expectation that
   $XDG_CACHE_HOME is cleaned eventually even if the application is not
   run regularly.
 * If a user just stops using an application and removes the package,
   then we should clean the cache. This works very nicely with an opt-
   out solution, as the tmpfiles.d config is removed and the default
   configuration kicks in and cleans things up.

And yes, I do agree that it may well be a bit painful at the point when
the switch is flipped. I wouldn't expect that to happen for another 1-2 
years after the specification changed though; and I expect that some
distributions would play safe und would wait even longer.

So yeah, I don't feel that sticking to the status-quo of never deleting
application caches is sane. I am happy to reconsider my position. But I
really don't find it a very convincing counter argument that the
transition may be painful in a few cases.

Benjamin

On Wed, 2020-02-26 at 13:45 +0100, Benjamin Berg wrote:
> Hi,
> 
> so I looked at gsd-housekeeping the other day. With systemd-
> tempfiles,
> it only has two purposes these days:
> 
>  1. Cleaning $XDG_CACHE_HOME/thumbnails after 30 days
>  2. Cleaning the trash directories after a configurable time
> 
> Currently it also tries to clean /tmp and /var/tmp, but doing so is
> really dangerous compared to just leaving it up to systemd-tempfiles
> (I
> have filed an MR to disable the logic if we are systemd booted).
> 
> 
> Now, systemd-tempfiles can already clean up everything except for the
> trash. And considering that $XDG_CACHE_HOME is non-essential by
> definition, I think it might be sane to use systemd-tempfiles not
> only
> to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
> future. So I am thinking that we could do the following:
> 
>1. Specify that we use the systemd tmpfiles.d configuration format
> for
>   cleaning $XDG_CACHE_HOME. Also specify that $XDG_CACHE_HOME
> will be
>   cleaned automatically after e.g. 30 days unless otherwise
> configured
>   by an application.
>2. Add some reference to this to the thumbnail specification.
>3. Tell application maintainers that they need to ship a
> configuration
>   if they want to keep files longer (likely candidates are e.g.
> email
>   clients).
>4. As a start, add a "xdg-thumbnails.conf" systemd-tempfiles
>   configuration to systemd that cleans $XDG_CACHE_HOME/thumbnails
>   after 30 days.
>5. After a grace period, add "xdg-cache.conf" to clean
> $XDG_CACHE_HOME
>   and remove "xdg-thumbnails.conf" again (similar to how
>   /usr/lib/tmpfiles.d/tmp.conf does it for /var/tmp and /tmp)
> 
> Is it reasonable to standardise on the systemd tmpfiles.d format?
> Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> default?
> Other thoughts?
> 
> Benjamin
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ___
> xdg mailing list
> xdg@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/xdg


signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

RE: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bollinger, John C]

On Wednesday, February 26, 2020 8:51 AM, Benjamin Berg wrote:

> [...] I *do* want applications to explicitly ship an opt-out file if they do 
> not want any cleaning ever. And threatening with automatic clean-up is 
> probably a good idea, because otherwise no one will bother.

This is simply wrongheaded.  Applications should not be required to opt-out, 
period, and it is reasonable for them to rely on not having to opt out.  It is 
moreover absolutely unacceptable to threaten people in order to obtain 
compliance with an arbitrary policy, as you propose to do.  It would be fine to 
provide a facility for explicit opt-out, and some applications might make use 
of it for extra assurance.  It is probably right that most will not bother, but 
that should not be regarded as a problem.

> So, I would still propose to write the requirement to ship a configuration 
> file and the possibility of defaulting to clean after e.g. 30 days into the 
> specification.

On the contrary.  If this generalized facility is implemented then it should 
provide a clear promise that only cache contents explicitly configured for 
automatic cleanup will be affected, with the possible pseudo-exception that 
areas that are already subject to such cleanup (e.g. thumbnails) will continue 
to be cleaned up by default.  I would expect the "by default" part of that to 
take the form of opt-in configuration being shipped and installed by default, 
presumably in $XDG_CONFIG_DIR, as opposed to the behavior being built directly 
into the software.

> That does not say anything about how long the grace period for such a change 
> would be. And while I personally think we should flip the default eventually, 
> we could still decide to never actually do so. Or leave the decision up to 
> distributions and administrators.

If it makes you feel any better, you can make the promise that opt-in is 
required a weak one, so that you can imagine that it would be on the table to 
flip the policy at some point in the future.  In reality, however, I don't see 
any reason to think that it would ever be reasonable to require opt-out under 
any circumstances.


John Bollinger




Email Disclaimer: www.stjude.org/emaildisclaimer
Consultation Disclaimer: www.stjude.org/consultationdisclaimer
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bastien Nocera]

On Wed, 2020-02-26 at 15:51 +0100, Benjamin Berg wrote:
> On Wed, 2020-02-26 at 14:18 +, Simon McVittie wrote:
> > On Wed, 26 Feb 2020 at 14:39:06 +0100, Benjamin Berg wrote:
> > > I don't think that $XDG_CACHE_HOME is designed to be used
> > > directly by
> > > users. And if it is an application which generates those
> > > repositories,
> > > then again, it can just drop-in the appropriate configuration to
> > > prevent cleaning.
> > 
> > It isn't designed to be used directly by users, but it *is*
> > designed
> > to be used (or at least usable) by programs/scripts that those
> > users write
> > locally, which is quite a fine line.
> > 
> > I think going straight from "$XDG_CACHE_HOME is not systematically
> > cleaned"
> > to "$XDG_CACHE_HOME is cleaned unless you opt-out" breaks the
> > principle of
> > least astonishment for existing applications (and libraries) that
> > cache
> > things that are "expensive" to recreate.
> > 
> > A safer route to this would seem to be defaulting to *not* dropping
> > caches, and having applications whose caches are suitable for time-
> > based
> > cleaning opt-in to it by installing tmpfiles.d fragments.
> 
> Oh, I fully agree. Though I *do* want applications to explicitly ship
> an opt-out file if they do not want any cleaning ever. And
> threatening
> with automatic clean-up is probably a good idea, because otherwise no
> one will bother.

Why would they need to bother? There's nothing that requires this to be
regularly cleaned rather than cleaned when needed. You have absolutely
no way to know whether the cached data is resource-intensive to
recreate or not.

Leave it up to an app like Baobab to guide users to clean this _when
needed_, maybe have applications offer up more metadata about what they
cache, but the automatic cleaning is really not that interesting.

> So, I would still propose to write the requirement to ship a
> configuration file and the possibility of defaulting to clean after
> e.g. 30 days into the specification. That does not say anything about
> how long the grace period for such a change would be. And while I
> personally think we should flip the default eventually, we could
> still
> decide to never actually do so. Or leave the decision up to
> distributions and administrators.
> 
> > If the vast majority of $XDG_CACHE_HOME users end up opting-in to
> > cleanup,
> > then the default perhaps can be reconsidered later.
> > 
> > > > > Is it reasonable to standardise on the systemd tmpfiles.d
> > > > > format?
> > 
> > It seems as good a format as any: there's a specification that
> > other
> > projects can implement if they are running on non-Linux or don't
> > like
> > systemd for whatever reason, and reusing the design work that the
> > systemd
> > developers have put into that seems better than inventing a
> > parallel
> > "desktop tmpfiles" specification.
> > 
> > There is a shell-script reimplementation of most of tmpfiles.d
> >  but it doesn't currently
> > support
> > age-based cleaning or the --user mode.
> > 
> > I believe tmpfiles.d is intended to be idempotent, so if the
> > systemd
> > implementation and a parallel implementation end up both running
> > for
> > whatever reason, the practical effect is likely to be the same as
> > if
> > only one runs?
> 
> Nice. I had not seen opentmpfiles yet. The format does indeed seem
> simple enough so that at least the relevant subset should be easily
> implementable. And I suspect that the systemd implementation could
> also
> be split out and used elsewhere.
> 
> And yes, two cleaners should just result in a bit of excessive IO
> with
> no further negative impact.
> 
> Benjamin
> ___
> xdg mailing list
> xdg@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/xdg

___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

On Wed, 2020-02-26 at 14:18 +, Simon McVittie wrote:
> On Wed, 26 Feb 2020 at 14:39:06 +0100, Benjamin Berg wrote:
> > I don't think that $XDG_CACHE_HOME is designed to be used directly by
> > users. And if it is an application which generates those repositories,
> > then again, it can just drop-in the appropriate configuration to
> > prevent cleaning.
> 
> It isn't designed to be used directly by users, but it *is* designed
> to be used (or at least usable) by programs/scripts that those users write
> locally, which is quite a fine line.
> 
> I think going straight from "$XDG_CACHE_HOME is not systematically cleaned"
> to "$XDG_CACHE_HOME is cleaned unless you opt-out" breaks the principle of
> least astonishment for existing applications (and libraries) that cache
> things that are "expensive" to recreate.
> 
> A safer route to this would seem to be defaulting to *not* dropping
> caches, and having applications whose caches are suitable for time-based
> cleaning opt-in to it by installing tmpfiles.d fragments.

Oh, I fully agree. Though I *do* want applications to explicitly ship
an opt-out file if they do not want any cleaning ever. And threatening
with automatic clean-up is probably a good idea, because otherwise no
one will bother.

So, I would still propose to write the requirement to ship a
configuration file and the possibility of defaulting to clean after
e.g. 30 days into the specification. That does not say anything about
how long the grace period for such a change would be. And while I
personally think we should flip the default eventually, we could still
decide to never actually do so. Or leave the decision up to
distributions and administrators.

> If the vast majority of $XDG_CACHE_HOME users end up opting-in to cleanup,
> then the default perhaps can be reconsidered later.
> 
> > > > Is it reasonable to standardise on the systemd tmpfiles.d format?
> 
> It seems as good a format as any: there's a specification that other
> projects can implement if they are running on non-Linux or don't like
> systemd for whatever reason, and reusing the design work that the systemd
> developers have put into that seems better than inventing a parallel
> "desktop tmpfiles" specification.
> 
> There is a shell-script reimplementation of most of tmpfiles.d
>  but it doesn't currently support
> age-based cleaning or the --user mode.
> 
> I believe tmpfiles.d is intended to be idempotent, so if the systemd
> implementation and a parallel implementation end up both running for
> whatever reason, the practical effect is likely to be the same as if
> only one runs?

Nice. I had not seen opentmpfiles yet. The format does indeed seem
simple enough so that at least the relevant subset should be easily
implementable. And I suspect that the systemd implementation could also
be split out and used elsewhere.

And yes, two cleaners should just result in a bit of excessive IO with
no further negative impact.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bastien Nocera]

On Wed, 2020-02-26 at 15:24 +0100, Benjamin Berg wrote:
> Hi,
> 
> On Wed, 2020-02-26 at 15:14 +0100, Bastien Nocera wrote:
> > Flatpak'ed applications also use a different cache directory, like:
> > $ export | grep CACHE
> > declare -x XDG_CACHE_HOME="/home/hadess/.var/app//cache"
> > 
> > I'd say that it might better left well alone, and have something
> > like
> > Baobab better signal what each directory is/which application it
> > belongs to, to clean it up.
> 
> This specification change would only affect the $XDG_CACHE_HOME as
> set
> at login time by pam_systemd (i.e. ~/.cache).
> 
> So in principle, Flatpak'ed applications are not affected at all.
> However, the logical extension of this proposal would be that
> appropriate configurations are shipped inside each Flatpak. Then we
> also have a cleanup mechanism there that does not depend on the
> application to be run regularly.
> 
> I had not thought about that yet. I could imagine Flatpak collecting
> the information and doing the simple re-write and dropping it into
> ~/.config/tmpfiles.d or $XDG_RUNTIME_DIR/tmpfiles.d at login time.
> And
> Flatpak itself could again ship a default to clean .var/app/*/cache
> by 
> default.

In this case, an opt-in would be less surprising than an opt-out.

___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

Hi,

On Wed, 2020-02-26 at 15:14 +0100, Bastien Nocera wrote:
> Flatpak'ed applications also use a different cache directory, like:
> $ export | grep CACHE
> declare -x XDG_CACHE_HOME="/home/hadess/.var/app//cache"
> 
> I'd say that it might better left well alone, and have something like
> Baobab better signal what each directory is/which application it
> belongs to, to clean it up.

This specification change would only affect the $XDG_CACHE_HOME as set
at login time by pam_systemd (i.e. ~/.cache).

So in principle, Flatpak'ed applications are not affected at all.
However, the logical extension of this proposal would be that
appropriate configurations are shipped inside each Flatpak. Then we
also have a cleanup mechanism there that does not depend on the
application to be run regularly.

I had not thought about that yet. I could imagine Flatpak collecting
the information and doing the simple re-write and dropping it into
~/.config/tmpfiles.d or $XDG_RUNTIME_DIR/tmpfiles.d at login time. And
Flatpak itself could again ship a default to clean .var/app/*/cache by 
default.

Benjamin

> > > 
> > > > Is it reasonable to standardise on the systemd tmpfiles.d
> > > > format?
> > > > Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> > > > default?
> > > 
> > > I'm guessing that's a no.
> > > 
> > > As for thumbnails, you'd probably get away with checking whether
> > > atime
> > > is actually set on that mount and cleaning up the ones that
> > > haven't
> > > been used.
> > 
> > Benjamin
> 
> 


signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Simon McVittie]

On Wed, 26 Feb 2020 at 14:39:06 +0100, Benjamin Berg wrote:
> I don't think that $XDG_CACHE_HOME is designed to be used directly by
> users. And if it is an application which generates those repositories,
> then again, it can just drop-in the appropriate configuration to
> prevent cleaning.

It isn't designed to be used directly by users, but it *is* designed
to be used (or at least usable) by programs/scripts that those users write
locally, which is quite a fine line.

I think going straight from "$XDG_CACHE_HOME is not systematically cleaned"
to "$XDG_CACHE_HOME is cleaned unless you opt-out" breaks the principle of
least astonishment for existing applications (and libraries) that cache
things that are "expensive" to recreate.

A safer route to this would seem to be defaulting to *not* dropping
caches, and having applications whose caches are suitable for time-based
cleaning opt-in to it by installing tmpfiles.d fragments.

If the vast majority of $XDG_CACHE_HOME users end up opting-in to cleanup,
then the default perhaps can be reconsidered later.

> > > Is it reasonable to standardise on the systemd tmpfiles.d format?

It seems as good a format as any: there's a specification that other
projects can implement if they are running on non-Linux or don't like
systemd for whatever reason, and reusing the design work that the systemd
developers have put into that seems better than inventing a parallel
"desktop tmpfiles" specification.

There is a shell-script reimplementation of most of tmpfiles.d
 but it doesn't currently support
age-based cleaning or the --user mode.

I believe tmpfiles.d is intended to be idempotent, so if the systemd
implementation and a parallel implementation end up both running for
whatever reason, the practical effect is likely to be the same as if
only one runs?

smcv
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Thomas Kluyver]

On Wed, Feb 26, 2020, at 1:30 PM, Benjamin Berg wrote:
> I am *not* proposing to nuke these directories. I am proposing to nuke
> them by default, and ask applications like evolution, jhbuild and
> others to ship their own configuration.

I suspect that the practical upshot of this would be that ~all applications 
using XDG_CACHE_HOME get bug reports at some time in the next couple of years, 
waste some time working out what's going on, grumble about it, and then write 
whatever configuration disables this behaviour to revert to the way things were 
before (and presumably, the way they are on other platforms).

XDG_RUNTIME_DIR is specified to have periodic cleanup unless files have the 
sticky bit set, although this is on a scale of hours rather than days. As an 
application author, this was a source of bugs until we eventually gave up using 
the directory entirely.

> And I do see a huge advantage in requiring applications to specify the 
> clean-up behaviour they need or want.

Why not take that as the starting point? Provide a way for apps to indicate to 
the system when it makes sense to clean up their cache files, but don't punish 
those that don't know about it by randomly deleting their cache.

Thomas
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bastien Nocera]

On Wed, 2020-02-26 at 14:30 +0100, Benjamin Berg wrote:
> On Wed, 2020-02-26 at 14:22 +0100, Bastien Nocera wrote:
> > On Wed, 2020-02-26 at 13:45 +0100, Benjamin Berg wrote:
> > > Now, systemd-tempfiles can already clean up everything except for
> > > the
> > > trash. And considering that $XDG_CACHE_HOME is non-essential by
> > > definition, I think it might be sane to use systemd-tempfiles not
> > > only
> > > to clean the thumbnails but the entirety of $XDG_CACHE_HOME in
> > > the
> > > future.
> > 
> > It's not "non-essential", it's a cache, which can be regenerated,
> > but
> > it might be utterly costly to do so. Eg. there are 10 gigs of
> > "cached"
> > evolution mails in my ~/.cache, 5 gigs of jhbuild builddirs.
> > 
> > Nuking it is a last ditch scenario. You'd avoid backing it up on
> > space
> > constrained storage, but you'd want to avoid having to regenerate
> > that
> > cache in most cases.
> 
> I am *not* proposing to nuke these directories. I am proposing to
> nuke
> them by default, and ask applications like evolution, jhbuild and
> others to ship their own configuration.
> 
> This matches the behaviour of /tmp and /var/tmp on systemd managed
> systems. In the simplest case, all evolution needs to do is ship a
> one
> line file with:
> 
> x %C/evolution
> 
> This file can even be installed to the users $XDG_CONFIG_DIR for
> applications that might not be able to do it globally.

That sounds like a recipe for disaster, if you have an older version of
evolution and a newer version of the host OS.

Flatpak'ed applications also use a different cache directory, like:
$ export | grep CACHE
declare -x XDG_CACHE_HOME="/home/hadess/.var/app//cache"

I'd say that it might better left well alone, and have something like
Baobab better signal what each directory is/which application it
belongs to, to clean it up.

> > 
> > > Is it reasonable to standardise on the systemd tmpfiles.d format?
> > > Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> > > default?
> > 
> > I'm guessing that's a no.
> > 
> > As for thumbnails, you'd probably get away with checking whether
> > atime
> > is actually set on that mount and cleaning up the ones that haven't
> > been used.
> 
> Benjamin

___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

On Wed, 2020-02-26 at 10:58 -0300, Soni L. wrote:
> On 2020-02-26 10:39 a.m., Benjamin Berg wrote:
> > [SNIP]
> > I don't think that $XDG_CACHE_HOME is designed to be used directly by
> > users. And if it is an application which generates those repositories,
> > then again, it can just drop-in the appropriate configuration to
> > prevent cleaning.
> > [SNIP]
> > Am I missing something obvious?
> 
> Oh sorry, I should've been more specific. I use it in the program I make 
> https://ganarchy.autistic.space/project/385e734a52e13949a7a5c71827f6de920dbfea43/
> 
> > > Besides, what if you nuke it *while* the thing is running? That
> > > doesn't sound like a good idea.
> > 
> > This seems like a rather unlikely corner case. In particular as the
> > atime is taken into account by tmpfiles.d. So there should be no need
> > to worry about files being deleted that have been used in the last 30
> > days or so.
> 
> I believe git won't necessarily touch old object files such as those 
> from really old commits, so that might still break stuff. Unless I'm 
> misunderstanding something here and it's meant to be all-or-nothing.

Right, but in your usecase, all you need to do is create a file (at
installation or startup when running from source) in:
  $XDG_CONFIG_DIR/tmpfiles.d/ganarchy.conf
or
  /us/share/user-tmpfiles.d/ganarchy.conf
with the single line
  x %C/ganarchy
or something very similar.

Really, all this does make it explicit that you do not want automatic
cache cleaning for the corresponding directory.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

On Wed, 2020-02-26 at 14:46 +0100, Bardot Jérôme wrote:
> On 26/02/2020 14:30, Benjamin Berg wrote:
> > On Wed, 2020-02-26 at 14:22 +0100, Bastien Nocera wrote:
> > > On Wed, 2020-02-26 at 13:45 +0100, Benjamin Berg wrote:
> > > > Now, systemd-tempfiles can already clean up everything except for the
> > > > trash. And considering that $XDG_CACHE_HOME is non-essential by
> > > > definition, I think it might be sane to use systemd-tempfiles not only
> > > > to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
> > > > future.
> > > 
> > > It's not "non-essential", it's a cache, which can be regenerated, but
> > > it might be utterly costly to do so. Eg. there are 10 gigs of "cached"
> > > evolution mails in my ~/.cache, 5 gigs of jhbuild builddirs.
> > > 
> > > Nuking it is a last ditch scenario. You'd avoid backing it up on space
> > > constrained storage, but you'd want to avoid having to regenerate that
> > > cache in most cases.
> > 
> > I am *not* proposing to nuke these directories. I am proposing to nuke
> > them by default, and ask applications like evolution, jhbuild and
> > others to ship their own configuration.
> > 
> 
> I’m not sure a destructive by default behavior is a good thing even on
> a cache directories.
> 
> I’m pretty sure this will lead to a lost of data.

Oh, I am sure there will be some pain because there will always be
someone does not get the memo. In the same way that e.g. disallowing
processes to linger after the user logs out caused some pain.

However, I do think it the problem is manageable. And I do see a huge
advantage in requiring applications to specify the clean-up behaviour
they need or want. So the primary goal here is to get applications to
define the clean-up behaviour they want. After that the secondary goal
is to define a better default behaviour for all other cases.

Benjamin

> > This matches the behaviour of /tmp and /var/tmp on systemd managed
> > systems. In the simplest case, all evolution needs to do is ship a one
> > line file with:
> > 
> > x %C/evolution
> > 
> > This file can even be installed to the users $XDG_CONFIG_DIR for
> > applications that might not be able to do it globally.
> > 
> Not all GNU/linux comes with systemd. Protocol and specification over
> tools/implementation specific. If it come restrictive it’s not good.
> Exept for security stuff and even in this case it’s not always a good
> choice. (for ie : 2fa when only choice is a phone)
> 
> > > 
> > > > Is it reasonable to standardise on the systemd tmpfiles.d format?
> > > > Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> > > > default?
> > > 
> > > I'm guessing that's a no.
> > > 
> > > As for thumbnails, you'd probably get away with checking whether atime
> > > is actually set on that mount and cleaning up the ones that haven't
> > > been used.
> > 
> > Benjamin
> > 
> > 
> > ___
> > xdg mailing list
> > xdg@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/xdg
> > 
> 
> ___
> xdg mailing list
> xdg@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/xdg


signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Soni L.]

On 2020-02-26 10:39 a.m., Benjamin Berg wrote:

On Wed, 2020-02-26 at 10:25 -0300, Soni L. wrote:
> I use XDG_CACHE_HOME to store git repos. While it's definitely
> possible to nuke them to save space... it'll just re-fetch everything
> next time it runs.

I don't think that $XDG_CACHE_HOME is designed to be used directly by
users. And if it is an application which generates those repositories,
then again, it can just drop-in the appropriate configuration to
prevent cleaning.

The only reason I see right now for a user to store data in
$XDG_CACHE_HOME is if their backup system does not easily allow
excluding further directories. But that seems like a rather odd reason
for a user to put data into $XDG_CACHE_HOME.

Am I missing something obvious?


Oh sorry, I should've been more specific. I use it in the program I make 
https://ganarchy.autistic.space/project/385e734a52e13949a7a5c71827f6de920dbfea43/




> Besides, what if you nuke it *while* the thing is running? That
> doesn't sound like a good idea.

This seems like a rather unlikely corner case. In particular as the
atime is taken into account by tmpfiles.d. So there should be no need
to worry about files being deleted that have been used in the last 30
days or so.


I believe git won't necessarily touch old object files such as those 
from really old commits, so that might still break stuff. Unless I'm 
misunderstanding something here and it's meant to be all-or-nothing.




Benjamin

> On 2020-02-26 9:45 a.m., Benjamin Berg wrote:
> > Hi,
> > 
> > so I looked at gsd-housekeeping the other day. With systemd-

> > tempfiles,
> > it only has two purposes these days:
> > 
> >  1. Cleaning $XDG_CACHE_HOME/thumbnails after 30 days

> >  2. Cleaning the trash directories after a configurable time
> > 
> > Currently it also tries to clean /tmp and /var/tmp, but doing so is

> > really dangerous compared to just leaving it up to systemd-
> > tempfiles (I
> > have filed an MR to disable the logic if we are systemd booted).
> > 
> > 
> > Now, systemd-tempfiles can already clean up everything except for

> > the
> > trash. And considering that $XDG_CACHE_HOME is non-essential by
> > definition, I think it might be sane to use systemd-tempfiles not
> > only
> > to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
> > future. So I am thinking that we could do the following:
> > 
> >1. Specify that we use the systemd tmpfiles.d configuration

> > format for
> >   cleaning $XDG_CACHE_HOME. Also specify that $XDG_CACHE_HOME
> > will be
> >   cleaned automatically after e.g. 30 days unless otherwise
> > configured
> >   by an application.
> >2. Add some reference to this to the thumbnail specification.
> >3. Tell application maintainers that they need to ship a
> > configuration
> >   if they want to keep files longer (likely candidates are e.g.
> > email
> >   clients).
> >4. As a start, add a "xdg-thumbnails.conf" systemd-tempfiles
> >   configuration to systemd that cleans
> > $XDG_CACHE_HOME/thumbnails
> >   after 30 days.
> >5. After a grace period, add "xdg-cache.conf" to clean
> > $XDG_CACHE_HOME
> >   and remove "xdg-thumbnails.conf" again (similar to how
> >   /usr/lib/tmpfiles.d/tmp.conf does it for /var/tmp and /tmp)
> > 
> > Is it reasonable to standardise on the systemd tmpfiles.d format?

> > Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> > default?
> > Other thoughts?
> > 
> > Benjamin
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > ___

> > xdg mailing list
> > xdg@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/xdg
>  
> ___

> xdg mailing list
> xdg@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/xdg


___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bardot Jérôme]

On 26/02/2020 14:30, Benjamin Berg wrote:
> On Wed, 2020-02-26 at 14:22 +0100, Bastien Nocera wrote:
>> On Wed, 2020-02-26 at 13:45 +0100, Benjamin Berg wrote:
>>> Now, systemd-tempfiles can already clean up everything except for the
>>> trash. And considering that $XDG_CACHE_HOME is non-essential by
>>> definition, I think it might be sane to use systemd-tempfiles not only
>>> to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
>>> future.
>>
>> It's not "non-essential", it's a cache, which can be regenerated, but
>> it might be utterly costly to do so. Eg. there are 10 gigs of "cached"
>> evolution mails in my ~/.cache, 5 gigs of jhbuild builddirs.
>>
>> Nuking it is a last ditch scenario. You'd avoid backing it up on space
>> constrained storage, but you'd want to avoid having to regenerate that
>> cache in most cases.
> 
> I am *not* proposing to nuke these directories. I am proposing to nuke
> them by default, and ask applications like evolution, jhbuild and
> others to ship their own configuration.
> 

I’m not sure a destructive by default behavior is a good thing even on
a cache directories.

I’m pretty sure this will lead to a lost of data.

> This matches the behaviour of /tmp and /var/tmp on systemd managed
> systems. In the simplest case, all evolution needs to do is ship a one
> line file with:
> 
> x %C/evolution
> 
> This file can even be installed to the users $XDG_CONFIG_DIR for
> applications that might not be able to do it globally.
> 
Not all GNU/linux comes with systemd. Protocol and specification over
tools/implementation specific. If it come restrictive it’s not good.
Exept for security stuff and even in this case it’s not always a good
choice. (for ie : 2fa when only choice is a phone)

>> 
>>> Is it reasonable to standardise on the systemd tmpfiles.d format?
>>> Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
>>> default?
>>
>> I'm guessing that's a no.
>>
>> As for thumbnails, you'd probably get away with checking whether atime
>> is actually set on that mount and cleaning up the ones that haven't
>> been used.
> 
> Benjamin
> 
> 
> ___
> xdg mailing list
> xdg@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/xdg
> 



0x053A41EF03878A98.asc
Description: application/pgp-keys
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

On Wed, 2020-02-26 at 10:25 -0300, Soni L. wrote:
> I use XDG_CACHE_HOME to store git repos. While it's definitely
> possible to nuke them to save space... it'll just re-fetch everything
> next time it runs.

I don't think that $XDG_CACHE_HOME is designed to be used directly by
users. And if it is an application which generates those repositories,
then again, it can just drop-in the appropriate configuration to
prevent cleaning.

The only reason I see right now for a user to store data in
$XDG_CACHE_HOME is if their backup system does not easily allow
excluding further directories. But that seems like a rather odd reason
for a user to put data into $XDG_CACHE_HOME.

Am I missing something obvious?

> Besides, what if you nuke it *while* the thing is running? That
> doesn't sound like a good idea.

This seems like a rather unlikely corner case. In particular as the
atime is taken into account by tmpfiles.d. So there should be no need
to worry about files being deleted that have been used in the last 30
days or so.

Benjamin

> On 2020-02-26 9:45 a.m., Benjamin Berg wrote:
> > Hi,
> > 
> > so I looked at gsd-housekeeping the other day. With systemd-
> > tempfiles,
> > it only has two purposes these days:
> > 
> >  1. Cleaning $XDG_CACHE_HOME/thumbnails after 30 days
> >  2. Cleaning the trash directories after a configurable time
> > 
> > Currently it also tries to clean /tmp and /var/tmp, but doing so is
> > really dangerous compared to just leaving it up to systemd-
> > tempfiles (I
> > have filed an MR to disable the logic if we are systemd booted).
> > 
> > 
> > Now, systemd-tempfiles can already clean up everything except for
> > the
> > trash. And considering that $XDG_CACHE_HOME is non-essential by
> > definition, I think it might be sane to use systemd-tempfiles not
> > only
> > to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
> > future. So I am thinking that we could do the following:
> > 
> >1. Specify that we use the systemd tmpfiles.d configuration
> > format for
> >   cleaning $XDG_CACHE_HOME. Also specify that $XDG_CACHE_HOME
> > will be
> >   cleaned automatically after e.g. 30 days unless otherwise
> > configured
> >   by an application.
> >2. Add some reference to this to the thumbnail specification.
> >3. Tell application maintainers that they need to ship a
> > configuration
> >   if they want to keep files longer (likely candidates are e.g.
> > email
> >   clients).
> >4. As a start, add a "xdg-thumbnails.conf" systemd-tempfiles
> >   configuration to systemd that cleans
> > $XDG_CACHE_HOME/thumbnails
> >   after 30 days.
> >5. After a grace period, add "xdg-cache.conf" to clean
> > $XDG_CACHE_HOME
> >   and remove "xdg-thumbnails.conf" again (similar to how
> >   /usr/lib/tmpfiles.d/tmp.conf does it for /var/tmp and /tmp)
> > 
> > Is it reasonable to standardise on the systemd tmpfiles.d format?
> > Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> > default?
> > Other thoughts?
> > 
> > Benjamin
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > ___
> > xdg mailing list
> > xdg@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/xdg
>  
> ___
> xdg mailing list
> xdg@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/xdg


signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

On Wed, 2020-02-26 at 14:22 +0100, Bastien Nocera wrote:
> On Wed, 2020-02-26 at 13:45 +0100, Benjamin Berg wrote:
> > Now, systemd-tempfiles can already clean up everything except for the
> > trash. And considering that $XDG_CACHE_HOME is non-essential by
> > definition, I think it might be sane to use systemd-tempfiles not only
> > to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
> > future.
> 
> It's not "non-essential", it's a cache, which can be regenerated, but
> it might be utterly costly to do so. Eg. there are 10 gigs of "cached"
> evolution mails in my ~/.cache, 5 gigs of jhbuild builddirs.
> 
> Nuking it is a last ditch scenario. You'd avoid backing it up on space
> constrained storage, but you'd want to avoid having to regenerate that
> cache in most cases.

I am *not* proposing to nuke these directories. I am proposing to nuke
them by default, and ask applications like evolution, jhbuild and
others to ship their own configuration.

This matches the behaviour of /tmp and /var/tmp on systemd managed
systems. In the simplest case, all evolution needs to do is ship a one
line file with:

x %C/evolution

This file can even be installed to the users $XDG_CONFIG_DIR for
applications that might not be able to do it globally.

> 
> > Is it reasonable to standardise on the systemd tmpfiles.d format?
> > Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> > default?
> 
> I'm guessing that's a no.
> 
> As for thumbnails, you'd probably get away with checking whether atime
> is actually set on that mount and cleaning up the ones that haven't
> been used.

Benjamin


signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Soni L.]

I use XDG_CACHE_HOME to store git repos. While it's definitely possible 
to nuke them to save space... it'll just re-fetch everything next time 
it runs.


Besides, what if you nuke it *while* the thing is running? That doesn't 
sound like a good idea.


On 2020-02-26 9:45 a.m., Benjamin Berg wrote:

Hi,

so I looked at gsd-housekeeping the other day. With systemd-tempfiles,
it only has two purposes these days:

  1. Cleaning $XDG_CACHE_HOME/thumbnails after 30 days
  2. Cleaning the trash directories after a configurable time

Currently it also tries to clean /tmp and /var/tmp, but doing so is
really dangerous compared to just leaving it up to systemd-tempfiles (I
have filed an MR to disable the logic if we are systemd booted).


Now, systemd-tempfiles can already clean up everything except for the
trash. And considering that $XDG_CACHE_HOME is non-essential by
definition, I think it might be sane to use systemd-tempfiles not only
to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
future. So I am thinking that we could do the following:

1. Specify that we use the systemd tmpfiles.d configuration format for
   cleaning $XDG_CACHE_HOME. Also specify that $XDG_CACHE_HOME will be
   cleaned automatically after e.g. 30 days unless otherwise configured
   by an application.
2. Add some reference to this to the thumbnail specification.
3. Tell application maintainers that they need to ship a configuration
   if they want to keep files longer (likely candidates are e.g. email
   clients).
4. As a start, add a "xdg-thumbnails.conf" systemd-tempfiles
   configuration to systemd that cleans $XDG_CACHE_HOME/thumbnails
   after 30 days.
5. After a grace period, add "xdg-cache.conf" to clean $XDG_CACHE_HOME
   and remove "xdg-thumbnails.conf" again (similar to how
   /usr/lib/tmpfiles.d/tmp.conf does it for /var/tmp and /tmp)

Is it reasonable to standardise on the systemd tmpfiles.d format?
Is it OK to clean $XDG_CACHE_HOME after a fixed time period by default?
Other thoughts?

Benjamin











___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg


___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Re: Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Bastien Nocera]

On Wed, 2020-02-26 at 13:45 +0100, Benjamin Berg wrote:
> Hi,
> 
> so I looked at gsd-housekeeping the other day. With systemd-
> tempfiles,
> it only has two purposes these days:
> 
>  1. Cleaning $XDG_CACHE_HOME/thumbnails after 30 days
>  2. Cleaning the trash directories after a configurable time
> 
> Currently it also tries to clean /tmp and /var/tmp, but doing so is
> really dangerous compared to just leaving it up to systemd-tempfiles
> (I
> have filed an MR to disable the logic if we are systemd booted).

That's fine.

> 
> Now, systemd-tempfiles can already clean up everything except for the
> trash. And considering that $XDG_CACHE_HOME is non-essential by
> definition, I think it might be sane to use systemd-tempfiles not
> only
> to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
> future.

It's not "non-essential", it's a cache, which can be regenerated, but
it might be utterly costly to do so. Eg. there are 10 gigs of "cached"
evolution mails in my ~/.cache, 5 gigs of jhbuild builddirs.

Nuking it is a last ditch scenario. You'd avoid backing it up on space
constrained storage, but you'd want to avoid having to regenerate that
cache in most cases.


Is it reasonable to standardise on the systemd tmpfiles.d format?
> Is it OK to clean $XDG_CACHE_HOME after a fixed time period by
> default?

I'm guessing that's a no.

As for thumbnails, you'd probably get away with checking whether atime
is actually set on that mount and cleaning up the ones that haven't
been used.

> Other thoughts?
> 
> Benjamin
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ___
> xdg mailing list
> xdg@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/xdg

___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg

Cleaning of $XDG_CACHE_HOME and $XDG_CACHE_HOME/thumbnails

[Benjamin Berg]

Hi,

so I looked at gsd-housekeeping the other day. With systemd-tempfiles,
it only has two purposes these days:

 1. Cleaning $XDG_CACHE_HOME/thumbnails after 30 days
 2. Cleaning the trash directories after a configurable time

Currently it also tries to clean /tmp and /var/tmp, but doing so is
really dangerous compared to just leaving it up to systemd-tempfiles (I
have filed an MR to disable the logic if we are systemd booted).


Now, systemd-tempfiles can already clean up everything except for the
trash. And considering that $XDG_CACHE_HOME is non-essential by
definition, I think it might be sane to use systemd-tempfiles not only
to clean the thumbnails but the entirety of $XDG_CACHE_HOME in the
future. So I am thinking that we could do the following:

   1. Specify that we use the systemd tmpfiles.d configuration format for
  cleaning $XDG_CACHE_HOME. Also specify that $XDG_CACHE_HOME will be
  cleaned automatically after e.g. 30 days unless otherwise configured
  by an application.
   2. Add some reference to this to the thumbnail specification.
   3. Tell application maintainers that they need to ship a configuration
  if they want to keep files longer (likely candidates are e.g. email
  clients).
   4. As a start, add a "xdg-thumbnails.conf" systemd-tempfiles
  configuration to systemd that cleans $XDG_CACHE_HOME/thumbnails
  after 30 days.
   5. After a grace period, add "xdg-cache.conf" to clean $XDG_CACHE_HOME
  and remove "xdg-thumbnails.conf" again (similar to how
  /usr/lib/tmpfiles.d/tmp.conf does it for /var/tmp and /tmp)

Is it reasonable to standardise on the systemd tmpfiles.d format?
Is it OK to clean $XDG_CACHE_HOME after a fixed time period by default?
Other thoughts?

Benjamin












signature.asc
Description: This is a digitally signed message part
___
xdg mailing list
xdg@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/xdg