Re: Why is /var/lib/apt/lists not in /var/cache?
On Mon, Dec 18, 2023 at 09:05:22AM -0500, Stefan Monnier wrote: > > I would imagine that it's due to the FHS (Filesystem Hierarchy Standard) > > which defines what the various directories on a "typical Linux system" are > > for. "man hier", for example, tells me that: > > > > * /var/cache - Data cached for programs. > > > > * /var/lib - Variable state information for programs. > > These leave open the question of what "data cached for the program" mean > and what "Variable state" means. > > Another way to look at it is in terms of what it implies. > For me, `/var/lib` data is data that is important to keep, e.g. data you > definitely don't want to delete/lose without a very good reason, whereas > `/var/cache` is data that's handy to have but not indispensable because > we can reconstruct it (or something close enough) from other sources, > which means that it doesn't have to be included in backups and it can > be deleted if it strikes your fancy (typically when you're in urgent need > of a tiny bit more disk space). [...] Most probably you are right, especially since TLDP only says you can delete /var/cache's content safely across reboots [1]. The apt lists will be regenerated after doing an apt-get update, so... OTOH it'd be only a true-real-honestly-promised cache if every apt command did an apt-get update whenever it found that file missing. Cheers [1] https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/var.html -- t signature.asc Description: PGP signature
Re: Why is /var/lib/apt/lists not in /var/cache?
> I would imagine that it's due to the FHS (Filesystem Hierarchy Standard) > which defines what the various directories on a "typical Linux system" are > for. "man hier", for example, tells me that: > > * /var/cache - Data cached for programs. > > * /var/lib - Variable state information for programs. These leave open the question of what "data cached for the program" mean and what "Variable state" means. Another way to look at it is in terms of what it implies. For me, `/var/lib` data is data that is important to keep, e.g. data you definitely don't want to delete/lose without a very good reason, whereas `/var/cache` is data that's handy to have but not indispensable because we can reconstruct it (or something close enough) from other sources, which means that it doesn't have to be included in backups and it can be deleted if it strikes your fancy (typically when you're in urgent need of a tiny bit more disk space). > system, so apt's state is more like the state of the repositories. But both the "files that contain the list of .deb files" and "the .deb files" are part of the state of the repositories :-) Or, stated in another way, this way to look at the problem degenerates into philosophical questions. So I think the question "does it have to be included in the backup" is a better guide. Stefan
Re: Why is /var/lib/apt/lists not in /var/cache?
On 16/12/2023 15:59, Stefan Monnier wrote: AFAICT, all of `/var/lib/apt/lists` is made of files fetched from repositories, which APT will re-fetch if missing. So, it sounds to me like it belongs in `/var/cache/apt/lists`, really. What am I missing? Or is it just a historical accident? Stefan "whose `/var/lib/apt/lists` is a symlink into /`var/cache`" I would imagine that it's due to the FHS (Filesystem Hierarchy Standard) which defines what the various directories on a "typical Linux system" are for. "man hier", for example, tells me that: * /var/cache - Data cached for programs. * /var/lib - Variable state information for programs. So, apt seems to be doing the right thing here. /var/cache is where the cached packages are stores, but /var/lib is where the state of the repositories is stored. Admittedly, the state of the repositories is cached information, but recall that apt is essentially just a network fetcher/cacher. dpkg would be the ultimate arbiter of the state of the system, so apt's state is more like the state of the repositories. OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Why is /var/lib/apt/lists not in /var/cache?
>> That was a typo. it's `/var/cache/plocate/plocate.db`, sorry. > My plocate.db is in /var/lib/plocate/, as is bookworm's. > Is that changing in the future? Hmm... I could swear that I saw it in /var/cache but every machine I look at has it in /var/lib, indeed. [ GNU locate puts its DB in `/var/cache/locate/`, apparently, OTOH. ] Anyway, thanks for all the suggestions. From what I can tell, there is no strong reason to put the lists in `/var/lib` rather than `/var/cache`. I suspect this is mostly a historical accident (at some point, erasing `/var/lib/apt/lists` caused problems because that dir wasn't automatically recreated if missing, so you'd have to `mkdir` manually as well as create the lock inside of it). Stefan
Re: Why is /var/lib/apt/lists not in /var/cache?
On Sun 17 Dec 2023 at 15:33:30 (-0500), Stefan Monnier wrote: > >> That seems similar to things like `locate` failing if you remove > >> `/var/log/plocate/plocate.db` (until that DB is rebuilt). > > > > It's tricky to discern your point as /var/log/ is not involved. > > That was a typo. it's `/var/cache/plocate/plocate.db`, sorry. My plocate.db is in /var/lib/plocate/, as is bookworm's. Is that changing in the future? $ ls -l /var/lib/*locate/ /var/lib/mlocate/: total 19344 -rw-r- 1 root mlocate 19807421 Aug 28 07:43 mlocate.db /var/lib/plocate/: total 20632 -rw-r--r-- 1 root root 183 Nov 17 2021 CACHEDIR.TAG -rw-r- 1 root plocate 21121629 Dec 16 16:06 plocate.db $ Cheers, David.
Re: Why is /var/lib/apt/lists not in /var/cache?
>> That seems similar to things like `locate` failing if you remove >> `/var/log/plocate/plocate.db` (until that DB is rebuilt). > > It's tricky to discern your point as /var/log/ is not involved. That was a typo. it's `/var/cache/plocate/plocate.db`, sorry. Stefan
Re: Why is /var/lib/apt/lists not in /var/cache?
On Sun 17 Dec 2023 at 01:06:28 (-0500), Stefan Monnier wrote: > > Some packages will stay the same for years, but in the past week > > I can see four occasions when changes in list contents have occurred > > on oldstable. So there's little similarity. > > The question is not really whether "apt/lists" is similar to > "apt/archives", but whether the content of "apt/lists" is appropriate > for `/var/cache`. I didn't bring up apt/archives, but my first words posted were: "This may not answer your question". > AFAIK nothing bad will happen to your system if you delete > `/var/lib/apt/lists`. It will still run as before and you can still > install packages as before. As you said. > The only thing that seems to be impacted is > that some APT operations will temporarily work in a kind of "degraded" > mode until the next `apt update`. As I said. On how significant that difference is, we beg to differ. > That seems similar to things like `locate` failing if you remove > `/var/log/plocate/plocate.db` (until that DB is rebuilt). It's tricky to discern your point as /var/log/ is not involved. Cheers, David.
Re: Why is /var/lib/apt/lists not in /var/cache?
> Some packages will stay the same for years, but in the past week > I can see four occasions when changes in list contents have occurred > on oldstable. So there's little similarity. The question is not really whether "apt/lists" is similar to "apt/archives", but whether the content of "apt/lists" is appropriate for `/var/cache`. AFAIK nothing bad will happen to your system if you delete `/var/lib/apt/lists`. It will still run as before and you can still install packages as before. The only thing that seems to be impacted is that some APT operations will temporarily work in a kind of "degraded" mode until the next `apt update`. That seems similar to things like `locate` failing if you remove `/var/log/plocate/plocate.db` (until that DB is rebuilt). Stefan
Re: Why is /var/lib/apt/lists not in /var/cache?
On Sat 16 Dec 2023 at 12:50:51 (-0500), Stefan Monnier wrote: > David Wright [2023-12-16 11:30:01] wrote: > > On Sat 16 Dec 2023 at 10:59:48 (-0500), Stefan Monnier wrote: > >> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from > >> repositories, which APT will re-fetch if missing. > >> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really. > >> What am I missing? Or is it just a historical accident? > > > > This may not answer your question, but when you fetch new lists, > > they aren't the same as what was there before, and don't actually > > match your system until you upgrade it. > > AFAIK those lists aren't supposed to "match your system", are they? > IIUC they just keep a local copy of the repository's list. Whether it's considered necessary or desirable depends on the user and what they use the lists for (apart from the obvious). > Admittedly if I `apt update` today, and then a week from now I delete > the lists and do `apt update` again I won't get the same files. > But similar, if I delete one of the files in > `/var/cache/apt/archive`, there's no guarantee that I'll be able to > re-download this exact file next week (tho I guess > https://snapshot.debian.org/ might still make it possible if you want it > badly enough). Some packages will stay the same for years, but in the past week I can see four occasions when changes in list contents have occurred on oldstable. So there's little similarity. Cheers, David.
Re: Why is /var/lib/apt/lists not in /var/cache?
Max Nikulin [2023-12-17 09:10:29] wrote: > On 16/12/2023 22:59, Stefan Monnier wrote: >> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from >> repositories, which APT will re-fetch if missing. >> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really. > APT running by a regular user is unable to write to /var/cache/apt. But neither is it able to write to `/var/lib/apt/lists`, AFAICT. Stefan
Re: Why is /var/lib/apt/lists not in /var/cache?
On 16/12/2023 22:59, Stefan Monnier wrote: AFAICT, all of `/var/lib/apt/lists` is made of files fetched from repositories, which APT will re-fetch if missing. So, it sounds to me like it belongs in `/var/cache/apt/lists`, really. APT running by a regular user is unable to write to /var/cache/apt. E.g. DNF creates cache in the user home directory in such cases. I like APT approach since (at least with default settings) search queries work without delay while repository data is being updated. (I am realizing that I may get stale results.) So for APT lists are a bit more than just cache.
Re: Why is /var/lib/apt/lists not in /var/cache?
David Wright [2023-12-16 11:30:01] wrote: > On Sat 16 Dec 2023 at 10:59:48 (-0500), Stefan Monnier wrote: >> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from >> repositories, which APT will re-fetch if missing. >> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really. >> What am I missing? Or is it just a historical accident? > > This may not answer your question, but when you fetch new lists, > they aren't the same as what was there before, and don't actually > match your system until you upgrade it. AFAIK those lists aren't supposed to "match your system", are they? IIUC they just keep a local copy of the repository's list. Admittedly if I `apt update` today, and then a week from now I delete the lists and do `apt update` again I won't get the same files. But similar, if I delete one of the files in `/var/cache/apt/archive`, there's no guarantee that I'll be able to re-download this exact file next week (tho I guess https://snapshot.debian.org/ might still make it possible if you want it badly enough). Stefan
Re: Why is /var/lib/apt/lists not in /var/cache?
On Sat 16 Dec 2023 at 10:59:48 (-0500), Stefan Monnier wrote: > AFAICT, all of `/var/lib/apt/lists` is made of files fetched from > repositories, which APT will re-fetch if missing. > So, it sounds to me like it belongs in `/var/cache/apt/lists`, really. > What am I missing? Or is it just a historical accident? This may not answer your question, but when you fetch new lists, they aren't the same as what was there before, and don't actually match your system until you upgrade it. Cheers, David.
