Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
> Am 17.01.2022 um 05:16 schrieb Chris Murphy : > > On Sun, Jan 16, 2022 at 3:59 PM Peter Boy wrote: >> >> >> >>> Am 14.01.2022 um 23:51 schrieb Fabio Valentini : >>> >>> >>> Wait, I thought this change was about making the path consistent >>> within Fedora variants? >> >> The question still is whether this is actually useful and beneficial. > > If you value Fedora having a snapshot and rollback scheme of some > kind, it's useful and beneficial. If you don't, then the change is > neutral because it has not a single technical downside presented so > far - just emotive ones. The loss of LSB / FHS compliance is by no means "just emotive", nor is the consequence of probably having to adapt an unknown number of scripts, backup routines, etc. . As you stated in a previous post, LVM snapshot is „under-developed“ and therefore currently not useful. Is there another snapshot & rollback tool in Fedora repository I can use out of the box (after I moved rpm db to /usr)? Unfortunately, I don’t know one. Currently I would have to create a LVM snapshot and a manually crafted selective backup of files and subdirectories. And test all the stuff. Not particularly practical. I guess, an achievement of a truly viable snapshot and rollback system for file-based distributions requires far a greater number of relocations than just the rpm db. The article of a (presumably) Suse employee I’m trying to retrieve (see an earlier post of mine) offers a proposal for this very purpose (including a backwards compatibility link system). This would eventually end up in an FHS 4.x. Moving the rpm db alone adds only disadvantages to file-based distributions, not a single advantage. And it is not neutral either. In the end, we don't really need to do anything. If I understand correctly, rpm-ostree is already implementing the change anyway, without any change voting, and everyone else will continue as before, following LSB/FHS and the current Fedora guidelines. > Again if you see no value in snapshots/rollbacks, you don't see the > advantage. If you like the idea, then you'd also necessarily come to > realize that some pressure on organizing files into locations with > compatible life cycles, so those locations can be independently rolled > back. I see a lot of value in snapshot systems. And I see not only „some“ pressure (or better requirements) to reorganise files into different locations. The rpm db is far from sufficient. The proposal mentioned above did exactly that, but rpm db did not end up in /usr but together with parts of /var and /etc somewhere else I don’t remember in detail. And if we make adjustments to achieve a snapshot/rollback system, then we should do it right and not stop halfway unfinished. Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
> Am 17.01.2022 um 05:17 schrieb Chris Murphy : > > On Sun, Jan 16, 2022 at 4:34 PM Peter Boy wrote: >> >> >> @Neal, I remember a Suse employee made once (about a year ago) a proposal to >> slightly modify FHS to better separate distro specific from local specific >> content, especially in /etc. Unfortunately I can't find the link again. Do >> you happen to know if that was related to Suse's snapshot+rollback scheme? > > Maybe this: > https://github.com/lnussel/lnussel.github.io/blob/fs/_posts/2020-12-16-fslayout.md > Yes, that the right direction. I’ll start my search from that. Thanks Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Sunday, January 16, 2022 11:16:57 PM EST Chris Murphy wrote: > On Sun, Jan 16, 2022 at 3:59 PM Peter Boy wrote: > > > Am 14.01.2022 um 23:51 schrieb Fabio Valentini : > > > > > > > > > Wait, I thought this change was about making the path consistent > > > within Fedora variants? > > > > The question still is whether this is actually useful and beneficial. > > If you value Fedora having a snapshot and rollback scheme of some > kind, it's useful and beneficial. If you don't, then the change is > neutral because it has not a single technical downside presented so > far - just emotive ones. I'm a little concerned about what this increased writing to /usr, which many people have on a SSD, will do. I have everything that gets written to with any regularity on spinning disks. Everything else is SSD. I have to agree with Chris Adams that rpms own things on many top level directories. I have rpms that own programs/files that live on /opt and other top level directories. I also agree that most programs cannot recreate their directory paths and consider it's absense to be a catastrophic failure. I also see things like kmail which uses mariadb as it's storage. It occassionally contains migration scripts to change the format of the database. It never has a migration script to go backwards. We do the same thing with fapolicyd and its lmdb backing storage. We migrate forward, but we never allow backwards migration. To roll back fapolicyd, you'd need to snapshot /var and roll it back. But since /var has the mail spool and other accumulated data, you'd risk losing lots of stuff if /var was rolled back. I'd think the solution for image based distros is to put the rpmdb in /usr/ lib/rpm and bind mount it to /var/lib/rpm. Doing it this way means no changes are needed. -Steve ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sun, Jan 16, 2022 at 5:05 PM Chris Adams wrote: > > Once upon a time, Ben Cotton said: > > == Benefit to Fedora == > > > > * The RPM database primarily describes the state of `/usr`. Storing > > the databases in `/usr` will more easily facilitate OS rollback, > > without affecting `/var`. > > Rolling back to the start... this statement is only marginally true. > And "primarily" is a hand-wavy hedge that just pretends the rest doesn't > matter. > > Ony my Fedora 35 desktop, I have things in RPM under a lot of top-level > directories: > > $ rpm -qla | grep '^/.*/' | cut -d/ -f2 | sort | uniq -c > 99 boot >2998 etc > 14792 lib > 43 lib64 > 97 opt > 5 root > 43 run > 10 sbin > 442175 usr >2036 var > > Some of those are RPMs that should probably be updated, but some of that > won't go away. There's important stuff in /var and /etc, and even for > the stuff that could be blown away and recreated, most programs aren't > set up (or don't have permission) to recreate their directories. I'm not expecting to do filesystem rollbacks to revert Firefox or SQLite. I expect if I have some kind of boot or startup problem, the system is a candidate for a rollback. In such a failure, I haven't run any other programs yet that may have upgraded a database. Could modification to something in /etc result in startup failure? Yep. Is a full system rollback really called for here? It'll work, but it's a hammer. So maybe services need to learn to be a little suspicious of /etc configs, and fail safe? Maybe system critical items should be able to fall back to /usr/etc? -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Sun, Jan 16, 2022 at 9:36 PM Chris Adams wrote: > > Once upon a time, Chris Murphy said: > > If you value Fedora having a snapshot and rollback scheme of some > > kind, it's useful and beneficial. If you don't, then the change is > > neutral because it has not a single technical downside presented so > > far - just emotive ones. > > It's only beneficial for snapshots if you believe that only /usr > matters, which I don't believe is true. It might be true that it's the only one that should matter, if you care to have the benefit of a clear cut snapshot and rollback design. This is the /etc/fstab for (open)SUSE Tumbleweed. $mainfsuuid / btrfs defaults 0 0 $mainfsuuid /varbtrfs subvol=/@/var 0 0 $mainfsuuid /usr/local btrfs subvol=/@/usr/local 0 0 $mainfsuuid /srvbtrfs subvol=/@/srv 0 0 $mainfsuuid /root btrfs subvol=/@/root0 0 $mainfsuuid /optbtrfs subvol=/@/opt 0 0 $mainfsuuid /home btrfs subvol=/@/home0 0 $mainfsuuid /boot/grub2/x86_64-efi btrfs subvol=/@/boot/grub2/x86_64-efi 0 0 $mainfsuuid /boot/grub2/i386-pc btrfs subvol=/@/boot/grub2/i386-pc 0 0 $mainfsuuid /.snapshots btrfs subvol=/@/.snapshots 0 0 $swapfsuuid swapswap defaults 0 0 $espfsuuid /boot/efi vfat utf8 0 2 It lacks a separate line for /var/tmp/rpm, which it would otherwise need, because the rpmdb is in /usr. It looks like this because /usr isn't the only thing that's important. These carve outs exist because it was easier to do with Btrfs subvolumes (effectively bind mounts behind the scenes) than (a) with individual file systems or (b) committing to reorganizing the filesystem hierarchy. Now (open)SUSE is doing the hard work to reorganize the filesystem hierarchy where RPM only touches /usr. If we're feeling bold and maybe even brazen, maybe we'd pull everything out of /usr /var /etc that can be called 'the system'. Specifically, that which is system critical for the purpose of booting and startup. Things that get us to a working login prompt, but not beyond that (or at least not much beyond it). Stick all of that into a new top-level directory called "system". Within that could be usr/ var/ etc/ if you want. This "system" subvolume or LV is what would be subject to snapshot and rollback. This could be done with metadata defining what individual files are "system critical" so that 'dnf history rollback' could do a fine grained revert for such those files. But that presumes we can even boot to a system sufficiently working that this command could succeed. I think it's better to have a system that's prepared for the possibility an update can introduce trouble, and has the ability to detect failure and reboot to a previous known working state. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
Once upon a time, Chris Murphy said: > If you value Fedora having a snapshot and rollback scheme of some > kind, it's useful and beneficial. If you don't, then the change is > neutral because it has not a single technical downside presented so > far - just emotive ones. It's only beneficial for snapshots if you believe that only /usr matters, which I don't believe is true. It is not neutral, because it does change things from long-standing defaults. It's also a change away from upstream, which normally I thought Fedora preferred to avoid unless there's a significantly compelling reason. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Sun, Jan 16, 2022 at 4:34 PM Peter Boy wrote: > > > > > Am 17.01.2022 um 00:09 schrieb Neal Gompa : > > > > On Fri, Jan 14, 2022 at 5:51 PM Fabio Valentini > > wrote: > >> > >> …. > > > > openSUSE originally did the move because standard openSUSE has a > > snapshot+rollback scheme and tracking the rpmdb is straightforward in > > /usr with all the other system state data. This benefited them for the > > development and release of openSUSE MicroOS a couple years later. > > > > @Neal, I remember a Suse employee made once (about a year ago) a proposal to > slightly modify FHS to better separate distro specific from local specific > content, especially in /etc. Unfortunately I can't find the link again. Do > you happen to know if that was related to Suse's snapshot+rollback scheme? Maybe this: https://github.com/lnussel/lnussel.github.io/blob/fs/_posts/2020-12-16-fslayout.md -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Sun, Jan 16, 2022 at 3:59 PM Peter Boy wrote: > > > > > Am 14.01.2022 um 23:51 schrieb Fabio Valentini : > > > > > > Wait, I thought this change was about making the path consistent > > within Fedora variants? > > The question still is whether this is actually useful and beneficial. If you value Fedora having a snapshot and rollback scheme of some kind, it's useful and beneficial. If you don't, then the change is neutral because it has not a single technical downside presented so far - just emotive ones. This change also doesn't put pressure on future decisions. It doesn't favor particular ways of snapshotting or rollback mechanisms, doesn't care about what volume manager is being used, etc. > All the arguments for this move that I have read so far explain benefits > related to an image based distribution. I have not seen any advantages of > this move for a file based distribution like Server or Workstation. Again if you see no value in snapshots/rollbacks, you don't see the advantage. If you like the idea, then you'd also necessarily come to realize that some pressure on organizing files into locations with compatible life cycles, so those locations can be independently rolled back. The change is already employed in traditional RPM based (open)SUSE a couple of years, the links to those discussions are in the change proposal. >And when I look at tools like LVM snapshot, I can't see any either. And I >couldn’t find something in a short online search. But maybe I missed >something? Then I would be grateful for some info/links. If anything it benefits LVM more, because each additional carveout on LVM requires a separate file system. On btrfs it's cheap to add subvolumes since they take up no space and instead all share a single storage pool, though it organizationally adds complexity. Would putting /var/lib/rpm on a dedicated LVM LV actually be entertained? I think it wouldn't. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
Once upon a time, Neal Gompa said: > On Sun, Jan 16, 2022 at 6:47 PM Chris Adams wrote: > > Once upon a time, Neal Gompa said: > > > It benefits LVM-based OS snapshotting equally well. However, the > > > tooling for LVM snapshots are under-developed, so it's not used so > > > much for this. > > > > How? Since the standard install uses a single root filesystem that > > includes /usr and /var, I can't see any benefit to it there either. > > It does *now*, but it's not necessarily desirable to keep it that way > if you want this capability. For Btrfs, shuffling subvolumes is cheap. > For LVM, shuffling LVs is a bit more work. I know about LVM snapshots - I used them for years to get point-in-time backups. But that doesn't explain a benefit to splitting /, /usr, and /var into separate filesystems/LVs/whatever. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
Once upon a time, Ben Cotton said: > == Benefit to Fedora == > > * The RPM database primarily describes the state of `/usr`. Storing > the databases in `/usr` will more easily facilitate OS rollback, > without affecting `/var`. Rolling back to the start... this statement is only marginally true. And "primarily" is a hand-wavy hedge that just pretends the rest doesn't matter. Ony my Fedora 35 desktop, I have things in RPM under a lot of top-level directories: $ rpm -qla | grep '^/.*/' | cut -d/ -f2 | sort | uniq -c 99 boot 2998 etc 14792 lib 43 lib64 97 opt 5 root 43 run 10 sbin 442175 usr 2036 var Some of those are RPMs that should probably be updated, but some of that won't go away. There's important stuff in /var and /etc, and even for the stuff that could be blown away and recreated, most programs aren't set up (or don't have permission) to recreate their directories. You can't snapshot just /usr, make software changes, and then roll it back, because programs in /usr have expectations about other directories. The whole OS doesn't exclusively exist in /usr. What if that /usr change was an updated version of MariaDB or PostgresSQL that updated the DB file format, or even just a program using one of those DBs that had a schema update? The only way I see any benefit to trying to rearrange the RPM database would be to split it up somehow, where it could be spread across filesystems (but that still has the same consistency issue of rolling back /usr without making the same rollback to /var and /etc). -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Sun, Jan 16, 2022 at 6:47 PM Chris Adams wrote: > > Once upon a time, Neal Gompa said: > > It benefits LVM-based OS snapshotting equally well. However, the > > tooling for LVM snapshots are under-developed, so it's not used so > > much for this. > > How? Since the standard install uses a single root filesystem that > includes /usr and /var, I can't see any benefit to it there either. It does *now*, but it's not necessarily desirable to keep it that way if you want this capability. For Btrfs, shuffling subvolumes is cheap. For LVM, shuffling LVs is a bit more work. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
Once upon a time, Neal Gompa said: > It benefits LVM-based OS snapshotting equally well. However, the > tooling for LVM snapshots are under-developed, so it's not used so > much for this. How? Since the standard install uses a single root filesystem that includes /usr and /var, I can't see any benefit to it there either. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
> Am 17.01.2022 um 00:09 schrieb Neal Gompa : > > On Fri, Jan 14, 2022 at 5:51 PM Fabio Valentini wrote: >> >> …. > > openSUSE originally did the move because standard openSUSE has a > snapshot+rollback scheme and tracking the rpmdb is straightforward in > /usr with all the other system state data. This benefited them for the > development and release of openSUSE MicroOS a couple years later. > @Neal, I remember a Suse employee made once (about a year ago) a proposal to slightly modify FHS to better separate distro specific from local specific content, especially in /etc. Unfortunately I can't find the link again. Do you happen to know if that was related to Suse's snapshot+rollback scheme? Thanks Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Sun, Jan 16, 2022 at 5:59 PM Peter Boy wrote: > > > > > Am 14.01.2022 um 23:51 schrieb Fabio Valentini : > > > > > > Wait, I thought this change was about making the path consistent > > within Fedora variants? > > The question still is whether this is actually useful and beneficial. > > All the arguments for this move that I have read so far explain benefits > related to an image based distribution. I have not seen any advantages of > this move for a file based distribution like Server or Workstation. And when > I look at tools like LVM snapshot, I can't see any either. And I couldn’t > find something in a short online search. But maybe I missed something? Then I > would be grateful for some info/links. > It benefits LVM-based OS snapshotting equally well. However, the tooling for LVM snapshots are under-developed, so it's not used so much for this. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Fri, Jan 14, 2022 at 5:51 PM Fabio Valentini wrote: > > On Fri, Jan 14, 2022 at 7:16 PM Colin Walters wrote: > > > > > > > > On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote: > > > > > The path "/usr/lib/sysimage/rpm" does look very out-of-place in > > > non-image-based systems, so *if* we want to move the rpmdb to a place > > > that's consistent across all our Editions, it should also be a > > > location name that makes sense across all Editions. > > > > I don't think we should discount alignment with OpenSUSE. When they > > originally proposed /usr/lib/sysimage I started to write a bikeshed reply > > email (like many that have been posted here) around why /usr/share was a > > bit better but then I said to myself: "You know what? I don't care as long > > as we get it in /usr. Since they're driving the change, and there really > > isn't any technical compelling reason to do something else, it's fine." > > > > Also, proliferation of these paths has a cost; see e.g. > > https://github.com/openSUSE/libsolv/pull/386 > > Though in practice *most* cases will be fine just chasing a symlink from > > /var/lib/rpm. > > Wait, I thought this change was about making the path consistent > within Fedora variants? > I understand that converging on the same path as OpenSUSE makes sense, > but does that mean we should not consider if there's a better > alternative? ... > And the "sysimage" path makes even less sense to me in the OpenSUSE > context, since they don't have an OSTree based variant at all (unless > I am mistaken)? > openSUSE originally did the move because standard openSUSE has a snapshot+rollback scheme and tracking the rpmdb is straightforward in /usr with all the other system state data. This benefited them for the development and release of openSUSE MicroOS a couple years later. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
> Am 14.01.2022 um 23:51 schrieb Fabio Valentini : > > > Wait, I thought this change was about making the path consistent > within Fedora variants? The question still is whether this is actually useful and beneficial. All the arguments for this move that I have read so far explain benefits related to an image based distribution. I have not seen any advantages of this move for a file based distribution like Server or Workstation. And when I look at tools like LVM snapshot, I can't see any either. And I couldn’t find something in a short online search. But maybe I missed something? Then I would be grateful for some info/links. In the case that there is no benefit for file based distributions, they only suffer the "malefit" of losing FHS compliance and possibly having to adapt administration scripts and routines. Then it would make more sense to move the RPM database only for image based distributions (which has obviously already been done anyway). Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Fri, Jan 14, 2022 at 3:51 PM Fabio Valentini wrote: > > On Fri, Jan 14, 2022 at 7:16 PM Colin Walters wrote: > > > > > > > > On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote: > > > > > The path "/usr/lib/sysimage/rpm" does look very out-of-place in > > > non-image-based systems, so *if* we want to move the rpmdb to a place > > > that's consistent across all our Editions, it should also be a > > > location name that makes sense across all Editions. > > > > I don't think we should discount alignment with OpenSUSE. When they > > originally proposed /usr/lib/sysimage I started to write a bikeshed reply > > email (like many that have been posted here) around why /usr/share was a > > bit better but then I said to myself: "You know what? I don't care as long > > as we get it in /usr. Since they're driving the change, and there really > > isn't any technical compelling reason to do something else, it's fine." > > > > Also, proliferation of these paths has a cost; see e.g. > > https://github.com/openSUSE/libsolv/pull/386 > > Though in practice *most* cases will be fine just chasing a symlink from > > /var/lib/rpm. > > Wait, I thought this change was about making the path consistent > within Fedora variants? CoreOS/Silverblue/Kinoite ls -l /var/lib/rpm lrwxrwxrwx. 1 root root 19 Dec 6 12:32 /var/lib/rpm -> ../../usr/share/rpm $ ls -l /usr/share/rpm total 52684 -rw-r--r--. 3 root root 53915648 Dec 31 1969 rpmdb.sqlite -rw-r--r--. 5 root root32768 Jan 14 18:10 rpmdb.sqlite-shm -rw-r--r--. 1 root root0 Jan 11 18:51 rpmdb.sqlite-wal $ ls -l /usr/lib/sysimage/rpm lrwxrwxrwx. 3 root root 15 Dec 6 12:23 /usr/lib/sysimage/rpm -> ../../share/rpm This symlink is present on Silverblue but I don't see it on a stock CoreOS image before it's been booted and subject to setup by ignition. $ ls -l /var/lib/rpm -> ../../usr/share/rpm > I understand that converging on the same path as OpenSUSE makes sense, > but does that mean we should not consider if there's a better > alternative? ... I think it's fine to consider alternatives, but the tradeoff might be that (open)SUSE sticks with what they've got. I have no idea but that could be a question posed to them before deciding this. > And the "sysimage" path makes even less sense to me in the OpenSUSE > context, since they don't have an OSTree based variant at all (unless > I am mistaken)? Absent the rpmdb, and ignoring all other considerations like storage and performance cost, could RPM learn to reconstruct the rpmdb? What would it need locally to do that? Would it need to keep rpm package headers and compare to what's installed? Some variation on rpm -Va? The point would be to keep /var/lib/rpm/rpmdb* allow it to be disposable, but with consequences. If rpm packages can be disembodied from their payload, just leaving the metadata, seems like it's plausible to reconstruct an rpmdb from checksums of what's installed versus would could be installed. This raises questions like, garbage collection of all these rpm files, would they be removed when the package is removed? -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Fri, Jan 14, 2022 at 7:16 PM Colin Walters wrote: > > > > On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote: > > > The path "/usr/lib/sysimage/rpm" does look very out-of-place in > > non-image-based systems, so *if* we want to move the rpmdb to a place > > that's consistent across all our Editions, it should also be a > > location name that makes sense across all Editions. > > I don't think we should discount alignment with OpenSUSE. When they > originally proposed /usr/lib/sysimage I started to write a bikeshed reply > email (like many that have been posted here) around why /usr/share was a bit > better but then I said to myself: "You know what? I don't care as long as we > get it in /usr. Since they're driving the change, and there really isn't any > technical compelling reason to do something else, it's fine." > > Also, proliferation of these paths has a cost; see e.g. > https://github.com/openSUSE/libsolv/pull/386 > Though in practice *most* cases will be fine just chasing a symlink from > /var/lib/rpm. Wait, I thought this change was about making the path consistent within Fedora variants? I understand that converging on the same path as OpenSUSE makes sense, but does that mean we should not consider if there's a better alternative? ... And the "sysimage" path makes even less sense to me in the OpenSUSE context, since they don't have an OSTree based variant at all (unless I am mistaken)? Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Thu, Jan 13, 2022, at 6:05 PM, Fabio Valentini wrote: > The path "/usr/lib/sysimage/rpm" does look very out-of-place in > non-image-based systems, so *if* we want to move the rpmdb to a place > that's consistent across all our Editions, it should also be a > location name that makes sense across all Editions. I don't think we should discount alignment with OpenSUSE. When they originally proposed /usr/lib/sysimage I started to write a bikeshed reply email (like many that have been posted here) around why /usr/share was a bit better but then I said to myself: "You know what? I don't care as long as we get it in /usr. Since they're driving the change, and there really isn't any technical compelling reason to do something else, it's fine." Also, proliferation of these paths has a cost; see e.g. https://github.com/openSUSE/libsolv/pull/386 Though in practice *most* cases will be fine just chasing a symlink from /var/lib/rpm. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Fri, Jan 14, 2022, at 2:46 AM, Chris Murphy wrote: > > What about /var/lib/selinux? It's owned by the selinux-policy-targeted > package. Even though the files may not change often, it probably needs > to be snapshot and rolled back with revision matching for /usr and > rpmdb. Yep, welcome to https://bugzilla.redhat.com/show_bug.cgi?id=1290659 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
Dne 13. 01. 22 v 15:26 Colin Walters napsal(a): On Thu, Jan 13, 2022, at 7:52 AM, Vít Ondruch wrote: Actually, shouldn't rpm-ostree carry around some copy of the RPM database, which would describe the state of /usr and once the update is successful (or snapshot active?), merge it into the main system RPM database? Apparently, something like this is already happening e.g. for /etc content if I understand it correctly. Or this is the direction in which the /boot will be handled eventually. Or possibly it could be just some linked (possibly just read only) database. I think you are confusing/conflating image based updates and client side snapshots (and relatedly, client side offline updates, which is what https://github.com/OpenSUSE/transactional-update). They are related, but different. I think you are doing unnecessary difference between those. In my view, there is no difference if /usr is read-only, network mounted, expanded from tarball or managed by rpm-ostree. The point is that the location, while being available on the user system, is not modifiable by RPM. The only requirement is that RPM is be able to provides metadata about that location. In this case, it makes perfect sense if the /usr contains some metadata in RPM consumable format (be it RPM database, but that is technical detail, because it could be its simpler version for RO filesystems). Now should there be different locations really managed by RPM, such as /opt, there is no reason why there should not be the system RPM database, which would store location about that parts of the system. To cover this scenario for rpm-ostree, there needs to be done something complex, in your words: If one chooses to engage client side layering (or overrides), rpm-ostree will (each time an upgrade or change is performed) indeed regenerate the rpm database using the "pristine" base image's version of the rpmdb. And I say this is wrong concept (understandably given by technical limitation). RPM in this case should keep using its system database and understand that /usr was delivered by different means, it can't manage the /usr content, but there is the metadata available. In this case, when the base image was updated, the RPM database would not need to be regenerated. However, the base image update could be clocked when e.g. RPM discovered, that it would broke the system dependencies. Vít OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Friday, 14 January 2022 at 00:05, Fabio Valentini wrote: > On Wed, Jan 12, 2022 at 9:45 AM Chris Murphy wrote: > > > > Given the choice, I prefer rpm only touches /usr, which includes > > /usr/var and /usr/etc. > > I must say that, of all the bad suggestions in the multiple threads > this proposal has spawned, I dislike using "/usr/var/" the least. > > The path "/usr/lib/sysimage/rpm" does look very out-of-place in > non-image-based systems, so *if* we want to move the rpmdb to a place > that's consistent across all our Editions, it should also be a > location name that makes sense across all Editions. > > A path that contains "sysimage" conveys to me that this is somehow > only related to image based variants. Something under /usr/var would > at least be an generic, non-confusing path. +1 to that. /usr/lib/sysimage is definitely a non-obvious location for rpmdb, while /usr/var/lib at least suggests it's something akin to /var/lib. Regards, Dominik -- Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Thu, Jan 13, 2022 at 8:49 AM David Cantrell wrote: > > On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote: > >I don't see how /state solves the problem, rather than just > >rearranging the chairs. > > I've seen the rearranging the chairs comment in multiple locations now. If > the location doesn't matter, why can't the RPM db stay in /var/lib/rpm? It's all about consequences, not that it can't stay in /var/lib/rpm. The rpmdb could go in its own subvolume "rpmdb" mounted at /var/lib/rpm, and snapshot it at the same time as subvolume "root" which contains /etc /usr /opt. Each snapshot is a revision. And a way to mount matching revisions is needed if there's more than one snapshot to rollback. e.g. root-x86-64:fedora36.0 root-x86-64:fedora36.1 rpmdb-x86-64:fedora36.0 rpmdb-x86-64:fedora36.1 When booting root-x86-64:fedora36.1 the helper needs to 'mount -o subvol=rpmdb-x86-64:fedora36.1 --uuid=$fsuuid /var/lib/rpm' When booting root-x86-64:fedora36.0, the helper needs to 'mount -o subvol=rpmdb-x86-64:fedora36.0 --uuid=$fsuuid /var/lib/rpm' And for that matter, it could be state-x86-64:fedora36.0 and .1, mounted at /state. The helper could rewrite /etc/fstab, or develop a "discoverable subvolumes" systemd generator that assembles things per a heuristic. https://lists.freedesktop.org/archives/systemd-devel/2021-November/047063.html It's true that this change proposal doesn't go far enough. The issues with /var go beyond /var/lib/rpm. What about /var/lib/selinux? It's owned by the selinux-policy-targeted package. Even though the files may not change often, it probably needs to be snapshot and rolled back with revision matching for /usr and rpmdb. It could be done with a subvolume or another file system with LVM. There might be 6 of these carve outs. Instead of possibly 6+ file systems for LVM thin setups, there could be two: "system" is subject to snapshots and rollbacks, "variable" is not. The same "discoverable subvolumes" systemd-generator idea proposes working with plain directories using the same naming scheme, and would use bind mounts to just reassemble everything from those two file systems onto the FHS layout. Including read-only /usr. This was brought up on devel@ list a year ago and discussed at length, explains the difficulties with the many carve outs approach (open)SUSE has been using, and the various pros and consequences to go about simplifying it. https://github.com/lnussel/lnussel.github.io/blob/fs/_posts/2020-12-16-fslayout.md > These are valid points. My suggestion of /state (or another named top-level > directory...I saw /meta suggested somewhere, maybe LWN...but I kind of don't > want to imply references to Facebook) is on the basis of RPM operating with a > single database on a given system. The idea of multiple RPM databases is > interesting and I think there's value in discussing that, but that would be a > separate item for discussion. > > Right now, a system needs to support installation from the vendor, third party > repos, and local system packages. All with a single RPM database. So I > suggested /state to get that information out of /var but still keep it tied to > the host in question. I don't think /state is a bad idea. Maybe it suffers the same issue as this change proposal, in that it doesn't go far enough? > The /opt tree introduces another wrinkle. It's one that's never really gained > a lot of use in the Linux world, but where it has it's been treated as both a > place where vendors can deliver software and a separate /usr/local. I think > it's reasonable to assume vendor-delivered things in /opt can be thought of as > more independent than what's in /usr, but that's just an assumption. My most > recent views of things found in /opt from third party vendors shows that they > are self-contained trees built with the mindset of "I don't trust what the > system will provide me". I guess the main issue here for snapshots and rollbacks is, would it be ok to put /opt and /usr on the same lifecycle? They get snapshot and rolled back together? Or should /opt be a 3rd party playground and perhaps not even be subject to snapshots and rollbacks, it's entirely managed by rpm only? I don't know the answer to that. But I kinda like the idea of separate /opt with its own rpmdb. That way 'rm -rf /opt/*' is a valid way to reset the 3rd party location without messing up rpmdb because it too is cleaned up. > To move the RPM database in to /usr for the rpm-ostree snapshot and rollback > case would also imply restrictions around what a user can and can't do with > RPM on their system. If that's the case, then I would say rpm should stop > being a general purpose tool for the user and become a backend only tool run > by the OS. Let the users create new third party packaging systems to install > things on their own system. I'm not sure about this. -- Chris Murphy ___ devel mailing list --
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Wed, Jan 12, 2022 at 9:45 AM Chris Murphy wrote: > > Given the choice, I prefer rpm only touches /usr, which includes > /usr/var and /usr/etc. I must say that, of all the bad suggestions in the multiple threads this proposal has spawned, I dislike using "/usr/var/" the least. The path "/usr/lib/sysimage/rpm" does look very out-of-place in non-image-based systems, so *if* we want to move the rpmdb to a place that's consistent across all our Editions, it should also be a location name that makes sense across all Editions. A path that contains "sysimage" conveys to me that this is somehow only related to image based variants. Something under /usr/var would at least be an generic, non-confusing path. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Thu, Jan 13, 2022 at 8:34 AM David Cantrell wrote: > > On Mon, Jan 10, 2022 at 01:24:20PM -0500, Colin Walters wrote: > > > >No, this is not about developers tar'ing up `/usr` by hand. It's about > >cleanly separating who owns what, and what its lifecycle is, which is > >critcially important for both "image based" updates as well as "local client > >side snapshots". Both these approaches end up creating more than one copy of > >certain files. > > I was trying to give an additional example of what people may do with /usr and > the expectations around it. /usr contains data that comes from the > distribution. We drop packages in place and we get files from that. The RPM > database is generated and updated on the target system and represents the > actions of RPM on that system. This is why I categorize the RPM database > differently than the rest of what is in /usr. Does rpm-ostree has the rpmdb in the wrong place? Or is the OSTree model OK because of where rpmdb was created, or how it's delivered, not because of the intrinsic nature of the rpmdb file? > >On what basis are you making this claim "unnecessary and sort of out of > >place"? > > See above. The RPM database contains data generated by RPM on the system in > question, it's not delivered to the user in a package or some other form. It is delivered to the user in some other form on rpm-ostree systems, not generated on the system in question. Does the location or delivery mechanism of the rpmdb alter the appropriateness of /usr as a location? Why does rpmdb's origin act as a material fact in determining where it should be stored rather than its intrinsic nature? I do not think the FHS helps us resolve where rpmdb goes because the FHS's own history of changes shows how often it made the wrong call. It follows reality rather than leading it. By nature it describes a universe. One system. Not many systems, not a multiverse. FHS: "/usr/lib includes object files and libraries. On some systems, it may also include internal binaries that are not intended to be executed directly by users or shell scripts." It is in effect anything that has to do with system function. There's a boat load of man pages, images, source files, in /usr. Clearly I can add to /usr outside of RPM by compiling and installing the kernel from upstream git. Binaries subsequently appear in /usr as a result of that action, but not in rpmdb. Therefore parity between the binaries in /usr and what's reflected in rpmdb is not an argument for whether a thing is appropriate for /usr or not. Nor where rpmdb is built or how it's delivered. 5.8. /var/lib : Variable state information 5.8.1. Purpose "State information is generally used to preserve the condition of an application (or a group of inter-related applications) between invocations and between different instances of the same application. State information should generally remain valid after a reboot, should not be logging output, and should not be spooled data." The rpmdb is most often *invariant*. Or more correctly, its variance is directly associated with some other variance, like a package being added, modified, or removed. It doesn't get modified as a matter of course like a VM image, or http database, or logs. Since the FHS provides no guidance on /state, I have to agree that /state is a better location than /var on the face of it. Chances are /var was never really a very good location for rpmdb in the first place, but it only becomes obvious upon system snapshots creating a multiverse of systems. There isn't a single system on a system with system snapshots. There are now multiple systems of different revisions. And that suggests multiple rpmdb's of different revisions to describe them. Or perhaps a single rpmdb that is snapshot aware, containing information about all system snapshots. So I think we are better off trying to understand the relative consequences of /state versus /usr for rpmdb than appealing to any higher authority (like the FHS), or feelings. I can't act on either the FHS or feelings, they're both unreliable. > >At least from my PoV, the rpmdb is by default read-only (except to > >rpm-ostree) along with the rest of /usr, so you can't just rm -rf it alone. > > Except removing a package would write that change to the rpmdb. Removing a package that contains files in /usr involves writes to /usr regardless of where rpmdb is located, because any modification of /usr means /usr must be read-write. Add, modify or remove. Even our default mount option, relatime, results in writes to /usr every day just by using the system. > >We have years of investment in rpm-ostree in the current design. For > >example, the tooling supports `rpm-ostree db diff` which shows you the delta > >between the current and pending rpmdb. How would this new directory work? > >How would it better solve problems that are today solved with the location in > >/usr? And do you even have a sense of how much work would creating for
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Tue, Jan 11, 2022 at 11:00:28AM +0200, Panu Matilainen wrote: On 1/10/22 23:53, Chris Murphy wrote: On Mon, Jan 10, 2022 at 9:20 AM David Cantrell wrote: On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr == Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip] Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal. Going back to the original discussions on moving rpmdb... Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html I don't see how /state solves the problem, rather than just rearranging the chairs. The problem with /usr/something is that the rpmdb is not specific to /usr contents at all, and unlike any other content in there, so putting it there just *feels so wrong*. That's what /state or /sysimage or, as we now have, /var supposedly solves. +1 I thought I'd suggested something at / level back then (in https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html and/or https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but seems like memory is failing me :) Maybe I thought it would seem too outrageous to FHS believers to bother. The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense. For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs. Also +1 Just FWIW, I would quit my whining about this right there if it went to a new toplevel directory instead because it just *feels* right unlike /usr. Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place. Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt? If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable. If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help. For one, /state (or whatever toplevel directory) allows for the fact that there are write-operations to rpmdb that do not touch any external files while maintaining read-only /usr. Such as rpmdb --rebuilddb, or rpm --import. And like mentioned in the original discussion and now here, although the discussion is on rpmdb, it's not the only data of this kind. Indeed. We have an opportunity here to get data that fits this categorization in to a more correct location on the system. rpmdb is a big one, but certainly not the only one. To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes? I don't understand the question. Rpm tracks and cares about all content it knows about equally, regardless of the path. /usr is NOT special in any way to rpm, it's just that most of *distro* content ends up in there but a huge number of packages have content spread across /etc too. Yes, the special semantics around /usr is from the rpm-ostree side and not rpm. I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Mon, Jan 10, 2022 at 02:53:52PM -0700, Chris Murphy wrote: On Mon, Jan 10, 2022 at 9:20 AM David Cantrell wrote: On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: >https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr > >== Summary == >Currently, the RPM databases is located in `/var`. Let's move it to >`/usr`. The move is already under way in rpm-ostree-based >installations, and in (open)SUSE. >[snip] Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal. Going back to the original discussions on moving rpmdb... Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html I don't see how /state solves the problem, rather than just rearranging the chairs. I've seen the rearranging the chairs comment in multiple locations now. If the location doesn't matter, why can't the RPM db stay in /var/lib/rpm? Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place. Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt? If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable. If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help. To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes? I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on. These are valid points. My suggestion of /state (or another named top-level directory...I saw /meta suggested somewhere, maybe LWN...but I kind of don't want to imply references to Facebook) is on the basis of RPM operating with a single database on a given system. The idea of multiple RPM databases is interesting and I think there's value in discussing that, but that would be a separate item for discussion. Right now, a system needs to support installation from the vendor, third party repos, and local system packages. All with a single RPM database. So I suggested /state to get that information out of /var but still keep it tied to the host in question. The /opt tree introduces another wrinkle. It's one that's never really gained a lot of use in the Linux world, but where it has it's been treated as both a place where vendors can deliver software and a separate /usr/local. I think it's reasonable to assume vendor-delivered things in /opt can be thought of as more independent than what's in /usr, but that's just an assumption. My most recent views of things found in /opt from third party vendors shows that they are self-contained trees built with the mindset of "I don't trust what the system will provide me". To move the RPM database in to /usr for the rpm-ostree snapshot and rollback case would also imply restrictions around what a user can and can't do with RPM on their system. If that's the case, then I would say rpm should stop being a general purpose tool for the user and become a backend only tool run by the OS. Let the users create new third party packaging systems to install things on their own system. -- David Cantrell Red Hat, Inc. | Boston, MA | EST5EDT ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Mon, Jan 10, 2022 at 01:24:20PM -0500, Colin Walters wrote: On Mon, Jan 10, 2022, at 11:19 AM, David Cantrell wrote: On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr == Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip] Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal. Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. No, this is not about developers tar'ing up `/usr` by hand. It's about cleanly separating who owns what, and what its lifecycle is, which is critcially important for both "image based" updates as well as "local client side snapshots". Both these approaches end up creating more than one copy of certain files. I was trying to give an additional example of what people may do with /usr and the expectations around it. /usr contains data that comes from the distribution. We drop packages in place and we get files from that. The RPM database is generated and updated on the target system and represents the actions of RPM on that system. This is why I categorize the RPM database differently than the rest of what is in /usr. See - https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-hysteresis/ - http://lists.rpm.org/pipermail/rpm-maint/2021-December/018770.html Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place. I struggle to how to even respond to this. Multiple independent groups who *actually work* on image based updates and/or client side snapshots all generally agree that the rpmdb should be in /usr. That's where this Change came in. On what basis are you making this claim "unnecessary and sort of out of place"? See above. The RPM database contains data generated by RPM on the system in question, it's not delivered to the user in a package or some other form. The change proposal is about moving the RPM database to /usr. I am offering my comments on why I think /usr is the incorrect location for this data. "OK, but you can do tar -X" The tar example was simply that, an example. I feel the categorization of system data is more important here. Panu brought this up on the referenced RPM mailing list thread years ago. The original /var location for the RPM database was fine, but we're at a point where we kind of have multiple categories of things historically found in /var. Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine". Yes, we call this "factory reset". https://github.com/coreos/fedora-coreos-tracker/issues/399 I am not sure where the terminology came from, but it is widely used when talking about e.g. mobile phones today. Technically you could remove the RPM database and the system still function, but arguably would still be broken since you really want the RPM database. This use case of removing the RPM database and still having a functioning system is really only useful for data recovery scenarios. You're ultimately going to reinstall. Probably. At least from my PoV, the rpmdb is by default read-only (except to rpm-ostree) along with the rest of /usr, so you can't just rm -rf it alone. Except removing a package would write that change to the rpmdb. So maybe the question is more "what is the correct location for data like the RPM database?" First, how is that data different from the rest of /var? It is host-specific, No. An image based updates model (both ostree and container images) ships a pristine copy that is bit-for-bit the same on every system. Container runtimes tend to make it mutable by default of course, but that doesn't change the fact that it's not by default host specific. We have other Fedora editions besides rpm-ostree. This change affects everyone. I would like to see Fedora introduce a new top-level directory called: /state We have years of investment in rpm-ostree in the current design. For example, the tooling supports `rpm-ostree db diff` which shows you the delta between the current and pending rpmdb. How would this new directory work? How would it better solve problems
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Thu, Jan 13, 2022, at 7:52 AM, Vít Ondruch wrote: > Actually, shouldn't rpm-ostree carry around some copy of the RPM > database, which would describe the state of /usr and once the update is > successful (or snapshot active?), merge it into the main system RPM > database? Apparently, something like this is already happening e.g. for > /etc content if I understand it correctly. Or this is the direction in > which the /boot will be handled eventually. Or possibly it could be just > some linked (possibly just read only) database. I think you are confusing/conflating image based updates and client side snapshots (and relatedly, client side offline updates, which is what https://github.com/OpenSUSE/transactional-update). They are related, but different. It's important to emphasize that rpm-ostree systems are by default, *image based*. For example, every single Fedora CoreOS system running the latest "stable" commit has *bit for bit identical /usr*, including the RPM database in /usr/share/rpm. All SELinux labeling was computed server side. All %post scripts were run on the build server. We perform integration testing on that image before it ever hits any user's disk. See https://github.com/coreos/fedora-coreos-tracker/issues/1066 for example. (This integration testing aspect is not true of Silverblue or IoT, because they just ship what bodhi ships, which is its own big problem) If one chooses to engage client side layering (or overrides), rpm-ostree will (each time an upgrade or change is performed) indeed regenerate the rpm database using the "pristine" base image's version of the rpmdb. What you are talking about seems more related to client side snapshots and/or client side offline updates. So in your phrasing above I'd say something more like "proposed dnf/snapper tooling" or so, and not rpm-ostree which works in a different way. > IOW, imagine, that we still keep the system read/write RPM database in > /var and we teach RPM to use additional database(s). So RPM would know > that there might be some additional database e.g. in /usr/var/ ... The > system database would not know nothing about content of /usr, but once > /usr was mounted, `rpm -q` would provide the information from the linked > RPM database. I am having trouble understanding what you are thinking here. If you are interested in this, I'd recommend taking some time trying out an existing system that is doing some of these things; either an OpenSUSE transactional-update system or an rpm-ostree system for example. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
Dne 12. 01. 22 v 22:05 Chris Murphy napsal(a): On Wed, Jan 12, 2022 at 2:04 AM Panu Matilainen wrote: On 1/12/22 10:45, Chris Murphy wrote: On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen wrote: For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs. If rpm is constrained to /usr, then its database can properly be somewhere in /usr If rpm is unconstrained, with transactions touching any of /usr /var /opt /etc, the resulting paradigm is inherently complicated, and an additional top-level directory doesn't help make it less complicated. WTF is this talk about constraining? There's no such constraint in rpm, and never will. I'm not talking about changing the nature of the beast. I'm talking about how it's used as a tool in a distribution with a lot of other requirements and tradeoffs. Expanding rpm is also a valid path. We have an example "DNF snapper plug-in" that demonstrates at least DNF *could* get into the business of managing snapshots and rollbacks, if it wants that responsibility, if it's helpful and worth the effort, and if the resources exist to do it. That's an interesting discussion. And Fedora packages are not constrained there either, a vast percentage places stuff in /etc, touches directory tree in /var and then there's /boot and the software collections and third party software. /boot might need its own manager(s). I'm strongly leaning toward bootupd and fwupd being the only things that touch /boot and everything else is proscribed. /boot is a small but hot mess right now. RPM, grub, grubby, dracut, all can touch /boot and in the Boot Loader Spec context there's multiple OS's sharing /boot. About the only thing I like better is linux as the bootloader, e.g. Linux Boot, so we have real device and file system drivers. Anything that manages /boot needs to needs understand snapshots, and which kernel+initramfs can boot which snapshots. Either top-level directories have their own databases, so rpm knows what's in them even if one is rolled back but not another; or there's one database to describe all of these top-level directories which then need to share a single (sub)volume so they're all snapshot and rolled back together. Either way, there's additionally a need for carve outs for things that shouldn't be rolled back. Erm. Packages don't live conveniently inside a single top level tree, they can't be split up that way. Right. I view this as a criticism, not a benefit though. Neither rpm nor the FHS are organizing things in a way that helps us do rollbacks. They're impediments we keep having to take other shortcuts and workarounds to achieve that goal, and as yet they still have too many shortcomings. It really isn't to be underestimated the importance of multiple independent parties coming to the same kinds of conclusions. Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc. But if left unconstrained, having databases inside the directories they describe is more reliable than treating the database as a sidecar file that can much more easily become disconnected with one or more top-level directories it ostensibly describes the contents of. Here seems to be another SMALL undocumented dependency of this change: completing the /usrmove thing to cover the whole world including /opt, /etc, /var, and presumably /boot as well because packages put stuff in it. Rpm wont care if you want to move content from /etc to /usr/etc and ditto for /var, /opt and /boot which is also used by packages, ie to complete to complete the great /usrmove thing started a decade ago. But that's quite a hidden agenda in this change if you ask me. This change proposal is narrow in scope. It imagines a future snapshot+rollback regime of some kind, but isn't depending on any particular file system or way of achieving it. I personally like the transactional updates approach. I like what rpm-ostree is doing, updating the system root out of band rather than updating the currently running system. Actually, shouldn't rpm-ostree carry around some copy of the RPM database, which would describe the state of /usr and once the update is successful (or snapshot active?), merge it into the main system RPM database? Apparently, something like this is already happening e.g. for /etc content if I understand it correctly. Or this is the direction in which the /boot will be handled eventually. Or possibly it could be just some linked (possibly just read only) database. IOW, imagine, that we still keep the system read/write RPM database in /var and we teach RPM to use additional database(s). So RPM would know that there might be some additional database e.g. in /usr/var/ ... The system database would not know nothing about content of /usr, but once /usr was mounted, `rpm -q`
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Wed, Jan 12, 2022 at 2:57 PM Neal Gompa wrote: > > On Wed, Jan 12, 2022 at 3:54 AM Chris Murphy wrote: > > > > On Wed, Jan 12, 2022 at 1:36 AM Panu Matilainen wrote: > > > > > > > So by moving the rpmdb to /usr, it's basically saying that `rpm > > > > --import` should change. > > > > > > This doesn't seem to be documented as a dependency of this move... > > > > > > > Added. > > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Dependencies > > > > No. This is nonsense. In current Fedora, this doesn't matter. In a > hypothetical future model of a transactional system, we'd want to move > repo configs and gpg key files from /etc to /usr at the same time. > > Right now, this is out of scope because the DNF team is not ready for that > work. Ahh yeah fair enough. There are further discussions before we actually get to snapshot and rollbacks, not least of which is things like 'dnf history' can't show you an accurate history if there's been an external (filesystem) rollback of the system root that does not rollback /var/lib/dnf. Ideally we'd never have to rollback anything in /var, but possibly with an option for selective restore. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Wed, Jan 12, 2022 at 3:54 AM Chris Murphy wrote: > > On Wed, Jan 12, 2022 at 1:36 AM Panu Matilainen wrote: > > > > > So by moving the rpmdb to /usr, it's basically saying that `rpm --import` > > > should change. > > > > This doesn't seem to be documented as a dependency of this move... > > > > Added. > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Dependencies > No. This is nonsense. In current Fedora, this doesn't matter. In a hypothetical future model of a transactional system, we'd want to move repo configs and gpg key files from /etc to /usr at the same time. Right now, this is out of scope because the DNF team is not ready for that work. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Wed, Jan 12, 2022 at 2:04 AM Panu Matilainen wrote: > > On 1/12/22 10:45, Chris Murphy wrote: > > On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen wrote: > >> For many practical purposes it's probably just rearranging the chairs, > >> but a separate top-level directory describing the *system* state seems > >> instinctively *much* more correct solution to it than stuffing it > >> somewhere deep inside a loosely related fs. > > > > If rpm is constrained to /usr, then its database can properly be > > somewhere in /usr > > > > If rpm is unconstrained, with transactions touching any of /usr /var > > /opt /etc, the resulting paradigm is inherently complicated, and an > > additional top-level directory doesn't help make it less complicated. > > WTF is this talk about constraining? There's no such constraint in rpm, > and never will. I'm not talking about changing the nature of the beast. I'm talking about how it's used as a tool in a distribution with a lot of other requirements and tradeoffs. Expanding rpm is also a valid path. We have an example "DNF snapper plug-in" that demonstrates at least DNF *could* get into the business of managing snapshots and rollbacks, if it wants that responsibility, if it's helpful and worth the effort, and if the resources exist to do it. That's an interesting discussion. > And Fedora packages are not constrained there either, a > vast percentage places stuff in /etc, touches directory tree in /var and > then there's /boot and the software collections and third party software. /boot might need its own manager(s). I'm strongly leaning toward bootupd and fwupd being the only things that touch /boot and everything else is proscribed. /boot is a small but hot mess right now. RPM, grub, grubby, dracut, all can touch /boot and in the Boot Loader Spec context there's multiple OS's sharing /boot. About the only thing I like better is linux as the bootloader, e.g. Linux Boot, so we have real device and file system drivers. Anything that manages /boot needs to needs understand snapshots, and which kernel+initramfs can boot which snapshots. > > Either top-level directories have their own databases, so rpm knows > > what's in them even if one is rolled back but not another; or there's > > one database to describe all of these top-level directories which then > > need to share a single (sub)volume so they're all snapshot and rolled > > back together. Either way, there's additionally a need for carve outs > > for things that shouldn't be rolled back. > > Erm. Packages don't live conveniently inside a single top level tree, > they can't be split up that way. Right. I view this as a criticism, not a benefit though. Neither rpm nor the FHS are organizing things in a way that helps us do rollbacks. They're impediments we keep having to take other shortcuts and workarounds to achieve that goal, and as yet they still have too many shortcomings. It really isn't to be underestimated the importance of multiple independent parties coming to the same kinds of conclusions. > > Given the choice, I prefer rpm only touches /usr, which includes > > /usr/var and /usr/etc. > > > > But if left unconstrained, having databases inside the directories > > they describe is more reliable than treating the database as a sidecar > > file that can much more easily become disconnected with one or more > > top-level directories it ostensibly describes the contents of. > > Here seems to be another SMALL undocumented dependency of this change: > completing the /usrmove thing to cover the whole world including /opt, > /etc, /var, and presumably /boot as well because packages put stuff in it. > > Rpm wont care if you want to move content from /etc to /usr/etc and > ditto for /var, /opt and /boot which is also used by packages, ie to > complete to complete the great /usrmove thing started a decade ago. But > that's quite a hidden agenda in this change if you ask me. This change proposal is narrow in scope. It imagines a future snapshot+rollback regime of some kind, but isn't depending on any particular file system or way of achieving it. I personally like the transactional updates approach. I like what rpm-ostree is doing, updating the system root out of band rather than updating the currently running system. I like the idea these updates can happen in a container, if they fail for any reason we just remove that bad tree - it's a completely disposable thing until it successfully updates and maybe we can even automate some simple tests (it gets to X milestone in the boot process) before we activate it as the current root. I like the idea of users being able to follow the trail of breadcrumbs of how systems boot. My strong bias from many years in Fedora QA is we deal every cycle with boot/startup failure in some form or another, and the less obscure less fragile more self-describing and standard that process can be, the easier it is to avoid, find and fix problems. We also know that (open)SUSE has proven a
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Wed, Jan 12, 2022, at 4:04 AM, Panu Matilainen wrote: > > Here seems to be another SMALL undocumented dependency of this change: > completing the /usrmove thing to cover the whole world including /opt, > /etc, /var, and presumably /boot as well because packages put stuff in it. There are very few packages that put things in /boot. For example, fwupd used to be in that set, but moved into a "self updater" model where the binary goes in /usr, and then it copies itself into the ESP instead of having yum/rpm do it. Now, rpm-ostree also does this with /boot: https://github.com/coreos/rpm-ostree/blob/210bf148342a9545c9841ae6d8403354ce49b672/src/libpriv/rpmostree-util.cxx#L134 And then we have a sister project https://github.com/coreos/bootupd/ that is only shipped in FCOS today (but would make sense to use on everything that uses rpm-ostree) which is scoped explicitly to only handle stuff on the ESP (and eventually, stuff like grub on the MBR and other architectures too). Correct handling of /boot is obviously essential for transactional updates; ostree is entirely designed around a "strong binding" of (kernel, userspace) pairs. Handling kernels in /boot for "client side snapshots" is something most projects in this space do out of band, as far as I've seen. Again, it's really that 90% of the data in the rpmdb is for /usr. We recently changed the kernel RPM to stick the kernel binary in /usr/lib/modules/$kver and only *copy* it from there to strengthen this model. And this move is pushing things farther along in that direction. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On 1/12/22 10:45, Chris Murphy wrote: On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen wrote: The problem with /usr/something is that the rpmdb is not specific to /usr contents at all, and unlike any other content in there, so putting it there just *feels so wrong*. That's what /state or /sysimage or, as we now have, /var supposedly solves. I thought I'd suggested something at / level back then (in https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html and/or https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but seems like memory is failing me :) Maybe I thought it would seem too outrageous to FHS believers to bother. The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense. For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs. If rpm is constrained to /usr, then its database can properly be somewhere in /usr If rpm is unconstrained, with transactions touching any of /usr /var /opt /etc, the resulting paradigm is inherently complicated, and an additional top-level directory doesn't help make it less complicated. WTF is this talk about constraining? There's no such constraint in rpm, and never will. And Fedora packages are not constrained there either, a vast percentage places stuff in /etc, touches directory tree in /var and then there's /boot and the software collections and third party software. Either top-level directories have their own databases, so rpm knows what's in them even if one is rolled back but not another; or there's one database to describe all of these top-level directories which then need to share a single (sub)volume so they're all snapshot and rolled back together. Either way, there's additionally a need for carve outs for things that shouldn't be rolled back. Erm. Packages don't live conveniently inside a single top level tree, they can't be split up that way. Many if not most span at least /etc and /usr, and /usr may be further split into separate mounts (eg /usr/local or /usr/share). And many things which install to /opt also place things into /usr for desktop integration. We have an example of the latter. (open)SUSE's current layout. There's a 10 line /etc/fstab, 9 of which are various Btrfs subvolumes to achieve the necessary carve out. It's sufficiently complex that the direction they're looking to go in is one in which rpm's only touch /usr. There's /usr/etc for read-only systemd defaults. And local customizations go in /etc. Much like rpm-ostree. Projects within two rpm-based distros independently came to realize that unconstrained rpm is difficult. Where rpm-ostree decided to do the hard work of actually becoming aware of the reality there's multiple system roots. I don't understand the question. Rpm tracks and cares about all content it knows about equally, regardless of the path. /usr is NOT special in any way to rpm, it's just that most of *distro* content ends up in there but a huge number of packages have content spread across /etc too. I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on. Oh, multiple rpmdbs. It's a while since that last turned up. It gets tossed around as a solution but to me it looks like it brings more problems than it solves. Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc. But if left unconstrained, having databases inside the directories they describe is more reliable than treating the database as a sidecar file that can much more easily become disconnected with one or more top-level directories it ostensibly describes the contents of. Here seems to be another SMALL undocumented dependency of this change: completing the /usrmove thing to cover the whole world including /opt, /etc, /var, and presumably /boot as well because packages put stuff in it. Rpm wont care if you want to move content from /etc to /usr/etc and ditto for /var, /opt and /boot which is also used by packages, ie to complete to complete the great /usrmove thing started a decade ago. But that's quite a hidden agenda in this change if you ask me. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Wed, Jan 12, 2022 at 1:36 AM Panu Matilainen wrote: > > > So by moving the rpmdb to /usr, it's basically saying that `rpm --import` > > should change. > > This doesn't seem to be documented as a dependency of this move... > Added. https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Dependencies -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Tue, Jan 11, 2022 at 2:00 AM Panu Matilainen wrote: > The problem with /usr/something is that the rpmdb is not specific to > /usr contents at all, and unlike any other content in there, so putting > it there just *feels so wrong*. That's what /state or /sysimage or, as > we now have, /var supposedly solves. > > I thought I'd suggested something at / level back then (in > https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html > and/or > https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but > seems like memory is failing me :) Maybe I thought it would seem too > outrageous to FHS believers to bother. > > The point was though, that the rpmdb is not at all the only data of this > kind and so having a dedicated home makes sense. > > For many practical purposes it's probably just rearranging the chairs, > but a separate top-level directory describing the *system* state seems > instinctively *much* more correct solution to it than stuffing it > somewhere deep inside a loosely related fs. If rpm is constrained to /usr, then its database can properly be somewhere in /usr If rpm is unconstrained, with transactions touching any of /usr /var /opt /etc, the resulting paradigm is inherently complicated, and an additional top-level directory doesn't help make it less complicated. Either top-level directories have their own databases, so rpm knows what's in them even if one is rolled back but not another; or there's one database to describe all of these top-level directories which then need to share a single (sub)volume so they're all snapshot and rolled back together. Either way, there's additionally a need for carve outs for things that shouldn't be rolled back. We have an example of the latter. (open)SUSE's current layout. There's a 10 line /etc/fstab, 9 of which are various Btrfs subvolumes to achieve the necessary carve out. It's sufficiently complex that the direction they're looking to go in is one in which rpm's only touch /usr. There's /usr/etc for read-only systemd defaults. And local customizations go in /etc. Much like rpm-ostree. Projects within two rpm-based distros independently came to realize that unconstrained rpm is difficult. Where rpm-ostree decided to do the hard work of actually becoming aware of the reality there's multiple system roots. > I don't understand the question. Rpm tracks and cares about all content > it knows about equally, regardless of the path. /usr is NOT special in > any way to rpm, it's just that most of *distro* content ends up in there > but a huge number of packages have content spread across /etc too. > > > I think rpm can't remain > > static for all time. It either needs to become aware of multiple root > > trees, and even mix and match top-level directories to create variable > > roots. And possibly even manage these things. Or it needs to constrain > > its reach to /usr and /opt. Whether /usr and /opt are tied together, > > or can be mix and match with their own rpmdb's, I have no strong > > opinion on. > > Oh, multiple rpmdbs. It's a while since that last turned up. It gets > tossed around as a solution but to me it looks like it brings more > problems than it solves. Given the choice, I prefer rpm only touches /usr, which includes /usr/var and /usr/etc. But if left unconstrained, having databases inside the directories they describe is more reliable than treating the database as a sidecar file that can much more easily become disconnected with one or more top-level directories it ostensibly describes the contents of. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On 1/11/22 17:18, Colin Walters wrote: On Tue, Jan 11, 2022, at 4:00 AM, Panu Matilainen wrote: The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense. You mentioned dnf/yum/PackageKit data; there's two kinds of that. One is e.g. /var/cache/dnf which does *not* move. It's just a cache. (Now there is this whole other very interesting thread around "why don't we include a cache of the rpm-md inside e.g. ostree or container images? And the reason why not is we don't want to have to respin all images every time a package not in the image changes) dnf does have its own outside-of-rpm state database in /var/lib/dnf which is closer to this. The situation with that is messier. AIUI this proposal so far is not calling for moving this. Where it lives and how it's versioned has strong implications for the people who want to support snapshot/rollback. But it's not relevant for rpm-ostree, which does not use this part of libdnf. We maintain our own little "state file" - for lots more detail on this, see https://github.com/coreos/rpm-ostree/issues/2326 (And it's important to note that rpm-ostree's origin file has almost nothing *unless* one explicitly engages client-side layering/overrides) For one, /state (or whatever toplevel directory) allows for the fact that there are write-operations to rpmdb that do not touch any external files while maintaining read-only /usr. Such as rpmdb --rebuilddb, `rpmdb --rebuilddb` is basically about supporting switching from e.g. bdb to sqlite, right? On the rpm-ostree side at least, the default format comes from the base image; there's no reason to directly support `rpmdb --rebuilddb` as something any normal user or admin would need. No, it's a database maintenance/repair operation. Converting between formats is merely a convenient side-effect. or rpm --import. OK yes, this (along with rpms that install into /opt as you mentioned) are I think the biggest examples of "rpmdb has stuff not about /usr". rpm --import actually encapsulates really well all the problems and benefits with everything we're trying to do. I have a big related blurb here https://github.com/rpm-software-management/libdnf/issues/43 But as I understand it, the creation of /etc/pki/rpm-gpg was at least partially motivated by the fact that in the traditional RPM model, the fact that GPG keys are stored in the rpmdb meant there was no way to update them "in band" of a default rpm operation. Today Fedora ships these GPG keys as an RPM which hence contain files, and when you type `yum upgrade` (or rpm-ostree upgrade) you get updates to those files, the same as other files. Now, as noted in the issue PackageKit (whose code was the precursor to libdnf, which has the code that rpm-ostree uses but AFAIK still not current dnf which has this logic in Python) auto-imports all of them. I am not completely sure how updating the rpmdb with new e.g. Fedora GPG key works. It might be part of system-upgrade or something? And then this all relates to a higher level goal we have with "image based updates", which is avoiding (or minimizing as much as possible) per-system hysteresis or "unmanaged state", particularly opaque (hard to see/diagnose/inspect) state. This set of trusted GPG keys stored in the rpmdb is both. The set of keys will just keep growing across in-place upgrades, depending on the initial Fedora version installed. And wh And this is security-relevant state, it's the set of trusted keys for RPM. Now, I am sure a number of sysadmins do understand that the rpmdb contains GPG keys. I'd bet a whole lot don't. I definitely think that it's confusing to have both /etc/pki/rpm-gpg *and* keys stored in the rpmdb. Of the two, I think the former - i.e. text files one can edit with standard tools - is much easier to understand and work with. It's also already in a place that is designed for users to edit in `/etc`. So by moving the rpmdb to /usr, it's basically saying that `rpm --import` should change. This doesn't seem to be documented as a dependency of this move... That said, this design could also clearly use some "systemd style" config model. ostree already supports /usr/share/ostree/trusted.gpg.d which is designed for keys shipped by the OS vendor. But, what's really tricky about this is we want to support in-place updates from previous versions (i.e. at least N-1), but hopefully not too old versions. Well, this is leading to a tangent so I'll cut off this sub-thread. The TL;DR for me is: I think everyone agrees that moving the rpmdb as it is today to /usr is not 100% a perfect fit. But it's a ~90% fit - almost all the raw data is just headers which are clearly immutable data generated elsewhere. And by making this change, we're basically saying we'll fix the remaining 10% of cases. Note for the people who are trying to do "client side snapshot/rollback" (i.e.
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Tue, Jan 11, 2022, at 4:00 AM, Panu Matilainen wrote: > The point was though, that the rpmdb is not at all the only data of this > kind and so having a dedicated home makes sense. You mentioned dnf/yum/PackageKit data; there's two kinds of that. One is e.g. /var/cache/dnf which does *not* move. It's just a cache. (Now there is this whole other very interesting thread around "why don't we include a cache of the rpm-md inside e.g. ostree or container images? And the reason why not is we don't want to have to respin all images every time a package not in the image changes) dnf does have its own outside-of-rpm state database in /var/lib/dnf which is closer to this. The situation with that is messier. AIUI this proposal so far is not calling for moving this. Where it lives and how it's versioned has strong implications for the people who want to support snapshot/rollback. But it's not relevant for rpm-ostree, which does not use this part of libdnf. We maintain our own little "state file" - for lots more detail on this, see https://github.com/coreos/rpm-ostree/issues/2326 (And it's important to note that rpm-ostree's origin file has almost nothing *unless* one explicitly engages client-side layering/overrides) > For one, /state (or whatever toplevel directory) allows for the fact > that there are write-operations to rpmdb that do not touch any external > files while maintaining read-only /usr. Such as rpmdb --rebuilddb, `rpmdb --rebuilddb` is basically about supporting switching from e.g. bdb to sqlite, right? On the rpm-ostree side at least, the default format comes from the base image; there's no reason to directly support `rpmdb --rebuilddb` as something any normal user or admin would need. > or rpm --import. OK yes, this (along with rpms that install into /opt as you mentioned) are I think the biggest examples of "rpmdb has stuff not about /usr". rpm --import actually encapsulates really well all the problems and benefits with everything we're trying to do. I have a big related blurb here https://github.com/rpm-software-management/libdnf/issues/43 But as I understand it, the creation of /etc/pki/rpm-gpg was at least partially motivated by the fact that in the traditional RPM model, the fact that GPG keys are stored in the rpmdb meant there was no way to update them "in band" of a default rpm operation. Today Fedora ships these GPG keys as an RPM which hence contain files, and when you type `yum upgrade` (or rpm-ostree upgrade) you get updates to those files, the same as other files. Now, as noted in the issue PackageKit (whose code was the precursor to libdnf, which has the code that rpm-ostree uses but AFAIK still not current dnf which has this logic in Python) auto-imports all of them. I am not completely sure how updating the rpmdb with new e.g. Fedora GPG key works. It might be part of system-upgrade or something? And then this all relates to a higher level goal we have with "image based updates", which is avoiding (or minimizing as much as possible) per-system hysteresis or "unmanaged state", particularly opaque (hard to see/diagnose/inspect) state. This set of trusted GPG keys stored in the rpmdb is both. The set of keys will just keep growing across in-place upgrades, depending on the initial Fedora version installed. And wh And this is security-relevant state, it's the set of trusted keys for RPM. Now, I am sure a number of sysadmins do understand that the rpmdb contains GPG keys. I'd bet a whole lot don't. I definitely think that it's confusing to have both /etc/pki/rpm-gpg *and* keys stored in the rpmdb. Of the two, I think the former - i.e. text files one can edit with standard tools - is much easier to understand and work with. It's also already in a place that is designed for users to edit in `/etc`. So by moving the rpmdb to /usr, it's basically saying that `rpm --import` should change. That said, this design could also clearly use some "systemd style" config model. ostree already supports /usr/share/ostree/trusted.gpg.d which is designed for keys shipped by the OS vendor. But, what's really tricky about this is we want to support in-place updates from previous versions (i.e. at least N-1), but hopefully not too old versions. Well, this is leading to a tangent so I'll cut off this sub-thread. The TL;DR for me is: I think everyone agrees that moving the rpmdb as it is today to /usr is not 100% a perfect fit. But it's a ~90% fit - almost all the raw data is just headers which are clearly immutable data generated elsewhere. And by making this change, we're basically saying we'll fix the remaining 10% of cases. Note for the people who are trying to do "client side snapshot/rollback" (i.e. the people whose names are attached to the Change), the rpmdb is still mutable by default. And I think their idea is is that by doing a "snapshot, then upgrade in place via traditional rpm/yum methods"
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On 1/10/22 23:53, Chris Murphy wrote: On Mon, Jan 10, 2022 at 9:20 AM David Cantrell wrote: On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr == Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip] Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal. Going back to the original discussions on moving rpmdb... Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html I don't see how /state solves the problem, rather than just rearranging the chairs. The problem with /usr/something is that the rpmdb is not specific to /usr contents at all, and unlike any other content in there, so putting it there just *feels so wrong*. That's what /state or /sysimage or, as we now have, /var supposedly solves. I thought I'd suggested something at / level back then (in https://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html and/or https://lists.rpm.org/pipermail/rpm-maint/2017-October/006699.html) but seems like memory is failing me :) Maybe I thought it would seem too outrageous to FHS believers to bother. The point was though, that the rpmdb is not at all the only data of this kind and so having a dedicated home makes sense. For many practical purposes it's probably just rearranging the chairs, but a separate top-level directory describing the *system* state seems instinctively *much* more correct solution to it than stuffing it somewhere deep inside a loosely related fs. Just FWIW, I would quit my whining about this right there if it went to a new toplevel directory instead because it just *feels* right unlike /usr. Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place. Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt? If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable. If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help. For one, /state (or whatever toplevel directory) allows for the fact that there are write-operations to rpmdb that do not touch any external files while maintaining read-only /usr. Such as rpmdb --rebuilddb, or rpm --import. And like mentioned in the original discussion and now here, although the discussion is on rpmdb, it's not the only data of this kind. To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes? I don't understand the question. Rpm tracks and cares about all content it knows about equally, regardless of the path. /usr is NOT special in any way to rpm, it's just that most of *distro* content ends up in there but a huge number of packages have content spread across /etc too. I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on. Oh, multiple rpmdbs. It's a while since that last turned up. It gets tossed around as a solution but to me it looks like it brings more problems than it solves. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Mon, Jan 10, 2022 at 9:20 AM David Cantrell wrote: > > On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: > >https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr > > > >== Summary == > >Currently, the RPM databases is located in `/var`. Let's move it to > >`/usr`. The move is already under way in rpm-ostree-based > >installations, and in (open)SUSE. > >[snip] > > Moving the RPM database to /usr feels incorrect to me, but we should move it > to gain the improvements as noted in the feature proposal. Going back to the original discussions on moving rpmdb... Preferred is a new top-level location in /usr, .e.g /usr/sysimage/rpm. Next is "least worst" in /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006764.html http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html And the convergence was on /usr/lib/sysimage/rpm http://lists.rpm.org/pipermail/rpm-maint/2017-October/006785.html I don't see how /state solves the problem, rather than just rearranging the chairs. > Numerous followups have noted the requirement that /usr contain read-only > content, that it be shareable across hosts, and similar concepts. While this > may or may not doable now like we could in the past, the bigger thing to me is > around the understanding of what /usr contains. It is generally understood > that /usr contains [most of] the installed system. What I think is a bigger > requirement or expection now is that one can tar up /usr and transport it to > another system or virtual machine or container and expect that it will > _probably_ work maybe with a bit of tinkering. This is a really valuable > thing to have for developers. Moving the RPM database to this tree adds a > component that is unnecessary and sort of out of place. Should /usr be independently portable? And is that with a version matched /opt, or can there be mix and match revisions of /usr and /opt? If /usr is to be truly portable and have e.g. 'rpm query, verify, remove, reinstall' work as expected, you need the metadata (the database) representing its state to always come along for the ride. Either the database is already in /usr, or you have to make sure /usr and /state are inseparable. If /usr and /state are inseparable, and if rpm can also describe anything in /etc or /var or /opt, then all or part of those directories are also inseparable from /state. And thus /usr. So I think /state doesn't help. To what degree do rpm and dnf intend to touch locations outside of /usr *and care* about tracking those changes? I think rpm can't remain static for all time. It either needs to become aware of multiple root trees, and even mix and match top-level directories to create variable roots. And possibly even manage these things. Or it needs to constrain its reach to /usr and /opt. Whether /usr and /opt are tied together, or can be mix and match with their own rpmdb's, I have no strong opinion on. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
> Am 10.01.2022 um 18:57 schrieb Alexander Sosedkin : > > On Mon, Jan 10, 2022 at 5:20 PM David Cantrell wrote: >> >> On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: >>> https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr >>> >>> == Summary == >>> Currently, the RPM databases is located in `/var`. Let's move it to >>> `/usr`. The move is already under way in rpm-ostree-based >>> installations, and in (open)SUSE. >>> [snip] >> ... > > The whole proposal is kinda sad to read. > It's 2022 and we're catering to filesystem-level rollbacks. > Filesystem rollbacks *are* a gigantic unsubtle hammer > that should not be used in an automated manner, > or better yet, not used at all. > As much as you don't do filesystem rollbacks > to undo paragraph deletions in Libreoffice Writer, > you shouldn't do filesystem rollbacks to fix your system. > That's your package manager's / configuration manager's job. Well, os-tree is doing filesystem rollback / -rollforward by architecture. If you want to feed rpm into it, it may be tricky. But this argues for just moving rpmdb for rpm-oss-tree, since os-tree has its own logic anyway. No need and no advantage to do it for non-rpm-os-tree systems as well. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Mon, Jan 10, 2022, at 11:19 AM, David Cantrell wrote: > On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: >>https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr >> >>== Summary == >>Currently, the RPM databases is located in `/var`. Let's move it to >>`/usr`. The move is already under way in rpm-ostree-based >>installations, and in (open)SUSE. >>[snip] > > Moving the RPM database to /usr feels incorrect to me, but we should move it > to gain the improvements as noted in the feature proposal. > > Numerous followups have noted the requirement that /usr contain read-only > content, that it be shareable across hosts, and similar concepts. While this > may or may not doable now like we could in the past, the bigger thing to me is > around the understanding of what /usr contains. It is generally understood > that /usr contains [most of] the installed system. What I think is a bigger > requirement or expection now is that one can tar up /usr and transport it to > another system or virtual machine or container and expect that it will > _probably_ work maybe with a bit of tinkering. This is a really valuable > thing to have for developers. No, this is not about developers tar'ing up `/usr` by hand. It's about cleanly separating who owns what, and what its lifecycle is, which is critcially important for both "image based" updates as well as "local client side snapshots". Both these approaches end up creating more than one copy of certain files. See - https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-hysteresis/ - http://lists.rpm.org/pipermail/rpm-maint/2021-December/018770.html > Moving the RPM database to this tree adds a > component that is unnecessary and sort of out of place. I struggle to how to even respond to this. Multiple independent groups who *actually work* on image based updates and/or client side snapshots all generally agree that the rpmdb should be in /usr. That's where this Change came in. On what basis are you making this claim "unnecessary and sort of out of place"? > > "OK, but you can do tar -X" > > The tar example was simply that, an example. I feel the categorization of > system data is more important here. Panu brought this up on the referenced > RPM mailing list thread years ago. The original /var location for the RPM > database was fine, but we're at a point where we kind of have multiple > categories of things historically found in /var. > > Reading comments and talking to people, the long standing understanding of > /var is still "that's stuff you can rm -rf and the system will still work > fine". Yes, we call this "factory reset". https://github.com/coreos/fedora-coreos-tracker/issues/399 I am not sure where the terminology came from, but it is widely used when talking about e.g. mobile phones today. > Technically you could remove the RPM database and the system still > function, but arguably would still be broken since you really want the RPM > database. This use case of removing the RPM database and still having a > functioning system is really only useful for data recovery scenarios. You're > ultimately going to reinstall. Probably. At least from my PoV, the rpmdb is by default read-only (except to rpm-ostree) along with the rest of /usr, so you can't just rm -rf it alone. > So maybe the question is more "what is the correct location for data like the > RPM database?" First, how is that data different from the rest of /var? It > is host-specific, No. An image based updates model (both ostree and container images) ships a pristine copy that is bit-for-bit the same on every system. Container runtimes tend to make it mutable by default of course, but that doesn't change the fact that it's not by default host specific. > I would like to see Fedora introduce a new top-level directory called: > > /state We have years of investment in rpm-ostree in the current design. For example, the tooling supports `rpm-ostree db diff` which shows you the delta between the current and pending rpmdb. How would this new directory work? How would it better solve problems that are today solved with the location in /usr? And do you even have a sense of how much work would creating for the rpm-ostree stack at least with a new toplevel directory with new, as yet ill-defined semantics? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Mon, Jan 10, 2022 at 5:20 PM David Cantrell wrote: > > On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: > >https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr > > > >== Summary == > >Currently, the RPM databases is located in `/var`. Let's move it to > >`/usr`. The move is already under way in rpm-ostree-based > >installations, and in (open)SUSE. > >[snip] > > Moving the RPM database to /usr feels incorrect to me, but we should move it > to gain the improvements as noted in the feature proposal. > > Numerous followups have noted the requirement that /usr contain read-only > content, that it be shareable across hosts, and similar concepts. While this > may or may not doable now like we could in the past, the bigger thing to me is > around the understanding of what /usr contains. It is generally understood > that /usr contains [most of] the installed system. What I think is a bigger > requirement or expection now is that one can tar up /usr and transport it to > another system or virtual machine or container and expect that it will > _probably_ work maybe with a bit of tinkering. This is a really valuable > thing to have for developers. Moving the RPM database to this tree adds a > component that is unnecessary and sort of out of place. > > "OK, but you can do tar -X" > > The tar example was simply that, an example. I feel the categorization of > system data is more important here. Panu brought this up on the referenced > RPM mailing list thread years ago. The original /var location for the RPM > database was fine, but we're at a point where we kind of have multiple > categories of things historically found in /var. > > Reading comments and talking to people, the long standing understanding of > /var is still "that's stuff you can rm -rf and the system will still work > fine". That's one interesting definition of "fine" if it's OK with wiping the system's (arguably) *only* non-disposable directory. > Technically you could remove the RPM database and the system still > function, but arguably would still be broken since you really want the RPM > database. This use case of removing the RPM database and still having a > functioning system is really only useful for data recovery scenarios. You're > ultimately going to reinstall. Probably. > > So maybe the question is more "what is the correct location for data like the > RPM database?" First, how is that data different from the rest of /var? It > is host-specific, it's updated by tools we run all the time, and it's > stateful. Losing it would render the system not really useful, though the > system would still function. Am I missing anything here? > > As Lennart noted, we have lots of examples in /usr of changing data. But I'd > say the difference between those examples (library symlinks, cache files, etc) > and the RPM database is that the loss of something like a library symlink or > cache file can be repaired easily but if you lose the RPM database, the system > is in an unrepairable state. > > As another example... If you use Bitlbee, this would be like losing your > account XML file in /var/lib/bitlbee. Sure, bitlbee still works, but that > file is important for Bitlbee to work for you. You have to remember to hang > on to that if you wipe /var or reinstall or whatever. I'd say the Bitlbee > files in /var/lib/bitlbee also fit this slightly different /var data > definition, just as an example. > > "But what about the FHS?" > > Ah, yes, the FHS. So, I am a fan of the FHS. I actually don't care that it > doesn't change every week and is relatively stable. Nothing in the FHS > prevents the addition of new top level directories. > > I would prefer we steer this conversation in the direction of determining a > new top level location to store data that fits this category of "stateful but > variable". > > /srv was introduced to provide a consistent location for data in this category > for server daemons (except mail). > > /media was introduced to provide a consistent location for removable media > mount points since distributions all did things slightly differently. > > /run was introduced for what was traditionally in /var/run. > > "So what are you suggesting?" > > I would like to see Fedora introduce a new top-level directory called: > > /state > > That holds the RPM database and other variable and stateful data. This keeps > it out of the /usr tree and can serve as a location for future data in this > category. If I ever see a system with /state I'll automatically assume this is the only location that's not wiped on a reboot, invented because the admin gave up on denylisting useless stuff in /var. The whole proposal is kinda sad to read. It's 2022 and we're catering to filesystem-level rollbacks. Filesystem rollbacks *are* a gigantic unsubtle hammer that should not be used in an automated manner, or better yet, not used at all. As much as you don't do filesystem rollbacks to undo paragraph deletions in
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Mon, Jan 10 2022 at 11:19:40 AM -0500, David Cantrell wrote: I would like to see Fedora introduce a new top-level directory called: /state Huh, a new top-level directory is a pretty big hammer. Seems like it would be a lot easier to just have two different locations for the rpmdb if non-ostree users really do not want it under /usr? Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
> Am 10.01.2022 um 17:19 schrieb David Cantrell : > > On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: >> https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr >> >> == Summary == >> Currently, the RPM databases is located in `/var`. Let's move it to >> `/usr`. The move is already under way in rpm-ostree-based >> installations, and in (open)SUSE. >> [snip] > > Moving the RPM database to /usr feels incorrect to me, but we should move it > to gain the improvements as noted in the feature proposal. > > ... > "But what about the FHS?" > > Ah, yes, the FHS. So, I am a fan of the FHS. I actually don't care that it > doesn't change every week and is relatively stable. Nothing in the FHS > prevents the addition of new top level directories. > > I would prefer we steer this conversation in the direction of determining a > new top level location to store data that fits this category of "stateful but > variable". > > /srv was introduced to provide a consistent location for data in this category > for server daemons (except mail). > > /media was introduced to provide a consistent location for removable media > mount points since distributions all did things slightly differently. > > /run was introduced for what was traditionally in /var/run. > > "So what are you suggesting?" > > I would like to see Fedora introduce a new top-level directory called: > >/state > > That holds the RPM database and other variable and stateful data. This keeps > it out of the /usr tree and can serve as a location for future data in this > category. I would like to second David Cantrell's suggestion. For Fedora Server it seems to me essential to strictly follow FHS. And from an "enterprise" perspective, it is essential that FHS indeed does not change every week. A /state directory is not included in FHS, sure. But at least it does not blatantly violate existing and accepted regulations. And the fact that rpm-ostree based installations have already made this change is not a relevant argument for FHS based installations. Best Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
New top-level dir: /state [WAS: Re: F36 Change: Relocate RPM database to /usr (System-Wide Change] proposal)
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr == Summary == Currently, the RPM databases is located in `/var`. Let's move it to `/usr`. The move is already under way in rpm-ostree-based installations, and in (open)SUSE. [snip] Moving the RPM database to /usr feels incorrect to me, but we should move it to gain the improvements as noted in the feature proposal. Numerous followups have noted the requirement that /usr contain read-only content, that it be shareable across hosts, and similar concepts. While this may or may not doable now like we could in the past, the bigger thing to me is around the understanding of what /usr contains. It is generally understood that /usr contains [most of] the installed system. What I think is a bigger requirement or expection now is that one can tar up /usr and transport it to another system or virtual machine or container and expect that it will _probably_ work maybe with a bit of tinkering. This is a really valuable thing to have for developers. Moving the RPM database to this tree adds a component that is unnecessary and sort of out of place. "OK, but you can do tar -X" The tar example was simply that, an example. I feel the categorization of system data is more important here. Panu brought this up on the referenced RPM mailing list thread years ago. The original /var location for the RPM database was fine, but we're at a point where we kind of have multiple categories of things historically found in /var. Reading comments and talking to people, the long standing understanding of /var is still "that's stuff you can rm -rf and the system will still work fine". Technically you could remove the RPM database and the system still function, but arguably would still be broken since you really want the RPM database. This use case of removing the RPM database and still having a functioning system is really only useful for data recovery scenarios. You're ultimately going to reinstall. Probably. So maybe the question is more "what is the correct location for data like the RPM database?" First, how is that data different from the rest of /var? It is host-specific, it's updated by tools we run all the time, and it's stateful. Losing it would render the system not really useful, though the system would still function. Am I missing anything here? As Lennart noted, we have lots of examples in /usr of changing data. But I'd say the difference between those examples (library symlinks, cache files, etc) and the RPM database is that the loss of something like a library symlink or cache file can be repaired easily but if you lose the RPM database, the system is in an unrepairable state. As another example... If you use Bitlbee, this would be like losing your account XML file in /var/lib/bitlbee. Sure, bitlbee still works, but that file is important for Bitlbee to work for you. You have to remember to hang on to that if you wipe /var or reinstall or whatever. I'd say the Bitlbee files in /var/lib/bitlbee also fit this slightly different /var data definition, just as an example. "But what about the FHS?" Ah, yes, the FHS. So, I am a fan of the FHS. I actually don't care that it doesn't change every week and is relatively stable. Nothing in the FHS prevents the addition of new top level directories. I would prefer we steer this conversation in the direction of determining a new top level location to store data that fits this category of "stateful but variable". /srv was introduced to provide a consistent location for data in this category for server daemons (except mail). /media was introduced to provide a consistent location for removable media mount points since distributions all did things slightly differently. /run was introduced for what was traditionally in /var/run. "So what are you suggesting?" I would like to see Fedora introduce a new top-level directory called: /state That holds the RPM database and other variable and stateful data. This keeps it out of the /usr tree and can serve as a location for future data in this category. -- David Cantrell Red Hat, Inc. | Boston, MA | EST5EDT ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
That addresses my concern. I'd assumed that it would occur as part of an RPM update, rather than as a triggered out of band event. It makes rebooting mandatory and slightly risky, especially if /usr overflows halfway through, the process, but that becomes far, far less likely with systemd doing it at boot time running at the same time other RPM updates. I do hope the scripting will exit gracefully if /usr overflows or is unwriteable for other reasons, such as being part of the "immutable" Linux concept. Nico Kadel-Garcia On Sat, Jan 8, 2022 at 8:30 PM Chris Murphy wrote: > > On Sat, Jan 8, 2022 at 6:07 PM Nico Kadel-Garcia wrote: > > > > On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy wrote: > > > > > > On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini > > > wrote: > > > > > > > > On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa wrote: > > > > > > > > > > On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini > > > > > wrote: > > > > > > > > > > > > On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton > > > > > > wrote: > > > > > > > > > > > > > > > > > > > (snip) > > > > > > > > > > > > > > > > > > > > == Detailed Description == > > > > > > > === Current location === > > > > > > > /var/lib/rpm > > > > > > > > > > > > > > === New location === > > > > > > > /usr/lib/sysimage/rpm > > > > > > > > > > > > > > /var/lib/rpm will be a symlink pointing to > > > > > > > /usr/lib/sysimage/rpm > > > > > > > > > > > > I did not find a mention of this in the thread or in the Change > > > > > > proposal, so I'll ask: > > > > > > How do you plan to handle the directory -> symlink replacement on > > > > > > upgrade? > > > > > > > > > > > > As far as I can tell, those always required special treatment via > > > > > > %pretrans scriptlets or something, and even the method currently > > > > > > recommended by the Packaging Guidelines seems to be broken due to > > > > > > the > > > > > > way dnf / RPM verifies validity of transactions. > > > > > > > > > > > > Additionally, that "special" handling will probably need to stay in > > > > > > the RPM package's .spec file for years, given that upgrades from > > > > > > Fedora 34 to 36 will need to be supported. > > > > > > > > > > > > > > > > This is documented in the Change: > > > > > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibility_impact > > > > > > > > > > The part that's probably missing there is that the upgraded package > > > > > will release ownership of /var/lib/rpm and own the > > > > > /usr/lib/sysimage/rpm directory. > > > > > > > > > > The configuration will change in the upgrade, and then an rpmdb > > > > > rebuild on reboot will finalize the transition, as rpmdb rebuilds are > > > > > done by loading the rpmdb in memory, renaming the directory, > > > > > recreating it, and re-initializing the database files from memory. > > > > > > > > > > This avoids the pitfalls you've described with the pretrans stuff. > > > > > > > > Oh, great. Looks like I'm ~tired~ and missed that this is in the > > > > Change proposal after all ... > > > > Thanks for confirming that you found a way to handle upgrades. > > > > > > I did a manual proof of concept with the pseudo-sequence in the change > > > proposal on a Fedora Rawhide VM. And it did work, and continues to do > > > both GNOME Software and dnf updates OK. This is a sample size of 1, so > > > there's more work to do to make sure it can be done safely, using the > > > rpm sqlite upgrade as a guide. But I can write up the steps I used, so > > > that anyone can test before we have it wired up. > > > > > > We have considered not applying the change to upgrades. Strictly > > > speaking the release criterion say we only support upgrades from > > > *clean installs* of the current two releases. But in practice quite a > > > lot of users depend on reliable upgrades for *many* releases, and they > > > get mad when things break even when it's been 10+ releases since they > > > did a clean install. And also, Workstation edition PRD "upgrade > > > process should give a result that is the same as an original install". > > > That is a tall order, but so long as it's safe, it's probably better > > > to apply this change to upgrades. If we run into issues or establish > > > the risk is too high, it's not such a big deal to apply the change to > > > only new clean installs. > > > > I'm personally concerned that the RPM transactions may not be handled > > atomically if the /var/lib/rpm/ is migrated in the midst of an RPM > > update. I'm not personally sure if RPM and Berkeley DB will handle > > that correctly if there's any issues with the migration, particularly > > if /usr/ overflows in the process of other bulky updates such as > > libreoffice. Part of my work involves looking for where multiple > > things can go wrong at once. > > Sqlite applies here, not bdb. > > But also note from that change, the rpmdb conversion to sqlite was not > done during the system upgrade, but by a systemd unit. >
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sat, Jan 8, 2022 at 6:07 PM Nico Kadel-Garcia wrote: > > On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy wrote: > > > > On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini wrote: > > > > > > On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa wrote: > > > > > > > > On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini > > > > wrote: > > > > > > > > > > On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton wrote: > > > > > > > > > > > > > > > > (snip) > > > > > > > > > > > > > > > > > == Detailed Description == > > > > > > === Current location === > > > > > > /var/lib/rpm > > > > > > > > > > > > === New location === > > > > > > /usr/lib/sysimage/rpm > > > > > > > > > > > > /var/lib/rpm will be a symlink pointing to > > > > > > /usr/lib/sysimage/rpm > > > > > > > > > > I did not find a mention of this in the thread or in the Change > > > > > proposal, so I'll ask: > > > > > How do you plan to handle the directory -> symlink replacement on > > > > > upgrade? > > > > > > > > > > As far as I can tell, those always required special treatment via > > > > > %pretrans scriptlets or something, and even the method currently > > > > > recommended by the Packaging Guidelines seems to be broken due to the > > > > > way dnf / RPM verifies validity of transactions. > > > > > > > > > > Additionally, that "special" handling will probably need to stay in > > > > > the RPM package's .spec file for years, given that upgrades from > > > > > Fedora 34 to 36 will need to be supported. > > > > > > > > > > > > > This is documented in the Change: > > > > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibility_impact > > > > > > > > The part that's probably missing there is that the upgraded package > > > > will release ownership of /var/lib/rpm and own the > > > > /usr/lib/sysimage/rpm directory. > > > > > > > > The configuration will change in the upgrade, and then an rpmdb > > > > rebuild on reboot will finalize the transition, as rpmdb rebuilds are > > > > done by loading the rpmdb in memory, renaming the directory, > > > > recreating it, and re-initializing the database files from memory. > > > > > > > > This avoids the pitfalls you've described with the pretrans stuff. > > > > > > Oh, great. Looks like I'm ~tired~ and missed that this is in the > > > Change proposal after all ... > > > Thanks for confirming that you found a way to handle upgrades. > > > > I did a manual proof of concept with the pseudo-sequence in the change > > proposal on a Fedora Rawhide VM. And it did work, and continues to do > > both GNOME Software and dnf updates OK. This is a sample size of 1, so > > there's more work to do to make sure it can be done safely, using the > > rpm sqlite upgrade as a guide. But I can write up the steps I used, so > > that anyone can test before we have it wired up. > > > > We have considered not applying the change to upgrades. Strictly > > speaking the release criterion say we only support upgrades from > > *clean installs* of the current two releases. But in practice quite a > > lot of users depend on reliable upgrades for *many* releases, and they > > get mad when things break even when it's been 10+ releases since they > > did a clean install. And also, Workstation edition PRD "upgrade > > process should give a result that is the same as an original install". > > That is a tall order, but so long as it's safe, it's probably better > > to apply this change to upgrades. If we run into issues or establish > > the risk is too high, it's not such a big deal to apply the change to > > only new clean installs. > > I'm personally concerned that the RPM transactions may not be handled > atomically if the /var/lib/rpm/ is migrated in the midst of an RPM > update. I'm not personally sure if RPM and Berkeley DB will handle > that correctly if there's any issues with the migration, particularly > if /usr/ overflows in the process of other bulky updates such as > libreoffice. Part of my work involves looking for where multiple > things can go wrong at once. Sqlite applies here, not bdb. But also note from that change, the rpmdb conversion to sqlite was not done during the system upgrade, but by a systemd unit. https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb Same for this change proposal. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sat, Jan 8, 2022 at 8:07 PM Nico Kadel-Garcia wrote: > > On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy wrote: > > > > On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini wrote: > > > > > > On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa wrote: > > > > > > > > On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini > > > > wrote: > > > > > > > > > > On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton wrote: > > > > > > > > > > > > > > > > (snip) > > > > > > > > > > > > > > > > > == Detailed Description == > > > > > > === Current location === > > > > > > /var/lib/rpm > > > > > > > > > > > > === New location === > > > > > > /usr/lib/sysimage/rpm > > > > > > > > > > > > /var/lib/rpm will be a symlink pointing to > > > > > > /usr/lib/sysimage/rpm > > > > > > > > > > I did not find a mention of this in the thread or in the Change > > > > > proposal, so I'll ask: > > > > > How do you plan to handle the directory -> symlink replacement on > > > > > upgrade? > > > > > > > > > > As far as I can tell, those always required special treatment via > > > > > %pretrans scriptlets or something, and even the method currently > > > > > recommended by the Packaging Guidelines seems to be broken due to the > > > > > way dnf / RPM verifies validity of transactions. > > > > > > > > > > Additionally, that "special" handling will probably need to stay in > > > > > the RPM package's .spec file for years, given that upgrades from > > > > > Fedora 34 to 36 will need to be supported. > > > > > > > > > > > > > This is documented in the Change: > > > > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibility_impact > > > > > > > > The part that's probably missing there is that the upgraded package > > > > will release ownership of /var/lib/rpm and own the > > > > /usr/lib/sysimage/rpm directory. > > > > > > > > The configuration will change in the upgrade, and then an rpmdb > > > > rebuild on reboot will finalize the transition, as rpmdb rebuilds are > > > > done by loading the rpmdb in memory, renaming the directory, > > > > recreating it, and re-initializing the database files from memory. > > > > > > > > This avoids the pitfalls you've described with the pretrans stuff. > > > > > > Oh, great. Looks like I'm ~tired~ and missed that this is in the > > > Change proposal after all ... > > > Thanks for confirming that you found a way to handle upgrades. > > > > I did a manual proof of concept with the pseudo-sequence in the change > > proposal on a Fedora Rawhide VM. And it did work, and continues to do > > both GNOME Software and dnf updates OK. This is a sample size of 1, so > > there's more work to do to make sure it can be done safely, using the > > rpm sqlite upgrade as a guide. But I can write up the steps I used, so > > that anyone can test before we have it wired up. > > > > We have considered not applying the change to upgrades. Strictly > > speaking the release criterion say we only support upgrades from > > *clean installs* of the current two releases. But in practice quite a > > lot of users depend on reliable upgrades for *many* releases, and they > > get mad when things break even when it's been 10+ releases since they > > did a clean install. And also, Workstation edition PRD "upgrade > > process should give a result that is the same as an original install". > > That is a tall order, but so long as it's safe, it's probably better > > to apply this change to upgrades. If we run into issues or establish > > the risk is too high, it's not such a big deal to apply the change to > > only new clean installs. > > I'm personally concerned that the RPM transactions may not be handled > atomically if the /var/lib/rpm/ is migrated in the midst of an RPM > update. I'm not personally sure if RPM and Berkeley DB will handle > that correctly if there's any issues with the migration, particularly > if /usr/ overflows in the process of other bulky updates such as > libreoffice. Part of my work involves looking for where multiple > things can go wrong at once. We're not doing the migration during the transaction itself. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sat, Jan 8, 2022 at 7:53 PM Chris Murphy wrote: > > On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini wrote: > > > > On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa wrote: > > > > > > On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini > > > wrote: > > > > > > > > On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton wrote: > > > > > > > > > > > > > (snip) > > > > > > > > > > > > > > == Detailed Description == > > > > > === Current location === > > > > > /var/lib/rpm > > > > > > > > > > === New location === > > > > > /usr/lib/sysimage/rpm > > > > > > > > > > /var/lib/rpm will be a symlink pointing to > > > > > /usr/lib/sysimage/rpm > > > > > > > > I did not find a mention of this in the thread or in the Change > > > > proposal, so I'll ask: > > > > How do you plan to handle the directory -> symlink replacement on > > > > upgrade? > > > > > > > > As far as I can tell, those always required special treatment via > > > > %pretrans scriptlets or something, and even the method currently > > > > recommended by the Packaging Guidelines seems to be broken due to the > > > > way dnf / RPM verifies validity of transactions. > > > > > > > > Additionally, that "special" handling will probably need to stay in > > > > the RPM package's .spec file for years, given that upgrades from > > > > Fedora 34 to 36 will need to be supported. > > > > > > > > > > This is documented in the Change: > > > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibility_impact > > > > > > The part that's probably missing there is that the upgraded package > > > will release ownership of /var/lib/rpm and own the > > > /usr/lib/sysimage/rpm directory. > > > > > > The configuration will change in the upgrade, and then an rpmdb > > > rebuild on reboot will finalize the transition, as rpmdb rebuilds are > > > done by loading the rpmdb in memory, renaming the directory, > > > recreating it, and re-initializing the database files from memory. > > > > > > This avoids the pitfalls you've described with the pretrans stuff. > > > > Oh, great. Looks like I'm ~tired~ and missed that this is in the > > Change proposal after all ... > > Thanks for confirming that you found a way to handle upgrades. > > I did a manual proof of concept with the pseudo-sequence in the change > proposal on a Fedora Rawhide VM. And it did work, and continues to do > both GNOME Software and dnf updates OK. This is a sample size of 1, so > there's more work to do to make sure it can be done safely, using the > rpm sqlite upgrade as a guide. But I can write up the steps I used, so > that anyone can test before we have it wired up. > > We have considered not applying the change to upgrades. Strictly > speaking the release criterion say we only support upgrades from > *clean installs* of the current two releases. But in practice quite a > lot of users depend on reliable upgrades for *many* releases, and they > get mad when things break even when it's been 10+ releases since they > did a clean install. And also, Workstation edition PRD "upgrade > process should give a result that is the same as an original install". > That is a tall order, but so long as it's safe, it's probably better > to apply this change to upgrades. If we run into issues or establish > the risk is too high, it's not such a big deal to apply the change to > only new clean installs. I'm personally concerned that the RPM transactions may not be handled atomically if the /var/lib/rpm/ is migrated in the midst of an RPM update. I'm not personally sure if RPM and Berkeley DB will handle that correctly if there's any issues with the migration, particularly if /usr/ overflows in the process of other bulky updates such as libreoffice. Part of my work involves looking for where multiple things can go wrong at once. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sat, Jan 8, 2022 at 7:53 AM Fabio Valentini wrote: > > On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa wrote: > > > > On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini wrote: > > > > > > On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton wrote: > > > > > > > > > > (snip) > > > > > > > > > > > == Detailed Description == > > > > === Current location === > > > > /var/lib/rpm > > > > > > > > === New location === > > > > /usr/lib/sysimage/rpm > > > > > > > > /var/lib/rpm will be a symlink pointing to > > > > /usr/lib/sysimage/rpm > > > > > > I did not find a mention of this in the thread or in the Change > > > proposal, so I'll ask: > > > How do you plan to handle the directory -> symlink replacement on upgrade? > > > > > > As far as I can tell, those always required special treatment via > > > %pretrans scriptlets or something, and even the method currently > > > recommended by the Packaging Guidelines seems to be broken due to the > > > way dnf / RPM verifies validity of transactions. > > > > > > Additionally, that "special" handling will probably need to stay in > > > the RPM package's .spec file for years, given that upgrades from > > > Fedora 34 to 36 will need to be supported. > > > > > > > This is documented in the Change: > > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibility_impact > > > > The part that's probably missing there is that the upgraded package > > will release ownership of /var/lib/rpm and own the > > /usr/lib/sysimage/rpm directory. > > > > The configuration will change in the upgrade, and then an rpmdb > > rebuild on reboot will finalize the transition, as rpmdb rebuilds are > > done by loading the rpmdb in memory, renaming the directory, > > recreating it, and re-initializing the database files from memory. > > > > This avoids the pitfalls you've described with the pretrans stuff. > > Oh, great. Looks like I'm ~tired~ and missed that this is in the > Change proposal after all ... > Thanks for confirming that you found a way to handle upgrades. I did a manual proof of concept with the pseudo-sequence in the change proposal on a Fedora Rawhide VM. And it did work, and continues to do both GNOME Software and dnf updates OK. This is a sample size of 1, so there's more work to do to make sure it can be done safely, using the rpm sqlite upgrade as a guide. But I can write up the steps I used, so that anyone can test before we have it wired up. We have considered not applying the change to upgrades. Strictly speaking the release criterion say we only support upgrades from *clean installs* of the current two releases. But in practice quite a lot of users depend on reliable upgrades for *many* releases, and they get mad when things break even when it's been 10+ releases since they did a clean install. And also, Workstation edition PRD "upgrade process should give a result that is the same as an original install". That is a tall order, but so long as it's safe, it's probably better to apply this change to upgrades. If we run into issues or establish the risk is too high, it's not such a big deal to apply the change to only new clean installs. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sat, Jan 8, 2022 at 3:39 PM Neal Gompa wrote: > > On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini wrote: > > > > On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton wrote: > > > > > > > (snip) > > > > > > > > == Detailed Description == > > > === Current location === > > > /var/lib/rpm > > > > > > === New location === > > > /usr/lib/sysimage/rpm > > > > > > /var/lib/rpm will be a symlink pointing to > > > /usr/lib/sysimage/rpm > > > > I did not find a mention of this in the thread or in the Change > > proposal, so I'll ask: > > How do you plan to handle the directory -> symlink replacement on upgrade? > > > > As far as I can tell, those always required special treatment via > > %pretrans scriptlets or something, and even the method currently > > recommended by the Packaging Guidelines seems to be broken due to the > > way dnf / RPM verifies validity of transactions. > > > > Additionally, that "special" handling will probably need to stay in > > the RPM package's .spec file for years, given that upgrades from > > Fedora 34 to 36 will need to be supported. > > > > This is documented in the Change: > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibility_impact > > The part that's probably missing there is that the upgraded package > will release ownership of /var/lib/rpm and own the > /usr/lib/sysimage/rpm directory. > > The configuration will change in the upgrade, and then an rpmdb > rebuild on reboot will finalize the transition, as rpmdb rebuilds are > done by loading the rpmdb in memory, renaming the directory, > recreating it, and re-initializing the database files from memory. > > This avoids the pitfalls you've described with the pretrans stuff. Oh, great. Looks like I'm ~tired~ and missed that this is in the Change proposal after all ... Thanks for confirming that you found a way to handle upgrades. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sat, Jan 8, 2022 at 8:58 AM Fabio Valentini wrote: > > On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton wrote: > > > > (snip) > > > > > == Detailed Description == > > === Current location === > > /var/lib/rpm > > > > === New location === > > /usr/lib/sysimage/rpm > > > > /var/lib/rpm will be a symlink pointing to > > /usr/lib/sysimage/rpm > > I did not find a mention of this in the thread or in the Change > proposal, so I'll ask: > How do you plan to handle the directory -> symlink replacement on upgrade? > > As far as I can tell, those always required special treatment via > %pretrans scriptlets or something, and even the method currently > recommended by the Packaging Guidelines seems to be broken due to the > way dnf / RPM verifies validity of transactions. > > Additionally, that "special" handling will probably need to stay in > the RPM package's .spec file for years, given that upgrades from > Fedora 34 to 36 will need to be supported. > This is documented in the Change: https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Upgrade.2Fcompatibility_impact The part that's probably missing there is that the upgraded package will release ownership of /var/lib/rpm and own the /usr/lib/sysimage/rpm directory. The configuration will change in the upgrade, and then an rpmdb rebuild on reboot will finalize the transition, as rpmdb rebuilds are done by loading the rpmdb in memory, renaming the directory, recreating it, and re-initializing the database files from memory. This avoids the pitfalls you've described with the pretrans stuff. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 4:03 PM Ben Cotton wrote: > (snip) > > == Detailed Description == > === Current location === > /var/lib/rpm > > === New location === > /usr/lib/sysimage/rpm > > /var/lib/rpm will be a symlink pointing to > /usr/lib/sysimage/rpm I did not find a mention of this in the thread or in the Change proposal, so I'll ask: How do you plan to handle the directory -> symlink replacement on upgrade? As far as I can tell, those always required special treatment via %pretrans scriptlets or something, and even the method currently recommended by the Packaging Guidelines seems to be broken due to the way dnf / RPM verifies validity of transactions. Additionally, that "special" handling will probably need to stay in the RPM package's .spec file for years, given that upgrades from Fedora 34 to 36 will need to be supported. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 1/3/22 22:55, Chris Murphy wrote: Does anyone know what /var/lib/rpm-state/gconf is used for? Owning package is GConf2-3.2.6-31.fc35.x86_64 On my Fedora 35 Workstation installation, it's empty. So no obvious conflict with the change proposal, but I'd like to make sure it's not something that if used is going to get mad if there's an rpmdb state change that this location isn't privy to. See https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/#_saving_state_between_scriptlets, the /var/lib/rpm-state/ directory is what Fedora recommends for communicating state from one scriptlet to another. Not used or endorsed by rpm itself, and not related to the database at all. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Mon, Jan 3, 2022 at 1:55 PM Chris Murphy wrote: > > Does anyone know what /var/lib/rpm-state/gconf is used for? Owning > package is GConf2-3.2.6-31.fc35.x86_64 > OK nevermind. GConf2 is dead upstream, and the only thing I have dragging it in is pdfmod. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
Does anyone know what /var/lib/rpm-state/gconf is used for? Owning package is GConf2-3.2.6-31.fc35.x86_64 On my Fedora 35 Workstation installation, it's empty. So no obvious conflict with the change proposal, but I'd like to make sure it's not something that if used is going to get mad if there's an rpmdb state change that this location isn't privy to. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wednesday, December 29, 2021 12:47:43 PM EST Gordon Messmer wrote: > On 12/29/21 07:26, Vitaly Zaitsev via devel wrote: > > On 29/12/2021 16:01, Ben Cotton wrote: > >> Currently, the RPM databases is located in `/var`. Let's move it to > >> `/usr`. The move is already under way in rpm-ostree-based > >> installations, and in (open)SUSE. > > > > It will break FHS compatibility. /usr must contain read-only data. > > If /usr really is read-only, then it probably doesn't matter where the > rpmdb is, since packages can't be installed (generally). Not only cases where /usr is readonly, what about the case where /usr is on a SSD because it should not be changing much? I have /var on a spinning disk because by definition it is where things live that change frequently. -Steve ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 1/3/22 15:36, Lennart Poettering wrote: On Mo, 03.01.22 14:15, Florian Weimer (fwei...@redhat.com) wrote: * Lennart Poettering: Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable? Maybe if RPM is used to install software under /opt? I'd argue that conceptually /opt and /usr should be seen as the same thing for RPM: i.e. during update time both are writable and outside of updates they are both to be considered read-only from RPM's PoV. Hence it should be fine too put the rpmdb in /usr/ too in that case. Or with other words: if RPM is used to install something in /opt, it should be fine to require that both /usr and /opt are writable then. It does seem a bit controversial to require /usr of all things to be writable to be install into /opt, /etc, /boot and whatnot. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
For the record, I obviously support this change. Responding to a few threads: On Wed, Dec 29, 2021, at 10:16 AM, Peter Robinson wrote: > How does this work on RO /usr files systems? I thought data in /usr > was supposed to be static/ It works for rpm-ostree because it's > updated at tree creation time. I think you know this but it's worth elaborating on here; rpm-ostree supports client-side layering (and overrides too) and even live updates - and those all operate by default while maintaining `/usr` read-only from the perspective of the rest of the system. The way this works ultimately is quite simple; the underlying filesystem is writable, we just remount it writable *in a private mount namespace*. So even when you do e.g. `rpm-ostree install -A usbguard`, there is no point where *other* processes can write. People are focusing a bit too much on "read-only" in this thread - it's more about "lifecycle binding" and versioning the binaries together with metadata about the binaries. On Thu, Dec 30, 2021, at 1:57 PM, Chris Murphy wrote: >> What happens if /var/lib is read-only? Changing (fixing?) this would >> be a pre-requisite to this proposal, we don't want 'dnf list' to break. Why would /var/lib be read-only, but /usr be writable? > Why should it be a prerequisite? In all Fedora editions and spins with > dnf, /usr and /var are read-write. In the case of rpm-ostree based > editions and spins, they don't include dnf. Remember rpm-ostree links to libdnf, and significant code is hence shared. That's part of the ASCII-art architecture diagram in the docs https://coreos.github.io/rpm-ostree/ The way I'd say this is it aligns "traditional" dnf systems with what rpm-ostree has been doing for many years now, and that will help share *even more* code and logic in the future. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 12/29/21 17:01, Ben Cotton wrote: Upstream RPM accept the change, but institutionally don't like the loss or weakening of a [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html very well known location] for the database, and [http://lists.rpm.org/pipermail/rpm-ecosystem/2021-December/000781.html anticipate complaints]. Just noticed this in the change text... To clarify the upstream position: there are no plans to change the upstream default, the level of acceptance is merely that if Fedora wants to change it I'm not going to stand in front of the Change truck. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 1/3/22 14:57, Lennart Poettering wrote: On Mo, 03.01.22 11:57, Panu Matilainen (pmati...@redhat.com) wrote: On 12/30/21 09:02, Chris Murphy wrote: On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel wrote: I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely: https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layout The FHS describes /usr here: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18 as "/usr is shareable, read-only data" which clearly does not apply to a database that changes. In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed.. The rpmdb has traditionally been like that, but it doesn't mean it will stay that way forever more. There are all manner of currently unimplemented use-cases which would require changing the database outside a direct install/update/erase context. Many of those use-cases are related to files and would fall under "but you need writable fs for that anyway" but not all. Of course it'll always be *mostly* read-only data because of the nature of the data, compared to a general purpose db in /var. Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable? There seems to be this strange underlying assumption that all packaged content lives in /usr when that's not at all the case. To install a kernel, or a config-only package (under etc), or 3rd party software putting stuff under /opt, or... you need a writable rpmdb. Ditto for 'rpm --import'. But the kind of thing I had in mind when making that comment initially is eg ability to update file states in the rpmdb to reflect local changes from eg network mounting .. well, the typical example would be /usr but in this case that gets a little strange. In general, "data set by users" is the common case, whether a policy to flag a given package as unremovable or non-updatable, or add a "reason" (dependency or user-installed), or other annotations. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Mo, 03.01.22 14:15, Florian Weimer (fwei...@redhat.com) wrote: > * Lennart Poettering: > > > Can you provide an example for such feature requests? i.e. where the > > rpmdb should be writable even though /usr is assumed to be immutable? > > Maybe if RPM is used to install software under /opt? I'd argue that conceptually /opt and /usr should be seen as the same thing for RPM: i.e. during update time both are writable and outside of updates they are both to be considered read-only from RPM's PoV. Hence it should be fine too put the rpmdb in /usr/ too in that case. Or with other words: if RPM is used to install something in /opt, it should be fine to require that both /usr and /opt are writable then. Lennart -- Lennart Poettering, Berlin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
* Lennart Poettering: > Can you provide an example for such feature requests? i.e. where the > rpmdb should be writable even though /usr is assumed to be immutable? Maybe if RPM is used to install software under /opt? Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Mo, 03.01.22 11:57, Panu Matilainen (pmati...@redhat.com) wrote: > On 12/30/21 09:02, Chris Murphy wrote: > > On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel > > wrote: > > > > > > I don't see how this is FHS compliant, which in turn would make > > > it non-compliant with Fedora Packaging Guidelines, namely: > > > > > > > > > https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layout > > > > > > The FHS describes /usr here: > > > > > > https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18 > > > > > > as "/usr is shareable, read-only data" which clearly does not > > > apply to a database that changes. > > > > In practice it is read-only data, except when software is being > > installed or updated. The FHS is a PITA sometimes, but it's not > > advocating for systems that can't be updated or changed.. > > > > The rpmdb has traditionally been like that, but it doesn't mean it will stay > that way forever more. There are all manner of currently unimplemented > use-cases which would require changing the database outside a direct > install/update/erase context. Many of those use-cases are related to files > and would fall under "but you need writable fs for that anyway" but not all. > Of course it'll always be *mostly* read-only data because of the nature of > the data, compared to a general purpose db in /var. Can you provide an example for such feature requests? i.e. where the rpmdb should be writable even though /usr is assumed to be immutable? Lennart -- Lennart Poettering, Berlin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Do, 30.12.21 07:05, Fedora Development ML (devel@lists.fedoraproject.org) wrote: > As I demonstrated later in my email the contents of /var/lib/rpm > do change at other times though. Note that there are other things in /usr that are similar to the RPM db in the "mostly-read-only-but-not-quite" sense, i.e. files that are generated locally during updates, but are static outside of updates. The kernel modules databases for example, i.e. /lib/modules/`uname -r`/*.dep. Or fontconfig for mime db stuff iirc. ldconfig creates so symlinks. And there's more. I think having that is even fine. Key must be though that the stuff that is generated locally and stored in /usr is: a) never updated outside of the update operations b) operates on input that is exclusively from the vendor, too (and not dependent on local admin's config choices) As long as these rules apply the /usr/ tree remains sharable between multiple systems, and hence it should be fine if some files stored there are not written by RPM itself but by tools invoked by it that operate on data also installed by it. I think one of those days we should make the first rule explicit btw, and by default overmount /usr/ with a read-only bind mount of itself, that RPM then remounts writable immediately before doing an update and remounts back afterwards. (or alternatively, rpm toggles the r/o bit on the btrfs subvolume if /usr/ is one). Key point I am trying to make here: having stuff in /usr that is not directly RPM payload is not a new thing, it's a common thing we already have been doing since a long time, and it's OK. I welcome if RPM would put its database in /usr/ as well. Lennart -- Lennart Poettering, Berlin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 12/30/21 09:02, Chris Murphy wrote: On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel wrote: I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely: https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layout The FHS describes /usr here: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18 as "/usr is shareable, read-only data" which clearly does not apply to a database that changes. In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed.. The rpmdb has traditionally been like that, but it doesn't mean it will stay that way forever more. There are all manner of currently unimplemented use-cases which would require changing the database outside a direct install/update/erase context. Many of those use-cases are related to files and would fall under "but you need writable fs for that anyway" but not all. Of course it'll always be *mostly* read-only data because of the nature of the data, compared to a general purpose db in /var. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Sat, Jan 01, 2022 at 12:52:57PM -0500, Matthew Miller wrote: > > Given that the "shared-memory file contains no persistent content", > > it seems like "rpmdb.sqlite-shm" could be a symlink to > > /dev/shm/rpmdb.sqlite-shm", or to some other tmpfs location. > It does seem like maybe it could. I'd definitely like someone who is an > expert in area this to figure out if that's right before we try any such > thing. That said, from https://sqlite.org/walformat.html#the_wal_index_or_shm_file: Since the content of the shm file does not need to be preserved across a crash, the shm file is never fsync()-ed to disk. In fact, if there were a mechanism by which SQLite could tell the operating system to never persist the shm file to disk but always hold it in cache memory, SQLite would use that mechanism to avoid any unnecessary disk I/O associated with the shm file. However, no such mechanism exists in standard posix. ... sounds promising. _However_, I'm not sure if this is something best done at the filesystem level (what recreates it at as a symlink if the file is removed?) or in rpm or DNF — or in SQLite itself, if they'd accept something that isn't "standard posix". -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 01:41:01PM -0800, Gordon Messmer wrote: > Given that the "shared-memory file contains no persistent content", > it seems like "rpmdb.sqlite-shm" could be a symlink to > /dev/shm/rpmdb.sqlite-shm", or to some other tmpfs location. It does seem like maybe it could. I'd definitely like someone who is an expert in area this to figure out if that's right before we try any such thing. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Fri, Dec 31, 2021 at 4:32 AM Richard W.M. Jones wrote: > > On Thu, Dec 30, 2021 at 12:00:06PM +0100, Vitaly Zaitsev via devel wrote: > > On 29/12/2021 18:47, Gordon Messmer wrote: > > >If /usr really is read-only, then it probably doesn't matter where > > >the rpmdb is, since packages can't be installed (generally). > > > > dnf opens these database files for writing, even for the simple `dnf list`. > > If so this is definitely a bug. (However "dnf list" and "dnf download" > seem to work as non-root, so I guess it must fall back to read-only?) Made a note in the change to investigate this. What I'm seeing with stat are ctime and/or mtime updates of the -shm and -wal files. I'm not sure either should be changing. I'm using noatime, but I expect atime updates probably happen with all these files normally. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Fri, Dec 31, 2021 at 4:30 AM Richard W.M. Jones wrote: > > On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr > > * Other developers: > > ** changes in SElinux policy > > Please can you make sure two bugs are filed against libguestfs and > supermin components to track this change (if it happens). Made a note for this in the scope > Will the symlink also exist on new installs, or only on upgrades? Both. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 31/12/2021 12:32, Richard W.M. Jones wrote: However "dnf list" and "dnf download" seem to work as non-root, so I guess it must fall back to read-only? Without root and `-C` command-line option it will download all metadata to your $HOME. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 12:00:06PM +0100, Vitaly Zaitsev via devel wrote: > On 29/12/2021 18:47, Gordon Messmer wrote: > >If /usr really is read-only, then it probably doesn't matter where > >the rpmdb is, since packages can't be installed (generally). > > dnf opens these database files for writing, even for the simple `dnf list`. If so this is definitely a bug. (However "dnf list" and "dnf download" seem to work as non-root, so I guess it must fall back to read-only?) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 10:01:57AM -0500, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr > * Other developers: > ** changes in SElinux policy Please can you make sure two bugs are filed against libguestfs and supermin components to track this change (if it happens). We now use librpm to parse the RPM database so I think we're OK from the inspection side, but we do use the RPM database's real location in two places: [supermin] To test if a package has been installed/upated/removed so that we can rebuild our cache [libguestfs] To build a "phony" Fedora image for testing with a fake RPM database. > == Upgrade/compatibility impact == > Change will be applied to offline upgrades, similar to the RPM sqlite > database change. A systemd service will move the rpmdb from /var to > /usr, then create a symlink pointing to /usr from /var. Will the symlink also exist on new installs, or only on upgrades? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 12/30/21 12:22, Matthew Miller wrote: So for a read-only filesystem, if the db has wal enabled, the files have to exist. (it's possible to disable WAL mode by running the sqlite 'PRAGMA journal_mode=DELETE;' command when the partition is writable beforehand, but I didn't see how to alter the mode while it is readonly) FWIW, Sqlite has guidance here: Given that the "shared-memory file contains no persistent content", it seems like "rpmdb.sqlite-shm" could be a symlink to /dev/shm/rpmdb.sqlite-shm", or to some other tmpfs location. Also: https://www.sqlite.org/walformat.html#the_wal_index_or_shm_file ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 05:53:14PM +0900, Dominique Martinet wrote: > So for a read-only filesystem, if the db has wal enabled, the files have > to exist. (it's possible to disable WAL mode by running the sqlite 'PRAGMA > journal_mode=DELETE;' command when the partition is writable beforehand, > but I didn't see how to alter the mode while it is readonly) FWIW, Sqlite has guidance here: https://www.sqlite.org/wal.html#readonly Older versions of SQLite could not read a WAL-mode database that was read-only. In other words, write access was required in order to read a WAL-mode database. This constraint was relaxed beginning with SQLite version 3.22.0 (2018-01-22). On newer versions of SQLite, a WAL-mode database on read-only media, or a WAL-mode database that lacks write permission, can still be read as long as one or more of the following conditions are met: 1. The -shm and -wal files already exists and are readable 2. There is write permission on the directory containing the database so that the -shm and -wal files can be created. 3. The database connection is opened using the immutable query parameter. Even though it is possible to open a read-only WAL-mode database, it is good practice to converted to PRAGMA journal_mode=DELETE prior to burning an SQLite database image onto read-only media. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 2:14 PM Stephen John Smoogen wrote: > > > > On Thu, 30 Dec 2021 at 13:58, Chris Murphy wrote: >> >> On Thu, Dec 30, 2021 at 1:31 AM Zbigniew Jędrzejewski-Szmek >> wrote: >> > >> > On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote: >> > > On Wed, Dec 29, 2021 at 03:17:42PM +, Tom Hughes via devel wrote: >> > > > At this point somebody will no doubt argue that /usr changes on a >> > > > package update and that the RPM database is a static definition of >> > > > the currently installed OS files, but evidence says otherwise: >> > > > >> > > > % ls -l /var/lib/rpm >> > > > >> > > > total 378M >> > > > >> > > > -rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite >> > > > -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm >> > > > -rw-r--r--. 1 root root0 Dec 28 16:08 rpmdb.sqlite-wal >> > > > >> > > > >> > > > While "Dec 28 16:08" is indeed the last time I updated that machine >> > > > it seems one of the files has changed more recently - no idea what >> > > > triggered that but clearly the files are not static between updates. >> > > >> > > That is a sqlite write-ahead log shared memory file used to coordinate >> > > access between concurrent clients. Someone who knows more about the >> > > depths >> > > of DNF and RPM than me will need to comment, but it looks like `dnf list` >> > > touches it -- even though `rpm -qa` doesn't. >> > >> > $ sudo strace -efile dnf list >> > ... >> > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", >> > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 >> > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", >> > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 >> > ... >> > >> > What happens if /var/lib is read-only? Changing (fixing?) this would >> > be a pre-requisite to this proposal, we don't want 'dnf list' to break. >> >> Why should it be a prerequisite? In all Fedora editions and spins with >> dnf, /usr and /var are read-write. In the case of rpm-ostree based >> editions and spins, they don't include dnf. I agree dnf should >> tolerate read-only rpmdb files, but I'm not following the logic >> leading to it being a prerequisite (must tolerate rather than should >> tolerate). > > > I think that at least it needs to be ok'd that they can and will work on it > to fix. The dnf team may have other things on their queue that they need to > focus on this release so having to redesign their plumbing to deal with moved > files may not be possible. that leaves F36 shipped with a broken dnf and no > way anyone can update or run anything. > I've already validated this with DNF on openSUSE. This change was executed in openSUSE for the rpmdb in 2017, and the DNF state information in early 2021. DNF handled both transitions fine, since libsolv gets the rpmdb path from rpm configuration. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, 30 Dec 2021 at 13:58, Chris Murphy wrote: > On Thu, Dec 30, 2021 at 1:31 AM Zbigniew Jędrzejewski-Szmek > wrote: > > > > On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote: > > > On Wed, Dec 29, 2021 at 03:17:42PM +, Tom Hughes via devel wrote: > > > > At this point somebody will no doubt argue that /usr changes on a > > > > package update and that the RPM database is a static definition of > > > > the currently installed OS files, but evidence says otherwise: > > > > > > > > % ls -l /var/lib/rpm > > > > > > > > total 378M > > > > > > > > -rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite > > > > -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm > > > > -rw-r--r--. 1 root root0 Dec 28 16:08 rpmdb.sqlite-wal > > > > > > > > > > > > While "Dec 28 16:08" is indeed the last time I updated that machine > > > > it seems one of the files has changed more recently - no idea what > > > > triggered that but clearly the files are not static between updates. > > > > > > That is a sqlite write-ahead log shared memory file used to coordinate > > > access between concurrent clients. Someone who knows more about the > depths > > > of DNF and RPM than me will need to comment, but it looks like `dnf > list` > > > touches it -- even though `rpm -qa` doesn't. > > > > $ sudo strace -efile dnf list > > ... > > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 > > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 > > ... > > > > What happens if /var/lib is read-only? Changing (fixing?) this would > > be a pre-requisite to this proposal, we don't want 'dnf list' to break. > > Why should it be a prerequisite? In all Fedora editions and spins with > dnf, /usr and /var are read-write. In the case of rpm-ostree based > editions and spins, they don't include dnf. I agree dnf should > tolerate read-only rpmdb files, but I'm not following the logic > leading to it being a prerequisite (must tolerate rather than should > tolerate). > I think that at least it needs to be ok'd that they can and will work on it to fix. The dnf team may have other things on their queue that they need to focus on this release so having to redesign their plumbing to deal with moved files may not be possible. that leaves F36 shipped with a broken dnf and no way anyone can update or run anything. -- Stephen J Smoogen. Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 1:31 AM Zbigniew Jędrzejewski-Szmek wrote: > > On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote: > > On Wed, Dec 29, 2021 at 03:17:42PM +, Tom Hughes via devel wrote: > > > At this point somebody will no doubt argue that /usr changes on a > > > package update and that the RPM database is a static definition of > > > the currently installed OS files, but evidence says otherwise: > > > > > > % ls -l /var/lib/rpm > > > > > > total 378M > > > > > > -rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite > > > -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm > > > -rw-r--r--. 1 root root0 Dec 28 16:08 rpmdb.sqlite-wal > > > > > > > > > While "Dec 28 16:08" is indeed the last time I updated that machine > > > it seems one of the files has changed more recently - no idea what > > > triggered that but clearly the files are not static between updates. > > > > That is a sqlite write-ahead log shared memory file used to coordinate > > access between concurrent clients. Someone who knows more about the depths > > of DNF and RPM than me will need to comment, but it looks like `dnf list` > > touches it -- even though `rpm -qa` doesn't. > > $ sudo strace -efile dnf list > ... > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 > ... > > What happens if /var/lib is read-only? Changing (fixing?) this would > be a pre-requisite to this proposal, we don't want 'dnf list' to break. Why should it be a prerequisite? In all Fedora editions and spins with dnf, /usr and /var are read-write. In the case of rpm-ostree based editions and spins, they don't include dnf. I agree dnf should tolerate read-only rpmdb files, but I'm not following the logic leading to it being a prerequisite (must tolerate rather than should tolerate). -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 01:14:40PM -0500, Matthew Miller wrote: > I'd be supportive of an effort to restart the FHS process with buy-in from > openSUSE (and ideally Debian, Arch, FreeBSD...). And probably systemd too, > from a plain pragmatic point of view. > > I think we'd want to either ask Linux Foundation to split it back from the > LSB into its own thing — or else explicitly fork into a new standard. Wait, I hit send before finishing my thought, which is: we also need a mechanism which reflects reality rather than an abstract design. Like, sure, if we were going to create the whole thing from scratch now with no history, it probably wouldn't look like it does. And distros (even Fedora) have always just made exceptions. It'd be better to have a working, living standard that makes room for that than one which everyone ignores. Which is to say: I _don't_ think the process should be to get the FHS restarted and updated and in agreement before moving forward, _especially_ when the proposal is already aligned with what openSUSE is doing, and what we're already doing ourselves with IoT/CoreOS/Kinoite/Silverblue. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 10:12:20AM -0500, Neal Gompa wrote: > > say, not as I do' route. I would say we either need to drop FHS > > compliance or use an amended copy and work with the SuSE people on > > making that a shared standard. > > That's pretty much what has been going on. I've been working in > openSUSE for a few years now to rationalize the FHS configuration in > openSUSE to more closely match Fedora. As a result, today the only I'd be supportive of an effort to restart the FHS process with buy-in from openSUSE (and ideally Debian, Arch, FreeBSD...). And probably systemd too, from a plain pragmatic point of view. I think we'd want to either ask Linux Foundation to split it back from the LSB into its own thing — or else explicitly fork into a new standard. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 10:09 AM Stephen John Smoogen wrote: > > > > On Wed, 29 Dec 2021 at 21:20, Matthew Miller wrote: >> >> On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote: >> > F36 but needs to be worked through the proper channels of 'upstream'. Get >> > the FHS updated and fixed, work out that the change actually is going to be >> >> >> Pretty sure that's a non-starter. FHS 3.0 was released in 2015, and the >> whole thing has been effectively dead since. >> > > I agree that the FHS and LSB are practically dead, but it is heavily used in > our work. FHS compliance is one of the first things package reviews get > dinged on and it doesn't help if we start down the 'Do as I say, not as I do' > route. I would say we either need to drop FHS compliance or use an amended > copy and work with the SuSE people on making that a shared standard. > That's pretty much what has been going on. I've been working in openSUSE for a few years now to rationalize the FHS configuration in openSUSE to more closely match Fedora. As a result, today the only differences between Red Hat and SUSE distributions are in /usr/lib/sysimage/rpm and usage of /srv in packages. I do not think we'll adopt their usage of /srv anytime soon, but RPM-OSTree variants already are working to move rpmdb from /usr/share/rpm to /usr/lib/sysimage/rpm based on upstream feedback. This Change basically makes it so we use the same rpmdb path regardless of Fedora variant. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, 29 Dec 2021 at 21:20, Matthew Miller wrote: > On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote: > > F36 but needs to be worked through the proper channels of 'upstream'. Get > > the FHS updated and fixed, work out that the change actually is going to > be > > > Pretty sure that's a non-starter. FHS 3.0 was released in 2015, and the > whole thing has been effectively dead since. > > I agree that the FHS and LSB are practically dead, but it is heavily used in our work. FHS compliance is one of the first things package reviews get dinged on and it doesn't help if we start down the 'Do as I say, not as I do' route. I would say we either need to drop FHS compliance or use an amended copy and work with the SuSE people on making that a shared standard. -- Stephen J Smoogen. Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
It doesn't happen me since a long time, but rpmdb can be corrupted, and it must be repaired without altering /usr. For me, rpmdb is variable data so it must be placed at /var/lib according to FHS. And because my sysadmin OCDs it would be disturbing for me relocating it at /usr :-D -- Francisco J. Tsao Santín http://gattaca.es 1024D/71CF4D62 42 F1 53 35 EF 98 98 8A FC 6C 56 B3 4C A7 7D FB___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 29/12/2021 18:47, Gordon Messmer wrote: If /usr really is read-only, then it probably doesn't matter where the rpmdb is, since packages can't be installed (generally). dnf opens these database files for writing, even for the simple `dnf list`. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
Zbigniew Jędrzejewski-Szmek wrote on Thu, Dec 30, 2021 at 08:29:54AM +: > $ sudo strace -efile dnf list > ... > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 > openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", > O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 > ... > > What happens if /var/lib is read-only? Changing (fixing?) this would > be a pre-requisite to this proposal, we don't want 'dnf list' to break. That's easy enough to test. Apparently, of the files already exist the open calls work without problem. If the files don't (e.g. cleared by running 'sqlite3 rpmdb.sqlite vacuum'), then sqlite apparently tries to create them and fails horribly -- but so does rpm -qa: # rpm -qa error: sqlite failure: CREATE TABLE IF NOT EXISTS 'Packages' (hnum INTEGER PRIMARY KEY AUTOINCREMENT,blob BLOB NOT NULL): unable to open database file error: cannot open Packages index using sqlite - No such file or directory (2) error: cannot open Packages database in /var/lib/rpm or for that matter plain sqlite3: # sqlite3 rpmdb.sqlite 'select * from Packages;' Error: unable to open database file So for a read-only filesystem, if the db has wal enabled, the files have to exist. (it's possible to disable WAL mode by running the sqlite 'PRAGMA journal_mode=DELETE;' command when the partition is writable beforehand, but I didn't see how to alter the mode while it is readonly) Anyway, I don't think vacuum ever runs automatically so the files should always be present and it probably should just work™. FWIW I've been running with my rpmdb in /usr (through a bind mount) for a few years precisely to keep the rpmdb in sync with the rest of /usr for snapshots -- it's pretty much necessary to run rpm -q on old snapshots and figure which version of what was where easily. Unless btrfs somehow becomes able to tie multiple subvolume snapshots together I think it's a good change. -- Dominique ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Thu, Dec 30, 2021 at 02:27:04AM -0500, Matthew Miller wrote: > On Wed, Dec 29, 2021 at 03:17:42PM +, Tom Hughes via devel wrote: > > At this point somebody will no doubt argue that /usr changes on a > > package update and that the RPM database is a static definition of > > the currently installed OS files, but evidence says otherwise: > > > > % ls -l /var/lib/rpm > > > > total 378M > > > > -rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite > > -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm > > -rw-r--r--. 1 root root0 Dec 28 16:08 rpmdb.sqlite-wal > > > > > > While "Dec 28 16:08" is indeed the last time I updated that machine > > it seems one of the files has changed more recently - no idea what > > triggered that but clearly the files are not static between updates. > > That is a sqlite write-ahead log shared memory file used to coordinate > access between concurrent clients. Someone who knows more about the depths > of DNF and RPM than me will need to comment, but it looks like `dnf list` > touches it -- even though `rpm -qa` doesn't. $ sudo strace -efile dnf list ... openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-wal", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 4 openat(AT_FDCWD, "/var/lib/rpm/rpmdb.sqlite-shm", O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC, 0644) = 5 ... What happens if /var/lib is read-only? Changing (fixing?) this would be a pre-requisite to this proposal, we don't want 'dnf list' to break. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 8:52 AM Stephen John Smoogen wrote: > > > > > On Wed, 29 Dec 2021 at 10:19, Tom Hughes via devel > wrote: >> >> I don't see how this is FHS compliant, which in turn would make >> it non-compliant with Fedora Packaging Guidelines, namely: >> > > > I am in agreement here and think that this is NOT a change to be made in F36 > but needs to be worked through the proper channels of 'upstream'. Get the FHS > updated and fixed, work out that the change actually is going to be stuck to > in SuSE and not rolled back and then push it to Fedora. It's actually /usr/lib not /usr that applies here. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s06.html And it's been worked through proper upstream channels for 4+ years. Location http://lists.rpm.org/pipermail/rpm-maint/2017-October/006722.html FHS question http://lists.rpm.org/pipermail/rpm-maint/2017-October/006697.html There's a bunch of back and forth throughout. The rpmdb isn't really variable data. It's static data that describes other static data. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 03:17:42PM +, Tom Hughes via devel wrote: > At this point somebody will no doubt argue that /usr changes on a > package update and that the RPM database is a static definition of > the currently installed OS files, but evidence says otherwise: > > % ls -l /var/lib/rpm > > total 378M > > -rw-r--r--. 1 root root 378M Dec 28 16:08 rpmdb.sqlite > -rw-r--r--. 1 root root 32K Dec 29 09:27 rpmdb.sqlite-shm > -rw-r--r--. 1 root root0 Dec 28 16:08 rpmdb.sqlite-wal > > > While "Dec 28 16:08" is indeed the last time I updated that machine > it seems one of the files has changed more recently - no idea what > triggered that but clearly the files are not static between updates. That is a sqlite write-ahead log shared memory file used to coordinate access between concurrent clients. Someone who knows more about the depths of DNF and RPM than me will need to comment, but it looks like `dnf list` touches it -- even though `rpm -qa` doesn't. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 30/12/2021 07:02, Chris Murphy wrote: On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel wrote: I don't see how this is FHS compliant, which in turn would make it non-compliant with Fedora Packaging Guidelines, namely: https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layout The FHS describes /usr here: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18 as "/usr is shareable, read-only data" which clearly does not apply to a database that changes. In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed.. As I demonstrated later in my email the contents of /var/lib/rpm do change at other times though. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 8:19 AM Tom Hughes via devel wrote: > > I don't see how this is FHS compliant, which in turn would make > it non-compliant with Fedora Packaging Guidelines, namely: > > > https://docs.fedoraproject.org/en-US/packaging-guidelines/#_filesystem_layout > > The FHS describes /usr here: > >https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html#purpose18 > > as "/usr is shareable, read-only data" which clearly does not > apply to a database that changes. In practice it is read-only data, except when software is being installed or updated. The FHS is a PITA sometimes, but it's not advocating for systems that can't be updated or changed.. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 7:36 PM Matthew Miller wrote: > > On Wed, Dec 29, 2021 at 12:57:25PM -0500, Neal Gompa wrote: > > of the RPM database. Once the RPM database is moved, it becomes > > possible to split /var into its own subvolume and make it trivial to > > configure a full boot-to-snapshot system leveraging the technologies > > we have available to us. > > The Benefit to Fedora section alludes to this, but doesn't really spell it > out... since this seems like a pretty compelling benefit, it probably > should. Added. There's more hurdles to jump, just so no one thinks snapshots and rollbacks are showing up in Fedora 36. There's a quandary with /boot which likewise is married to /usr via the vmlinuz kernel modules. As we snapshot /usr how do we retain vmlinuz such that we can still boot those older snapshots? And then how do we do it considering Boot Loader Spec expects "boot" (or $BOOT) volume to be shared among multiple distros? Related is the "discoverable sub-volumes specification" discussion, so that we have a way of self-described assembly of systems during startup instead of having to depend on fstab, or any other database for that matter, to know how a system should boot. https://lists.freedesktop.org/archives/systemd-devel/2021-November/047059.html -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 12:57:25PM -0500, Neal Gompa wrote: > of the RPM database. Once the RPM database is moved, it becomes > possible to split /var into its own subvolume and make it trivial to > configure a full boot-to-snapshot system leveraging the technologies > we have available to us. The Benefit to Fedora section alludes to this, but doesn't really spell it out... since this seems like a pretty compelling benefit, it probably should. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote: > F36 but needs to be worked through the proper channels of 'upstream'. Get > the FHS updated and fixed, work out that the change actually is going to be Pretty sure that's a non-starter. FHS 3.0 was released in 2015, and the whole thing has been effectively dead since. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, 2021-12-29 at 12:57 -0500, Neal Gompa wrote: > On Wed, Dec 29, 2021 at 12:48 PM Gordon Messmer > wrote: > > > > On 12/29/21 07:26, Vitaly Zaitsev via devel wrote: > > > On 29/12/2021 16:01, Ben Cotton wrote: > > > > Currently, the RPM databases is located in `/var`. Let's move it > > > > to > > > > `/usr`. The move is already under way in rpm-ostree-based > > > > installations, and in (open)SUSE. > > > > > > It will break FHS compatibility. /usr must contain read-only data. > > > > > > If /usr really is read-only, then it probably doesn't matter where > > the > > rpmdb is, since packages can't be installed (generally). > > > > Moreover, for systems where /usr is read-only and/or shared > > (especially > > stateless systems), having the rpmdb on /usr seems like the most > > rational place for it, if one expects to be able to use rpm to query > > the > > package list. Otherwise, there is an implicit coupling of /usr and > > /var/lib/rpmdb that requires two mounts rather than one. > > Bingo. It's generally accepted that the RPM database changes when the > system state changes. And if the system state is not allowed to > change, the rpmdb should not either. The bigger problem is that having > the RPM database in /var makes it much harder to correctly implement a > boot-to-snapshot scheme for Fedora Linux that reasonably maintains > system state properly once /var is carved out. you just need change to change the default --dbpath of rpm [1] , i.e, I suggest instead change default location of rpm , change the dbpath configuration for the atomic images, is just one idea . [1] man rpm --dbpath DIRECTORY Use the database in DIRECTORY rather than the default path /var/lib/rpm > The reason that /var > *isn't* carved out by default with our Btrfs configuration is because > of the RPM database. Once the RPM database is moved, it becomes > possible to split /var into its own subvolume and make it trivial to > configure a full boot-to-snapshot system leveraging the technologies > we have available to us. > > > > -- > 真実はいつも一つ！/ Always, there's only one truth! > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure -- Sérgio M. B. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On 12/29/21 13:21, Sebastian Crane wrote: Why would you need to move the rpmdb? Users probably aren't installing rpm packages in containers at run time (particularly if /usr is read-only); installation typically happens when building the container image, at which point /usr isn't read-only. I do actually install RPM package inside containers, but in my case I'm using containers more as short-lived virtual machines for testing than as a deployment mechanism. That said, I don't think that this nullifies your point, as I'm not using a read-only /usr inside these containers. Yeah, that was poorly worded. Sorry. What I meant was simply that containers don't seem to be a unique use case. If a container has a writable /usr, then that should be a suitable location for the rpmdb. And if the container's /usr is read-only, then the rpmdb doesn't need to be in /var. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 05:06:37PM -0500, Stephen John Smoogen wrote: > On Wed, 29 Dec 2021 at 16:16, Michel Alexandre Salim < > sali...@fedoraproject.org> wrote: > > > All that said, this isn't the first time this has happened. It is the > reason why various large changes usually require groups to sort of make a > sample sausage for people to eat before they have to sit through watching > the larger sausage made. Because we are all going to have to do that while > any of these changes are working through the grinder. > That's good feedback, thanks. It can probably be done in both this case and fs-verity: - for the RPMDB change: have a COPR repo with the patched RPM and the systemd service, with clear instructions how to test Potential issue: if someone enables this without reading the instructions, and did not create the symlink either manually or via the service, this can break their system. Probably have to name it something like 'rpmdb-move-READ-INSTRUCTIONS-FIRST' or something scary like that - for fsverity: having a COPR with the plugin and some re-signed packages would probably work, so people can get familiar with it Best regards, -- Michel Alexandre Salim profile: https://keyoxide.org/mic...@michel-slm.name signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, Dec 29, 2021 at 3:36 PM Stephen John Smoogen wrote: > > > > On Wed, 29 Dec 2021 at 13:51, Gordon Messmer wrote: >> >> On 12/29/21 09:59, Stephen John Smoogen wrote: >> > The modern day case where /usr is read-only is inside a container and >> > you put an overlay or using some sort of linking to /var which is >> > read-write in case of reboots. >> >> >> Right, that makes sense. >> >> >> > To me this is like saying 'move everything into /usr but because its >> > volitile move it back into /var but in a sub-directory from where it >> > was so you can keep an image running.' In this case, this doesn't >> > sound like any savings and more of a headache of why did it corrupt >> > this time. >> >> >> But this doesn't. Why would you need to move the rpmdb? Users probably >> aren't installing rpm packages in containers at run time (particularly >> if /usr is read-only); installation typically happens when building the >> container image, at which point /usr isn't read-only. >> > Most of the containers I am dealing with are > Grab the base image, > Create a layer, and add the images you want, > Test and deploy the layered image. > Update that image over time. > > Theoretically people should build the thing from scratch every time but > instead you get someone downloading the base image which they have gotten an > OK to use, then adding the stuff they need, and then running with that for > YEARS because the person who built the first one left long ago and no one > wants to break the paycheck program again. This is a very, very old problem: I was dealing with it with OS images 20 years ago. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Relocate RPM database to /usr (System-Wide Change proposal)
On Wed, 29 Dec 2021 at 16:16, Michel Alexandre Salim < sali...@fedoraproject.org> wrote: > On Wed, Dec 29, 2021 at 10:51:44AM -0500, Stephen John Smoogen wrote: > > Also please realise that the community can eat only so many changes per > > release no matter how much you want to otherwise. You can probably get > this > > OR the fs-verity in some future release, but not both in the same release > > and trying to fight for both is going to mean you get neither. In fact, > at > > a certain point too many changes start making it that any CR with certain > > names attached will get derailed because people are sure there is some > sort > > of agenda or conspiracy which they have to fight to stop. > > I'm not sure I understand this point. Chris is proposing this change, > and Neal and I listed because I volunteered to help him implement some of > the > changes. > > And Chris is not on the fs-verity change, and neither is Neal. > > My apologies.. I lost track of who was on each of these, and I did not double check before posting. That is my fault 100%. I should have just said I have Change Fatigue with all the proposed changes and have left it at that. > I do hope everyone give each other the benefit of the doubt and not jump > to assuming there is a secret agenda or conspiracy afoot -- I see this > raised against the DIGLIM proposal too, at least once -- but if it helps > make it clear that there is no agenda being pushed, I'll take my name > off this change so there is no confusion. > > My main issue is that people are seeing agenda's, and it isn't just the usual suspects of people who say NO to many changes. After rereading the changes to make sure I know who is on them, the main problem I am seeing is that there is nothing to sell why the standard Fedora contributor will want any of them in their OS. There is not even a 'Hey we did this already for this rebuild and you can play with it to see if it makes sense for you'. There does not seem to be any community involvement beforehand and nothing for people but to 'react against' the change. Add in the number of them and those reactions become looking for 'hidden' agendas etc. All that said, this isn't the first time this has happened. It is the reason why various large changes usually require groups to sort of make a sample sausage for people to eat before they have to sit through watching the larger sausage made. Because we are all going to have to do that while any of these changes are working through the grinder. > Best regards, > > -- > Michel Alexandre Salim > profile: https://keyoxide.org/mic...@michel-slm.name > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > -- Stephen J Smoogen. Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
