Re: Announcing creation of Fedora Source-git SIG
On Wed, 2021-04-14 at 10:45 +0200, Tomas Tomecek wrote: ... > We are looking for maintainers of Fedora Linux packages who'd be > interested in being early adopters and give us feedback during the > development process. You don't need to do any coding unless you want > to :) I'd like to register interest in this topic on behalf of myself and Facebook. I'm the maintainer for one (very recently added) package. The source-git workflow has appeal, especially in context of other SIGs - namely CentOS Hyperscale which I'm part of. Looking forward to the meeting! Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change: Package information on ELF objects (System-Wide Change proposal)
Sorry for not responding to this in my previous reply. On Wed, 2021-04-14 at 15:29 +, Zbigniew Jędrzejewski-Szmek wrote: > I wanted to investigate this, but unfortunately, it's hard to check > right now, because all builds are non-reproducible (in the sense of > reproducible-builds.org), because we include the mtime of build > products in rpm metadata, so pretty much all binary rpms are > different. I'm thinking this isn't that important. In most current RPMs, the mtimes for files are in two places: 1. In the (main) rpm header 2. in the cpio header for the file in the payload. I can talk about the effect on RPMCow: the mtime isn't part of the identity of the file - it's just a content hash. When the files are actually installed, then the resulting inode is touch'ed to the right time. Therefore I think it's moot (MOOt?) from a CoW perspective, the reuse can happen. Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change: Package information on ELF objects (System-Wide Change proposal)
On Wed, 2021-04-14 at 15:29 +, Zbigniew Jędrzejewski-Szmek wrote: > Unfortunately this doesn't work for two important cases: > - when a binary or shared library has been replaced on disk. E.g. > it is fairly common for packages to crash on upgrade, and the crash > could be in the _old_ code. When the metadata is loaded in a section, > we get it all nice and dandy in the coredump. If it's in an xattr, > we don't or even worse, get outdated info. That's fair - if it were possible to get an fd during dump, we could use fgetxattr. If not, we can use /proc/$pid/exe - even when deleted you can interact with it: [malmond@malmond-x1 ~]$ ls -l /proc/$$/exe lrwxrwxrwx. 1 malmond malmond 0 Apr 14 15:45 /proc/364665/exe -> '/home/malmond/testbash (deleted)' [malmond@malmond-x1 ~]$ attr -l /proc/$$/exe Attribute "selinux" has a 54 byte value for /proc/364665/exe (this is me copying bash, executing it, then deleting it). My thinking is this could go in systemd-coredump as it's invoked when dumping core anyway. Libraries are accessible from /proc/$pid/map_files/$range. > - it doesn't work for non-rpm stuff. I'm confused about this - I had put forth an idea for how to make rpm create this when installing packages (so it works with older or third party packages) but the same xattr could be created for any packaging system. Can you clarify what is rpm dependent here? Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change: Package information on ELF objects (System-Wide Change proposal)
On Mon, 2021-04-12 at 23:10 +0200, Lennart Poettering wrote: > Or in other words: packaging metadata are sources too. If they change > (and a version bump constitutes a change) the output might change, > and > that's expected. What's key really is that the only things that can > effect generated output are the build/packaging environment and the > sources, but not parameters outside of that, such as the actual > wallclock. The main way that packaging "interferes" with the source is when patches are applied - the original timestamp of a tarball (for example) isn't complete enough to use for $SOURCE_DATE_EPOCH. That's fair. > > > My concern centers around the Copy on Write (CoW) use case - when > > packages are updated, some files changes, and some may stay the > > same. > > Where they are the same, we can save I/O and possibly download time > > long term. > > Reproducible builds the way they are defined do not address such > file-level CoW optimization so much. They do address CoW optimization > on a package level much more however: i.e. the same package build > will > have the same files in them, no matter what. > > Or to say this differently: if you want reproducible to work the way > ou think it should work, you'd have to start by convincing the > uptream > maintainers to kill $SOURCE_DATE_EPOCH and similar concepts, but good > luck with that. I think we should be careful to de-couple these two things. Just because $SOURCE_DATE_EPOCH is likely to affect a lot of binaries is not proof that all binaries will. I remain concerned that this proposal forces the issue and for every single version of every single ELF binary *must* be different, even if they really didn't change. The pattern I see is more automation and faster, smaller release cycles, and this forcing downloads and writes of binaries that really didn't change their code. I have just thought of an alternative proposition: for ELF objects (and ELF objects only): rpm could automatically, and systematically record the metadata in an xattr. This would work on images without rpmdb, works on most filesystem types, be serialized in archives. Most interestingly this could be implemented as an rpm plugin, and would work retroactively for packages that were built before this proposal. It could also be made to work for other packaging systems, and the tooling that reads it wouldn't need to know the original packaging system. Thoughts? Matthew ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change: Package information on ELF objects (System-Wide Change proposal)
On Mon, 2021-04-12 at 15:46 -0400, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects Putting packaging info into a binary guarantees that each successive package containing ELF binaries will not contain exactly the same binaries, even if there are no changes. Now, what I just wrote there is predicated on "reproducible builds" where the same source (including deps, headers) and the same toolchain produce the same output. This may or may not be a thing. My concern is that we completely eliminate the possibility of binaries being unchanged. My concern centers around the Copy on Write (CoW) use case - when packages are updated, some files changes, and some may stay the same. Where they are the same, we can save I/O and possibly download time long term. My recommendation here is to (continue to?) log build ids, and resolve remotely if you don't have an rpmdb to consult. Build ids are opaque and meaningless to end users, but end users aren't the target. My expectation is that any data collection around crashes needs to aggregate, and build ids are good enough to identify packages, even after the fact. Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Mon, 2020-12-21 at 11:28 -0500, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/RPMCoW > > > == Summary == > > RPM Copy on Write provides a better experience for Fedora Users as it > reduces the amount of I/O and offsets CPU cost of package > decompression. RPM Copy on Write uses reflinking capabilities in > btrfs, which is the default filesystem in Fedora 33. > I've been communicating with the maintainer of RPM on the pull request and it's become clear that this likely depends on the creation of a public, supportable API for RPM. This is not achievable within the window for Fedora 34, so I'm withdrawing the change for Fedora 34 at this time. I will continue to work on this, and expect to re-submit for Fedora 35. Just a reminder for those interested: I'm giving a talk at CentOS Dojo on this topic on Friday at 17:00 CET[2] Regards, Matthew. [1] https://github.com/rpm-software-management/rpm/pull/1470#issuecomment-772410935 [2] https://hopin.com/events/centos-dojo-fosdem ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Self Introduction: Matthew Almond
Hi everyone, Some background on me: I've been working with Linux since 1995 in a variety of capacities. I'm fascinated by file systems, configuration and system management technologies. I'm the author of RPMCoW[1]. This was recently accepted for Fedora 34. I aim to provide the new features first as patches to rpm[2] and librepo[3] along with a new package dnf-plugin-cow[4] which I will submit shortly. Given the deadlines for Fedora 34, I expect to iterate the patches against the upstream sources[5][6] in parallel with the forks/PRs on the package sources. I'm presenting (virtually) this work at CentOS Dojo @ FOSDEM 2021[7]. I anticipate the talk will be recorded. To date my best explanations have been facilitated by use of diagrams and simple animation. I expect to eventually write a white paper on this topic, but as time is a precious resource, I am inclined to focus on making this available to all first. I am excited to finally contribute back to the community which has given so much. Cheers! - Matthew [1] https://fedoraproject.org/wiki/Changes/RPMCoW [2] https://src.fedoraproject.org/fork/malmond/rpms/rpm [3] https://src.fedoraproject.org/fork/malmond/rpms/librepo [4] https://github.com/facebookincubator/dnf-plugin-cow [5] https://github.com/rpm-software-management/rpm/pull/1470 [6] https://github.com/rpm-software-management/librepo/pull/222 [7] https://hopin.com/events/centos-dojo-fosdem ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Signed RPM Contents (late System-Wide Change)
On Tue, 2021-01-05 at 13:05 -0500, Ben Cotton wrote: > == Benefit to Fedora == > > Having all files signed with a verifiable key means that system > owners can use the kernel Integrity and Measurement Architecture > (IMA) to enforce only verified files can be executed, or define other > policies. > afaik IMA has other capabilities outside file verification and trust. There is another mechanism that is scoped to this problem space: fs- verity[1]. The main difference is that IMA lets you sign things, but can't be expected to verify reads because there's only one digest per file. By contrast, fs-verity uses a merkle tree so you can verify one section of a file (e.g. mmaped) without needing to read the whole thing. The thing that's not clear to me in this change is how IMA can be used > ... to enforce runtime policies to ensure execution of only trusted files. The only way I can think to do this is to make all the signed files immutable, which is a) really annoying because you can't add hard links and b) trivial for the root user to undo. Note: support for fs-verity in RPM exists today[2], but is really new: enabling it by default is probably premature without more real world testing. One of the big downsides is that it's dependent on filesystem support. IMA is xattr based, so broad support is easier. [1] https://www.kernel.org/doc/html/latest/filesystems/fsverity.html [2] https://github.com/rpm-software-management/rpm/pull/1203 Matthew ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Tue, 2021-01-05 at 18:18 +0100, Daniel Mach wrote: > Dne 24. 12. 20 v 22:54 Matthew Almond via devel napsal(a): > > Depends on how it got there, and what you asked for. Here's some > > examples: > > > > 1. cp foo.rpm /var/cache/dnf//Packages/ && dnf install foo > > ...will fail the librepo full file check, and it'll be re- > > downloaded. > > 2. dnf install /root/foo.rpm || rpm -i /root/foo.rpm > > (not actually tested) will likely fail with CPIO/payload error > > > > Note that tools like rpm2cpio and rpm2archive will also fail on > > transcoded rpms. I have an open task to make the dnf plugin not > > transcode with 'yumdownloader' or 'dnf download' (plugin) as those > > are > > reasonable command to run. I will look at making error messages > > better > > and/or making some of these use cases work. > > > > This concerns us (speaking for RPM and DNF people) a little bit. > If the transcoded RPM cannot be used as a regular RPM, it probably > should have a different identity, for example a different suffix > than > .rpm. RPM and DNF are designed for generic use cases. I see these > transcoded packages as a "cache" tailored for btrfs based systems > only. > It would be probably good to draw a border between them. > > If I understand it correctly, the transcoding happens on each host. > Have > you considered transcoding all RPMs in a repo on server instead? Or > would that be inefficient and increase network traffic too much? > The transcoded RPM is a valid RPM for the rpm program and most use cases. It's got all the original headers, so querying it (-qp) works perfectly, and it's still signed, and it's only produced in concert with dnf/librepo which validates that the file downloaded is the one described in the repo. Notably it doesn't work with rpm2cpio and rpm2archive (and yes, I'd like to have a better story there). You typically get these through 'dnf download'. When I implemented the plugin for yum, I was able to avoid transcoding, but on dnf it's not implemented yet. Signature *verification* partially works. Everything to do with signatures on just the header works (and the header describes the payload digest). There is one specific area which needs fixed: regular RPMs are read, digested, and signature verified before decompression. We need to guard against malicious compressed payloads that either perform a DoS on space/time, or worse (but more difficult) could exploit a bug in a decompression library. I am actively working on this. The bottom line is that in every place you'd expect to see an rpm, and use it later, you have exactly the same number of things in the same place. If you want to reflink clone the dnf cache into a container - it'll work (really well). If you want to clean up the cache in some selective way. The interface for the cache remains the same. On server side: no this doesn't make sense: you'd save on decompressing, but you'd use 2x the bandwidth and space on server side. You'll also need to keep the original rpms for clients that don't or can't use reflinking, so it's more like 2.5x the space, which I think is unreasonable. I do think there's some room in the future to think and talk about how repos could be changed to take better advantage of this. I got some feedback on RPM PR ( https://github.com/rpm-software-management/rpm/pull/1470#issuecomment-754025728 ) which is somewhere along the lines of what I was already thinking. That said, I'm aware that I'm being ambitious with this change request, and I'm focused on trying to integrate things that have been written/exist and can be demonstrated first. Hope these explanations help! Thanks for the feedback :) Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Delta RPMs in Fedora 34
On Mon, 2021-01-04 at 11:25 -0500, Matthew Miller wrote: > On Sun, Jan 03, 2021 at 03:25:29PM -0800, Kevin Fenzi wrote: > > I remember when drpms landed I heard people say they choose Fedora > > because of them. That may have changed over the years I guess. :) > > and there have been only 2 or 3 reports about how few drpms exist > > in the last few years (ie, most people didn't really notice). > > Hmmm, here's an idea: what if instead of nightly drpms, we made them > only > every two weeks, but always exactly two weeks, so that people > updating on a > specific cadence would get them? There's been a lot of interesting talk about the state and future of drpm. I'd like to propose we continue the conversation about that with a different subject line :) - Matthew ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Sun, 2021-01-03 at 16:16 -0500, Colin Walters wrote: > > On Sat, Jan 2, 2021, at 10:03 AM, Zbigniew Jędrzejewski-Szmek wrote: > > > I fail to see why this would be significantly better... > > I don't claim that the "separate temporary directory of unpacked > content" is *better* - just that it's as easy to implement *and* > doesn't require an RPM format change (with all the consequent pain) > or support for reflinks from the underlying filesystem. > > > The logic to > > handle the split rpm contents would seem to be more complicated > > than the > > rewrite with /usr/bin/rpm2extents. Other comments? > > Hard to really say for sure I guess without trying to write > both. Probably the biggest impediment is that changes like that > would end up needing to be split across the librpm + zypper/rpm- > ostree/dnf tools. It wasn't an accident really that for rpm-ostree > /usr/bin/rpm is read-only - we effectively squash those layers > togther and can thus make deep changes as a single unit. > > Anyways, none of this really *requires* reflinks in any way and so > calling the Change "RPMCoW" is misleading from that > perspective. "DnfParallelUnpack" would probably be a better title, > with a dependency on "RPMFormatCowReady" or something. And then my > point is that one could do "DnfParallelUnpack" without changing the > RPM format without much more complexity, if any. Early on in this project I looked at creating all the files during download in a temporary directory. It would work. It is more filesystem type agnostic. If moving the decompression to an earlier step were the sole goal, it's reasonable. The goal of RPMCoW is to write once, and re-use data multiple times. This comes up in a number of circumstances for this proposal: 1. Reflinking allows for de-duplication of file content. Today this is only within a single RPM. I am looking at changing rpm2extents to reuse data across (cached) rpms to achieve something kind of like delta rpm. That is: if you already have file X, you don't write it, you clone it from any other rpm. 2. Reflinking allows sharing of file contents, without side effects from the installed copy. Each copy is a real, distinct file, can be deleted and or modified. Only the differences cost something, and 99% of rpms files don't get modified. The net result is that the rpm cache costs very little. 3. If you can keep a rpm cache, you can reuse the data very quickly, either to build a new rootfs in a subdir/subvolume with the same or different packages, and you can use those files for containers. This sounds similar to using snapshots, but with snapshots you're operating on a filesystem at a time, and you can only go backwards. Here you can decide what you want, and you get maximum reuse automatically. By contrast "DnfParallelUnpack" by itself, without CoW, is less useful because you will need to re-fetch and re-decompress data. Lastly, I'd like to emphasize that I'm not trying to change the "normal rpm format". Doing so would orphan every previously built and signed rpm, and would present a serious backward compatibility problem. I aim to only change how they're downloaded and stored in the cache, locally, and consumed in rpm itself within the confines of hosts that (can) enable this. - Matthew ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Wed, 2020-12-23 at 19:23 -0500, James Cassell wrote: > # Resolve packaging request into a list of packages and operations > > # Download and '''decompress''' packages into a '''locally > > optimized''' rpm file > > Please verify the signature on the downloaded RPM before > decompressing it. (Do we do this already?) > We have an opportunity to do the verification during download, but I'm not keen on it for two major reasons: 1. the transcoder would need to open the rpmdb to perform the verification, adding a fair amount of complexity. 2. (more crucially) I observe that dnf downloads packages and signatures before asking whether to trust them. The order of events means we can't be confident that all signatures are in the rpmdb yet. My proposal is for enabling CoW with dnf. The code change to do transcoding is in librepo as part of the generic file download mechanism. librepo does verify downloads relative to the repo's recorded digest. The transcoder produces a different series of bits to be written to disk, so how could that verification work? Turns out the answer is easy: we see the original bits on the input to the transcoder, so we calculate the digest of the bits received from the yum server and record this in the footer of the transcoded rpm. I've modified lr_checksum_fd() in librepo to look for this before using the xattr cache or reading the whole file again. You can only locate that whole file digest if the footer itself is complete. The digest is actually a list of digests. The default value in createrepo_c is SHA256 (irrespective of what digest algorithm is used to identify/verify files in each rpm) and for now the dnf plugin passes "SHA256" to the transcoder statically. This is ultimately repo specific. I hope to eliminate the hard coding later if there's signal within librepo to choose the right digest algo for the specific repo. The job of actually verifying the signature falls through to rpm as it did before. As stated in the proposal: the headers (lead, signature, main header) are completely untouched, so the gpg based signature is still verified as before, and at the same point in time. > > # Install and/or upgrade packages sequentially using RPM files, > > using > > '''reference linking''' (reflinking) to reuse data already on disk. > > Sounds like a great improvement! Any real-world data on how much > time it saves, how much it changes disk usage, or how much SSD writes > it saves? > Forthcoming! I've got some numbers I've used internally at Facebook to talk about this. To do this I had to write another rpm plugin to measure how much time was spent on decompressing and writing data. I'm planning on improving this and open sourcing that too. The goal here is to produce some publicly reproducible numbers. > > The outcome is intended to be the same, but the order of operations > > is > > different. > > > > # Decompression happens inline with download. This has a positive > > effect on resource usage: downloads are typically limited by > > bandwidth. Decompression and writing the full data into a single > > file > > per rpm is essentially free. Additionally: if there is more than > > one > > download at a time, a multi-CPU system can be better utilized. All > > compression types supported in RPM work because this uses the rpm > > I/O > > functions. > > I referenced above, I think each chunk should also be verified before > decompressing. > This is certainly possible, but not implemented. My thinking here is that the full rpm file digest enforced for files downloaded with dnf/librepo also covers this. The only optimization possible here is for a damaged rpm to fail faster during transcode. I consider this a pretty minor optimization. > > # RPMs are cached on local storage between downloading and > > installation time as normal. This allows DNF to defer actual RPM > > installation to when all the RPM are available. This is unchanged. > > # The file format for RPMs is different with Copy on Write. The > > headers are identical, but the payload is different. There is also > > a > > footer. > > ## Files are converted (“transcoded”) locally during download using > > /usr/bin/rpm2extents (part of rpm codebase). The > > format > > is not intended to be “portable” - i.e. copying the files from the > > cache is not supported. > > I think these should be made to be portable. How many variants of > these are there? Would it be difficult to make the transcoder also > understand RPMs transcoded for a different > platform/setup? Eventually, I'd like to see additional signatures > added to the RPM for each of the variants so RPM itself can do the > verification at install time, avoiding a transcode to the "canonical" > format. (I suppose this might require a build-time or sign-time > transcode to each of the other variants.) Until then, I'd like to > ensure that the package signatures are being verified in a secure > manner, which would be necessary for the
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
> I currently download once and upgrade three different systems by > rsync-ing the cache. > > Do I understand that this will no longer be supported or work? That's an interesting question. Is sharing the cache directory from a single host intended to be shared like this? I am guessing no, but it may still be common. It should still work, with two caveats: 1. The files in the cache will be bigger, so a simple rsync will involve more I/O, and the destination filesystem will also need more space and I/O time. 2. The systems must be the same endianness (The transcoded format doesn't bother with network order, because it's not intended to be shared) 3. The page size must be the same for reflinking to work: This is actually worked out when the filesystem is created, and defaults to the system page size, and if not the same as the current page size, the filesystem isn't even guaranteed to mount (see --sectorsize option in mkfs.btrfs man page). In reality you're quite unlikely to share packages unless the architecture were the same, which would steer both endianness and page size to the same value. That said, I'm aware that aarch64 can be flexible in both ways. I'm covering my bases with my statement: I have thought about it, and don't think I'm in any position to make promises. For this proposal: we're talking about shipping the code that would allow this to be turned on. We're not talking about enabling it by default. We can't until we have good answers to questions like this. Thanks, Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
> On Mon, Dec 21, 2020, at 1:07 PM, Neal Gompa wrote: > > Yes it does. It avoids writing the compressed data and then copying it back > out > uncompressed, which is the same amount of savings as the reflink approach. > > (It's also equally incompatible with deltarpm) > > > No - static deltas exist, plus layered RPMs work on the wire the same. But > this isn't > really relevant here. > > > Adding a hardlink indeed requires updating inodes proportional to the number > of files, but > that's more an implementation of the transactional update approach, not of the > "download and unpack in parallel" part which is more what we're discussing > here. (Though they are entangled a bit) > > Anyways, I'd still stand by my summary that the much lower tech "files in > temporary directory that get rename()d" approach would be all of *more* > efficient on > disk, simpler to implement and much less disruptive than an RPM format > change. (The main > cost would be a new temporary directory path that would need cleanup as part > of e.g. `yum > clean` etc.) I'm replying to a bunch of topics in the same thread (via the web ui because I wasn't subscribed to the mailing list until today, yikes) On editions: I wrote fedora-workstation because that's the same one that has btrfs as root by default Zero byte files: I think reflinking is specifically fine here because reflinking is about contents, not inodes. A zero byte reflink should be a no-op (on the filesystem level, but I should check, if it's not, I can special case it easily enough). The process of installing files based on reflinks involves actually opening new files, then reflinking content. On small files and alignment/waste: I believe most mutable filesystems do "waste some space". I call it out here because it's explicitly in the file format, the same as in .tar (without compression) and it's because FICLONERANGE and the filesystems demand it. I account for it as (number of files) x (native block size) / 2 - i.e. assume 50% usage of the tail of every file. The block size of ppc64 is unfortunate, but I expect the same level of waste happens whether you're using reflinking or not. Talking about the topic more broadly: The hardlinking approach in rpm-ostree depends on either a completely read-only system, or the use of a layered filesystem like overlayfs. I think it's a completely valid approach, and to my understanding, is the technology that underpins Fedora CoreOS and Project Atomic. These are different distro builds and have specific use cases in mind. As I understand it, they also have very different management policies: they are intended to be managed in a specific way, and that updates seem to require a reboot. My hope for CoW for RPM is to bring a similar set of capabilities and benefits to Fedora, and eventually CentOS, RHEL without requiring any changes to how the system works or is managed. The new requirements are fairly simple: one filesystem for the rootfs and dnf cache, and that this filesystems supports reflinking. Today data deduplication is within a given rpm. Looking forwards, I would like to extend the rpm2extents processor to read and re-use other blocks from the dnf/rpm cache and then we get full system level de-duplication. I am really grateful for all this feedback, hopefully what I write makes sense - Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
>> === New process === >> # Resolve packaging request into a list of packages and operations >> # Download and '''decompress''' packages into a '''locally optimized''' rpm >> file >> # Install and/or upgrade packages sequentially using RPM files, using >> ''reference linking''' (reflinking) to reuse data already on disk. > This sound great because free space requirements can be reduced, > specially when installing new packages. I need to re-word this: the "reuse" of data is between the locally downloaded rpm and the installed destination. I do have a plan to investigate making rpm2extents enumerate the dnf/rpm cache (if you enable it) and reflink any shared data between rpms, saving writes. Today this proposal explains that disk space requirements during updates are expected to be higher. See https://fedoraproject.org/wiki/Changes/RPMCoW#Notes item 3. > I have experimented building very small appliances using btrfs > compression on things like /usr/share. So I think this could disrupt > this because if I am correct the extends will be first downloaded to a > temporary directory without compression enabled. There is also some confusion between compressed data in the rpm and the transcoded one, and filesystem level compression. This proposal affects the former, but not the latter. I'd caution against using btrfs specific attributes to disable compression the dnf/rpm cache directory tree, because then the extents written/shared to the installed file locations will also not be compressed. (this is my interpretation of what I expect to see with FICLONERANGE ioctl etc: it'd be slower if it honored filesystem level compression because it'd need to re-write the data.) > I am happy with an option to disable this behavior. I'm unclear on which behavior you're referring to. This proposal is add support for Copy on Write in Fedora, but not make it default at this time. Thanks, Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
> I cannot find it anywhere in rpm codebase. The current status section of the proposal describes this as pending two PRs, and in the dependencies list, they're enumerated. Most of the code is in https://github.com/malmond77/rpm/tree/cow and enabled through work in https://github.com/malmond77/librepo/tree/transcode_cow > Hmm, I, personally, see much better perfomance (and storage) improvements in > enabling %_minimize_writes > however there is still https://bugzilla.redhat.com/show_bug.cgi?id=1872141 to > be resolved before this got enabled by default. I'm curious about this so I'll look at it, but at first glance it seems tangential to this proposal. Thanks, Matthew. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
