Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
Apologies for this not progressing anywhere, but the time in between has confirmed that something like this will need a general purpose use-case in rpm itself so that it can be regularly tested. We'll be exploring this area in the future, but this isn't the time, we need to focus on v6. I'm closing this now because it doesn't do anybody any good to have old PR's hanging around. I've backed up your patch for possible future use/reference. Thanks for your work on this! -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-2033653423 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
Closed #2416. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#event-12335589169 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
Also we still need a way for the plugin to claim ownership of a package to trigger the reflinking during install. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1598860857 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
Thanks for the valuable comments. I will look into creating the new FA_REFLINK file action. For the transcoding part, if we unpack the package files in a directory, I think we still need to keep the package header somewhere for rpm to process it, but this could indeed be a good approach. I will look into it as well. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1598688831 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
I don’t consider any sort of streaming unpacking to be viable with the v4 file format. For it to be viable with v6, v6 would need to use some sort of Merkle tree to allow incremental verification of the signature. One option would be to pack a bunch of hashes (as binary, not hex) into a single tag data entry in the main header, with each hash being taken over a 1 mebibyte chunk of the payload. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1597752331 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
The more I think about it, the more I think it has to be rpm that is responsible for installing files. Delegating that to a plugin creates a support nightmare where we (as an upstream) no longer have the slightest clue what might be going on in case of problems, and requiring every bug report dialogue to start with "what plugins you have installed?" question. Of the file actions, only FA_CREATE is something that could be replaced by a reflink operation. That is trying to tell me something, but not sure what. I read again the Fedora CoW proposal and it seems to me like a really complicated way of trying to do less. If the download step just unpacked the files as-is into a cache directory (say, using the hash as a filename) the install step could then just reflink the entire file from there, and no need for this complicated page-aligned transcoding etc to recrete something resembling an rpm file but with broken checksums. The plugin would basically just need to supply an open fd back for a file when asked, and rpm could take care of the actual reflink + fallback. That would be a whole lot closer to something rpm itself might one day do: unpack files into a per-fs temporary directory before moving to final location to make transactions more robust. It'd also be easy to populate such a cache for given set of packages (even the entire distro) for super-fast chroot-installs and such. Of course there would need to be a way to signal this kind of operation to rpm. rpmPackageFilesInstall() actually already has logic to run without an attached payload for the restore operation, in which case it runs in the header, not payload order, which could probably serve this usage as well. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1596910504 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
@dmnks , FA_PLUGIN and FA_CUSTOM have effectively the same problem as previous versions of this patch had with RPMRC_PLUGIN_CONTENTS: there could be multiple plugins wanting to claim it, whereas there can only ever be exactly one thing to create a file. See https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-5046976 for some background. On one level allowing a plugin to make rpm reflink a file instead of unpacking one seems like a fairly reasonable thing actually, but trying to fit this into rpm does seem like cramming a square peg through a round hole kind of thing. Which basically tells you the rpm API isn't right for the job, because it was never planned to handle such a task. Which is of course what we're discussing here, but the "seems to be two different things" issue from my earlier comment is nagging at me more and more. -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1596680739 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
> What if we instead had a FA_REFLINK file action (supported in rpm core), and > the plugin would 1) set this for applicable files 2) supply the necessary > details such as what to actually reflink to? I like this idea. In the context of `rpmFileAction`, reflinking indeed can be seen as a file action that's different from the usual `FA_CREATE` which just unpacks and copies the file. (There even used to be a `FA_COPYIN` and `FA_COPYOUT` although I'm not sure what they did and they're obsolete nowadays anyway.) -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1593735668 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
This is indeed closer to what I had in mind, but. It ends up leaking way way too many rpm internals (fsm and all) to the plugins - plugins which need to get by with just public rpm APIs. (This may have been an issue with the earlier versions as well, I don't remember or maybe I never got that far) Externalizing the fsmFileInstall() level of detail does that I guess. We need to come up with something else that doesn't require spilling rpm's guts on the street. It all gets muddied up because it's trying to be two things: support an alternate payload format AND support reflink on that. The former shouldn't require exposing too much extra (think, say, a theoretical tar archive plugin instead of cpio), the latter seems to be more problematic. What if we instead had a FA_REFLINK file action (supported in rpm core), and the plugin would 1) set this for applicable files 2) supply the necessary details such as what to actually reflink to? Since you wouldn't then have to try an replace fsmFileInstall() at all, it also shouldn't need to expose as much. I haven't really thought it through, much less tried, so consider that mostly just food for thought - would something along those lines seem useful? -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1590791031 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
@pmatilai @DemiMarie Did you get a chance to have a look at this? -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416#issuecomment-1479478779 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
@rphibel pushed 1 commit. 8a7cd8139b759d1711273839a6502333e3c0f914 Create content handler plugin hook -- View it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416/files/5965d2be5b80eea738b767a5b48c5bff50c87da3..8a7cd8139b759d1711273839a6502333e3c0f914 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
@rphibel pushed 1 commit. 5965d2be5b80eea738b767a5b48c5bff50c87da3 Create content handler plugin hook -- View it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416/files/82b6cfe2cfdc981cc7fcda4ecd473c8729aa6920..5965d2be5b80eea738b767a5b48c5bff50c87da3 You are receiving this because you are subscribed to this thread. Message ID: ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Create content handler plugin hook (PR #2416)
### Description This change creates a new `content_handler` plugin hook. This hook can be used by plugins to claim responsability for the package payload. The plugin will then be responsible for: - reading the payload - verifying it - installing the files The main motivation for this change is to enable features like [RPMCoW](https://fedoraproject.org/wiki/Changes/RPMCoW) where the package payload is not in a `cpio` format. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/2416 -- Commit Summary -- * Create content handler plugin hook -- File Changes -- M include/rpm/rpmte.h (62) M include/rpm/rpmts.h (12) M include/rpm/rpmtypes.h (10) M lib/depends.c (16) M lib/fsm.c (101) M lib/fsm.h (14) M lib/rpmplugin.h (12) M lib/rpmplugins.c (28) M lib/rpmplugins.h (9) M lib/rpmte.c (38) M lib/transaction.c (34) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/2416.patch https://github.com/rpm-software-management/rpm/pull/2416.diff -- Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/2416 You are receiving this because you are subscribed to this thread. Message ID: rpm-software-management/rpm/pull/2...@github.com ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
