Re: F41 Change Proposal: OpenSSL Deprecate Engine (system-wide)
> Hi Zbigniew! > > On Tue, Apr 2, 2024 at 1:15 PM Zbigniew Jędrzejewski-Szmek < > zbyszek(a)in.waw.pl wrote: > > > Thanks. In the period between the proposal was written and published the > TPM2 provider has landed in Fedora. > PKCS#11 provider is already here for a while. The fact that such packages are physically present is not enough - they need to implement all the needed features, and they need to be mature enough to just work out of the box. Neither of these are true today, and providers just do not work for very simple use cases like signing a UKI with a yubikey. At the very least a couple more years of development and testing is needed before they are anywhere near ready to drop support for engines, that actually do work out of the box. Not to mention third party engines that are specific to internal/private build systems - if any such system runs Fedora as the build host, they'd have to migrate to Debian/Ubuntu to keep working. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F41 Change Proposal: Disable openSSL Engine Support (system-wide)
For an example of missing critical functionality, see this comment: https://github.com/systemd/systemd/pull/29539#issuecomment-1760243611 Aside from that, trying to use the pkcs11 and tpm2 providers just ended up with unintelligible errors being vomited on the console. No, I did not keep a copy of those tests, and no, I am not going to try again, as I most definitely do not have the time nor the interest to become an openssl or pkcs11 expert. This stuff needs to "just work" by default, without spending days tinkering, and engines do that just fine, because they have been developed for years. Providers are just too new, and those that exist need a few more years as optional features for enthusiasts and experts to adopt and fix all the bugs, add all the needed features, and make sure they work out of the box. We are not there yet, and jumping the gun is not going to help anybody. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F41 Change Proposal: Disable openSSL Engine Support (system-wide)
There are 2 major issues with this: 1) A lot of site-specific build systems implement signing via private/local/proprietary engines, which means those build systems will no longer be able to run on Fedora (and if this spreads to CentOS/RHEL, those too) 2) Even open source providers are still mostly broken, missing core functionality, and largely in a "developers preview" state and years of work away from being anywhere close to stability and reliability of engines. When adding engines support to various systemd tools recently, I tried to use the tpm2 and pkcs11 providers, and just gave up, as there was simply no way to make them work, they are simply not fit for purpose at this stage. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F40 Change Proposal: Unified Kernel Support Phase Two (System-Wide)
> Gerd Hoffmann this AFAIU means that we also need shim in the boot chain if we want to > support these addons. Only if you want to use certs in MOK to verify them, otherwise it's not necessary. The protocol is just LoadImage which every firmware also provides and checks against DB. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 proposal: BiggerESP (Self-Contained Change proposal)
> On Thu, May 11, 2023 at 6:26 AM Luca Boccassi > I definitely am, considering how often there's no space and how easy > it is to brick many systems through it. Right, and that's the only other way to convey this information to the bootloader. And that's one of the reasons why in sd-boot we use the filename of the UKI to do boot counting, rather than an NV var. There's really nothing else that can be used on an average UEFI system - either NV storage or ESP. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 proposal: BiggerESP (Self-Contained Change proposal)
> On Thu, May 11, 2023 at 3:33 AM Zbigniew Jędrzejewski-Szmek > > We already don't do boot counting from the bootloader side. That > happens in the operating system. boot counting in the bootloader is a good thing to have, as that's the only place to catch a few significant failures, and also it's the place where the decision is made. If you are worried about ESP storage space usage, NV storage (and wear) should be even more concerning. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 proposal: BiggerESP (Self-Contained Change proposal)
> On Wed, May 10, 2023 at 5:34 PM Chris Murphy wrote: > > > In this idea, the dm-verity parition/file would only be accessed from the > initrd, once we have enough kernel to ability to interact with physical > storage, understand partitions, initialize dm-verity, and read *a* > partition, but potentially not enough to read from a bluetooth keyboard, > show a graphical prompt, mount / from the network, etc. > > What dm-verity provides here is a way to trust content without requiring it > to be read ahead of time, checksumed, signature checked and put into ram. We already support dm-verity for the rootfs, in both dracut and mkosi-initrd, not sure what's the issue here? > The elephant in the room is whether the "desktops in the default > installation" UKI requires piles of graphics firmware. It might not be at > all small in that case. We support optional extensions with UKIs exactly for this reason. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 proposal: BiggerESP (Self-Contained Change proposal)
> On Wed, May 10, 2023 at 2:24 PM Owen Taylor > fsverity is separate from fscrypt. We can apply filesystem authentication > today. fsverity does not protect metadata, and most importantly it does not protect the filesystem superblock. It has its uses, but this is not it. > No. It initializes the whole operating system, and then pivots the > user-space later. That's why we have to everything in initramfs. > UKIs attempt to standardize the early-stage image without attempting > to solve this problem, because a two-stage boot process requires > changing how we think about operating system initialization. > > In Windows, the Windows Boot Manager loads the NT > kernel stub from the NTFS volume, which then loads the minimal > operating system environment, and bootstraps the full Windows > experience. The Windows Boot Manager has just enough to handle > BitLocker and NTFS, and then transfers the rest to Windows itself. It is really not that different than the initrd approach. Just the storage is different, but that's easier when you own both filesystems implementations. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 proposal: BiggerESP (Self-Contained Change proposal)
> On Wed, May 10, 2023 at 03:52:51PM -0000, Luca Boccassi wrote: > > If the idea to allow a UKI to contain multiple alternate, signed, > cmdline line profiles gets accepted [1], then a "rescue" image > won't neccessarily need to be a separate image anymore. There could > just be an alternative cmdline that caused the initrd to launch in > a "rescue" / "safe" mode, and that would be nicely complemented by > an A/B scheme to cope with bad kernel upgrades. > > With regards, > Daniel > > [1] https://github.com/systemd/systemd/issues/24539 Yes, we should definitely have that. Had started scoping how to build it and encode it, but got distracted by other squirrels. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 proposal: BiggerESP (Self-Contained Change proposal)
> On Tue, 2023-05-09 at 18:32 +0200, Lennart Poettering wrote: > > It sounds reasonable for sure. > The only concern is, given Microsoft creates at most 500MB ESP > partitions, are we sure all UEFI systems out there will not choke on a > 1GB one? > > Can't we reduce the number of kernels by having *only* one UKI and a > rescue one that can be used to restore the previous working UKI from > /root if the active one fails? > > Or perhaps just have always 2 UKI (current, and former working). > Do we actually need a separate dedicated rescue UKI? Can't rescue be > implemented by booting the previously working UKI with a "rescue" > command line option ? Word of caution on 'rescue' images: MSFT just had to essentially render 10 years of recovery/install media unbootable due to the black lotus vulnerability. It was not (and still is not) pretty. When there's signatures and verifications involved, you really want an upgradable system. But if you set that whole infrastructure up, there's really not that much difference left with an A/B scheme. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Unified Kernel Support Phase 1 (System-Wide Change proposal)
> On 12/22/22 15:39, Lennart Poettering wrote: > > > > Well, the thing with a chain of trust is the fact that the only chain > the user can trust is the one that he himself or the host device he owns > and operates generated that trust of chain, from link 0 in that chain. ( > And we all know how browsers handle self signed certificates who are no > less secure than those issued ) > > If the user does not generate or otherwise have control over *all* the > links in the trust chain, that chain cant be considered trusted now can > it, which in turn begs the question why partake in this industry > security theater which may brick or otherwise make the end users life > more miserable or even exclude certain types of devices, if in the end > of the day, the host or the end user is not "secure" for it. > > Are those efforts truly for the end user or just to meet some > industry/government requirements ( some governments require backdoor > entrance(s) from vendors for "lawful inspection", backdoor(s) that might > be implement or otherwise supported in the trust chain itself if the > host or user has not full control over that chain ). For the vast, vast, vast majority of users a chain of trust that anchors to the device manufacturer is more than enough, and solves real-world problems with rootkits and such things. End users have to trust the hardware vendors anyway. For the tiny minority of use cases where that is not enough (and I'm pretty sure nobody here is actually interesting enough to fall into that group) rolling your own PK is always possible. Seriously, this "debate" is no longer a debate, it's been done and dusted in the 15 years or so secure boot has been a thing, it's been over for a long while and it's really time to stop rehashing tired and debunked nonsense. We have more important things to do. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Unified Kernel Support Phase 1 (System-Wide Change proposal)
> On Thu, Dec 22, 2022 at 12:42:38PM -0600, Dennis Gilmore wrote: > Different architectures have different boot loaders. In particular, s390x and > ppc64el are very different. The proposal is to add support for UKIs in grub2, > so > that we will cover UEFI and non-UEFI machines that use grub2. For other > architectures, we can in principle do the same thing… After all, the UKI is > just > a binary with a few sections appended. But it seems to early to think such > details far ahead. As extensively discussed, the support for separate initrds > is > not going anyway and the proposal only targets a small subset of the amd64 > space. Also, doesn't u-boot support those architectures? The latest u-boot UEFI interface seems to work quite well when I tried, including support for secure boot. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Unified Kernel Support Phase 1 (System-Wide Change proposal)
> On 22/12/2022 21:29, Chris Murphy wrote: > > I don't think so. Power outage is a very common problem in some countries. > > I still remember how unreliable FAT32 was in the Windows 9x era. You > needed to run a scandisk check after every power failure or pressing the > reset button. And sometimes your documents or other files disappeared. I > really don't want a repeat of this. As mentioned many times already, vfat here is not used to keep files open for continuous editing or things like that, where that experience might be repeated. It's single-block or as-atomic-as-it-can-be single-file swap (and with newer kernels it looks like there's actual atomic rename too). So "files disappearing" due to power outages are extremely unlikely in this particular use case. The worst case scenario is that you end up with both old-and-new UKIs, which means you can still boot (and there's no "valuable" data to be lost here, everything that goes in the ESP can be regenerated on the next boot). On the other hand reimplementing filesystem drivers in the bootloader can result in broken filesystems or worse, security vulnerabilities. This is something that actually happens and keeps happening. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Unified Kernel Support Phase 1 (System-Wide Change proposal)
> On 12/22/22 12:00, Luca Boccassi wrote: > > As Neal mentioned earlier, these mechanisms are often broken. Not only > does some firmware wind up bricking the machine when they are used, they > also may not support removing the key used to sign Windows. If the > attacker can boot Windows and measured boot is not in use, they have won. Nah, it's the other way around. Linux is severely behind Windows, and I mean really severely. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Unified Kernel Support Phase 1 (System-Wide Change proposal)
> On Thu, Dec 22, 2022 at 12:00 PM Luca Boccassi > Your concept only works in *some* enterprise hardware where it's even > possible to have full control without breaking something. Even in my > server gear, I can't do that without breaking the network firmware. If > I can't safely distrust Microsoft reliably, then it's already broken. > > Firmware trust has been broken since the very beginning, and it was > broken by design in the PC world. "My" (not really mine, but whatever) concept works on pretty much every single x86 consumer device sold in the past ~12 years, in every public cloud provider, a gazillion arm64 use cases, and more. It's only "broken" if by that you mean that an entire class of very real and very much alive in the wild malware infections are no longer possible. Of course if you do things like deleting the 3rd party UEFI CA on machines that require option roms to work without allow-listing or re-signing, then things break, but that has absolutely nothing to do with the "concept" - if you delete a trust anchor, objects trusted by it are no longer trusted in the absence of an alternative chain, that's pretty much expected behaviour. Obviously things can always be improved, and we are working on that, SBAT is the first step in that direction. But saying that the trust chain model is broken is simply untrue. > Consumer PC equipment is even worse off because of how Microsoft's > Windows requirements influence how UEFI implementations work. You meant to say "light years ahead" here - it is not even funny anymore how far behind Linux is to Windows w.r.t. security, especially in the boot process. We have been playing catch-up for 10 years and are nowhere near done. It is very fortunate for the ecosystem at large that there are people who actually understand the threat models pushing the industry forward, because if it was up to the hardware vendors computer security would still be where it was in the mid 90s. But we are working on it, and making progress - in some technical concepts we are even ahead of them right now (eg: signed TPM policies, FIDO2 for luks, Windows doesn't have anything like that), but only in PoCs. Now we need the wide adoption, and this proposal is one timid step forward in that direction. The fact that in 2022 there is still no mainstream distro that has closed the glaring security gaping hole of writable, untrusted initrd (yes some distros have non-default specialized flavours for this, but it's niche) should be a source of embarrassment for anybody who works on OS development. It is a difficult problem, but by no means impossible, and it really ought to have been fixed at least for the generic use case by now. We need to get this sorted at long last. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Unified Kernel Support Phase 1 (System-Wide Change proposal)
> On Thu, Dec 22, 2022 at 10:51 AM Lennart Poettering > > Basically, I'm saying that the model of trust is flawed because users > are unable to work with it. > > And besides, each level up is a smaller scope from the previous. A > cert trusted by shim can execute anything the firmware trusts, a cert > trusted by grub can only execute things it trusts, and finally a cert > trusted by the OS can only execute things in its context. Once we > reach the OS-level, we don't need pre-boot trust anymore. So enrolling > certificates to trust kernel modules/sysexts/etc. should not require > going down the trust levels. The OS should be able to establish its > own trust to those pieces or reject them independently. It should > certainly trust everything the lower levels trust, but there's no > reason to not allow the higher levels to establish their own scoped > trust. > > This is the flaw we have right now: we can't do that. Of course there's a reason to only allow a fully validated trust chain - not only that, but it's the very basic of the entire model, and without it the entire premise falls flat on its face. The way to enroll your own certs is via firmware-mediated mechanisms such as shim+mok, and via built-in trusted keyring. Adding arbitrary trust anchors at the OS level completely ignores the foundational principle of the whole thing. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: future of dual booting Windows and Fedora, redux
> * Vitaly Zaitsev via devel: > > > But they also say this: > > | The default state of Secure Boot has a wide circle of trust which can > | result in customers trusting boot components they may not need. Since > | the Microsoft 3rd Party UEFI CA certificate signs the bootloaders for > | all Linux distributions, trusting the Microsoft 3rd Party UEFI CA > | signature in the UEFI database increase[]s the attack surface of > | systems. A customer who intended to only trust and boot a single Linux > | distribution will trust all distributions–much more than their desired > | configuration. > > And this is an accurate description of the situation. > > Unfortunately, Fedora promoted this broken model with pervasive > cross-distribution/cross-OS trust as well. People are generally quick > to criticize those who control a PKI, but very few organizations are > willing to step up to hold the key material for the key of last resort > because of the risk inherent to that. Consequently, pretty much all > distributions hide behind the Microsoft key, instead of running their > own PKI and working with OEMs to get it accepted by the firmware. I mean there are hundreds of distributions, and hundreds if not thousands of OEM. How could that even work? _maybe_ major OEMs would pick up the phone if it's Redhat, Canonical and maybe SUSE who are calling. What about everyone else? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: report on the "ELF package notes" status
00078 FDO_PACKAGING_METADATA Packaging
Metadata:
{"type":"rpm","name":"3mux","version":"1.0.1-4.fc36","architecture":"x86_64","osCpe":"cpe:/o:fedoraproject:fedora:36"}
Displaying notes found in: .note.ABI-tag
OwnerData sizeDescription
GNU 0x0010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.2.0
Displaying notes found in: .note.go.buildid
OwnerData sizeDescription
Go 0x0053 GO BUILDID description data: 62
56 61 67 68 7a 68 5f 53 6e 72 77 70 51 48 49 73 6f 36 41 2f 4a 35 53 4b 51 55
4b 46 59 4c 43 48 41 52 71 61 7a 6d 6c 73 2f 30 73 49 6b 69 77 39 62 6b 32 31
4f 75 7a 39 69 50 77 56 4d 2f 5f 39 74 55 41 36 31 6c 36 37 75 42 57 67 74 6a
2d 48 66 38
Displaying notes found in: .gnu.build.attributes
OwnerData sizeDescription
GA$3a1 0x0010 OPENApplies to region
from 0x4f000 to 0x4f396
GA$3a1 0x0010 OPENApplies to region
from 0x4f3a0 to 0x1ceb21
--
Kind regards,
Luca Boccassi
signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: report on the "ELF package notes" status
On Fri, 2022-02-11 at 20:01 +0100, Zbigniew Jędrzejewski-Szmek wrote: > Hi! > > I did some quick'n'dirty statistics of how many ELF files successfully gained > a .package.note section. > > packages: 28742, see [1] for the list (*) > ELF files: 72464, see [2] > ELF files with .package.note: 47939, see [3] > ELF files without: 24525, see [4] > > It turns out that many of those are special files. I filtered > out '\.(mod|o|h|cmxs|go|syso|c32|fas|wcx|wdx|dsx|wlx|wfx|out|dyn.*none)$' > (**). > Most of those seem irrelevant, in particular .o hasn't been linked, > so it can't have the notes section… > > ELF files after filtering: 15723, see [5] > packages for those filtered files: 4659, see [6] > > I did some quick analysis, and the reasons why the notes section is > missing differ: > - 460 packages don't have the .fc36 tag, so they weren't rebuilt > - some packages I checked were built before the mass rebuild and > got the .fc36 disttag, but failed in the subsequent mass rebuild > - opt-outs: > 895 ghc packages > 210+ ocaml packages > 197 R packages > 30 ruby packages > 61 python packages > > That still leaves quite a lot of packages without the notes, but I think > more manual analysis would be necessary to figure out the reasons. > > Coverage: > packages: 1 - 4659/28742: 84% (***) > ELF files, excl. irrelevant: 47939 / (47939 + 15723): 75% > ELF files under /usr/lib64/: 1-8114/(8114+30320): 79% > ELF files under /usr/bin/ and /usr/sbin/: 1-5769/(5769+16321): 74% > > I think this gives pretty good coverage, esp. for libraries, but > more remains. > > [1] https://in.waw.pl/~zbyszek/fedora/f36-package-grep-4.txt > [2] https://in.waw.pl/~zbyszek/fedora/f36-elf.txt > [3] https://in.waw.pl/~zbyszek/fedora/f36-elf-noted.txt > [4] https://in.waw.pl/~zbyszek/fedora/f36-elf-unnoted.txt > [5] https://in.waw.pl/~zbyszek/fedora/f36-elf-unnoted-filtered.txt > [6] https://in.waw.pl/~zbyszek/fedora/f36-elf-unnoted-rpms.txt > > (*) Some -data and -langpack- files were excluded to avoid unnecessary > downloads. > > (**) .h is because I did some earlier grep wrong, so some ELFheader.h > was included in the list :( > > (***) an applies-to-orangies comparison, because there are many noarch > packages in the total number. > > Zbyszek Fantastic work and very interesting summary, thank you! -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Package notes feature causing build paths to be embedded
> On Fri, Feb 4, 2022 at 9:07 AM Simo Sorce > Wait what? pkgconf (our pkg-config implementation) builds on even the > weird esoteric architectures like ppc32 and m68k. What are we missing > here? Yeah, and the older implementation in Debian builds and runs fine even on architectures that have more letters in their abbreviated names than users: https://buildd.debian.org/status/package.php?p=pkg-config I mean, if it can run on a Motorola CPU, I'm pretty sure it can run anywhere :-) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Package notes feature causing build paths to be embedded
> On 03. 02. 22 16:36, Simo Sorce wrote: > > I've just tried to build python-gssapi with notes enabled after krb5 was > fixed > and it builds fine. > > See https://src.fedoraproject.org/rpms/python-gssapi/pull-request/4 It looks like it's not the first time that krb5-config causes issues because it picks up state from the build env: https://bugzilla.redhat.com/show_bug.cgi?id=1997021 https://bugzilla.redhat.com/show_bug.cgi?id=1204646 https://krbdev.mit.edu/rt/Ticket/Display.html?id=8159 We had similar problems long ago with libpcap's own tool, before the project made available a pkg-config file, which is much simpler and nicer (and doesn't fall apart when cross-compiling). Is there any reason to keep using these specialized binaries over pkg-config these days? Or is it just a matter of doing the work to gradually switch over? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
> On Wed, Dec 15, 2021, at 1:45 PM, Luca Boccassi wrote: > > Hmm. Some interesting stuff going on there but I would have started with a > new SELinux > access vector. That'd allow fine-grained constraints, e.g. disallowing > `init_t` but > allowing `unconfined_service_t`. Possibly also landlock should be able to > hook into this. > IOW it's not clear to me that a new LSM is the best thing for the ecosystem > here. > > But bigger picture I'd agree that fs-verity is significantly stronger when > coupled > with such a policy - strong enough to block exploits like the runc one: > https://unit42.paloaltonetworks.com/breaking-docker-via-runc-explaining-c... We use this in production, and those kind of constraints are just not enough, because there's too many obvious ways around them. What we needed is for the kernel to enforce that only signed code shall run, and thus IPE was born, so that when coupled with dm-verity/fs-verity it allows to enforce a policy like that. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
> We don't have a proof of concept for the LSM module. I agree with you that in > practice > it would probably need to implement some kind of "list of files we care > about", > but I do not have an intelligent opinion about that. > > Based on Roberto's comment in a different sub-thread, there could be some > ongoing work > integrating with IMA that might be past the hypothetical stage? IPE supports fs-verity and dm-verity, and allows to write a policy such as "only allow execution of binaries/libraries from fs-verity verified files or dm-verity verified volumes): https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd3...@linux.microsoft.com/T/ https://microsoft.github.io/ipe/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
> On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote: > > Any file covered by fs-verity is immutable after installation. So you > cannot modify the contents, the kernel refuses. But you can just > replace the file (like during an upgrade), and of course copy and edit > in a different location. If replaced, no fs-verity checking is done > any more by the kernel. There was some talk about high-level solution > to prevent such files from being executed, e.g. an LSM module, but no > details... (Thinking about this, it would be pretty hard, because the > LSM would need to be smart enough to know which files are installed > through rpm, and which files are not. I would love to hear more details > about what is planned here.) > > Zbyszek There is such an LSM that supports fs-verity (and dm-verity), being reviewed right now: IPE https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd3...@linux.microsoft.com/T/ https://microsoft.github.io/ipe/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Mon, Nov 15, 2021 at 10:33 AM Luca Boccassi > Please mention RPMCoW directly. Mentioned and also linked the mailing list post you linked above as a reference, let me know if there's other changes I can do. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Mon, Nov 15, 2021 at 10:18 AM Luca Boccassi > Huh, I guess it was. :/ Does that paragraph address your concerns? If so, I can update it to mention RPMCoW directly, for future reference. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Mon, Nov 8, 2021 at 2:06 PM David Cantrell wrote: > > Last cycle, I brought up the problem that it being part of the ELF > data destroys a lot of the value of the RPMCoW change[1] that is also > in development for this release. I'm disappointed that the Change > authors didn't care to resolve that issue for this iteration. > > [1]: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o... Unless I'm mistaken, this is addressed indirectly by this paragraph, no? https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects#Concerns_about_additional_changes_to_files > ELF files in Fedora already vary between different package versions. The > official version of the package is embedded in the .gnu_debuglink section. > And since that varies between rebuilds, .note.gnu.build-id link which is > calculated over all sections also varies. Thus every file has two sections > that vary, and we're adding a third. So package content is already changing if any part of the source package changes, if I understand correctly. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Thu, Nov 04, 2021 at 10:26:20AM +, Zbigniew Jędrzejewski-Szmek wrote: > > But a binary representation of that could strip it down into ~50%. Would it though? Have you tested and checked it to make that determination? Can you share the code to reproduce it? The values necessarily have to be strings, since the data varies unpredictably. The longest data fields are most likely going to be CPE and debuginfod URLs, package name and package version, which can be in the worst cases a few dozen bytes each. And to avoid completely throwing extensibility and flexibility to the wind, which is one of the key aspects of the proposals, the keys have to be strings too, so that each producer can add new fields if they want to, and users/support/maintainers can read them just fine without knowing all possible keys in advance. So at most you'd save from the json's quotes, commas and colons - that's 6 bytes per field. With 5 fields, that's 30 bytes saved without considering the space needed for the header encoding sizes and whatnot, on a total of ~200 bytes, with a considerable worse usability (custom format needing its own parsers and builders instead of well-known and understood json). It doesn't seem worth the trouble. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Thu, Oct 28, 2021 at 09:30:18PM +0200, Lennart Poettering wrote: > > I was thinking more about this proposal over the past weekend and > where I keep ending up is that this is really optimizing for a small > use case by touching ELF metadata all over the system. And that > strikes me as pretty invasive, so is it worth the tradeoffs and risks > and such? Well, whether it's small or not is subjective, I think. CBL-Mariner was the first distro to add support for this, and I can tell you with certainty that it's not small at all for the internal use cases where this is employed. I mean build-ids too, if everything goes well and nothing ever crashes or needs debugged, are never needed. So one could say that even adding that to the ELF header is touching everything for a small use case. But I don't think it would be fair, because even if most of the times things are just working and you don't need to debug anything, it's when things start to go very wrong that you want to have as much help and as many tools at your disposal as possible. Even if it was a small use case, support engineers and maintainers have such a hard job already, it seems to me it would be worth making some of it easier if the cost is just a few KBs in disk space in the typical scenario. > Debugging is a pain, and anything to make that easier is better. It > has been stated multiple times that the information needs to be in the > ELF header because containers and images may lack an RPM database. > Fair, but what about the users that both want a container and image > without the RPM database and systemd-coredump? They still have all of > their ELF files with this information that they removed in other ways. > Do we provide those users with a script to strip .gnu.notes from > everything or is that even a use case of concern? > > Efforts to get the system very small for container and image use has > been a goal for a while. And sure we're not talking about a lot of > data, but that's now. The size of everything only grows, so is that > something to consider with the implementation of this feature? Note that systemd-coredump is on the host, not in the container. This is one of the points of the proposal, and mentioned in the wiki: core files are collected and analyzed on the host, not in container guests, so one of the reasons an rpm/dpkg/whatever database won't help is because they don't necessarily match. So users don't have to include systemd-coredump in their container images at all, before or after this proposal. Moreover, the difference in size in a typical small container between an rpm/dpkg/whatever database and the overhead of these notes is several orders of magnitudes - several megabytes vs several kilobytes. Again this comparison is done on the wiki: https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects#Why_not_just_use_the_rpm_database.3F Finally, if one is really convinced that they don't need this and want to strip it out, there's not even the need for a script, objcopy supports removing notes out of the box with objcopy --remove-section. > Another thing I thought about were reproducible builds. Does this > impact reproducible builds and if so, how do we handle that? It does not impact that, this is covered on the wiki: https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects#Won.27t_this_affect_the_Reproducible_Builds_effort.3F > I would feel more comfortable with this proposal if the data for > systemd-coredump was not part of the ELF metadata. Or if it > absolutely must be part of the ELF metadata, users should know how it > can be removed. I would also vote for a format other than JSON, but > that's just me. At MSFT we tried several options, but as far as we could tell the only way to be 100% certain that the metadata is always included automatically in the corefile if present in the binary is by using an allocated ELF PT_NOTE. Note that systemd-coredump is just one of the possible consumers - again at MSFT we have other internal tooling consuming it as well. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Thu, Oct 28, 2021 at 07:37:27PM +, Zbigniew Jędrzejewski-Szmek wrote: > > These are reasonable examples that demonstrate how developers and > users could benefit from the change proposal. Could more things like > this be added to: > > https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects... Copied as a paragraph here: https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects#Why_do_this_in_Fedora.3F ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On 10/29/21 3:53 PM, Lennart Poettering wrote: > > Does there need to be any parsing at all? WireGuard avoids the problem > by only using fixed-size fields, so one only needs to check that the > field is of the correct length. Qubes OS uses the same solution in > at least its GUI protocol. > > Sincerely, > > Demi Marie Obenour (she/her/hers) Different vendors have different requirements, so one of the goals was to be very specific about the elf format so that it's easy on the tooling (specific note name, id, owner, alignment, padding, readonly, section), but very open ended on the payload so that each vendor can add as many or as few key/value pairs as they need/cam afford. Furthermore, if you start asking questions like "what's the longest version a package can have", the answer can be extremely surprising - some time ago someone checked the pathological corner cases in Debian and it was like a hundred characters or so (!!) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Thu, Oct 28, 2021 at 06:39:38PM -0000, Luca Boccassi wrote: > > It depends on how wide of a net you cast. Since package naming is > user-controlled and distribution-wide rules are enforced by disparate > build systems and environments, an NVR (or NEVRA) is not unique. It's > close to unique, but it can't be guaranteed. Sure, if one squints hard enough and goes looking for trouble and to intentionally create conflicts, you can do it. No good reason to do intentionally though. And most importantly, the build-ids are going nowhere, so disambiguating is always going to be possible even in the most extreme of corner cases. I mean even build-ids, if you really want to intentionally make a mess, nothing stops you from duplicating one. They are not signed or assigned from a central authority. But it would be pretty weird, no? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Thu, Oct 28, 2021 at 2:40 PM Luca Boccassi > It is not enough. It's not enough in *any* distribution, because > people can (re)build and deploy the same NEVRA. You *need* a build-id > to guarantee uniqueness of the binary. If the NVR is the same but the > build has been modified, the build-id changes. > > Debian has the same problem, especially when someone uses an Ubuntu > package on Debian or vice-versa. NEVRAs are *not* globally unique. Maybe I'm misunderstanding the use case, but in Debian when a build of the same source is done again (eg: library ABI transition), the revision gets +bX appended, so the metadata changes. I know Suse does the same on OBS, there's both a build counter and a source checkout counter (ie, download the same source twice and it gets an incremental revision). But anyway, the build-id doesn't go anywhere anyway? It's there and collected too by systemd-coredump/coredumpctl. > I think you've already failed at that. This would not help solve that > problem, only guarantee that you need to. Well we use this system internally at $work already, so I know for a fact that it does help. In some cases one has the luxury of being able to ignore bug reports, in others, not so much. > Even if we did this, it will be a long, long, long time before there > will be interop between Fedora and Debian. Of course, that's understood. It's going to be a long and difficult road. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Mon, Oct 25, 2021 at 07:26:47PM +, Zbigniew Jędrzejewski-Szmek wrote: > > Thanks for revising the change proposal and filling in more details. > After reading through it, I have some questions: > > 1) The proposal notes that users tend to combine built packages from > different distributions. Even in the current environment, do we care > about those use cases without also getting a reproducer for Fedora? > For me, I feel that in a situation like that where a user has > submitted a bug report that implies using a binary from some other > distribution will lead me to ask "ok, but does this happen with the > packages provided in Fedora? If so, how can I reproduce the problem > locally?" So while these scenarios are described in the proposal, are > they something that Fedora is trying to support? There are and there will be some who do care, and whose life will be made oh so much easier. Maybe they will be Fedora developers, or maybe they will be just Fedora users. Maybe it's someone managing a bunch of containers, and one of them happens to be Fedora, so it won't be you receiving the bug report, but someone else. We are trying to enable a cross distro specification, and this is part of building a solid base upon which others can build on. That should be enough already, no? > 2) The proposal is built around using the package NVR to indicate > where it came from. But those names aren't unique. In some cases > it'll work, but in cases where the noted package cannot be found or > has been reaped or is just otherwise unavailable, you're back to > asking for a reproducer on a Fedora release, right? Does the NVR data > save much work over having build-ID plus debuginfod? That's not > rhetorical? I don't have many bug reports that are not resolvable by > just talking through a reproducer and seeing it happen locally, but I > know I'm not a control case. Isn't the combination of distro name + distro version + package name + package version + package arch enough to uniquely identify? Are there cases where there can be duplicates in Fedora? Speaking of the Debian case, the distro version isn't even needed, you won't have duplicates even across multiple releases. > 3) The proposal notes making crash reporting more user-readable. NVRs > instead of Build-IDs, for instance. Why can't systemd ask debuginfod > for that information for reporting? Why does this need to be embedded > in the ELF objects? If it's a value-add, then it could happen if > debuginfod is set up and just have it fall back on the current > reporting mechanism. First of all, we do not want to do network accesses from systemd-coredump, and keep any other system accesses to the bare minimum possible. It runs sandboxed, because core files are fundamentally unknown untrusted data, so the process doing that is restricted as much as possible. Secondly, internet access might be forbidden in the setup where a problem is happening. Thirdly, maybe it's the components that allow you to do network access in the first place that are crashing, and all you have is a serial console and desperation. Finally, as Mark said, with this scheme you can actually enable what you propose for the cases where it's possible, because as part of the metadata file you can include the debuginfod URL. Please think bigger picture than single distro: maybe the entity that created the binary uses the federated service so the build-id is enough, but maybe it does not. Fedora can be a trailblazer as always and be one of the first adopters (although CBL-Mariner beat you folks for first place :-) ) but our desired goal is very much to have this enabled cross-distro, so that a Fedora container on a Debian host or whatnot still works out of the box. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On Wed, Oct 27, 2021 at 02:10:37PM +0100, Daniel P. Berrangé wrote: > > It breaks libguestfs. It also breaks basic stuff like "what is > installed in this container?" "Is this file owned by RPM?" "Has > this > file been modified?" I think deleting the RPM database is broken, and > tools that do this should be corrected. > > Rich. That is not going to happen, because people doing it do not care about any of that in the slightest, and have very different needs. And that's ok, your requirements != everyone else's. And as mentioned already, dracut does exactly that for initrds in Fedora. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On 27/10/2021 21:38, Luca Boccassi wrote: > > I will ask additional information from user if the bug report has no > useful backtrace. Which you might get or not, and might be correct or not. This is what we experience daily - just because you are lucky and don't need it, it doesn't mean nobody else does. See: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DNCBX76FW5Y7OAA2BWXQZKSVX7LXS6MD/ > Most upstream developers are too busy and just close the bug report > without useful information such as a full crash core dump or a backtrace. Exactly, which is a bad state of affairs as bugs go unfixed and users are kept unhappy and developers are frustrated. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> Luca Boccassi wrote: > > The problem with your argument is that one "ridiculously negligible" > overhead and then another and then yet another etc. ends up accumulating and > we end up with minimum RAM and disk space requirements increased by a factor > of 10 (!) since the day Fedora was founded. > > And yes, I also complain about the other sources of bloat. They all add up, > and they are all a problem. (For the build flags, I have been arguing for > ages that we should build with -Os rather than -O2.) > > Kevin Kofler If I am reading this correctly, when Fedora was first released in 2003, common hard drive capacity was around 80 GB: https://upload.wikimedia.org/wikipedia/commons/a/a1/Hard_drive_capacity_over_time.png Today, 1 TB+ hard drives are common. Hence, even taking your x10 figure at face value, the growth of Fedora's requirements for disk space has not matched the growth of disk space availability, but it has stayed below, hence it's a win-win - you get the benefit and the cost is more than absorbed by hardware improvements. Furthermore, given an installation with the entire RPM universe installed (iirc) taking ~10 GBs, with a penalty of ~10 MBs, you'd need approximately 9000 (nine thousand) proposals "like this one here and there" to get the x10 you speak of. If I am reading correctly, there are about ~50 fesco proposals per Fedora release (https://pagure.io/fesco/roadmap?status=all), so it would take 180 releases over 90 years to get there by accumulating proposals like this one. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> Dne 27. 10. 21 v 21:35 Luca Boccassi napsal(a): > > In repository in general (can be deb, zypper, local directory). Even the > offline systems > have some repository where they > get the packages from. > > Miroslav How do you know which one is it then? You have a core file from a container long gone. Do you use dnf? zypper? apt? To which repository do you point them to? Did it even have one? And even if you have all the information, which is far from certain, what if what's crashing is what allows you to contact the remote service in the first place and all that's working is a serial console? This might not be what you personally face daily, but it's what many others do. With the current system everything needs to align just right. Most often it's true and it works beautifully, but not always. In many deployments the requirements directly forbid some of the needed pieces. What we are trying to do is make sure the bare minimum you get in a core file is usable and actionable even when everything has gone horribly wrong, because that's when you need it the most. For a ~200 bytes per binary cost, which in other situations like for example changing compiler version/flags would be perceived as being negligible. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> Daniel P. Berrangé wrote: > > Well, my take is that it is really weird that the response to "I deleted the > metadata from my container and now I cannot query the very metadata I > deleted." (hardly a surprise!) is "Let us just duplicate the same metadata > somewhere else, bloating the files for all users, even those who did *not* > delete the data it turns out they need." I cannot follow that logic at all. > > Kevin Kofler Whether you follow it or not, it has happened, it is happening and it will keep happening, because for others it is perfectly logical and highly desirable. So one can either stay here and complain all day long that containers are bad and they are all doing them wrong, and if they only listened to reason everything would be just perfect, or one can do something to significantly improve the baseline for everybody at a cost so ridiculously negligible that if the same standard were applied to compiler updates or changing build flags or whatnot nothing would ever, ever change. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> On 27/10/2021 18:40, Daniel P. Berrangé wrote: > > As an upstream developer, I prefer only backtraces with debuginfo installed. As an upstream developer, you get what users send you, which might or might not be what you prefer. With this change, the bare minimum produced as a corefile is useful and actionable even when everything else has failed. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> Dne 25. 10. 21 v 21:09 Ben Cotton napsal(a): > > Why you hesitate to use network? When you allow network access then > debuginfod already do > that (more or less). There can be many reasons, for example: - privacy: it reveals what you are running to an external service - security: airgapped or severely restricted networks These are real-world, existing use cases. > ??? The build-id can be > always retrieved from the dnf repository. > > Miroslav Which dnf repository? What if it's not dnf but apt? Or zypper? What if it's none? How do you know? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> * Zbigniew Jędrzejewski-Szmek: > > > The general case of any statically linked code. It could be libgcc, > startup files, the non-shared bits of glibc, static-only libraries, or > header-only C++ libraries. > > Thanks, > Florian This would be indeed useful, but quite harder to do automagically I think? Is there rpmbuild magic that figures out that a static dependency is being used? If so, new fields could be used in the note section of the ELF object, eg: "staticLibfooName", "staticLibfooVersion", when the elf object is assembled. In the Debian world we have the Built-Using annotation that goes in the package metadata, I guess I'd use that as a source of info - but it does rely on the maintainer doing the manual annotation (for some languages like golang iirc it's done automatically, but we don't have anything for C/C++ iirc). ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Package information on ELF objects (System-Wide Change proposal)
> Hi, > > On Mon, 2021-10-25 at 15:09 -0400, Ben Cotton wrote: > > I like this idea. It will be really useful when multiple distros adopt > it. > > > It isn't immediately clear to me which of the key's will be included. > The format describes 6 standard ones: type, os, osVersion, name, > version, osCpe and debugInfoUrl. But the example given later only shows > type, name, version and osCpe. > > To me the debugInfoUrl is the most interesting because combined with > the already existing build-id note it allows automagic fetching of the > debuginfo and sources (and the debuginfod server could tell you the > type, name and version of the archive, so in theory you don't have to > include those fields, but they might still be useful if you cannot > access the debuginfod url). The general spec doesn't make any field mandatory intentionally, as different consumers/producers might have different needs. One of the advantages of using JSON as a format is that it's self-describing, so you don't need to. I guess from a distro-specific implementation point of view you'd want at least: - type - name - version - architecture As that allows you to pint-point with reasonable accuracy I think? Throw in the OS name (ID or CPE from os-release) to ensure there are no possible ambiguities. debuginfo server sounds also something that would be useful for a generic-purpose distro like Fedora or Debian, absolutely. > One ELF technicality I don't see described is the alignment and padding > of the ELF NOTE. Normally on GNU systems ELF NOTEs are aligned on 4 > bytes (for both 32/64 bit ELF files). But the recent .note.gnu.property > NOTEs are aligned on 4 bytes for 32bit systems, but 8 bytes on 64bit > systems (this is causing some interesting issues trying to combine > those NOTEs...). What alignment/padding will this new ELF note have? > > Thanks, > > Mark 4 bytes alignment is shown in the examples: https://systemd.io/COREDUMP_PACKAGE_METADATA/ But yes, it should be specified I think, I'll add an explicit mention of it. It should also be read-only, otherwise there are problems consuming it. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Storing package metadata in ELF objects
On Wed, 2021-05-19 at 16:58 +0200, Mark Wielaard wrote: > Hi Guillem, > > On Wed, 2021-05-19 at 02:19 +0200, Guillem Jover wrote: > > So this is where I guess I'm missing something. To be able to make > > sense of the coredumps there are two things that might end up being > > relevant, backtraces and source code. systemd-coredump might already > > emit a backtrace, and depending on the information provided it might > > be more or less useful. If one needs the actual debug symbols there's > > already some external querying/fetching required, and if distribution > > specific source code is required because many distributions patch > > upstream source, then even more querying/fetching will be required. > > > > Which is why I'm not seeing why this standalone and isolated metadata > > would be of much help by itself. As in, the way I see it, either the > > information from systemd (w/o the extra metadata) is sufficient to > > track down bugs, or that querying/fetching would be needed anyway, at > > which point the metadata can be inferred too then? > > Because without that metadata you cannot easily figure out where/how to > get the files needed to properly track down the bugs. But using that > metadata you can figure out where the debuginfod server is that can > provide all that information for the binaries/core files (or a distro > specific method if the distributor doesn't have a debuginfod server > yet). Yes, that is a good use case. Another use case is that the person (or bot) who first looks at the journal with the crash notification and the metadata might not necessarily be the one that does the full in-depth debugging. First level support will look at things and go "oh this is from package X at version Y, therefore team Z needs to chime in". Or it can see that it's version A.B.C, which is listed as known buggy, and ignore the issue. Or a myriad of other combinations. Then there are automated systems, parsing logs and creating tickets. Having the structured metadata available in the journal in the crash message means much more complex and useful querying and ticket creation systems can be set up. You _cannot_ just go and query random remote internet servers from these systems, they need to be network-isolated, as logs can have confidential data from customers. The main point is, there are many many things that happen between a crash and the team owning the component looking up debugging symbols and loading core files. This feature is very very useful for a whole lot of those things. > > Oh, I was thinking about those mixed environments, full chroots or > > stripped down containers, from different vendors, but affecting Debian > > installations. What I'm also probably missing here is how does the > > metadata help for a third-party that is not expected to track/keep/upload > > debug symbols nor source packages into some repository, because otherwise > > I'm not seeing how they'd make use of the cores (if they are insufficient > > by themselves) to debug issues? I guess my question back would be what > > would they do with the metadata if they do not have the debug symbols > > nor the sources readily available? Also assuming of course they exercise > > good practices such as never reusing the same package-version-arch tuple > > for different builds, and similar. :) > > Different builds will have different build-ids, so they can be kept > apart. But even if without the distributor having added debuginfod > meta-data and providing a server to fetch the extra symbols, debuginfo, > sources, the extra meta-data is useful to administrators. Just seeing > that crashes are associated with specific distro/version/packages helps > narrow down instabilities. Precisely - again this is a very much real use case. In my case, the engineers doing support and looking at the logs (and only at the logs - no access to the running systems is permitted) _want_ this, because it helps them. Being able to know at a glance to the journal exactly what is borken, with version info, is extremely valuable to them. Yes the version info might not be precise for a minority of use cases that override the binary version with something different than the source version, but that's fine as it's far and few, mostly affects metapackages, and even then it can be documented clearly that the reference is to the source version, not the binary one. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https:
Re: Storing package metadata in ELF objects
On Fri, 2021-05-14 at 12:41 +0200, Guillem Jover wrote: > On Sat, 2021-04-10 at 13:38:31 +0100, Luca Boccassi wrote: > > On Sat, 2021-04-10 at 13:29 +0100, Luca Boccassi wrote: > > > After an initial discussion [0], recently we have been working on a new > > > specification [0] to encode rich package-level metadata inside ELF > > > objects, so that it can be included automatically in generated coredump > > > files. The prototype to parse this in systemd-coredump and store the > > > information in systemd-journal is ready for testing and merged > > > upstream. We are now seeking further comments/opinions/suggestions, as > > > we have a few months before the next release and thus there's plenty of > > > time to make incompatible changes to the format and implementation, if > > > required. > > I've skimmed over the discussion at [0], and while having this data > seems like it might be "nice", I've to agree with the comments there > voicing that there does not really seem to be an actual need and the > overhead and additional work do not seem worth it, TBH, at least > in the Debian context. Hi Guillem, thanks for having a look, much appreciated! Just to clarify, the need is there - this is not an experimental exercise, but it is borne out of an actual need, and it is undergoing testing right now before deployment in a large scale production infrastructure. Not _everybody_ will need it, and not everywhere - that's absolutely fair, and discussions on whether the ovearhead is worth it for something that is not universally needed, but only in certain use cases, is perfectly reasonable and welcome. I know Zbigniew is going to try and get some raw numbers on the kind of overhead we are talking about, that will hopefully help frame the discussion with more precision. > > > The Fedora Wiki and the systemd.io document have more details, but to > > > make a long story short, a new .notes.package section with a JSON > > > payload will be included in ELF objects, encoding various package- > > > build-time information like distro name, package name, > > > etc. > > > > > > To summarize from the discussion, the main reasons why we believe this > > > is useful are as following: > > > > > > 1) minimal containers: the rpm database is not installed in the > > > containers. The information about build-ids needs to be stored > > > externally, so package name information is not available immediately, > > > but only after offline processing. The new note doesn't depend on the > > > rpm db in any way. > > In the Debian context, the build-ids data is going to be available > in the affected executables, and in debug symbols packages and the > Packages metaindices listing them, so there's no need for access to > any local dpkg database. Given that someone needing to install debug > packages will need access to those indices (either with outgoing network > access or with a repository mirror), these can be queried at that time. > Not to mention that AFAIR the Debian debug symbol servers make it > possible to query for specific build-ids. This is not strictly related to debug packages, though? In fact, on systems where this could be of most use you explicitly do _not_ install debug packages (or anything at all). Or even if you wanted to, you could not - corefiles are not handled inside the container, but outside. Even if you wanted to and were allowed to (which for many environments it's not the case), you can't install a Debian debug package on a CoreOS host or Mariner host or a Flatcar host. > > > 2) handling of a core from a container, where the container and host > > > have different distros > > How each distribution handles debug packages and debug symbols is > going to be different, so it seems there will be a need for explicit > handling of these, at which point the above mentioned querying can be > implemented as well, w/o the need to encode the packaging data inside > the executable. Again, matching to debug symbols is not the main goal here, build-id works for that. The main goal is to have useful metadata immediately available in all occasions, regardless of where the core was generated on the host, without reaching out to external services, so that it is directly included and collated in the system journal when the core file is handled. With a common metadata definition, there's no need to query or explicitly handle anything - this already works if you use systemd- coredump built from the main branch, and handle a core file from different containers running different distros with binaries having this metadata in the ELF file, and it just works. This is tested, not theoretical. > > > 3) self-built and
Re: Storing package metadata in ELF objects
On Thu, 2021-05-06 at 03:17 +0200, Mark Wielaard wrote: > Hi Luca, > > On Tue, 2021-05-04 at 14:43 +0100, Luca Boccassi wrote: > > On Fri, 2021-04-30 at 19:57 +0200, Mark Wielaard wrote: > > > Is there a list of default keys (and their canonical spelling, upper- > > > lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key > > > with as value an URL pointing to the debuginfod server URL where the > > > embedded build-id executable, debuginfo and sources can be found? > > > https://sourceware.org/elfutils/Debuginfod.html > > > > The "Implementation" section of the spec lists the "main" fields: > > > > https://systemd.io/COREDUMP_PACKAGE_METADATA/ > > > > (source for that is > > https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md > > ) > > > > Would you like to send a PR to update it and add that field? > > Sorry, I don't have a github account. But attached is a patch for to > document it and one for the package-notes generator to add an -- > debuginfod argument (maybe the distro should set a default value for > that?) Hopefully those patches could be applied somehow. Hi, Thanks, opened PRs with your commits: https://github.com/systemd/systemd/pull/19523 https://github.com/systemd/package-notes/pull/8 Yes, if the distro has a debuginfod server, it should definitely be included. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Storing package metadata in ELF objects
On Fri, 2021-04-30 at 19:57 +0200, Mark Wielaard wrote: > Hi, > > On Sat, 2021-04-10 at 18:44 +, Zbigniew Jędrzejewski-Szmek wrote: > > [I'm forwarding the mail from Luca who is not subscribed to fedora- > > devel] > > On Sat, Apr 10, 2021 at 01:38:31PM +0100, Luca Boccassi wrote: > > Cross-posting to the mailing lists of a few relevant projects. > > Note that in this version of the email the [N] references in your email > don't seem to point anywhere. I found an older variant of the same > email which contained: > > [0] https://github.com/systemd/systemd/issues/18433 > [1] https://systemd.io/COREDUMP_PACKAGE_METADATA/ > [2] https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects > [3] https://github.com/systemd/package-notes Sorry about that! Must have messed up the copy > > After an initial discussion [0], recently we have been working on a new > > specification [0] to encode rich package-level metadata inside ELF > > objects, so that it can be included automatically in generated coredump > > files. The prototype to parse this in systemd-coredump and store the > > information in systemd-journal is ready for testing and merged > > upstream. We are now seeking further comments/opinions/suggestions, as > > we have a few months before the next release and thus there's plenty of > > time to make incompatible changes to the format and implementation, if > > required. > > > > A proposal to use this by default for all packages built in Fedora 35 > > has been submitted [1]. > > > > The Fedora Wiki and the systemd.io document have more details, but to > > make a long story short, a new .notes.package section with a JSON > > payload will be included in ELF objects, encoding various package- > > build-time information like distro name, package > > name, > > etc. > > Is there a list of default keys (and their canonical spelling, upper- > lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key > with as value an URL pointing to the debuginfod server URL where the > embedded build-id executable, debuginfo and sources can be found? > https://sourceware.org/elfutils/Debuginfod.html The "Implementation" section of the spec lists the "main" fields: https://systemd.io/COREDUMP_PACKAGE_METADATA/ (source for that is https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md ) Would you like to send a PR to update it and add that field? > > To summarize from the discussion, the main reasons why we believe this > > is useful are as following: > > > > 1) minimal containers: the rpm database is not installed in the > > containers. The information about build-ids needs to be stored > > externally, so package name information is not available immediately, > > but only after offline processing. The new note doesn't depend on the > > rpm db in any way. > > > > 2) handling of a core from a container, where the container and host > > have different distros > > > > 3) self-built and external packages: unless a lot of care is taken to > > keep access to the debuginfo packages, this information may be lost. > > The new note is available even if the repository metadata gets lost. > > Users can easily provide equivalent information in a format that makes > > sense in their own environment. It should work even when rpms and debs > > and other formats are mixed, e.g. during container image creation. > > > > Other than in Fedora, we are already making the required code changes > > at Microsoft to use the same format for internally-built > > binaries, and for tools that parse core files and logs. > > > > Tools for RPM and DEB (debhelper) integration are also available [3]. > > > > > -- > > > Kind regards, > > > Luca Boccassi -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change: Package information on ELF objects (System-Wide Change proposal)
> On Tue, Apr 13, 2021 at 8:09 AM Zbigniew Jędrzejewski-Szmek > > The new metadata guarantees that the ELF data churns, though. For > example, if I bump the Release in a spec file for something unrelated > to the build, all the ELF blobs change. The current state means that > this is deduplicated in RPM CoW and a very cheap upgrade, since the > binaries weren't all touched. The content of the key:value pairs is entirely under your control though - if you don't want to include the spec release field because of the case where that would be the only change to the binary, then don't. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change: Package information on ELF objects (System-Wide Change proposal)
> On Wed, 2021-04-14 at 15:29 +, Zbigniew Jędrzejewski-Szmek wrote: > > That's fair - if it were possible to get an fd during dump, we could > use fgetxattr. If not, we can use /proc/$pid/exe - even when deleted > you can interact with it: > > [malmond@malmond-x1 ~]$ ls -l /proc/$$/exe > lrwxrwxrwx. 1 malmond malmond 0 Apr 14 15:45 /proc/364665/exe -> > '/home/malmond/testbash (deleted)' > [malmond@malmond-x1 ~]$ attr -l /proc/$$/exe > Attribute "selinux" has a 54 byte value for /proc/364665/exe > > (this is me copying bash, executing it, then deleting it). My thinking > is this could go in systemd-coredump as it's invoked when dumping core > anyway. Libraries are accessible from /proc/$pid/map_files/$range. > > > I'm confused about this - I had put forth an idea for how to make rpm > create this when installing packages (so it works with older or third > party packages) but the same xattr could be created for any packaging > system. Can you clarify what is rpm dependent here? > > Matthew. Hi, There's a few issues with using xattr, some minor and one major. The minor issues is that it's really not great when you are shipping stuff around - the source/transport/medium/archiving format might or might not support it. Having to deal with this for cross-building Linux binaries from Windows with SELinux labels I can assure it's a massive headache I'd rather not replicate :-) The major issue though is a different one, if related: one of the central, nicest capabilities of this proposal is that the ELF note gets _automatically_ included in the generated core file. No change required anywhere for that to happen. This means if you are running a fleet of headless systems, and you collect corefiles on each node and ship them off for offline analysis, all the metadata is nicely included, automagically. That's because the metadata is a property of the binary and assigned at the source, not of the system where it runs on and applied at the destination. If you use xattrs though, the metadata suddenly becomes a property of the system where the binary happens to run on. The core file won't have it. The only way to see it is to have access to the system. It's no longer nicely self-contained and "replicating". Also you are no longer guaranteed to _always_ have the metadata available as long as you add it at build time - suddenly, if your binary ends up installed on the 'wrong' system/container/whatever, because they are too old or don't have a package manager or whatever else, your metadata is gone. And just to clarify, these use cases are not theoretical - they are very much real and already working. And these are the main reasons the team that handles crashdumps at Microsoft suggested adding a '.note' to the ELF rather than a new header or other proposals. I realize this is less of a problem if you look at things exclusively from the closed-loop of a single distribution building its stuff and shipping its installations only, but with this spec and implementation we are trying hard to have a general, 'world-wide' solution that works across distros, version, flavours, etc etc. In this sense, the implementation I contributed to systemd-coredump is ""just"" a reference implementation of how to parse and use the spec, but it's by no means intended to be a systemd-only affair and an internal-only protocol, ending at the new journald fields. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Storing package metadata in ELF objects
On Sat, 2021-04-10 at 13:29 +0100, Luca Boccassi wrote: > Hello, > > Cross-posting to the mailing lists of a few relevant projects. > > After an initial discussion [0], recently we have been working on a new > specification [0] to encode rich package-level metadata inside ELF > objects, so that it can be included automatically in generated coredump > files. The prototype to parse this in systemd-coredump and store the > information in systemd-journal is ready for testing and merged > upstream. We are now seeking further comments/opinions/suggestions, as > we have a few months before the next release and thus there's plenty of > time to make incompatible changes to the format and implementation, if > required. > > A proposal to use this by default for all packages built in Fedora 35 > has been submitted [1]. > > The Fedora Wiki and the systemd.io document have more details, but to > make a long story short, a new .notes.package section with a JSON > payload will be included in ELF objects, encoding various package- > build-time information like distro name, package name, > etc. > > To summarize from the discussion, the main reasons why we believe this > is useful are as following: > > 1) minimal containers: the rpm database is not installed in the > containers. The information about build-ids needs to be stored > externally, so package name information is not available immediately, > but only after offline processing. The new note doesn't depend on the > rpm db in any way. > > 2) handling of a core from a container, where the container and host > have different distros > > 3) self-built and external packages: unless a lot of care is taken to > keep access to the debuginfo packages, this information may be lost. > The new note is available even if the repository metadata gets lost. > Users can easily provide equivalent information in a format that makes > sense in their own environment. It should work even when rpms and debs > and other formats are mixed, e.g. during container image creation. > > Other than in Fedora, we are already making the required code changes > at Microsoft to use the same format for internally-built > binaries, and for tools that parse core files and logs. > > Tools for RPM and DEB (debhelper) integration are also available [3]. Wrong Fedora list address - off to a great start already :-) (fixed now) -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
