Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants
On Thu, 2020-07-09 at 16:15 -0400, Simo Sorce wrote: > However I have had bad kernels, power outages, loss of battery power > (laptops on too long suspend) and other random reasons to force > reboot > a system. That has been the primary case of file system checks > through > my Fedora usage. And luckily so far I never had a loss of filesystem > or > data that way, fsck always ended up solving most of the issues, and > whenever I lost file they ended up being temporary files I did not > care > for. > > I do not think those failures are common in Facebook fleets, so I am > quite skeptical FB data and failure modes are representative of > Fedora > usage as a desktop/laptop OS and therefore of the behavior of btrfs > in > those cases. As someone on one of the teams at FB that has to deal with that, I can assure you all the scenarios you listed can and do happen, and they happen a lot. While we don't have the "laptop's out of battery" issue on the production side, we have plenty of power events and unplanned maintenances that can and will hit live machines and cut power off. Force reboots (triggered by either humans or automation) are also not at all uncommon. Rebuilding machines from scratch isn't free, even with all the automation and stuff we have, so if power loss or reboot events on machines using btrfs caused widespread corruption or other issues I'm confident we'd have found that out pretty early on. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
jq issue with integer handling logic
Hello, the jq package currently has an unfortunate issue with handling large integers: $ echo '{"a":9011153322235679}' | jq '.a' 9011153322235680 I reported this in https://bugzilla.redhat.com/show_bug.cgi?id=2008979 a while ago and put up a PR at https://src.fedoraproject.org/rpms/jq/pull-request/4 to fix it. I think it'd be nice to get this merged and built before Fedora 35 releases if possible. For reference, the equivalent PR has already been merged in CentOS Stream 9 (https://gitlab.com/redhat/centos-stream/rpms/jq/-/merge_requests/1). Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: jq issue with integer handling logic
On Wed, 2021-10-13 at 18:53 -0400, Neal Gompa wrote: > > I can pick this up as a provenpackager tomorrow if this has been > requested and accepted as an FE for F35 and the maintainers haven't > done anything yet. Thanks, I have proposed this as a Freeze Exception: https://bugzilla.redhat.com/show_bug.cgi?id=2008979#c2 Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: jq issue with integer handling logic
On Wed, 2021-10-13 at 23:00 +, Davide Cavalca via devel wrote: > On Wed, 2021-10-13 at 18:53 -0400, Neal Gompa wrote: > > > > I can pick this up as a provenpackager tomorrow if this has been > > requested and accepted as an FE for F35 and the maintainers haven't > > done anything yet. > > Thanks, I have proposed this as a Freeze Exception: > https://bugzilla.redhat.com/show_bug.cgi?id=2008979#c2 The FE for this has been accepted for F35: https://bugzilla.redhat.com/show_bug.cgi?id=2008979#c3 Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: jq issue with integer handling logic
On Thu, 2021-10-14 at 22:41 -0400, Neal Gompa wrote: > I've gone ahead and done it, it needs karma: > > * F35: > https://bodhi.fedoraproject.org/updates/FEDORA-2021-ab44a1d0c9 > > * F34: > https://bodhi.fedoraproject.org/updates/FEDORA-2021-cdc2cb4c5a > > * F33: > https://bodhi.fedoraproject.org/updates/FEDORA-2021-f727442f9b > Thanks! These all look good to me. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Thu, 2021-12-02 at 15:08 -0500, Frank Ch. Eigler wrote: > > > === Relationship with IMA === > > > > [https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another > > technology meant to provide detection of file alterations. IMA and > > fsverity operate very differently, and are somewhat complementary. > > [...] > > Do these two systems use the same per-file signature metadata in the > RPMs? Both fs-verity and IMA use file signatures, but they each have their own dedicated flags and signing flows in RPM (e.g. see https://github.com/rpm-software-management/rpm/blob/4afe2d14d33db82ccb41c0a8d5eb1a4db90762fc/rpmsign.c for the signing implementation). The signatures themselves are not interchangeable -- fs-verity's signature is based off the Merkle tree (which itself is block-based), while IMA measures the file as a whole. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote: > On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote: > ...snip... > > > > In the context of rpm, there are two parts to this: > > * at build time, we compute the Merkle tree for the files within a > > package, then sign it and ship it as part of the rpm metadata; > > This is some kind of seperate signing that happens at build time? > > Or it's added to the rpm metadata and covered by the normal package > signing if/when the package is signed? As part of the signing flow (e.g. via rpmsign), the Merkle tree is generated and a signature is computed from it, which is then added to the rpm metadata. > > * at run time, if the fsverity rpm plugin is enabled, rpm will > > install > > the fsverity signature key and enable fsverity on files that are > > installed. > > Is this signature key the fedora rpm package signing key? > Or some seperate fsverity key? fs-verity needs a RSA key/cert pair for file signing at package signature time. At package install time, the cert needs to be loaded in the appropriate kernel keyring. We've always used a dedicated keypair during testing -- I'm not actually sure if the package signing key could be reused for this, as it's a GPG key, but this is something we should follow up on. > > fs-verity works by using a Merkle tree to generate a checksum for > > every data block in the system, and reads will fail if a single > > data > > every data block? or every packaged in rpms data block? fs-verity only operates on files where it has been enabled via its ioctl (which, if you install the RPM plugin, is taken care of by RPM on your behalf). For those, fs-verity will checksum every data block whenever it's accessed and validate it still matches. > > block read fails it’s checksum. The signature of the the file is > > validated against a public key loaded into the kernel keyring. > > Because > > fsverity operates on block reads, its runtime cost is small (as it > > only needs to verify the block that is being accessed), and it can > > protect from alterations at any point in time. > > Is this via dm_verity kernel module? Or thats a completely seperate > thing? It's somewhat inspired by dm-verity, but it's a separate implementation, the only shared logic is the hash computation code in the kernel. > > == Scope == > > > > * Proposal owners > > ** btrfs kernel enablement work > > ([ > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146054090b0859b28fc39015c7704ccc3c3a347f > > landed in 5.15]); see this > > [ > > https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-btrfs/ > > blog post] for more details > > What does this mean for all the other filesystems? They would just be > slower since btrfs is computing the trees as part of it's normal > checksumming? fs-verity requires support in the underlying filesystem. If you're using a filesystem that doesn't support it and attempt to enable fs- verity on a file, the ioctl will fail. Note that this is only a concern at runtime, not at build time. > > ** koji integration: koji will need to add the fs-verity metadata > > to > > packages when signing them > > Well, koji doesn't sign packages. robosignatory listens for messages > on > the message bus for koji tagging and based on it's config, it asks > sigul > to sign rpms and import the signatures into koji. > > There's support in robosignatory to ask to sign files (used for the > short lived IMA stuff), but I suspect it would need a new ability for > this. > > Finally who is going to write this? Change owners? > Or do you expect robosignatory maintainers to do so? Thanks for clarifying! Yes, I think robosignatory is likely what we want here. We (the Change owners) expect to do the work, though we'll likely need some advice/help around code review, testing and deployment. > > * fs-verity requires filesystem support; currently support for ext4 > > and f2fs is already available; support for btrfs landed in 5.15 > > No xfs support? Correct, XFS doesn't support fs-verity at the moment (though it could be implemented if one wanted to). Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Fri, 2021-12-03 at 12:21 +0100, Vitaly Zaitsev via devel wrote: > On 02/12/2021 20:36, Ben Cotton wrote: > > Enable the use of fsverity for installed RPM files validation. > > -1. RPM already supports files validation and this feature will waste > file system space. To clarify: RPM does support files validation, but fs-verity is more than just that. With RPM, the validation only happens on install time, and when one runs rpm -V manually. With fs-verity, the validation happens on-demand whenever a block of a file that originated from an RPM is accessed. This means, for example, that if an attacker replaces /bin/ls on disk with a compromised one, the next time it's read from disk (e.g. because you ran it) you will see a validation failure and the syscall will be blocked, preventing the compromised code from being executed. About filesystem usage: unless you install rpm-plugin-fsverity (which is not and will not be installed by default), there is no disk space increase for verity-signed RPM packages. If you do install rpm-plugin- fsverity, some disk space will be used for the Merkle tree as described in the Change. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Thu, 2021-12-02 at 20:05 -0500, Josh Boyer wrote: > Yes, I saw that and I appreciate it. That's a comparison between the > two implementations. I am asking about what benefits and use cases > fs-verity solves in Fedora. Right now, the change simply says: > > "The main benefit is the ability to do block-level verification of > RPM-installed files. In turn, this can be used to implement > usecase-specific validation and verification policies depending on > the > environment requirements." > > which is also largely true of IMA. The IMA change went into more > detailed use cases, which perhaps may have been it's downfall. So > can > you describe what most Fedora users would use this for or benefit > from > it? Or if "most users" is not an applicable qualifier, can you at > least give some more detailed use cases that you would expect people > to use it for? Broadly speaking, fs-verity makes it possible to ensure that files that were installed via an RPM have not been modified. It is useful in environments where an attacker might be able to modify system files (say, replace /bin/ls with a compromised version) and you want to protect against that. For example, consider an appliance-like system placed in an untrusted location where you may not be able to control who has physical access (this could be a server, but it could also be a kiosk in an internet point or a school). In this scenario, fs-verity can be one of the building blocks to ensure and maintain system trust. This Change is mostly about putting in place the necessary plumbing for this to be at all possible. We'll try to expand the Change proposal and flesh out potential usecases a bit more. > OK. I guess I was looking for some side-by-side data comparisons in > the overhead between IMA metadata and fs-verity. "1/127th of the > original Merkel tree size" doesn't tell me much. > > Are there some test runs with numbers to show before/after data for > both the RPM size and installed FS usage? Perhaps with an example > install. The IMA change attempted to document this and seeing a 1.1% > average increase in RPM size was easier to understand. We've done some empirical testing on this (showing neglibible increases), but still need to gather more formal data. We'll try to prioritize that and add it to the Change once it's available. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Thu, 2021-12-02 at 19:10 -0500, Josh Boyer wrote: > On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel > wrote: > > > Correct, XFS doesn't support fs-verity at the moment (though it > > could > > be implemented if one wanted to). > > That means it would exclude Fedora Server and ELN as they are XFS > based. There isn't any impact on XFS-based editions by default. Building and signing RPMs with fs-verity works fine, as there's no filesystem dependency for that. Installing RPMs also works fine, with one caveat: if you're running a system with an unsupported filesystem and have rpm- plugin-fsverity (which is not installed by default), fs-verity will not be enabled for the installed files so there will be no verification (but the RPM installation itself should succeed). Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Fri, 2021-12-03 at 22:08 +, Richard W.M. Jones wrote: > I'm unclear about the threat model - this is an attacker who is > someone able to overwrite single files (eg. /bin/ls) but cannot turn > off the fs-verity system as a whole? > > Also if RPM can update /bin/ls then surely an attacker who can widely > compromise system files must also be able to update /bin/ls in the > same way? Once fs-verity is enabled for a given file (which, in the RPM case, happens at package installation time), it cannot be disabled, and the file becomes immutable. One can still rename() or unlink() it (and this is indeed how rpm is able to replace files when upgrading packages), but the actual contents cannot be altered. Where is this useful? For example, fs-verity can help in the scenario where an attacker has out-of-band access to the storage device (say, they pull a hard drive from a colo'd server or a sdcard from an embedded device, or they boot into a liveusb, or they access a VM image directly from a host). Let's say that happens, and the attacker changes a few blocks of /bin/ls on the device to make it run nefarious code. When you boot your system again, it would fail at exec() time because the Merkle tree wouldn't match. Let's say that instead the attacker mounts (or gains access to) your filesystem, unlinks /bin/ls and replaces it wholesale with a new copy (hence creating a new inode). The attacker doesn't have your signing key, so they can't resign the file and enable fs-verity on it (they could resign it with their own key, but unless they can then find a way to load its cert into the kernel keyring it won't do much good). To protect against this, you now have a few options: - you could use a LSM to enforce that exec() can only happen on files with valid fs-verity signatures; this would protect any binary - you could use a launcher booted from secure storage (say, a dm-verity volume, which could even be the initrd), and have this launcher perform the verification; this of course only protects against binaries executed from the launcher, but depending on your threat model it might be enough Like most security solutions, this isn’t a silver bullet and it’s not something that in and of itself would necessarily prevent all possible attacks. However, fs-verity can be a useful building block in a defense-in-depth approach against specific attacks, depending on your threat model. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Sat, 2021-12-04 at 09:37 -0500, Stephen John Smoogen wrote: > > Or just pad /usr/bin/rpm with some null characters at the end to break > its signature and also stop updates from happening. [Or the fs-verity > daemon which will report that these problems are occuring. ] If the attacker has filesystem access, this wouldn't work, as fs-verity makes the file immutable. If they have block level access to the underlying storage, they can alter the data blocks of the rpm binary, and that would indeed result in failure on the next exec as the signature wouldn't match. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Sat, 2021-12-04 at 23:46 +0100, Kevin Kofler via devel wrote: > Davide Cavalca via devel wrote: > > To clarify: RPM does support files validation, but fs-verity is > > more > > than just that. With RPM, the validation only happens on install > > time, > > and when one runs rpm -V manually. With fs-verity, the validation > > happens on-demand whenever a block of a file that originated from > > an > > RPM is accessed. This means, for example, that if an attacker > > replaces > > /bin/ls on disk with a compromised one, the next time it's read > > from > > disk (e.g. because you ran it) you will see a validation failure > > and > > the syscall will be blocked, preventing the compromised code from > > being > > executed. > > This means that there is a performance cost in addition to the disk > space > cost (because something has to compute those checksums each time the > file is > acessed). There's only a performance cost if fs-verity is actually enabled (which is not in scope for this proposal). The checksums are computed on- demand on a per-block basis, so you only end up checksumming the pages you're actually accessing. > It also means that it is harder for users to exercise their right > to modify the Free Software (because replacing or patching RPM- > installed > binaries will lead to them failing to execute). There's nothing stopping the user from loading their own key in the kernel keyring and then installing their own locally-build RPM that has been verity-signed with their own key. And again, this only becomes a concern if one is actually enabling fs-verity in the first place. > > Since the change also adds to the metadata in the RPM, that means > that it > also increases the size of the RPMs. With keepcache=1, this also > translates > to increased disk space use. But even if the user does not keep > cached RPMs, > the download sizes will increase, which can cost time and for some > users > even money. That's correct, there will be (modest) increase in the size of the RPMs. We're going to collect some more data to quantify this more concretely. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)
On Mon, 2021-12-13 at 16:00 +0100, Vít Ondruch wrote: > Would it be possible to document the editing of protected file in the > change proposal, probably including example of the best way to do it > (is > it possible to replace the file by symlink?) Or is there a way to > temporary enable the editing with some overlay? Is there any other way > to restore the original file except "dnf reinstall"? I've added this to the wiki: https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_the_user_modify_a_file_shipped_by_a_package_.28e.g._to_edit_a_script_while_debugging.29_.3F You could restore the original file via "dnf reinstall", or by moving it back into place (rename() and unlink() are allowed on fs-verity enabled files). Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Self Introduction: Davide Cavalca
Hello! I've been involved in the Fedora and CentOS communities for a few years now as part of the Operating Systems team at Facebook, and most recently helped drive the btrfs by default change proposal for Fedora 33. Myself, Filipe and Michel are kicking off an effort within Facebook to try and get more of our OSS software packaged in Fedora. There's a lot of software on github.com/{facebook,facebookincubator} that's likely of general interest, but it can be difficult to use due to the web of hard-to-build dependencies that is often required. We're hoping that by maintaining it in Fedora we'll be able to make it more accessible, both for users and for potential contributors. Our initial target is "below" ( https://github.com/facebookincubator/resctl#below), and we've already started working on a number of its dependencies (with a handful already available in Rawhide as of this week). Our plan is to initially co- maintain these packages between the three of us, and eventually expand as we build more internal awareness around Fedora and gather interest. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Mon, 2020-12-21 at 18:00 +0100, Tomasz Torcz wrote: > On Mon, Dec 21, 2020 at 11:28:51AM -0500, Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/RPMCoW > > # dnf-plugin-reflink (a new package): > > https://github.com/facebookincubator/dnf-plugin-cow/ > > Does not exists, but I've just noticed it mentioned in Current > Status > on Wiki: > 3.2 Github repo needs to be published Yeah, apologies for that, we wanted to get the Change proposal out asap to start the discussion and gather feedback, but a few of the pieces are still in the works. Specifically, the repo is currently pending internal review and should be out soon. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, 2020-12-21 at 10:57 -0600, Michael Catanzaro wrote: > On Mon, Dec 21, 2020 at 11:28 am, Ben Cotton > wrote: > > == Upgrade/compatibility impact == > > > > Existing systems running earlyoom will not be modified. One can > > transition to systemd-oomd via: > > > > sudo systemctl disable --now earlyoom > > sudo systemctl enable --now systemd-oomd > > Systems that were previously not running earlyoom will have > > systemd-oomd enabled by default. > > I suggest we enable systemd-oomd for everyone on upgrade, to follow > our > design goal of ensuring upgraded systems match fresh installs as > closely as practical. We had thought about that, but one concern was migrating custom configuration that one might have for earlyoom, which could be tricky. If we're ok with unconditionally migrating to oomd with its default config, that should be pretty straightforward to do. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Mon, 2020-12-21 at 12:54 -0400, Robert Marcano via devel wrote: > On 12/21/20 12:28 PM, Ben Cotton wrote: > > ... > > > > === New process === > > > > # Resolve packaging request into a list of packages and operations > > # Download and '''decompress''' packages into a '''locally > > optimized''' rpm file > > # Install and/or upgrade packages sequentially using RPM files, > > using > > '''reference linking''' (reflinking) to reuse data already on disk. > > This sound great because free space requirements can be reduced, > specially when installing new packages. > > I have experimented building very small appliances using btrfs > compression on things like /usr/share. So I think this could disrupt > this because if I am correct the extends will be first downloaded to > a > temporary directory without compression enabled. For CoW to be beneficial, the package cache should be on the same filesystem used for the bulk of the system. In this scenario, compression should work just fine, as long as it's enabled on the appropriate subvolumes. > I am happy with an option to disable this behavior. To be clear, for this Change we do not plan to enable CoW by default. If would be a user opt-in via the dnf-plugin-cow package. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Mon, 2020-12-21 at 12:48 -0500, Colin Walters wrote: > > On Mon, Dec 21, 2020, at 11:28 AM, Ben Cotton wrote: > > > > > > == Summary == > > > > RPM Copy on Write provides a better experience for Fedora Users as > > it > > reduces the amount of I/O and offsets CPU cost of package > > decompression. RPM Copy on Write uses reflinking capabilities in > > btrfs, which is the default filesystem in Fedora 33. > > A bunch of points here: > > - No, it's the default for one Edition. Others don't default to > it. And even for Workstation we can't *require* it because it's > definitely supported to use other filesystems and storage layouts. > > - Orthogonal to this, I'd also note that xfs supports reflinks too. > > Combining those I'd say instead e.g.: "Most Fedora Editions default > to a filesystem that support reflinks, e.g. btrfs or xfs" (actually I > think IoT defaults to ext4 for...probably they didn't consider it?) Thanks for surfacing this, we'll make the language clearer. About XFS: it should work, but we haven't tested it extensively, and this work has been developed primarily with btrfs in mind. > - When talking about RPMs we need to think about container images, > which use overlayfs by default, which defers to the underlying > filesystem for reflinks - so should be fine, but should be explicitly > written down (and tested) If reflinking isn't possible (which can also happen if e.g. the package cache and the system are on different filesystems) things work as normal, albeit with a performance penalty (because more I/O is required to install the package). I'll let Matthew weigh in on the other points you raised. Thanks for the feedback! Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable systemd-oomd by default for all variants (System-Wide Change)
On Mon, 2020-12-28 at 16:22 -0800, Kevin Fenzi wrote: > On Mon, Dec 28, 2020 at 07:44:04PM -, Tom Seewald wrote: > > > On Tue, Dec 22, 2020 at 6:17 PM Anita Zhang > > > > > > > > > > > Another variation on this theme: enable by default in Fedora 34 > > > Server > > > edition. And more broadly rolled out for Fedora 35. > > > > > > If it's broadly ready for Fedora 34, great. Otherwise, it seems > > > like a > > > good fit for Fedora Server edition, given sd-oomd's server origin > > > and > > > oomd2 been used in production for a number of years. It'd be a > > > significant headline feature for Server edition, especially > > > fitting > > > for the in-progress reboot of that project. Any thoughts from > > > Server > > > WG folks? > > > > I don't think enabling systemd-oomd for Fedora server by default > > makes a lot of sense right now. Why would we want to automatically > > enable systemd-oomd in cases where users either have to manually > > manage cgroups or risk a worse experience than what currently > > exists with earlyoom? If a user is already creating/managing > > cgroups themselves, then manually enabling systemd-oomd would be a > > minor extra step. But if the user isn't managing cgroups (which I > > believe is the common case), then that user would be pretty > > surprised if systemd-oomd wipes out a huge swath of running > > programs that happen to be in the same cgroup. > > > > With Gnome and KDE Plasma most of the cgroup creation is done > > automatically, so it makes more sense to enable systemd-oomd by > > default as those systems are already set up for systemd-oomd to > > work well. > > I'm a bit unclear on this and perhaps the change owners could > comment: > > This operates on slices right? So in a server context each service is > in > it's own slice. So, while it might kill say a httpd slice, other > services would be fine? That doesn't seem great if you are running > a deadicated webserver, but perhaps not so bad if you are running a > bunch of different services. This is correct. For server usecases, I can see services primarily getting started via systemd or via a container manager. In both cases, every service should get its own cgroup by default. Usecases I can see as being problematic: - stuff like runit would probably put all their charges under the same cgroup - all cronjobs will be under the cron cgroup (systemd timers would work just fine though) - if one starts a long running service in an unortodox way (say, using screen), that's likely going to end up accounted as part of their user sessions All in all, I think this should be a fairly safe change for servers, as long as we document properly the potential pitfalls. > Also, perhaps there's some way to teach the web server applications > to > put say different wsgi applications in different slices, but no idea > if > any work has gone into that. Depending on the web server, this shouldn't be too difficult to do (either by leveraging template services in systemd, or by writing some glue logic that leverages the run API to setup slices directly). Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: ELN composes on mirrors
Hey folks, wanted to update on the current status here. On the ELN side, we've agreed to reduce the compose frequency to make ELN easier to mirror; this is being tracked in https://github.com/fedora-eln/eln/issues/39 On the infra side, as part of https://pagure.io/fedora-infrastructure/issue/9730 we got an rsync endpoint at rsync://dl.fedoraproject.org/fedora-eln to expose ELN composes. However, this doesn't work properly due to the use of symlinks pointing to a different NFS volume that's not part of the rsync. There's a few options to move forward: - keep things as they are, and use something like lftp when mirroring to fill in the missing files by fetching those over HTTP; this works, but it will consume extra bandwidth and cause extra load - switch ODCS to do "full composes"; this would solve the problem, but it will take up a lot of extra disk space - setup a batch job to upload each ODCS compose to a S3 bucket as it's completed, which mirrors can then sync down; this solve the disk space issue, at the cost of taking up some extra upload bandwidth after each compose None of these are ideal tbh, so if folks have better ideas I'm all ears. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
ELN SIG Meeting Minutes (2021-06-04)
#fedora-meeting: Fedora ELN SIG (2021-06-04) Meeting started by dcavalca at 16:02:29 UTC. The full logs are available at https://meetbot.fedoraproject.org/fedora-meeting/2021-06-04/eln.2021-06-04-16.02.log.html . Meeting summary --- * init process (dcavalca, 16:03:10) * Old Business (dcavalca, 16:05:55) * New Business (dcavalca, 16:08:09) * Broken Compose and detection (dcavalca, 16:08:25) * LINK: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/toddlers/templates/cron_playtime.yml is the example I was pointed towards (dcavalca, 16:16:08) * Tracking side-tags in Rawhide (dcavalca, 16:20:54) * AGREED: focus on tracking regular side tags first, and tackle the long-term ones later on (dcavalca, 16:30:04) * ACTION: bookwar[m] setup the dummy pipeline so that we can iterate on the script (dcavalca, 16:30:45) * ELN branch request policy (dcavalca, 16:31:12) * ELN branch request policy draft: https://hackmd.io/OoLbOauKR7WexZ0Rdm-XiA (dcavalca, 16:31:41) * Clarifying guidance on conditionals (dcavalca, 16:35:10) * AGREED: Conditionals should be avoided where possible. The default state should be to assume the build will be performed for current Rawhide. (+6, 0, -0) (dcavalca, 16:47:39) * ACTION: Eighth_Doctor submit a PR to clarify the docs as agreed (dcavalca, 16:51:39) * Open Floor (dcavalca, 16:52:15) Meeting ended at 17:00:01 UTC. Action Items * bookwar[m] setup the dummy pipeline so that we can iterate on the script * Eighth_Doctor submit a PR to clarify the docs as agreed Action Items, by person --- * bookwar[m] * bookwar[m] setup the dummy pipeline so that we can iterate on the script * Eighth_Doctor * Eighth_Doctor submit a PR to clarify the docs as agreed * **UNASSIGNED** * (none) People Present (lines said) --- * dcavalca (78) * sgallagh (27) * tdawson (23) * bookwar[m] (19) * jforbes (18) * zodbot (16) * Eighth_Doctor (14) * michel (7) * cyberpear (3) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F35 Change: Make btrfs the default file system for Fedora Cloud (System-Wide Change proposal)
On Wed, 2021-06-09 at 19:53 +, Zbigniew Jędrzejewski-Szmek wrote: > Yeah, I think we have to accept that there won't be any kernel support > in RHEL in a timeframe that matters for Fedora, and the RHEL host will > not be > able to mount the images natively. So the questions for me are: > 1. I this a problem in practice? I.e. how often do people need to use > Fedora > images for containers on RHEL hosts? > 2. What workaround can we put in place? Something in EPEL or dkms > module with > btrfs in a copr repo? I've put together a Fedora-based libguestfs container that should address some of these issues and allow accessing and manipulating btrfs filesystems even on RHEL hosts that do not support it. It's currently in review at https://bugzilla.redhat.com/show_bug.cgi?id=1970071 and I'd appreciate any feedback you might have there. In addition to this, we're exploring the possiblity of developing a userspace btrfs-fuse implementation by leveraging the existing logic in brtfsprogs, which could also provide an alternative access option. On the CentOS side, the Hyperscale SIG is working on a btrfs-enabled kernel for CentOS Stream: https://pagure.io/centos-sig-hyperscale/sig/issue/7 . There's also a kmods SIG currently being proposed (https://wiki.centos.org/SpecialInterestGroup/Kmods) that could be a place for distributing a btrfs module for RHEL / CentOS stock kernels for the time being. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: ELN SIG First Meeting
On Mon, 2021-03-01 at 09:26 -0500, Stephen Gallagher wrote: > I'd like to encourage anyone interested in this meeting to submit > agenda topics by replying to this email. Currently the agenda One thing I'd be interested in exploring is the feasibility of extending ELN to cover EPEL as well. This would make it easier to keep EPEL consistent between major releases (as packages would get branched automatically). It would also make it possible to test the combined ELN + EPEL snapshot and find potential issues early on in the process. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: ELN SIG First Meeting
On Mon, 2021-03-01 at 12:47 -0800, Kevin Fenzi wrote: > I'm not sure exactly what you mean here... > > I think (please correct me if I'm wrong) you are wanting "all EPEL > packages" to also be built as part of ELN and shipped as some sort of > 'EPEL-ELN' ? Yes, the idea I had in mind was that each package that currenty has an "epel8" branch would also get an "epeln" branch that would be built against ELN. Then when ELN branches into CentOS Stream, the epeln branches could be used as the starting point for the corresponding EPEL release. > The main problem with this is that "all EPEL packages" is not a > defined > set like ELN is. EPEL is not a specific collection of packages. It's > packages which have maintainers that wish to maintain them. For this > (and other) reasons we have never mass branched epel, we have always > let > maintainers branch when they wish to support that branch. > > I'm not sure if there's a way around this... I suppose we could try > and > collect a list of packages where maintainer(s) always want to branch > them, but that would need to be a living document/list and would > probibly be hard to keep up to date. Yeah, that's a good point. To be clear, I was not proposing to do this for all Fedora packages, just for the ones that were already present in the current EPEL. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: ELN SIG First Meeting
On Mon, 2021-03-01 at 15:49 -0500, Stephen John Smoogen wrote: > That would require a lot of changes in both EPEL and in Fedora. In > Fedora there is a general expectation that if a 'branch' is active > then it is maintained by someone.. usually the primary maintainer. > Many Fedora maintainers are only interested in maintaining packages > in the latest release. THis is why the ELN packages are being > 'watched/maintained' by the ELN team and sig. Maintainership usually > means dealing with bug reports, build failures, etc which can take up > a good amount of time. That's a fair point. To be clear, I would not expect this to happen by itself -- it this idea turns out to be feasible, I would be happy to pitch in and/or try and find resources to help with this from my end. > This is part of the reason why EPEL packages are not auto-forked for > each EL release. Most maintainers are only interested in supporting > maybe EL-6 or EL7 and we need to find someone new to do the EL-8 > branch. Interesting, I had not realized this was the case. My (likely naive) assumption was that if there's an epelX branch for a given package, there would likey be an epelX+1 too, and when that wasn't the case it was mostly because the maintainer hasn't gotten around it yet or nobody had asked. My assumption was that by providing a readily updated "epeln" branch we would save the maintainer some of this work. Are you saying that for some packages it is expected that the maintainer would only care about say epel7 and not epel8? In that case, do we / would we track the maintainer for epel8 specifically somewhere, if one were to volunteer? > We would need to have a dedicated team of people to do this with ELN > related items. We would also need to have additional build and > storage resources to deal with these artifacts, not alone the growth > of 'just need this' packages. Is there an easy way to ballpark estimate the resource demand that an effort like this would require? > When I was trying to make EPEL-8 1:1 with EPEL-7 I found I was having > to add more and more packages just to get the new build 'requires:' > done. I stopped around a thousand 'new' packages to the tree when I > was reminded that we don't do automatic branching for EPEL. I really > don't know how many packages would be needed to make it work, but do > know it is a full time job to get set up and keep going. While ELN is > probably 4000 src.rpms we would be looking at needing to > build/rebuild double that for EPEL. Ah, that's fair. I hadn't thought about potentially neededing to branch extra packages to meet new build dependencies, but you're absolutely right, that would be a major issue here. I should probably expand on why I'm thinking about this in the first place. I want to use ELN as a proxy for the next CentOS Stream release to streamline its qualification on our infrastructure. The idea being that if we can continously test ELN on a small subset of production, that will detect potential issues early on, allow us to get ahead of policy changes that might require internal work, and hopefully surface bugs that we can report and fix upstream long before branch time. However, we use EPEL a lot in our infrastructure, to the point that I suspect it would be difficult to do a meaningful prod-like deployment with ELN alone. I could probably hack something around this, but I figured it'd be worth to see if we can build a generic solution instead that could benefit Fedora and CentOS upstream, rather than something Facebook specific. To be clear, I'm aware that this would require work and resources within Fedora, and I'd be happy to help with both as needed if this is deemed feasible. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: ELN SIG First Meeting
On Thu, 2021-03-04 at 08:54 +0100, Petr Pisar wrote: > V Wed, Mar 03, 2021 at 09:46:52AM -0800, Kevin Fenzi napsal(a): > > All that said, we could change this and just mass branch everything > > and > > leave it to maintainers to clean up/dead.package/retire things they > > no > > longer wish to maintain. That pushes more work on them tho (as well > > as > > more releng work to mass branch, build everything, etc). > > > It would be great to notify EPEL maintainers that there is a new EPEL > 9 and > that they could try packaging previous EPEL 8 packages for EPEL 9. > Either as > an e-mail message or a bug in Bugzilla. > > But branching EPEL 9 from EPEL 8 is not helpful. At least for me as > a packager. The reason is, as Kevin wrote, that you want to base EPEL > 9 on > the latest (or very recent) Fedora sources. In my opinion the > maintainers > would end up with merging rahide to epel9 branch overrideding all > epel8 > commits with the risk of creaping unwanted epel8 tweaks there. I > ground this > claim on the fact the RHEL 9 is based on Fedora 34 where you have > different > packaging standards and package set than in RHEL 8. Yeah, if we were to mass branch, I think that branching from Rawhide would be more useful than branching from the previous EPEL. Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: ELN SIG First Meeting
On Tue, 2021-03-02 at 08:38 -0500, Stephen John Smoogen wrote: > So mainly a package maintainer only worries about what is deployed at > their workplace. And I would guess from the size of unanswered bugs > and other things, some of these maintainers did a one-time build to > get what they wanted and went onto other things. Others would update > things until it could no longer build with older RHEL-6 or 7 code and > stop maintaining it. This is where i get most of my answers.. > someone asks 'why is it not in EL-?' and I go ask the maintainers. > Usually the Fedora maintainer is like 'I don't use EL and don't want > to unless paid to' and then I find on the list of co-maintainers who > might be an EPEL maintainer. This usually gets the answer of 'look we > are a EL-6 or EL-7 shop and I can only really do it for that. Can you > find someone else?' Then someone eventually volunteers, we find out > they aren't a packager and have to go look again. Eventually we get > someone who is a packager who gets added to the packaging list for > that package and takes over that branch. > > Since EPEL was not really a major thing the Fedora maintainers or the > base community have wanted to invest in, there has been no drive to > do more book-keeping than what is needed to keep the system going. We > could track EPEL/non-EPEL maintainers in the package database system > and now we can track some level of control in pagure I believe but it > has been pretty much 'is it there? good.' Thanks for the clarification, this is helpful. I think if we could better track who maintains which release that would definitely help down the road, both for this effort and in general. Another thing to consider is allowing maintainers to say "I don't care about EPEL" outright, and then streamlining the process of branching those packages for EPEL if someone interested does show up (e.g. via the EPEL Packagers SIG). > up to 8 full time people for basic packaging, builds, dealing with > bugs, etc. [This is going on a decision that packages being > 'maintained' are only at 'it built, ship it' quality. If we need > higher levels then you will need more staff.] > 4 people to do documentation, upkeep and other management duties of > keeping things on track > 4 people to work through QA items and actually figure out tests > relevant to EPEL. > 6-8 people to design and work with the koji upstream to fix the > fedora build system to work with two different modularities. > Currently MBS can only build a limited type of modules which is what > is breaking PHP and I think perl apps. Are these MBS limitations documented somewhere? I'm not terribly familiar with this part of the infra (nor with modularity in general). > The work plan would be that most of the first 6-8 months would be > that last group working on getting a build system which works for > EPEL. Because EPEL uses a koji 'hack' to get to the RHEL binaries it > does not 'know' really what those packages are in the same way it > does for packages it built itself. The same goes with MBS. This means > that you have to come up with a way to fool them into using the > proper modules when they are needed etc. HOWEVER you also have to not > break the Fedora build system at the same time so that fedora > packages can continue to build. At the middle of the initial period > it would be either done or found to be outside of easily possible and > some other system would be needed. At that point it is either finish > it up or build a new system. > > The package branching/building/qa can go on for a subset of packages > but some will be impossible because of the modularity problem. > > After either of the systems are in place, a planning period goes into > place on how to build out modules and for what. There are some things > which make sense (PHP, python, ruby) and then there are specific > complicated apps which make sense. However there need to be > consistent rules, documentation on how to do it, and training to make > them. Also a consistent testing process would need to be done. > > By the time this is done then it will be time to work on the next set > of EL branching > > [I had an older proposal which went over the above but undercut the > number of people needed.. when we tried this with EPEL-8 it was > overwhelming at 1/2 the staff numbers so I am hoping I am getting > this right at this level.] Ok, this would definitely be a major effort, both in terms of manpower and of disruption to the infrastructure. It seems unfeasible to go down this road without resourcing and wide buy-in that this is the way to go. > I do see the need for this, but I really think this might be the > straw which broke the camel's back. ELN is already getting a lot of > pushback because the tools send regular failure notices to > maintainers who have nothing to do with ELN. Adding in 10,000 more > packages would basically push a lot of maintainers past the breaking > point. There is also the fact that ELN-E
Re: ELN SIG First Meeting
On Sat, 2021-03-13 at 17:09 -0800, Troy Dawson wrote: > Sorry for coming late to the discussion. I took a week off and all > sorts of things happened while I was gone. > > I believe Kevin and Smooge, and possibly even you Davide got this > backwards. And I think if we do this right, this can be a thing. > > When we started ELN, one of the major promises was that it wouldn't > interfere with regular Fedora work. That your average Fedora > packager > that didn't care about ELN, could continue to not care about ELN and > nothing would change. > I believe we (ELN SIG) should extend the same courtesy to EPEL and > the > EPEL community and packagers. > > The email discussion went in the direction of all the work that EPEL > would need to do to create an ELN EPEL. But we (ELN SIG) shouldn't > have expected that. We should have expected to do all the work. > > So, if we flip this around, where everything is on ELN, how would > that work. > > We create a new Fedora target and tag: eln-extra (so people do NOT > confuse it with real EPEL) > eln-extra-build inherits from itself and eln-build > If a package is built against the eln-extra target, and it is > successful, it gets tagged with the eln-extra tag. > There is a daily (or some other time period) repo creation. No > images, just a repo, like epel. > There is a list of packages, similar to the list of packages used to > create the ELN list, on some github/gitlab/pagure repo. If you put a > package on that list, you associate your name with that package. > Just like ELN, when a package on the eln-extra list gets built in > rawhide, it get's built in eln-extra. In fact, it would be best if > we > just altered the ELN trigger/periodic scripts to look at this list > along with the regular ELN list. > > What are people's thoughts on this? > No extra work on EPEL. > If someone, or some company wants to test ELN and need packages not > in > ELN, they can add the packages to the list, with their name/company > associated with that package. > It would get built, put in the repo, and they can then run their ELN > test with the package they need. > > Thoughts? Thanks Troy for taking time to put this together. I like this plan: it solves my usecase and it doesn't put undue burden onto EPEL or the individual packagers, while also leaving open the possibility of leveraging eln-extra to seed the next EPEL release if a packager so desires. How would be manage in practice the maintainership of packages in eln- extra? Would it be recorded in dist-git (coupled with the relevant ACLs to allow pushing fixes if needed), or somewhere else? Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
ELN composes on mirrors
I had filed https://github.com/fedora-eln/eln/issues/33 a while ago to track documentation improvements around how to consume ELN composes. I'd like to take a step back and propose adding them to the mirror network, akin to what we already do for Rawhide. This would make it trivial to consume or mirror them locally (e.g. via rsync), without having to hit ODCS directly. The only downside I can think of is that it'd consume a bit more storage on the mirrors themselves. Thoughts? Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: ELN composes on mirrors
On Sun, 2021-03-14 at 13:47 -0700, Kevin Fenzi wrote: > The eln composes (at least as far as I know) are done via ODCS > (on demand compose service) and are already available on the master > mirrors: > > https://dl.fedoraproject.org/odcs/production/ > (Although not via rsync currently). > > The big problem with mirroring them back in the past was that they > changed too quick. There's a compose every 3 hours I think. That > wouldn't be nearly enough time for our mirror network to keep up. > > Given that I expect the number of people who would sync this content is > so small, perhaps we could just leave them on master mirrors? > (we can enable rsync if you want... just put in a ticket). > If that proves to be too much load, we could perhaps try and sync them > to a s3 bucket or some other location? I just dont think our normal > mirror network would be a good fit here. Thanks! I'm ok with consuming these from the master mirror. I've filed https://pagure.io/fedora-infrastructure/issue/9730 to get rsync enabled for that endpoint. Once that's sorted out I'll setup a periodic sync, (daily or weekly, at least at the beginning), and republish the composes on https://mirror.facebook.net. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: [ELN] Proposal: ELN Extra
On Fri, 2021-03-19 at 16:35 +0100, Miro Hrončok wrote: > With the current way of things (that could possibly change), when > EPEL 10 is > created, ELN is long gone in the RHEL 11 world. > > I could only imagine this scheme: > > ELN -> CentOS Stream N -> RHEL N > ELN Extra -> EPEL N Next -> EPEL N > > (With heroic efforts to align the arrows on the EPEL line to happen > very soon > after the arrows on the RHEL line.) > > Even if we somehow manage do this, what benefits does it bring over: > > ELN -> CentOS Stream N -> RHEL N > Rawhide/Branched -> EPEL N Next -> EPEL N > > ? There's two parts to this. On one hand, a package in eln-extra will be continuously built against ELN, and if the build breaks fixes will be pushed to Rawhide. This means that when the time comes to branch for the new EPEL, it's likely to work out of the box. The other point (and the one I'm specifically interested in) is that this makes it easier to do continuous testing using ELN. Specifically, my plan is to deploy ELN on a small number of systems and use it to spot potential issues and changes that would otherwise only show up when one starts testing the *next* CentOS Stream release. Ideally, this will make it easier to get stuff addressed and fixed in Rawhide, long before the next CentOS Stream even branches, which should results in benefits to both Fedora and CentOS Stream. However, to do this effectively I would also need to have a subset of EPEL available, as in reality we (as I suspect most people) always deploy CentOS Stream together with EPEL. So that's where the idea of having ELN builds for (a subset of) packages currently in EPEL started from, which then evolved in the eln-extra proposal that Troy posted here. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: fedora-create-review error message
On Sun, 2022-08-07 at 20:27 -0500, Richard Shaw wrote: > I'm working on a package review for libphidget22 (rename of > libphidget), but after typing in my bugzilla credentials I get the > following: > > 32000: The method 'Bug.get' is not supported without using API keys > and the the authentication header. See > https://bugzilla.redhat.com/docs/en/html/api/core/v1/general.html#authentication > for details on using the 'Authorization' header. at > /usr/share/perl5/vendor_perl/SOAP/Lite.pm line 2855.\n"> > > Haven't seen this before, but I haven't submitted a new package in a > while either. This should have been fixed in https://pagure.io/FedoraReview/c/7330e7f1b47986211aeb2ce1059731ec6a9a1171?branch=master but I don't think that's made it to a release yet. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Announcing LLVM Snapshot Packages for Fedora Linux
On Fri, 2021-10-08 at 12:13 +0200, Konrad Kleine wrote: > Dear Fedora packagers, developers and users, > > we have some good news for you: > > We are beginning to build nightly snapshot packages of LLVM for the > latest > versions of Fedora Linux (currently 34, 35 and rawhide) for a growing > list of > architectures. > > You can grab them here: > > https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/ This is excellent, thanks for putting it together! Would you be open to adding builds for EPEL and CentOS Stream? I saw some preliminary work in that direction in https://github.com/kwk/llvm-daily-fedora-rpms/commit/cc9d02dc300aeed583ccda960e7eb16962686e33 but it was later reverted. I'd be happy to help with testing that if needed (also adding the EPEL list in case other folks are interested). Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Announcing LLVM Snapshot Packages for Fedora Linux
On Tue, 2022-02-01 at 13:59 +0100, Konrad Kleine wrote: > Davide, > > thank you for your interest in this. May I ask what plans you have > using it for? We're investigating an integration into CentOS Stream. Hi Konrad, the immediate usecase for us is making it easier to do development on BPF upstream and run the test suite. CentOS Stream would actually work well in this case, as that's what we actually run, though I don't think the delta between targeting CentOS Stream and EPEL would be that significant. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: [EPEL-devel] RHEL moving to issues.redhat.com only long term
On Mon, 2022-03-07 at 12:44 -0500, Josh Boyer wrote: > Hi Fedora, CentOS, and EPEL Communities! > > As part of our continued 3 year major Red Hat Enterprise Linux > release > cadence, RHEL 9 development is starting to wrap up with the spring > 2022 release coming soon. That means planning for the next release > will start in earnest in the very near future. As some of you may > know, Red Hat has been using both Bugzilla and Jira via > issues.redhat.com for RHEL development for several years. Our > intention is to move to using issues.redhat.com only for the major > RHEL releases after RHEL 9. Thanks for sharing this Josh. Would you be able to expand on why Red Hat chose to use Jira specifically here, and what benefits do you forsee this switch will bring to CentOS down the road? Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: [Fedocal] Reminder meeting : ELN SIG
#fedora-meeting: Fedora ELN SIG (2022-04-22) Meeting started by dcavalca at 16:02:41 UTC. The full logs are available at https://meetbot.fedoraproject.org/fedora-meeting/2022-04-22/eln.2022-04-22-16.02.log.html . Meeting summary --- * init process (dcavalca, 16:03:23) * Old Business (dcavalca, 16:05:31) * LINK: https://github.com/fedora-eln/eln/issues/87 also qualifies, but is likely workaroundable (t184256, 16:13:41) * LINK: https://github.com/rhinstaller/kickstart-tests (jkonecny[m], 16:23:03) * ACTION: Eighth_Doctor davdunc dcavalca meet with adamw to discuss openqa in the cloud to help with ELN testing capacity (dcavalca, 16:41:22) * ACTION: jkonecny[m] look into enabling kickstart tests for ELN (dcavalca, 16:42:01) * ACTION: dcavalca file an issue to get a stable boot.iso symlink for the latest compose (dcavalca, 16:46:02) * New Business (dcavalca, 16:49:46) * LINK: https://download.copr.fedorainfracloud.org/results/@meta/drgn/fedora-eln-ppc64le/04294758-python-drgn/builder-live.log.gz is one example fyi (dcavalca, 16:55:52) Meeting ended at 17:00:45 UTC. Action Items * Eighth_Doctor davdunc dcavalca meet with adamw to discuss openqa in the cloud to help with ELN testing capacity * jkonecny[m] look into enabling kickstart tests for ELN * dcavalca file an issue to get a stable boot.iso symlink for the latest compose Action Items, by person --- * adamw * Eighth_Doctor davdunc dcavalca meet with adamw to discuss openqa in the cloud to help with ELN testing capacity * davdunc * Eighth_Doctor davdunc dcavalca meet with adamw to discuss openqa in the cloud to help with ELN testing capacity * dcavalca * Eighth_Doctor davdunc dcavalca meet with adamw to discuss openqa in the cloud to help with ELN testing capacity * dcavalca file an issue to get a stable boot.iso symlink for the latest compose * Eighth_Doctor * Eighth_Doctor davdunc dcavalca meet with adamw to discuss openqa in the cloud to help with ELN testing capacity * jkonecny[m] * jkonecny[m] look into enabling kickstart tests for ELN * **UNASSIGNED** * (none) People Present (lines said) --- * dcavalca (66) * jkonecny[m] (26) * SSmoogen (16) * zodbot (14) * Eighth_Doctor (11) * t184256 (9) * praiskup (8) * michel (4) * davdunc[m (3) * asosedkin (2) * jforbes (1) * davdunc (1) * adamw (1) * sgallagh (0) Generated by `MeetBot`_ 0.4 .. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Bug of NsCDE of Fedora
On Thu, 2022-05-19 at 17:05 +0800, 邓景元 wrote: > Bug of NsCDE of Fedora > Former report of update (not responded) > https://bugzilla.redhat.com/show_bug.cgi?id=2036232 > Bug reported (now NsCDE dose not work anymore due to no catching up > with update) > https://bugzilla.redhat.com/show_bug.cgi?id=2083654 > Non-responsive maintainer check > https://bugzilla.redhat.com/show_bug.cgi?id=2088326 I'm around, and fixing this is on my list. It'll probably take a few weeks though, as I'm going on vacation starting tomorrow and will be off the grid for a while. I've added Michel as a co-maintainer in the meantime, in case he has time to tackle this while I'm out. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Orphaned packages looking for new maintainers
On Thu, 2022-06-16 at 13:40 +0200, Miro Hrončok wrote: > ubertooth orphan 0 > weeks ago I took this and submitted a fixed Rawhide build. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Orphaned packages looking for new maintainers
On Mon, 2022-07-04 at 13:41 +0200, Miro Hrončok wrote: > rust-curve25519-dalek orphan, rust-sig 1 > weeks ago I took this one and just submitted an updated build for it. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Orphaned packages looking for new maintainers
On Tue, 2022-07-05 at 22:24 +0200, Fabio Valentini wrote: > On Tue, Jul 5, 2022 at 10:18 PM Davide Cavalca via devel > wrote: > > > > On Mon, 2022-07-04 at 13:41 +0200, Miro Hrončok wrote: > > > rust-curve25519-dalek orphan, rust- > > > sig 1 > > > weeks ago > > > > I took this one and just submitted an updated build for it. > > But why? Do you need it? Not right now, but I think I will down the road for a downward dependency, so I figured it'd be easier to sort it out now that to unretire it later. > Additionally, your update didn't actually fix any of the problems > that > caused it to be orphaned in the first place ... Yeah, the f36 and f35 updates aren't submitted yet, I'm getting a 500 error from bodhi at the moment, so I'll try again later. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Orphaned packages looking for new maintainers
On Wed, 2022-07-06 at 00:43 +0200, Miro Hrončok wrote: > On 06. 07. 22 0:35, Fabio Valentini wrote: > > > > Additionally, your update didn't actually fix any of the > > > > problems > > > > that > > > > caused it to be orphaned in the first place ... > > > > > > Yeah, the f36 and f35 updates aren't submitted yet, I'm getting a > > > 500 > > > error from bodhi at the moment, so I'll try again later. > > > > No, this isn't about f36 or f35. > > > > The broken dependencies that caused the package to be orphaned are > > still there after your most recent commit (i.e. fiat-crypto and > > packed_simd2 crates are not packaged yet and corresponding optional > > features have not been disabled). So I expect the original FTI bug > > to > > be re-opened / re-filed as soon as the script that checks package > > installability runs the next time. > > I just run it: > > https://bugzilla.redhat.com/show_bug.cgi?id=2104302 Thank you both, I'll get this sorted out. I've also filed https://pagure.io/fedpkg/issue/484 which will hopefully make this harder to mess up down the road. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Rust Stack Spring Cleaning (July 2022 Edition)
On Sat, 2022-07-09 at 00:26 +0200, Fabio Valentini wrote: > - dcavalca (2): rust-esphome, rust-rustcat Fixed, thanks for the reminder. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Rust Stack Spring Cleaning (July 2022 Edition)
On Sat, 2022-07-09 at 00:26 +0200, Fabio Valentini wrote: > If you want a scripted way of adding "@rust-sig" group to many > packages, you > can generate an API token on src.fedoraproject.org (with "Modify an > existing > project") access level, and use the simple Python script from this > GitHub gist: I finally found some time to clean up my script for this and put it up at https://pagure.io/fedora-sig-onboard . This takes care of adding the SIG to the package ACL, set the BZ assignee and add the package to Anitya with the right settings. It's pretty hacky and has minimal error checking at the moment, but hopefully it can be useful to others as well. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Announcing fmt library soversion bump
On Thu, 2022-07-14 at 12:58 -0500, Maxwell G via devel wrote: > +ntfs2btrfs I just rebuilt this one in the sidetag. Cheers Davide ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure