Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Wed, Nov 30, 2022, at 8:11 PM, Colin Walters wrote: > > BTW I wanted to give an update here specifically regarding the "dnf > image" bit - as of late, I've been working on a fresh new "bootc" CLI, > see https://github.com/ostreedev/ostree-rs-ext/pull/412 and > https://github.com/cgwalters/bootc and that may make more sense for the > client side. This is now more real - it's been moved to github.com/containers/bootc and I've updated the Change proposal at https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable to reflect this. This is all pretty new but - again I've been searching for a succinct way to describe all this stuff, and "bootc" seems to work much better than "ostree container" or "dnf image" etc. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Mon, Nov 21, 2022, at 10:20 AM, Jonathan Lebon wrote: > On Tue, Oct 25, 2022 at 12:43 PM Colin Walters wrote: >> - This proposal is explicitly trying to tie everything together. I think >> without the "bigger picture", it's actually *more* confusing. For example, >> just pushing the container images does little unless we invest in them as a >> derivation source and build tooling and docs around them. > > I agree it's helpful to discuss this at a higher level than the > individual pieces to make sense of it. But the proposal as is seems to > imply it will all be done in one cycle which I agree with Dusty is > very optimistic. I agree. > WDYT about introducing phases into the proposal? E.g. something like: > - phase 1 (f38): start pushing OSTree-based variants as container > images in Quay.io; users can opt into rebasing onto them > > - phase 2 (f39): automatically transition existing users to shipping > from Quay.io; users can opt out. > - phase 3 (f40): stop pushing OSTree repo updates entirely I'd agree that "stop pushing ostree repo updates" probably isn't going to happen in Fedora 38. Dunno, I guess we could try a f40 change that calls that out explicitly. > Some concerns about this have been raised upstream already around > whether hijacking the `dnf` command in that way could cause more > confusion than it's worth. ISTM like a cleaner approach would be to > build this out into `dnf` directly instead and have it drive > rpm-ostree. I agree with this longer term, though there's a *lot* of details here. We have the same problem with this that exists with dnf5 in that there are a *ton* of options that dnf exposes that we're unlikely to make work anytime soon. (A random example here is "-4 resolve to IPv4 addresses only" which seems tricky to plumb through the stack, particularly scoping in the container side). > It would be great to get feedback from the DNF maintainers about this > proposal and some of the ideas here. Agreed! I pinged them directly again but was hoping they'd see this Change and reply here. > I think this probably makes sense generally, but I'll note that for > Fedora CoreOS at least, the whole point is to have users' workloads > run in containers and decoupled from the host. E.g. we've gone to > great lengths to keep Python out for that reason (also, `cowsay` pulls > in Perl BTW :) ). Having non-finger compatibility with `dnf install -y > cowsay` was kind of a feature in that sense; it made users think twice > before falling into the same patterns as other systems. Now with this > and especially container layering, it gives more power to users but > weakens that messaging. This is true. But weakening that messaging (and making the technology more flexible) also *weakens the barriers* we've set up between "CoreOS" and other editions. (This topic gets confusing because we could be talking about the *build* side or the client side or both; I hope most people agree the CLI compatibility on the build side just makes sense) BTW I wanted to give an update here specifically regarding the "dnf image" bit - as of late, I've been working on a fresh new "bootc" CLI, see https://github.com/ostreedev/ostree-rs-ext/pull/412 and https://github.com/cgwalters/bootc and that may make more sense for the client side. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Tue, Oct 25, 2022 at 12:43 PM Colin Walters wrote: > - This proposal is explicitly trying to tie everything together. I think > without the "bigger picture", it's actually *more* confusing. For example, > just pushing the container images does little unless we invest in them as a > derivation source and build tooling and docs around them. I agree it's helpful to discuss this at a higher level than the individual pieces to make sense of it. But the proposal as is seems to imply it will all be done in one cycle which I agree with Dusty is very optimistic. > But you're certainly not wrong, it *is* a big proposal. Happy to make > changes, I'm mainly just saying there already are dedicated sub-proposal > trackers to discuss the details of those. WDYT about introducing phases into the proposal? E.g. something like: - phase 1 (f38): start pushing OSTree-based variants as container images in Quay.io; users can opt into rebasing onto them - phase 2 (f39): automatically transition existing users to shipping from Quay.io; users can opt out. - phase 3 (f40): stop pushing OSTree repo updates entirely This is just an example, we can break it down some other way. Also, it'd make sense to have the transition for each variant be driven by that variant's working group (obviously with collaboration between them). On that topic, it doesn't seem like we got any feedback from Fedora IoT at least. Also not sure where we stand for Fedora Silverblue and Kinoite. On Thu, Oct 13, 2022 at 3:09 PM Ben Cotton wrote: > * `rpm-ostree` is reworked to gain strong CLI compatibility with > `yum/dnf` commands (cliwrap) > * The new container native functionality is exposed via `dnf image` > * (TBD: Rebranding of rpm-ostree itself to `dnf-image` or something else) Some concerns about this have been raised upstream already around whether hijacking the `dnf` command in that way could cause more confusion than it's worth. ISTM like a cleaner approach would be to build this out into `dnf` directly instead and have it drive rpm-ostree. Then the relationship becomes much clearer, and there's an obvious path towards gaining parity. Conveniently with dnf5 ditching Python, this becomes more achievable for systems that don't want Python. Things like `install` and `update` can more or less be passthrough as is (some debate there about whether to provide "advice" on whether what you're trying to install should be in a container instead; more of that below). Things like `status` would be valid in "image-mode" only to start, but then we could build out a traditional version of it. It would be great to get feedback from the DNF maintainers about this proposal and some of the ideas here. On Tue, Nov 1, 2022 at 9:34 AM Colin Walters wrote: > I personally think "we'll actually just do what you asked for when you type > dnf -y install cowsay" is a notable leap here. I think this probably makes sense generally, but I'll note that for Fedora CoreOS at least, the whole point is to have users' workloads run in containers and decoupled from the host. E.g. we've gone to great lengths to keep Python out for that reason (also, `cowsay` pulls in Perl BTW :) ). Having non-finger compatibility with `dnf install -y cowsay` was kind of a feature in that sense; it made users think twice before falling into the same patterns as other systems. Now with this and especially container layering, it gives more power to users but weakens that messaging. I'm not sure if it's worth adding some artificial technological constraints to try to steer them back or just keeping it as a documentation thing would suffice. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Tue, Nov 01, 2022 at 09:18:19AM -0400, Colin Walters wrote: > > Would it make sense to have a Fedora Objective (at the Council level) around > > this? > > Probably? > > I would say actually that this proposal is not even the end. I cut out a > part due to objections/concerns, but I personally (still) want to get rid > of e.g. the Silverblue/Kinoite/CoreOS type "sub-brands". It's just e.g. > "Workstation, but in image+container-default mode" - where it even > matters. Otherwise one can just say "I run Fedora". This is revisiting > https://discussion.fedoraproject.org/t/its-more-fedora-than-it-is-fedora-noun/33864 > in effect - I personally think "we'll actually just do what you asked for > when you type dnf -y install cowsay" is a notable leap here. While I will carefully skirt the terminology issue, I love the vision. Let's talk more about making an Objective around this. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Mon, Oct 31, 2022, at 5:14 PM, Matthew Miller wrote: > On Tue, Oct 25, 2022 at 09:00:40AM -0400, Colin Walters wrote: >> Two things: >> >> - This proposal is explicitly trying to tie everything together. I think >> without the "bigger picture", it's actually *more* confusing. For example, >> just pushing the container images does little unless we invest in them as a >> derivation source and build tooling and docs around them. >> - At a very practical level, I find context switching in Mediawiki syntax to >> be rather tedious. I'd prefer to discuss the individual sections in the >> already extant tracker issues that accept Markdown and have a >> well-understood and used discussion mechanism. (Or of course, email here) >> >> But you're certainly not wrong, it *is* a big proposal. Happy to make >> changes, I'm mainly just saying there already are dedicated sub-proposal >> trackers to discuss the details of those. > > Would it make sense to have a Fedora Objective (at the Council level) around > this? Probably? I would say actually that this proposal is not even the end. I cut out a part due to objections/concerns, but I personally (still) want to get rid of e.g. the Silverblue/Kinoite/CoreOS type "sub-brands". It's just e.g. "Workstation, but in image+container-default mode" - where it even matters. Otherwise one can just say "I run Fedora". This is revisiting https://discussion.fedoraproject.org/t/its-more-fedora-than-it-is-fedora-noun/33864 in effect - I personally think "we'll actually just do what you asked for when you type dnf -y install cowsay" is a notable leap here. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Tue, Oct 25, 2022 at 09:00:40AM -0400, Colin Walters wrote: > Two things: > > - This proposal is explicitly trying to tie everything together. I think > without the "bigger picture", it's actually *more* confusing. For example, > just pushing the container images does little unless we invest in them as a > derivation source and build tooling and docs around them. > - At a very practical level, I find context switching in Mediawiki syntax to > be rather tedious. I'd prefer to discuss the individual sections in the > already extant tracker issues that accept Markdown and have a well-understood > and used discussion mechanism. (Or of course, email here) > > But you're certainly not wrong, it *is* a big proposal. Happy to make > changes, I'm mainly just saying there already are dedicated sub-proposal > trackers to discuss the details of those. Would it make sense to have a Fedora Objective (at the Council level) around this? -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Mon, Oct 24, 2022, at 11:45 PM, Dusty Mabe wrote: > There are a lot of things going on in this proposal: > > - shipping editions as container images in quay https://pagure.io/releng/issue/11047 > - migrating existing users to the new container image based updates (No tracker yet) > - overriding yum/dnf commands via cliwrap https://github.com/coreos/rpm-ostree/issues/2883 > - rebranding `rpm-ostree` https://github.com/coreos/rpm-ostree/issues/405 > > I feel like we'd have a more cohesive discussion if we tried to break this up > into smaller pieces so we can have a more focused discussion. These links were already mostly in the proposal, but I've added them again here for convenience. > As written this proposal is hard to swallow. Two things: - This proposal is explicitly trying to tie everything together. I think without the "bigger picture", it's actually *more* confusing. For example, just pushing the container images does little unless we invest in them as a derivation source and build tooling and docs around them. - At a very practical level, I find context switching in Mediawiki syntax to be rather tedious. I'd prefer to discuss the individual sections in the already extant tracker issues that accept Markdown and have a well-understood and used discussion mechanism. (Or of course, email here) But you're certainly not wrong, it *is* a big proposal. Happy to make changes, I'm mainly just saying there already are dedicated sub-proposal trackers to discuss the details of those. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On 10/24/22 23:45, Dusty Mabe wrote: > > > On 10/13/22 15:08, Ben Cotton wrote: >> >> An update will be shipped that will cause existing systems tracking >> the production ostree branches to be switched to the corresponding >> container image. Concretely for example, a system running >> `fedora:fedora/36/x86_64/silverblue` will execute `rpm-ostree rebase >> ostree-remote-registry:fedora:quay.io/fedora/fedora-silverblue:36`. > > I don't think I'm comfortable with making this switch by default for some > time (certainly not in the F38 time frame). We currently only have the > container image in place for one edition (Fedora CoreOS) and FCOS isn't > even ready (you mention the design issue below). > > This will probably take a few cycles. What we need to do now is come up > with a plan and then implement it over time. And to be clear. What will take a few cycles is fully adopting this proposal. There are certainly pieces of it that work already today. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On 10/13/22 15:08, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable > > This document represents a proposed Change. As part of the Changes > process, proposals are publicly announced in order to receive > community feedback. This proposal will only be implemented if approved > by the Fedora Engineering Steering Committee. > > > == Summary == > > Continue the work done in > https://fedoraproject.org/wiki/Changes/OstreeNativeContainer but in an > officially stable format, and expanded to cover more OSTree-based > editions. This goes "all in" on being container-native and > significantly changes the technology and user emphasis. > > == Owner == > > * Name: [[User:walters| Colin Walters]], [[User:jmarrero| Joseph > Marrero]], [[User:baude| Brent Baude]] > * Email: walt...@verbum.org, jmarr...@fedoraproject.org, > ba...@fedoraproject.org > > > == Detailed Description == > > * rpm-ostree's functionality to > [https://coreos.github.io/rpm-ostree/container/ boot and upgrade from > Docker/OCI container images] is declared stable > * Rework Fedora editions and spins (CoreOS, IoT, Silverblue, Kinoite, > etc) that use ostree to instead deliver via Docker/OCI container > images > * `rpm-ostree` is reworked to gain strong CLI compatibility with > `yum/dnf` commands (cliwrap) > * The new container native functionality is exposed via `dnf image` > * (TBD: Rebranding of rpm-ostree itself to `dnf-image` or something else) > * Support and documentation for generating derived images > * Support in Anaconda and other tools > * Support for generating bootable images There are a lot of things going on in this proposal: - shipping editions as container images in quay - migrating existing users to the new container image based updates - overriding yum/dnf commands via cliwrap - rebranding `rpm-ostree` I feel like we'd have a more cohesive discussion if we tried to break this up into smaller pieces so we can have a more focused discussion. As written this proposal is hard to swallow. > > > A top level goal is that users should not need to know or understand > ostree (or rpm-ostree); they only need to know containers and most key > functionality is exposed via the `yum/dnf` frontend with a few > extensions. > > > === Backwards compatibility === > > If you're a user of ostree today: no need to worry. Everything that > works today in ostree and rpm-ostree will continue to work for years > to come (it's shipped in RHEL9). However, it's expected that most > users will find the new model sufficiently compelling to switch fairly > quickly. > > === Stable Bootable OCI === > > More information about this is available in > [https://coreos.github.io/rpm-ostree/container/ the rpm-ostree > documentation]. This Change highlights that this functionality is > planned to be officially stable. > > === Changing to OCI over the wire === > > Today Fedora has an OSTree repository that contains updates for > editions such as Fedora CoreOS and Fedora Silverblue. > > These editions instead will become [https://github.com/opencontainers > OCI base images], available at e.g. > > * `quay.io/fedora/fedora-coreos:stable` > ([https://quay.io/repository/fedora/fedora-coreos this exists today]!) > * `quay.io/fedora/fedora-silverblue:38` (does not exist, but can soon) > > An update will be shipped that will cause existing systems tracking > the production ostree branches to be switched to the corresponding > container image. Concretely for example, a system running > `fedora:fedora/36/x86_64/silverblue` will execute `rpm-ostree rebase > ostree-remote-registry:fedora:quay.io/fedora/fedora-silverblue:36`. I don't think I'm comfortable with making this switch by default for some time (certainly not in the F38 time frame). We currently only have the container image in place for one edition (Fedora CoreOS) and FCOS isn't even ready (you mention the design issue below). This will probably take a few cycles. What we need to do now is come up with a plan and then implement it over time. > > Note that at the current time, Fedora does not sign container images. > This should change (likely to something based on > [cosign](https://github.com/containers/skopeo/pull/1701)), but in the > mean time, the ostree-container flow supports verifying the embedded > ostree-baesd GPG signatures and this will continue to be used. > > The Fedora OSTree repository will become read-only and stop receiving updates. As mentioned above. I highly doubt it's feasible to stop shipping updates there in a single cycle. > > Note that for Fedora CoreOS specifically, there is some outstanding > design discussion around the intersection with Cincinnati: > https://github.com/coreos/fedora-coreos-tracker/issues/1263 > > Wire efficiency > > Today the container images generated by rpm-ostree are "chunked" in a > reproducible fashion; for more information on this, see > > *
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On 10/14/22 18:09, Colin Walters wrote: On Thu, Oct 13, 2022, at 3:08 PM, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable I know there's a lot going on here, so I put together https://github.com/cgwalters/dnfimage-config as a demonstration system to show this all works today. (Though there's a lot left to do) To say this another way...I really, really wish I had a time machine to go back and announce *this* instead of doing Fedora Atomic Host way back in the day. If when Docker had come out (before Kubernetes, before podman, before CoreOS) I wish I'd said "hey this container stuff is cool, why don't we make bootable host operating systems configurable via containers too"). Oh well, better late than never! (And yes, we're not the first to this nowadays, but...first, this path gives a seamless upgrade for all the existing ostree-based systems out there, and second, you absolutely can continue to do "dnf install cowsay" or whatever client side on standalone systems like desktops and homelab servers, you aren't obligated to tie your an installed OS to external build infrastructure) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue I just wanted to throw in my two cents on this, that I absolutely love the idea. We all wish we had Way Back machines. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Thu, Oct 13, 2022, at 3:08 PM, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable I know there's a lot going on here, so I put together https://github.com/cgwalters/dnfimage-config as a demonstration system to show this all works today. (Though there's a lot left to do) To say this another way...I really, really wish I had a time machine to go back and announce *this* instead of doing Fedora Atomic Host way back in the day. If when Docker had come out (before Kubernetes, before podman, before CoreOS) I wish I'd said "hey this container stuff is cool, why don't we make bootable host operating systems configurable via containers too"). Oh well, better late than never! (And yes, we're not the first to this nowadays, but...first, this path gives a seamless upgrade for all the existing ostree-based systems out there, and second, you absolutely can continue to do "dnf install cowsay" or whatever client side on standalone systems like desktops and homelab servers, you aren't obligated to tie your an installed OS to external build infrastructure) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: Ostree Native Container (Phase 2, stable) (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 3:08 PM Ben Cotton wrote: > > == Contingency Plan == > * Contingency mechanism: Continue to ship things the way we ship them today > * Contingency deadline: Dunno > * Blocks release? No I suggest the branch day (2023-02-07) as the contingency deadline here. -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
