On 10/13/22 15:08, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable > > This document represents a proposed Change. As part of the Changes > process, proposals are publicly announced in order to receive > community feedback. This proposal will only be implemented if approved > by the Fedora Engineering Steering Committee. > > > == Summary == > > Continue the work done in > https://fedoraproject.org/wiki/Changes/OstreeNativeContainer but in an > officially stable format, and expanded to cover more OSTree-based > editions. This goes "all in" on being container-native and > significantly changes the technology and user emphasis. > > == Owner == > > * Name: [[User:walters| Colin Walters]], [[User:jmarrero| Joseph > Marrero]], [[User:baude| Brent Baude]] > * Email: walt...@verbum.org, jmarr...@fedoraproject.org, > ba...@fedoraproject.org > > > == Detailed Description == > > * rpm-ostree's functionality to > [https://coreos.github.io/rpm-ostree/container/ boot and upgrade from > Docker/OCI container images] is declared stable > * Rework Fedora editions and spins (CoreOS, IoT, Silverblue, Kinoite, > etc) that use ostree to instead deliver via Docker/OCI container > images > * `rpm-ostree` is reworked to gain strong CLI compatibility with > `yum/dnf` commands (cliwrap) > * The new container native functionality is exposed via `dnf image` > * (TBD: Rebranding of rpm-ostree itself to `dnf-image` or something else) > * Support and documentation for generating derived images > * Support in Anaconda and other tools > * Support for generating bootable images
There are a lot of things going on in this proposal: - shipping editions as container images in quay - migrating existing users to the new container image based updates - overriding yum/dnf commands via cliwrap - rebranding `rpm-ostree` I feel like we'd have a more cohesive discussion if we tried to break this up into smaller pieces so we can have a more focused discussion. As written this proposal is hard to swallow. > > > A top level goal is that users should not need to know or understand > ostree (or rpm-ostree); they only need to know containers and most key > functionality is exposed via the `yum/dnf` frontend with a few > extensions. > > > === Backwards compatibility === > > If you're a user of ostree today: no need to worry. Everything that > works today in ostree and rpm-ostree will continue to work for years > to come (it's shipped in RHEL9). However, it's expected that most > users will find the new model sufficiently compelling to switch fairly > quickly. > > === Stable Bootable OCI === > > More information about this is available in > [https://coreos.github.io/rpm-ostree/container/ the rpm-ostree > documentation]. This Change highlights that this functionality is > planned to be officially stable. > > === Changing to OCI over the wire === > > Today Fedora has an OSTree repository that contains updates for > editions such as Fedora CoreOS and Fedora Silverblue. > > These editions instead will become [https://github.com/opencontainers > OCI base images], available at e.g. > > * `quay.io/fedora/fedora-coreos:stable` > ([https://quay.io/repository/fedora/fedora-coreos this exists today]!) > * `quay.io/fedora/fedora-silverblue:38` (does not exist, but can soon) > > An update will be shipped that will cause existing systems tracking > the production ostree branches to be switched to the corresponding > container image. Concretely for example, a system running > `fedora:fedora/36/x86_64/silverblue` will execute `rpm-ostree rebase > ostree-remote-registry:fedora:quay.io/fedora/fedora-silverblue:36`. I don't think I'm comfortable with making this switch by default for some time (certainly not in the F38 time frame). We currently only have the container image in place for one edition (Fedora CoreOS) and FCOS isn't even ready (you mention the design issue below). This will probably take a few cycles. What we need to do now is come up with a plan and then implement it over time. > > Note that at the current time, Fedora does not sign container images. > This should change (likely to something based on > [cosign](https://github.com/containers/skopeo/pull/1701)), but in the > mean time, the ostree-container flow supports verifying the embedded > ostree-baesd GPG signatures and this will continue to be used. > > The Fedora OSTree repository will become read-only and stop receiving updates. As mentioned above. I highly doubt it's feasible to stop shipping updates there in a single cycle. > > Note that for Fedora CoreOS specifically, there is some outstanding > design discussion around the intersection with Cincinnati: > https://github.com/coreos/fedora-coreos-tracker/issues/1263 > > ==== Wire efficiency ==== > > Today the container images generated by rpm-ostree are "chunked" in a > reproducible fashion; for more information on this, see > > * https://github.com/ostreedev/ostree-rs-ext/issues/69 > * https://coreos.github.io/rpm-ostree/container/ > > However, longer term (as that first issue notes) we do want to add > [https://github.com/containers/image/pull/902 container deltas]. > Doing so will benefit non-host containers too. > > > > === dnf/yum CLI compatibility === > > rpm-ostree has a feature called > [cliwrap](https://coreos.github.io/rpm-ostree/cliwrap/) today. A key > motivation here is that in the switch to using containers as the > frontend, the project name "rpm-ostree" (which is very literal) no > longer makes sense. A further emphasis on "container native" will > strongly encourage writing e.g. a `Dockerfile` like this: > > > <pre> > FROM quay.io/fedora/fedora-silverblue:stable > RUN dnf -y install sway && ostree container commit > </pre> > > *Today*, one can spell this as `RUN rpm-ostree install -y sway && > ostree container commit` - but a toplevel goal here is to allow host > system customization using the same patterns and tools one uses to > build application containers, and to de-emphasize the "ostree backend" > aspect. > > To complete this story on the client side, what is today called > "ostree native containers" will be exposed via a new `dnf image` > subcommand on the client system. > > For example: > > > <pre> > $ dnf image rebase quay.io/examplecorp/customos:latest > </pre> > > As well as other offline/"image based" functionality such as `dnf > image rollback` and `dnf image apply-live`. > > However, as many people know, rpm-ostree does today support > "per-machine" package layering/overrides - and *that will continue to > work*. So users are not *required* to do builds on a remote server > even in "image mode". > > Concretely, it will also work to `sudo dnf -y install sway` inside a > root terminal on a Fedora Workstation for example. This will still be > a transactional operation. > > More: https://github.com/coreos/rpm-ostree/issues/2883 > > === Rebranding of rpm-ostree === > > This is a fundamental change in technology *and* positioning for > rpm-ostree (and for the projects that use it). The terminology of > "immutable" gets even fuzzier now (see also [this > blog](https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-hysteresis/)). > > It would make sense to rebrand the project due to this. No decision > has been made on this, but the choices are: > > - Keep as is > - Rebrand to dnf-image (or, reviving a past name, yum-image) > - Rebrand to something else > > More: https://github.com/coreos/rpm-ostree/issues/405 > > === Support for building derivative containers === > > Until now, the model for systems like Fedora CoreOS and Silverblue > etc. has included an emphasis on containerization, but avoiding > extensive customization of the host system. This will be weakened > somewhat - it will be fully supported to build derivative operating > system images that in turn contain software which itself should run > directly on the host system, and not in a container. > > A key aspect of this is that by generating a custom build, one binds > together the code and configuration as a transactionally updatable > unit. > > The role of things like Kickstart, cloud-init, and Ignition and other > system config management shrinks significantly. > > === Example 1: Kubernetes and kubelet === > > It is not supported upstream to run the kubelet in a container image > itself. Today, for OpenShift 4 we embed the kubelet as part of the > CoreOS builds. The [https://www.okd.io/ OKD] project > [https://github.com/openshift/okd-machine-os/blob/fa2248d332abc9926e68428dc6658e55ec6f350b/Dockerfile#L25 > already installs kubelet during a container build] deriving from the > base `fedora-coreos` image. It is possible that RHEL CoreOS will make > [https://github.com/openshift/os/issues/799 a similar split]. > > === Example 2: Agent software === > > Many organizations require agents (cloud agents, security scanners, > antivirus) which are not ready to be containerized. Often too, these > tools come in e.g. RPM form, and also expect to be configured (remote > logging server, etc.). This tooling allows to `dnf install` these > *and* embed the configuration, generating a standard container image > which can then be booted as an atomic unit. > > (Note that in the OpenShift context, we expect this to chain with the > first example - users instead derive from a `okd-node-base` image > which itself derives from `fedora-coreos`). > > === Example 3: Podman machine === > > The `[http://podman.io Podman]` project has a function called > `[https://github.com/containers/podman/blob/main/docs/source/markdown/podman-machine.1.md > machine]` where it creates a virtual machine and boots standard CoreOS > builds. Podman developers and users has long desired a custom FCOS > image that is focused on container deployment using Podman, including > certain QEMU packages that allow for cross-architecture development. > This approach will solve a series of problems including but not > limited to getting more timely release alignment between Podman and > FCOS. > > We also have a user base that wants to customize the VM image based on > restrictive environments ... bring your own virtual machine image. > > === Support in Anaconda and other tools === > > Anaconda support for custom images is tracked in this bug: > https://bugzilla.redhat.com/show_bug.cgi?id=2125655 > > But, as part of this Change, it should also work to generate an > Anaconda ISO which allows *user choice* for whether to install in > "image mode" or "traditional". > > === Support for generating bootable images === > > Instead of using an existing disk image/ISO (Anaconda, AWS AMI, etc.) > and then doing an in-place update to a generated container image, in > many cases will want to generate disk images using their container > image as input. > > For example, generating an ISO that embeds that container image as > input and will install it. > > In this proposal, we plan to ship a container image as part of Fedora > CoreOS that will do this: > https://github.com/coreos/fedora-coreos-tracker/issues/1151 > > But, it also makes sense to have a tool that e.g. does this and > generates an Anaconda ISO (we will inherently be doing this as part of > generating ostree-based Anaconda ISOs already!) - this would just be > supporting that in a clear external way too. One likely target for > this is [https://github.com/osbuild osbuild] and > [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/composing_a_customized_rhel_system_image/index > RHEL Image Builder]. > > > > == Feedback == > > > == Benefit to Fedora == > > The need to understand ostree goes away for most users; they > understand RPMs and containers. While retaining all the current > (rpm-)ostree features. > > == Scope == > * Proposal owners: > * Other developers: May affect e.g. gnome-software and similar tools > that talk to rpm-ostree. > > * Release engineering: [https://pagure.io/releng/issue/11047] > * Policies and guidelines: None. > * Trademark approval: N/A (not needed for this Change) > * Alignment with Objectives: None > > == Upgrade/compatibility impact == > > We will ship a systemd unit that transitions systems currently using > OSTree on the wire over to tracking the corresponding container > images. Note again, all features of rpm-ostree continue to work! For > example, if you have layered packages, that will continue to be > applied. > > == How To Test == > > Build container images, boot them. > > == User Experience == > > Users will not need to understand ostree (at least at a superficial > level), only container images. > > == Dependencies == > > Needs improvements in Anaconda in some cases, also would like to > increase alignment with dnf. Fetching container images uses `skopeo`, > maintained by containers team. > > == Contingency Plan == > * Contingency mechanism: Continue to ship things the way we ship them today > * Contingency deadline: Dunno > * Blocks release? No > > == Documentation == > > https://coreos.github.io/rpm-ostree/container/ and > https://github.com/coreos/coreos-layering-examples etc. > > == Release Notes == > > _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
