Re: [PATCH 06/12] RFC: automation: Add linux stubdom build and smoke test
On Fri, May 17, 2024 at 05:40:52PM -0700, Stefano Stabellini wrote: > On Thu, 16 May 2024, Marek Marczykowski-Górecki wrote: > > Add minimal linux-stubdom smoke test. It starts a simple HVM with > > linux-stubdom. The actual stubdom implementation is taken from Qubes OS > > and then stripped off Qubes-specific code. In particular, the remaining > > code does _not_ support: > > - direct kernel boot (implemented by relaying on specific guest disk > >laying in Qubes OS) > > - graphical console (used Qubes GUI agent injected into > >stubdomain's qemu) > > - audio input/output (used Qubes audio agent inside stubdomain) > > - USB passthrough (used qrexec <-> usbip proxy inside stubdomain) > > - setting up DHCP server (assumes guest addressing used in Qubes OS) > > > > For this smoke test, the relevant part is missing direct kernel boot, as > > that's used in other smoke tests. Solve this by preparing disk image > > with proper bootloader (grub) installed. Since the test script is > > running on arm64 to control x86_64 box, it cannot (easily) install grub > > directly. For this reason, prepare bootsector as part of the Xen build > > (which runs on x86_64) and then prepend do the disk image during the > > test (and adjust partitions table afterwards). > > I am not an expert on this, but do you think it would be possible to use > network boot and tftp instead of grub on emulated disk? That would not > require us to build neither /grub-core.img nor build_domU_disk(). Honestly, I don't know. I guess I'd need at least dnsmasq in dom0, and also iPXE for the domU (if not built already?). I can try for this test. But a later test (the PCI one) connects a network card and dom0 can't really setup own DHCP on that network. Additionally combining this with vif network for PXE might be confusing down the road. > I am trying to avoid grub-core.img and disk.img because I think direct > kernel boot or network boot are easier to maintain and more similar to > the other tests. If you see the ARM tests, they all use tftp boot. The ARM ones boot as dom0less, where there is only one boot mode for the system to start in. Here, we have two: xen+dom0 (which already does network boot), and then domU started by dom0. The latter would need either a separate DHCP server on a separate network (vif interface in dom0 should be fine), or some other way to separate dom0/domU boot mode. That said, the stubdom used in Qubes does support direct kernel boot. It is removed from this version, because it relies on specific disk layout (it reserves xvdd for this purpose). But I do want to bring this capability to the upstream version too at some point. > > Signed-off-by: Marek Marczykowski-Górecki > > --- > > The test is implemented using hardware runner, because some of the > > further tests will require it (for example PCI passthrough with > > stubdomain). But if there is strong desire to have stubdomain tested > > inside qemu tests (to be included in patchew runs), it is probably an > > option for this basic smoke test. > > Thanks for this amazing work. This is a great start, we can see how to > create more tests after merging this one. > > > > For now I'm keeping stubdomain code (build and glue scripts) in separate > > repository on my github account. This is far from ideal. What would be > > preferred option? New repository on xenbits? Or add directly into > > xen.git (stubdom directory)? Honestly, I'd rather avoid the latter, as > > from packager point of view those are mostly separate beings (similar to > > qemu, where many use distribution-provide one instead of the one bundled > > with Xen) and it's convenient to not need to rebuild stubdomain on every > > hypervisor change (like a security patch). > > My suggestion is to create repositories under gitlab.com/xen-project gitlab.com/xen-project/stubdom-dm-linux ? Initially I can create the repository under people/marmarek/. Is there any preference regarding git history? I see two options: 1. Preserve the current history, where there is a lot of qubes-specific work and on top a bunch of commits making it not qubes-specific (this is what is there now). 2. Start with fresh history and reference original repository (and the commit id) in the initial commit message. > > Another topic is QEMU version inside stubdomain. It needs to be a > > separate build due to vastly different configure options, so I cannot > > reuse the qemu binary built for dom0 (or distribution-provided one if > > Xen is configured to use it). But also, at this moment qemu for > > stubdomain needs few extra patches that are not upstream yet. > > What should be the proper solution here (after upstreaming all the > > patches)? > > It is fine to use a different QEMU version. For now, I would suggest to > keep the QEMU patches as patches to be applied, in the new repositories > under gitlab.com/xen-project Ok, makes sense. > > Generally, I try to add tests early, even though there is still some > > work t
Re: [PATCH 06/12] RFC: automation: Add linux stubdom build and smoke test
On Thu, 16 May 2024, Marek Marczykowski-Górecki wrote: > Add minimal linux-stubdom smoke test. It starts a simple HVM with > linux-stubdom. The actual stubdom implementation is taken from Qubes OS > and then stripped off Qubes-specific code. In particular, the remaining > code does _not_ support: > - direct kernel boot (implemented by relaying on specific guest disk >laying in Qubes OS) > - graphical console (used Qubes GUI agent injected into >stubdomain's qemu) > - audio input/output (used Qubes audio agent inside stubdomain) > - USB passthrough (used qrexec <-> usbip proxy inside stubdomain) > - setting up DHCP server (assumes guest addressing used in Qubes OS) > > For this smoke test, the relevant part is missing direct kernel boot, as > that's used in other smoke tests. Solve this by preparing disk image > with proper bootloader (grub) installed. Since the test script is > running on arm64 to control x86_64 box, it cannot (easily) install grub > directly. For this reason, prepare bootsector as part of the Xen build > (which runs on x86_64) and then prepend do the disk image during the > test (and adjust partitions table afterwards). I am not an expert on this, but do you think it would be possible to use network boot and tftp instead of grub on emulated disk? That would not require us to build neither /grub-core.img nor build_domU_disk(). I am trying to avoid grub-core.img and disk.img because I think direct kernel boot or network boot are easier to maintain and more similar to the other tests. If you see the ARM tests, they all use tftp boot. > Signed-off-by: Marek Marczykowski-Górecki > --- > The test is implemented using hardware runner, because some of the > further tests will require it (for example PCI passthrough with > stubdomain). But if there is strong desire to have stubdomain tested > inside qemu tests (to be included in patchew runs), it is probably an > option for this basic smoke test. Thanks for this amazing work. This is a great start, we can see how to create more tests after merging this one. > For now I'm keeping stubdomain code (build and glue scripts) in separate > repository on my github account. This is far from ideal. What would be > preferred option? New repository on xenbits? Or add directly into > xen.git (stubdom directory)? Honestly, I'd rather avoid the latter, as > from packager point of view those are mostly separate beings (similar to > qemu, where many use distribution-provide one instead of the one bundled > with Xen) and it's convenient to not need to rebuild stubdomain on every > hypervisor change (like a security patch). My suggestion is to create repositories under gitlab.com/xen-project > Another topic is QEMU version inside stubdomain. It needs to be a > separate build due to vastly different configure options, so I cannot > reuse the qemu binary built for dom0 (or distribution-provided one if > Xen is configured to use it). But also, at this moment qemu for > stubdomain needs few extra patches that are not upstream yet. > What should be the proper solution here (after upstreaming all the > patches)? It is fine to use a different QEMU version. For now, I would suggest to keep the QEMU patches as patches to be applied, in the new repositories under gitlab.com/xen-project > Generally, I try to add tests early, even though there is still some > work to do for proper stubdomain integration into upstream Xen, so any > cleanups and future changes (like the CDROM libxl patches by Jason > Andryuk) can be made with more confidence and reduce risk of > regressions. > > The patch is RFC only because of the stubdom repository location. > --- > automation/build/alpine/3.19-arm64v8.dockerfile | 2 +- > automation/build/alpine/3.19.dockerfile | 9 ++- > automation/gitlab-ci/build.yaml | 3 +- > automation/gitlab-ci/test.yaml| 8 +- > automation/scripts/build | 12 ++- > automation/scripts/qubes-x86-64.sh| 87 +++- > automation/tests-artifacts/alpine/3.19.dockerfile | 6 +- > 7 files changed, 123 insertions(+), 4 deletions(-) > > diff --git a/automation/build/alpine/3.19-arm64v8.dockerfile > b/automation/build/alpine/3.19-arm64v8.dockerfile > index 158cf465a9ff..12810f87ecc6 100644 > --- a/automation/build/alpine/3.19-arm64v8.dockerfile > +++ b/automation/build/alpine/3.19-arm64v8.dockerfile > @@ -47,3 +47,5 @@ RUN apk --no-cache add \ ># qubes test deps >openssh-client \ >fakeroot \ > + sfdisk \ > + e2fsprogs \ > diff --git a/automation/build/alpine/3.19.dockerfile > b/automation/build/alpine/3.19.dockerfile > index 0be6d7c85fe7..108284613987 100644 > --- a/automation/build/alpine/3.19.dockerfile > +++ b/automation/build/alpine/3.19.dockerfile > @@ -49,3 +49,12 @@ RUN apk --no-cache add \ >pixman-dev \ ># livepatch-tools deps >elfutils-dev \ > + # stubdom deps > + dracut-core \ > + quilt \ > + gnupg \ > +
[PATCH 06/12] RFC: automation: Add linux stubdom build and smoke test
Add minimal linux-stubdom smoke test. It starts a simple HVM with linux-stubdom. The actual stubdom implementation is taken from Qubes OS and then stripped off Qubes-specific code. In particular, the remaining code does _not_ support: - direct kernel boot (implemented by relaying on specific guest disk laying in Qubes OS) - graphical console (used Qubes GUI agent injected into stubdomain's qemu) - audio input/output (used Qubes audio agent inside stubdomain) - USB passthrough (used qrexec <-> usbip proxy inside stubdomain) - setting up DHCP server (assumes guest addressing used in Qubes OS) For this smoke test, the relevant part is missing direct kernel boot, as that's used in other smoke tests. Solve this by preparing disk image with proper bootloader (grub) installed. Since the test script is running on arm64 to control x86_64 box, it cannot (easily) install grub directly. For this reason, prepare bootsector as part of the Xen build (which runs on x86_64) and then prepend do the disk image during the test (and adjust partitions table afterwards). Signed-off-by: Marek Marczykowski-Górecki --- The test is implemented using hardware runner, because some of the further tests will require it (for example PCI passthrough with stubdomain). But if there is strong desire to have stubdomain tested inside qemu tests (to be included in patchew runs), it is probably an option for this basic smoke test. For now I'm keeping stubdomain code (build and glue scripts) in separate repository on my github account. This is far from ideal. What would be preferred option? New repository on xenbits? Or add directly into xen.git (stubdom directory)? Honestly, I'd rather avoid the latter, as from packager point of view those are mostly separate beings (similar to qemu, where many use distribution-provide one instead of the one bundled with Xen) and it's convenient to not need to rebuild stubdomain on every hypervisor change (like a security patch). Another topic is QEMU version inside stubdomain. It needs to be a separate build due to vastly different configure options, so I cannot reuse the qemu binary built for dom0 (or distribution-provided one if Xen is configured to use it). But also, at this moment qemu for stubdomain needs few extra patches that are not upstream yet. What should be the proper solution here (after upstreaming all the patches)? Generally, I try to add tests early, even though there is still some work to do for proper stubdomain integration into upstream Xen, so any cleanups and future changes (like the CDROM libxl patches by Jason Andryuk) can be made with more confidence and reduce risk of regressions. The patch is RFC only because of the stubdom repository location. --- automation/build/alpine/3.19-arm64v8.dockerfile | 2 +- automation/build/alpine/3.19.dockerfile | 9 ++- automation/gitlab-ci/build.yaml | 3 +- automation/gitlab-ci/test.yaml| 8 +- automation/scripts/build | 12 ++- automation/scripts/qubes-x86-64.sh| 87 +++- automation/tests-artifacts/alpine/3.19.dockerfile | 6 +- 7 files changed, 123 insertions(+), 4 deletions(-) diff --git a/automation/build/alpine/3.19-arm64v8.dockerfile b/automation/build/alpine/3.19-arm64v8.dockerfile index 158cf465a9ff..12810f87ecc6 100644 --- a/automation/build/alpine/3.19-arm64v8.dockerfile +++ b/automation/build/alpine/3.19-arm64v8.dockerfile @@ -47,3 +47,5 @@ RUN apk --no-cache add \ # qubes test deps openssh-client \ fakeroot \ + sfdisk \ + e2fsprogs \ diff --git a/automation/build/alpine/3.19.dockerfile b/automation/build/alpine/3.19.dockerfile index 0be6d7c85fe7..108284613987 100644 --- a/automation/build/alpine/3.19.dockerfile +++ b/automation/build/alpine/3.19.dockerfile @@ -49,3 +49,12 @@ RUN apk --no-cache add \ pixman-dev \ # livepatch-tools deps elfutils-dev \ + # stubdom deps + dracut-core \ + quilt \ + gnupg \ + libseccomp-dev \ + glib-static \ + gmp-dev \ + mpc1-dev \ + mpfr-dev \ diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index b186289bbd82..783a0687ba34 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -323,9 +323,11 @@ alpine-3.19-rootfs-export: image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.19 script: - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz +- cp /grub-core.img binaries/grub-core.img artifacts: paths: - binaries/initrd.tar.gz + - binaries/grub-core.img tags: - x86_64 @@ -353,6 +355,7 @@ alpine-3.19-gcc-debug: extends: .gcc-x86-64-build-debug variables: CONTAINER: alpine:3.19 +STUBDOM_LINUX: y debian-stretch-gcc-debug: extends: .gcc-x86-64-build-debug diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index f62d426a8d34..80d10eb7f476 100644 --- a/automation/gitlab-ci/test.yaml +++ b/auto