[libvirt PATCH v3 12/18] qemu: include nbdkit state in private xml

2022-10-20 Thread Jonathon Jongsma
Add xml to the private data for a disk source to represent the nbdkit process so that the state can be re-created if the libvirt daemon is restarted. Format: /path/to/nbdkit.pid /path/to/nbdkit.socket Signed-off-by: Jonathon Jongsma --- src/qemu/qemu_domain.c | 53

[libvirt PATCH v3 00/18] Use nbdkit for http/ftp/ssh network drives in libvirt

2022-10-20 Thread Jonathon Jongsma
This is the third version of this patch series. See https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information about the goal, but the summary is that RHEL does not want to ship the qemu storage plugins for curl and ssh. Handling them outside of the qemu process provides several

[libvirt PATCH v3 07/18] qemu: use file cache for nbdkit caps

2022-10-20 Thread Jonathon Jongsma
Add the virFileCache implementation for nbdkit capabilities to the qemu driver. This allows us to determine whether nbdkit is installed and which plugins are supported. it also has persistent caching and the capabilities are re-queried whenever something changes. Signed-off-by: Jonathon Jongsma

[libvirt PATCH v3 16/18] qemu: pass sensitive data to nbdkit via pipe

2022-10-20 Thread Jonathon Jongsma
Rather than passing passwords and cookies (which could contain passwords) to nbdkit via commandline arguments, use the alternate format that nbdkit supports where we can specify a file descriptor which nbdkit will read to get the password or cookies. Signed-off-by: Jonathon Jongsma ---

[libvirt PATCH v3 18/18] qemu: Monitor nbdkit process for exit

2022-10-20 Thread Jonathon Jongsma
Adds the ability to monitor the nbdkit process so that we can take action in case the child exits unexpectedly. When the nbdkit process exits, we pause the vm, restart nbdkit, and then resume the vm. This allows the vm to continue working in the event of a nbdkit failure. Eventually we may want

[libvirt PATCH v3 17/18] qemu: add test for authenticating a https network disk

2022-10-20 Thread Jonathon Jongsma
Signed-off-by: Jonathon Jongsma --- tests/qemunbdkitdata/disk-network-source-curl.args.1.pipe.1 | 1 + tests/qemunbdkitdata/disk-network-source-curl.args.disk1 | 4 +++- .../disk-network-source-curl.args.disk1.pipe.1780 | 2 +- .../disk-network-source-curl.args.disk1.pipe.1782

[libvirt PATCH v3 11/18] qemu: split qemuDomainSecretStorageSourcePrepare

2022-10-20 Thread Jonathon Jongsma
This prepares encryption secrets and authentication secrets. When we add nbdkit-backed network storage sources, we will not need to send authentication secrets to qemu, since they will be sent to nbdkit instead. So split this into two different functions. Signed-off-by: Jonathon Jongsma

[libvirt PATCH v3 10/18] tests: add ability to test various nbdkit capabilities

2022-10-20 Thread Jonathon Jongsma
Add new DO_TEST_CAPS_LATEST_NBDKIT macro to test xml2argv for various nbdkit capability scenarios. Signed-off-by: Jonathon Jongsma --- src/qemu/qemu_nbdkit.c | 20 +--- tests/qemuxml2argvtest.c | 11 +++ tests/testutilsqemu.c| 27 +++

[libvirt PATCH v3 14/18] tests: add tests for nbdkit invocation

2022-10-20 Thread Jonathon Jongsma
We were testing the arguments that were being passed to qemu when a disk was being served by nbdkit, but the arguments used to start nbdkit itself were not testable. This adds a test to ensure that we're invoking nbdkit correctly for various disk source definitions. For now, expect failure for

[libvirt PATCH v3 15/18] util: make virCommandSetSendBuffer testable

2022-10-20 Thread Jonathon Jongsma
Add a private function to peek at the list of send buffers in virCommand so that it is testable Signed-off-by: Jonathon Jongsma --- src/libvirt_private.syms | 1 + src/util/vircommand.c | 16 src/util/vircommand.h | 8 src/util/vircommandpriv.h | 4 4

[libvirt PATCH v3 13/18] qemu: use nbdkit to serve network disks if available

2022-10-20 Thread Jonathon Jongsma
For virStorageSource objects that contain an nbdkitProcess, start that nbdkit process to serve that network drive and then pass the nbdkit socket to qemu rather than sending the network url to qemu directly. Signed-off-by: Jonathon Jongsma --- src/qemu/qemu_block.c | 162

[libvirt PATCH v3 03/18] qemu: expand nbdkit capabilities

2022-10-20 Thread Jonathon Jongsma
In order to add caching of the nbdkit capabilities, we will need to compare against file modification times, etc. So look up this information when creating the nbdkit caps. Add a nbdkit_moddir build option to allow the builder to specify the location to look for nbdkit plugins and filters.

[libvirt PATCH v3 05/18] qemu: implement basic virFileCache for nbdkit caps

2022-10-20 Thread Jonathon Jongsma
Preparatory step for caching nbdkit capabilities. This patch implements the newData and isValid virFileCacheHandlers callback functions. Signed-off-by: Jonathon Jongsma --- src/qemu/qemu_nbdkit.c | 93 +- src/qemu/qemu_nbdkit.h | 4 ++ 2 files changed,

[libvirt PATCH v3 09/18] qemu: add functions to start and stop nbdkit

2022-10-20 Thread Jonathon Jongsma
Add some helper functions to build a virCommand object and run the nbdkit process for a given virStorageSource. Signed-off-by: Jonathon Jongsma --- src/qemu/qemu_nbdkit.c | 251 + src/qemu/qemu_nbdkit.h | 10 ++ 2 files changed, 261 insertions(+) diff

[libvirt PATCH v3 02/18] qemu: Add functions for determining nbdkit availability

2022-10-20 Thread Jonathon Jongsma
In future commits, we will optionally use nbdkit to serve some remote disk sources. This patch queries to see whether nbdkit is installed on the host and queries it for capabilities. The data will be used in later commits. Signed-off-by: Jonathon Jongsma --- po/POTFILES| 1 +

[libvirt PATCH v3 04/18] util: Allow virFileCache data to be any GObject

2022-10-20 Thread Jonathon Jongsma
Since the libvirt documentation suggests to prefer GObject over virObject, and since virObject is a GObject, change virFileCache to allow GObjects as data. Signed-off-by: Jonathon Jongsma --- src/util/virfilecache.c | 14 -- src/util/virfilecache.h | 2 +- 2 files changed, 9

[libvirt PATCH v3 08/18] qemu: Add qemuNbdkitProcess

2022-10-20 Thread Jonathon Jongsma
An object for storing information about a nbdkit process that is serving a specific virStorageSource. At the moment, this information is just stored in the private data of virStorageSource and not used at all. Future commits will use this data to actually start a nbdkit process. Signed-off-by:

[libvirt PATCH v3 06/18] qemu: implement persistent file cache for nbdkit caps

2022-10-20 Thread Jonathon Jongsma
Implement the loadFile and saveFile virFileCacheHandlers callbacks so that nbdkit capabilities are cached perstistently across daemon restarts. The format and implementation is modeled on the qemu capabilities, but simplified slightly. Signed-off-by: Jonathon Jongsma --- src/qemu/qemu_nbdkit.c

[libvirt PATCH v3 01/18] schema: allow 'ssh' as a protocol for network disks

2022-10-20 Thread Jonathon Jongsma
There was support in the code for parsing protocol='ssh' on network disk sources, but it was not present in the xml schema. Add this to the schema and mention it in the documentation. Signed-off-by: Jonathon Jongsma --- src/conf/schemas/domaincommon.rng | 1 +

Re: [libvirt PATCH] qemu: do not attempt to pass unopened vsock FD

2022-10-20 Thread Peter Krempa
On Thu, Oct 20, 2022 at 16:27:45 +0200, Ján Tomko wrote: > On normal vm startup, we open a file descriptor > for the vsock device in qemuProcessPrepareHost. > > However, when doing domxml-to-native, no file descriptors are open. > > Only pass the fd if it's not -1, to make domxml-to-native work.

[PATCH 5/6] storage: Add VIR_STORAGE_VOL_CREATE_VALIDATE flag

2022-10-20 Thread Peter Krempa
Allow users to request validation of the storage volume XML. Add new flag and virsh support. Signed-off-by: Peter Krempa --- docs/manpages/virsh.rst | 9 +++-- include/libvirt/libvirt-storage.h | 1 + tools/virsh-volume.c | 14 ++ 3 files changed, 22

[PATCH 4/6] conf: storage: Add support for validating storage vol XML to virStorageVolDefParse

2022-10-20 Thread Peter Krempa
Introduce the VIR_VOL_XML_PARSE_VALIDATE parser flag and wire it up into the validator. Signed-off-by: Peter Krempa --- src/conf/storage_conf.c | 3 ++- src/conf/storage_conf.h | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/src/conf/storage_conf.c

[PATCH 6/6] storage|test|vbox: Implement support for validating storage volume XMLs

2022-10-20 Thread Peter Krempa
Signed-off-by: Peter Krempa --- src/storage/storage_driver.c | 18 ++ src/test/test_driver.c | 16 src/vbox/vbox_storage.c | 8 ++-- 3 files changed, 32 insertions(+), 10 deletions(-) diff --git a/src/storage/storage_driver.c

[PATCH 3/6] nodedev|test: Implement support for validating node device XMLs

2022-10-20 Thread Peter Krempa
Signed-off-by: Peter Krempa --- src/node_device/node_device_driver.c | 10 ++ src/test/test_driver.c | 6 -- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/src/node_device/node_device_driver.c b/src/node_device/node_device_driver.c index

[PATCH 1/6] conf: node_device: Add 'validate' argument to virNodeDeviceDefParse

2022-10-20 Thread Peter Krempa
Allow callers to request XML validation against the schema. All callers for now pass 'false'. Signed-off-by: Peter Krempa --- src/conf/node_device_conf.c | 5 +++-- src/conf/node_device_conf.h | 3 ++- src/hypervisor/domain_driver.c | 6 +++---

[PATCH 2/6] nodedev: Add VIR_NODE_DEVICE_(CREATE|DEFINE)_XML_VALIDATE flags

2022-10-20 Thread Peter Krempa
The node device APIs which get XML from the user don't yet support XML validation flags. Introduce virNodeDeviceCreateXMLFlags and virNodeDeviceDefineXMLFlags with the appropriate flags and add virsh support for the new flags. Signed-off-by: Peter Krempa --- docs/manpages/virsh.rst |

[PATCH 0/6] Implement XML validation feature for node devices and storage volumes

2022-10-20 Thread Peter Krempa
In the last round of adding support for built-in validation the node device APIs and storage volume creation were not covered. Note that due to the close freeze date I've already marked the APIs for v8.10. Peter Krempa (6): conf: node_device: Add 'validate' argument to virNodeDeviceDefParse

[libvirt PATCH] qemu: do not attempt to pass unopened vsock FD

2022-10-20 Thread Ján Tomko
On normal vm startup, we open a file descriptor for the vsock device in qemuProcessPrepareHost. However, when doing domxml-to-native, no file descriptors are open. Only pass the fd if it's not -1, to make domxml-to-native work. https://bugzilla.redhat.com/show_bug.cgi?id=1777212 Signed-off-by:

Re: [libvirt PATCH] build: drop LINUGAS sorting rule

2022-10-20 Thread Daniel P . Berrangé
On Thu, Oct 20, 2022 at 03:06:23PM +0200, Ján Tomko wrote: > A recent merge request from Weblate adding a new file fails syntax-check > because it adds a new language at the end of LINGUAS, instead of sorting > it alphabetically. Rather than trying to work around it, drop this > pointless rule. >

[libvirt PATCH] build: drop LINUGAS sorting rule

2022-10-20 Thread Ján Tomko
A recent merge request from Weblate adding a new file fails syntax-check because it adds a new language at the end of LINGUAS, instead of sorting it alphabetically. Rather than trying to work around it, drop this pointless rule. Signed-off-by: Ján Tomko --- build-aux/syntax-check.mk | 8

Re: [libvirt PATCH 00/12] tools: provide virt-qemu-sev-validate for SEV(-ES) launch attestation

2022-10-20 Thread Daniel P . Berrangé
On Thu, Oct 20, 2022 at 08:18:20AM -0400, Cole Robinson wrote: > On 10/18/22 5:22 AM, Daniel P. Berrangé wrote: > > On Sun, Oct 16, 2022 at 03:06:17PM -0400, Cole Robinson wrote: > >> On 10/7/22 7:42 AM, Daniel P. Berrangé wrote: > >>> The libvirt QEMU driver provides all the functionality

Re: [libvirt PATCH 00/12] tools: provide virt-qemu-sev-validate for SEV(-ES) launch attestation

2022-10-20 Thread Cole Robinson
On 10/18/22 5:22 AM, Daniel P. Berrangé wrote: > On Sun, Oct 16, 2022 at 03:06:17PM -0400, Cole Robinson wrote: >> On 10/7/22 7:42 AM, Daniel P. Berrangé wrote: >>> The libvirt QEMU driver provides all the functionality required for >>> launching a guest on AMD SEV(-ES) platforms, with a

Re: [libvirt PATCH 02/12] tools: support validating SEV firmware boot measurements

2022-10-20 Thread Cole Robinson
On 10/20/22 8:11 AM, Cole Robinson wrote: > On 10/18/22 5:15 AM, Daniel P. Berrangé wrote: >> On Sun, Oct 16, 2022 at 02:54:47PM -0400, Cole Robinson wrote: >>> On 10/7/22 7:42 AM, Daniel P. Berrangé wrote: The virt-qemu-sev-validate program will compare a reported SEV/SEV-ES domain

Re: [libvirt PATCH 02/12] tools: support validating SEV firmware boot measurements

2022-10-20 Thread Cole Robinson
On 10/18/22 5:15 AM, Daniel P. Berrangé wrote: > On Sun, Oct 16, 2022 at 02:54:47PM -0400, Cole Robinson wrote: >> On 10/7/22 7:42 AM, Daniel P. Berrangé wrote: >>> The virt-qemu-sev-validate program will compare a reported SEV/SEV-ES >>> domain launch measurement, to a computed launch