[libvirt PATCH v8 33/37] qemu: implement keyfile auth for ssh disks with nbdkit

2023-08-31 Thread Jonathon Jongsma
For ssh disks that are served by nbdkit, we can support logging in with an ssh key file. Pass the path to the configured key file and the username to the nbdkit process. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/conf/domain_conf.c| 30

[libvirt PATCH v8 15/37] tests: add ability to test various nbdkit capabilities

2023-08-31 Thread Jonathon Jongsma
Add new DO_TEST_CAPS_LATEST_NBDKIT macro to test xml2argv for various nbdkit capability scenarios. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_nbdkit.c | 20 +--- tests/qemuxml2argvtest.c | 11 +++ tests/testutilsqemu.c| 26

[libvirt PATCH v8 21/37] util: make virCommandSetSendBuffer testable

2023-08-31 Thread Jonathon Jongsma
Add a private function to peek at the list of send buffers in virCommand so that it is testable Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/libvirt_private.syms | 1 + src/util/vircommand.c | 17 + src/util/vircommand.h | 8

[libvirt PATCH v8 12/37] qemu: Extract qemuDomainLogContext into a new file

2023-08-31 Thread Jonathon Jongsma
This will allow us to use it for nbdkit logging in upcoming commits. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- po/POTFILES| 1 + src/qemu/meson.build | 1 + src/qemu/qemu_domain.c | 247 ++ src/qemu/qemu_domain.h

[libvirt PATCH v8 35/37] qemu: implement ssh-agent auth for ssh disks with nbdkit

2023-08-31 Thread Jonathon Jongsma
It's not possible to use password-protected ssh keys directly with libvirt because libvirt doesn't have any way to prompt a user for the password. To accomodate password-protected key files, an administrator can add these keys to an ssh agent and then configure the domain with the path to the

[libvirt PATCH v8 34/37] schema: add ssh-agent configuration for ssh disks

2023-08-31 Thread Jonathon Jongsma
Add the ability to specify a path to a ssh-agent socket in order to use the ssh-agent to authenticate to remote ssh disks. Example configuration: ... ... Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa ---

[libvirt PATCH v8 29/37] qemu: implement password auth for ssh disks with nbdkit

2023-08-31 Thread Jonathon Jongsma
For ssh disks that are served by nbdkit, lookup the password from the configured secret and securely pass it to the nbdkit process using fd passing. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_nbdkit.c| 84 ++-

[libvirt PATCH v8 37/37] ci: add libnbd to build

2023-08-31 Thread Jonathon Jongsma
Signed-off-by: Jonathon Jongsma Reviewed-by: Erik Skultety --- ci/buildenv/almalinux-8.sh| 1 + ci/buildenv/centos-stream-8.sh| 1 + ci/buildenv/centos-stream-9.sh| 1 + ci/buildenv/debian-12-cross-aarch64.sh

[libvirt PATCH v8 10/37] qemu: add functions to start and stop nbdkit

2023-08-31 Thread Jonathon Jongsma
Add some helper functions to build a virCommand object and run the nbdkit process for a given virStorageSource. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_nbdkit.c | 250 + src/qemu/qemu_nbdkit.h | 10 ++ 2 files

[libvirt PATCH v8 14/37] qemu: log error output from nbdkit

2023-08-31 Thread Jonathon Jongsma
log stderr and stdout from nbdkit into its own log so that nbdkit-related issues can be debugged more easily. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_nbdkit.c | 16 +++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git

[libvirt PATCH v8 28/37] schema: add password configuration for ssh disk

2023-08-31 Thread Jonathon Jongsma
Right now, ssh network disks are not usable. There is some basic support in libvirt that is meant to support disk chains that have backing disks located at ssh urls, but there is no real way for a user to configure a ssh-based disk. This commit allows users to configure an ssh disk with password

[libvirt PATCH v8 25/37] qemu: Monitor nbdkit process for exit

2023-08-31 Thread Jonathon Jongsma
Adds the ability to monitor the nbdkit process so that we can take action in case the child exits unexpectedly. When the nbdkit process exits, we pause the vm, restart nbdkit, and then resume the vm. This allows the vm to continue working in the event of a nbdkit failure. Eventually we may want

[libvirt PATCH v8 27/37] qemu: try to connect to nbdkit early to detect errors

2023-08-31 Thread Jonathon Jongsma
When using nbdkit to serve a network disk source, the nbdkit process will start and wait for an nbd connection before actually attempting to connect to the (remote) disk location. Because of this, nbdkit will not report an error until after qemu is launched and tries to read from the disk. This

[libvirt PATCH v8 06/37] qemu: implement persistent file cache for nbdkit caps

2023-08-31 Thread Jonathon Jongsma
Implement the loadFile and saveFile virFileCacheHandlers callbacks so that nbdkit capabilities are cached perstistently across daemon restarts. The format and implementation is modeled on the qemu capabilities, but simplified slightly. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa

[libvirt PATCH v8 17/37] qemu: include nbdkit state in private xml

2023-08-31 Thread Jonathon Jongsma
Add xml to the private data for a disk source to represent the nbdkit process so that the state can be re-created if the libvirt daemon is restarted. Format: /path/to/nbdkit.pid /path/to/nbdkit.socket Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa ---

[libvirt PATCH v8 36/37] rpm: update spec file for for nbdkit support

2023-08-31 Thread Jonathon Jongsma
Require libnbd-devel when building the qemu driver, recommend nbdkit packages. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- libvirt.spec.in | 8 1 file changed, 8 insertions(+) diff --git a/libvirt.spec.in b/libvirt.spec.in index b471afebb1..744fa5c88d 100644 ---

[libvirt PATCH v8 13/37] qemu: move qemuProcessReadLog() to qemuLogContext

2023-08-31 Thread Jonathon Jongsma
This code can be used by the nbdkit implementation for reading back filtered log data for error reporting. Move it to qemuLogContext so that it can be shared. Renamed to qemuLogContextReadFiltered(). Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_logcontext.c | 65

[libvirt PATCH v8 09/37] qemu: query nbdkit module dir from binary

2023-08-31 Thread Jonathon Jongsma
Rather than hard-coding the nbdkit module directory, query the nbdkit binary for the location to these directories. nbdkit provides a --dump-config optiont that outputs this information and can be easily parsed. We can also get the version from this output rather than executing `nbdkit --version`

[libvirt PATCH v8 31/37] qemu: implement knownHosts for ssh disks with nbdkit

2023-08-31 Thread Jonathon Jongsma
For ssh disks that are served by nbdkit, use the configured value for knownHosts and pass it to the nbdkit process. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/conf/domain_conf.c| 8 ++ src/conf/storage_source_conf.c| 2 ++

[libvirt PATCH v8 24/37] qemu: Add Taint for nbdkit restart failure

2023-08-31 Thread Jonathon Jongsma
Since the restart handler will trigger at an arbitrary time (when the nbdkit process crashes, for instance), it's difficult to provide feedback to the user if the restart is unsuccessful. Rather than just relying on a warning in the log, taint the domain so that there will be a slightly more

[libvirt PATCH v8 32/37] schema: add keyfile configuration for ssh disks

2023-08-31 Thread Jonathon Jongsma
Authenticating via key file to an ssh server is often preferable to logging in via password. In order to support this functionality add a new xml element for ssh disks that allows the user to specify a keyfile and username. Example configuration: ... ...

[libvirt PATCH v8 18/37] util: secure erase virCommand send buffers

2023-08-31 Thread Jonathon Jongsma
All users of virCommandSetSendBuffer() are using it to send sensitive data to a child process. So, since these buffers contain sensitive information, clear it with virSecureErase(). Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/util/vircommand.c | 2 ++ 1 file changed, 2

[libvirt PATCH v8 30/37] schema: add configuration for host verification of ssh disks

2023-08-31 Thread Jonathon Jongsma
In order to make ssh disks usable, we need to be able to validate a remote host. To do this, add a xml element for ssh disks to allow the user to specify a location for a file that contains known host keys. Implementation to follow. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa ---

[libvirt PATCH v8 05/37] qemu: implement basic virFileCache for nbdkit caps

2023-08-31 Thread Jonathon Jongsma
Preparatory step for caching nbdkit capabilities. This patch implements the newData and isValid virFileCacheHandlers callback functions. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_nbdkit.c | 89 +- src/qemu/qemu_nbdkit.h

[libvirt PATCH v8 20/37] qemu: use nbdkit to serve network disks if available

2023-08-31 Thread Jonathon Jongsma
For virStorageSource objects that contain an nbdkitProcess, start that nbdkit process to serve that network drive and then pass the nbdkit socket to qemu rather than sending the network url to qemu directly. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_block.c

[libvirt PATCH v8 26/37] qemu: improve error handling when restarting nbdkit

2023-08-31 Thread Jonathon Jongsma
Change the return value for qemuNbdkitProcessRestart() and qemuNbdkitStorageSourceManageProcess() to return an error status. The main effect of this change is that when libvirt starts up and reconnects to an already-running domain with an nbdkit-backed disk, it will return an error if it fails to

[libvirt PATCH v8 22/37] tests: add tests for nbdkit invocation

2023-08-31 Thread Jonathon Jongsma
We were testing the arguments that were being passed to qemu when a disk was being served by nbdkit, but the arguments used to start nbdkit itself were not testable. This adds a test to ensure that we're invoking nbdkit correctly for various disk source definitions. Signed-off-by: Jonathon

[libvirt PATCH v8 04/37] util: Allow virFileCache data to be any GObject

2023-08-31 Thread Jonathon Jongsma
Since the libvirt documentation suggests to prefer GObject over virObject, and since virObject is a GObject, change virFileCache to allow GObjects as data. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/util/virfilecache.c | 14 -- src/util/virfilecache.h | 2 +-

[libvirt PATCH v8 23/37] qemu: add test for authenticating a https network disk

2023-08-31 Thread Jonathon Jongsma
Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- tests/qemunbdkitdata/disk-network-source-curl.args.disk1 | 4 +++- .../disk-network-source-curl.args.disk1.pipe.780 | 2 +- .../disk-network-source-curl.args.disk1.pipe.782 | 1 +

[libvirt PATCH v8 16/37] qemu: split qemuDomainSecretStorageSourcePrepare

2023-08-31 Thread Jonathon Jongsma
This prepares encryption secrets and authentication secrets. When we add nbdkit-backed network storage sources, we will not need to send authentication secrets to qemu, since they will be sent to nbdkit instead. So split this into two different functions. Signed-off-by: Jonathon Jongsma

[libvirt PATCH v8 07/37] qemu: use file cache for nbdkit caps

2023-08-31 Thread Jonathon Jongsma
Add the virFileCache implementation for nbdkit capabilities to the qemu driver. This allows us to determine whether nbdkit is installed and which plugins are supported. it also has persistent caching and the capabilities are re-queried whenever something changes. Signed-off-by: Jonathon Jongsma

[libvirt PATCH v8 03/37] qemu: expand nbdkit capabilities

2023-08-31 Thread Jonathon Jongsma
In order to add caching of the nbdkit capabilities, we will need to compare against file modification times, etc. So look up this information when creating the nbdkit caps. Add a nbdkit_moddir build option to allow the builder to specify the location to look for nbdkit plugins and filters.

[libvirt PATCH v8 19/37] qemu: pass sensitive data to nbdkit via pipe

2023-08-31 Thread Jonathon Jongsma
Rather than passing passwords and cookies (which could contain passwords) to nbdkit via commandline arguments, use the alternate format that nbdkit supports where we can specify a file descriptor which nbdkit will read to get the password or cookies. Signed-off-by: Jonathon Jongsma Reviewed-by:

[libvirt PATCH v8 11/37] Generalize qemuDomainLogContextNew()

2023-08-31 Thread Jonathon Jongsma
Allow to specify a basename for the log file so that qemuDomainLogContextNew() can be used to create log contexts for secondary loggers. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/qemu/qemu_domain.c | 5 +++-- src/qemu/qemu_domain.h | 3 ++- src/qemu/qemu_process.c | 2

[libvirt PATCH v8 08/37] qemu: Add qemuNbdkitProcess

2023-08-31 Thread Jonathon Jongsma
An object for storing information about a nbdkit process that is serving a specific virStorageSource. At the moment, this information is just stored in the private data of virStorageSource and not used at all. Future commits will use this data to actually start a nbdkit process. Signed-off-by:

[libvirt PATCH v8 02/37] qemu: Add functions for determining nbdkit availability

2023-08-31 Thread Jonathon Jongsma
In future commits, we will optionally use nbdkit to serve some remote disk sources. This patch queries to see whether nbdkit is installed on the host and queries it for capabilities. The data will be used in later commits. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa ---

[libvirt PATCH v8 00/37] Use nbdkit for http/ftp/ssh network drives in libvirt

2023-08-31 Thread Jonathon Jongsma
This is the eighth version of this patch series. See https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information. Note that testing this requires selinux policy changes which are not fully done, but there is a new policy in development that has allowed me to run with selinux in

[libvirt PATCH v8 01/37] schema: allow 'ssh' as a protocol for network disks

2023-08-31 Thread Jonathon Jongsma
There was support in the code for parsing protocol='ssh' on network disk sources, but it was not present in the xml schema. Add this to the schema. Signed-off-by: Jonathon Jongsma Reviewed-by: Peter Krempa --- src/conf/schemas/domaincommon.rng | 1 + tests/qemublocktest.c

[PATCH] virsh: Fix net-desc --config output

2023-08-31 Thread K Shiva Kiran
Fixes the following bug: Command: `net-desc --config [--title] my_network` Expected Output: Title/Description of persistent config Output: Title/Description of live config This was caused due to the usage of a single `flags` variable in `virshGetNetworkDescription()` which

[PATCH] virsh: Fix net-desc --config output

2023-08-31 Thread K Shiva Kiran
Fixes the following bug: Command: `net-desc --config [--title] my_network` Expected Output: Title/Description of persistent config Output: Title/Description of live config This was caused due to the usage of a single `flags` variable in `virshGetNetworkDescription()` which

Re: [libvirt PATCH 00/33] ci: Unify the GitLab CI jobs with local executions && adopt lcitool container executions

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:08PM +0200, Erik Skultety wrote: > Technically a v2 of: > https://listman.redhat.com/archives/libvir-list/2023-February/237552.html > > However, the approach here is slightly different and what that series said > about migration to lcitool container executions as a

Re: [libvirt PATCH 25/33] ci: helper: Add a job argparse subparser

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:33PM +0200, Erik Skultety wrote: > The idea behind this subcommand is to follow whatever build job we have > defined in the GitLab CI pipeline, so that we only have a single source > of truth for the recipes. Adds 'shell' as an extra option for > interactive container

Re: [libvirt PATCH 26/33] ci: helper: Add a required_deps higher order helper/decorator

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:34PM +0200, Erik Skultety wrote: > Since we'll depend on GitPython for repo cloning, we need to make sure > to emit a user friendly error if the module is not installed. This > patch introduces a helper which future patches will use as a decorator. > Inspiration for

Re: [libvirt PATCH 27/33] ci: helper: Add Python code hangling git clones

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:35PM +0200, Erik Skultety wrote: > This helper will be utilized by a future patch which will add the > lcitool container execution logic. The reason why the required_deps > decorator isn't being used here is because this is a property. > > Signed-off-by: Erik

Re: [libvirt PATCH 28/33] ci: helper: Add a helper to create a local repo clone Pythonic way

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:36PM +0200, Erik Skultety wrote: > A proper Python equivalent of 'git clone --local'. > > Signed-off-by: Erik Skultety > --- > ci/helper | 4 > 1 file changed, 4 insertions(+) Reviewed-by: Daniel P. Berrangé With regards, Daniel -- |: https://berrange.com

Re: [libvirt PATCH 29/33] ci: helper: Rework _lcitool_run method logic

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:37PM +0200, Erik Skultety wrote: > This method wasn't even utilized before this patch. This patch adds all > the necessary logic to successfully execute a container workload via > lcitool (which will later allow us to ditch ci/Makefile). Because > container executions

Re: [libvirt PATCH 30/33] ci: helper: Add an action to run the container workload via lcitool

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:38PM +0200, Erik Skultety wrote: > Just like with the other CLI sub-commands, add an action to run a > GitLab spec job in a local container via lcitool. > > Signed-off-by: Erik Skultety > --- > ci/helper | 4 > 1 file changed, 4 insertions(+) Reviewed-by:

Re: [libvirt PATCH 33/33] ci: helper: Drop the _make_run method

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:41PM +0200, Erik Skultety wrote: > We've successfully migrated over to lcitool to take care of the > container workload execution, so dropping this 'make' prep code is a > prerequisite of finally getting rid of the ci/Makefile script. > > Signed-off-by: Erik Skultety

Re: [libvirt PATCH 32/33] ci: helper: Drop the --meson-args/--ninja-args CLI options

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:40PM +0200, Erik Skultety wrote: > These originally allowed customizing the ci/Makefile script which was > the core of the local container executions. The problem was that > however flexible this may have been, it never mirrored what was being > done as part of the

Re: [libvirt PATCH 31/33] ci: helper: Drop original actions

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:39PM +0200, Erik Skultety wrote: > Previous patches added a single 'run' command parametrized with GitLab > job specs via '--job' that cover all of these original actions, adding > some more in the process. Drop the original actions as we don't need > them anymore. >

Re: [libvirt PATCH 24/33] ci: helper: Add --lcitool-path CLI option

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:32PM +0200, Erik Skultety wrote: > We'll soon be relying solely on lcitool so we need to be able to run it > from a user-provided location if it's not installed in a known > location. > > Signed-off-by: Erik Skultety > --- > ci/helper | 6 ++ > 1 file changed,

Re: [libvirt PATCH 23/33] ci: helper: Don't make ':' literal a static part of the image tag

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:31PM +0200, Erik Skultety wrote: > ':' is just a connecting character, we can add it to the appropriate > place later in the Python script later, but it doesn't make sense to be > part of the image 'tag' string. > > Signed-off-by: Erik Skultety > --- > ci/helper |

Re: [libvirt PATCH 22/33] ci: helper: Drop _lcitool_get_targets method

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:30PM +0200, Erik Skultety wrote: > This method unused anywhere, so drop it. > > Signed-off-by: Erik Skultety > --- > ci/helper | 4 > 1 file changed, 4 deletions(-) Reviewed-by: Daniel P. Berrangé With regards, Daniel -- |: https://berrange.com -o-

Re: [libvirt PATCH 21/33] .gitlab-ci.yml: Convert the potfile job to the build.sh usage

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:29PM +0200, Erik Skultety wrote: > Individual shell command executions are replaced by respective > functions in the ci/build.sh base script. This will make sure we use > the same recipes in GitLab jobs as well as in local executions. > > Signed-off-by: Erik Skultety

Re: [libvirt PATCH 20/33] .gitlab-ci.yml: Convert the codestyle job to the build.sh usage

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:28PM +0200, Erik Skultety wrote: > Individual shell command executions are replaced by respective > functions in the ci/build.sh base script. This will make sure we use > the same recipes in GitLab jobs as well as in local executions. > > Signed-off-by: Erik Skultety

Re: [libvirt PATCH 19/33] .gitlab-ci.yml: Convert the website build job to the build.sh usage

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:27PM +0200, Erik Skultety wrote: > Individual shell command executions are replaced by respective > functions in the ci/build.sh base script. This will make sure we use > the same recipes in GitLab jobs as well as in local executions. > > Signed-off-by: Erik Skultety

Re: [libvirt PATCH 18/33] .gitlab-ci.yml: Convert the cross build job to the build.sh usage

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:26PM +0200, Erik Skultety wrote: > Individual shell command executions are replaced by respective > functions in the ci/build.sh base script. This will make sure we use > the same recipes in GitLab jobs as well as in local executions. > > Signed-off-by: Erik Skultety

Re: [libvirt PATCH 17/33] .gitlab-ci.yml: Convert the native build job to the build.sh usage

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:25PM +0200, Erik Skultety wrote: > Individual shell command executions are replaced by respective > functions in the ci/build.sh base script. This will make sure we use > the same recipes in GitLab jobs as well as in local executions. > > Signed-off-by: Erik Skultety

Re: [libvirt PATCH 07/33] ci: build.sh: Add a helper function to create the dist tarball

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:15PM +0200, Erik Skultety wrote: > This helper function does not correspond to a particular GitLab job, it > just logically separates the necessary step of creating a dist tarball > from the RPM build job that takes over. > One notable change here is the need to

Re: [libvirt PATCH 06/33] ci: build.sh: Add a wrapper function over the 'build' job

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:14PM +0200, Erik Skultety wrote: > This helper is a shell function transcript of its original GitLab CI > counterpart. > > Signed-off-by: Erik Skultety > --- > ci/build.sh | 7 +++ > 1 file changed, 7 insertions(+) > > diff --git a/ci/build.sh b/ci/build.sh >

Re: [libvirt PATCH 15/33] ci: build.sh: Drop MESON_ARGS definition from global level

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:23PM +0200, Erik Skultety wrote: > Signed-off-by: Erik Skultety > --- > ci/build.sh | 10 -- > 1 file changed, 10 deletions(-) Reviewed-by: Daniel P. Berrangé With regards, Daniel -- |: https://berrange.com -o-

Re: [libvirt PATCH 14/33] ci: build.sh: Drop direct invocation of meson/ninja commands

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:22PM +0200, Erik Skultety wrote: > We've moved all invocations to the respective helper function which > we'll execute both from gitlab CI jobs and local environments so we > don't need to have them on the global level as it would also not work > with "sourcing" this

Re: [libvirt PATCH 13/33] ci: build.sh: Drop changing working directory to CI_CONT_DIR

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:21PM +0200, Erik Skultety wrote: > Firstly, this would mangle with "sourcing" this file in either > execution environment later down the road. Secondly, we won't need this > as future ci/helper patches will generate a throwaway script that will > take care of a

Re: [libvirt PATCH 12/33] ci: build.sh: Add a wrapper function over the 'website' job

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:20PM +0200, Erik Skultety wrote: > This helper is a shell function transcript of its original GitLab CI > counterpart. > > Signed-off-by: Erik Skultety > --- > ci/build.sh | 8 > 1 file changed, 8 insertions(+) > > diff --git a/ci/build.sh b/ci/build.sh >

Re: [libvirt PATCH 11/33] ci: build.sh: Add a wrapper function over the 'rpmbuild' job

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:19PM +0200, Erik Skultety wrote: > This helper is a shell function transcript of its original GitLab CI > counterpart. > > Signed-off-by: Erik Skultety > --- > ci/build.sh | 11 +++ > 1 file changed, 11 insertions(+) > > diff --git a/ci/build.sh

Re: [libvirt PATCH 10/33] ci: build.sh: Add a wrapper function over the 'potfile' job

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:18PM +0200, Erik Skultety wrote: > This helper is a shell function transcript of its original GitLab CI > counterpart. There's one notable difference such that we pass '-j1' to > the meson compile command otherwise we'd have to execute the 'run_build' > function

Re: [libvirt PATCH 09/33] ci: build.sh: Add a wrapper function over the 'codestyle' job

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:17PM +0200, Erik Skultety wrote: > This helper is a shell function transcript of its original GitLab CI > counterpart. > > Signed-off-by: Erik Skultety > --- > ci/build.sh | 9 + > 1 file changed, 9 insertions(+) > > diff --git a/ci/build.sh b/ci/build.sh

Re: [libvirt PATCH 04/33] ci: build.sh: Add a wrapper function over meson's setup

2023-08-31 Thread Daniel P . Berrangé
On Thu, Aug 31, 2023 at 05:37:47PM +0100, Daniel P. Berrangé wrote: > On Fri, Aug 25, 2023 at 07:55:12PM +0200, Erik Skultety wrote: > > The reason for this wrapper is that all job functions introduced in > > future patches will refer to this one instead of open-coding the same > > 'meson setup'

Re: [libvirt PATCH 03/33] ci: build.sh: Don't mention that MESON_ARGS are available via CLI

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:11PM +0200, Erik Skultety wrote: > Previous patches have removed the code that allowed injecting arbitrary > meson arguments, same for ninja args. > > Signed-off-by: Erik Skultety > --- > ci/build.sh | 4 +--- > 1 file changed, 1 insertion(+), 3 deletions(-)

Re: [libvirt PATCH 02/33] ci: build.sh: Add GIT_ROOT env helper variable

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:10PM +0200, Erik Skultety wrote: > We'll use this one in many of the job functions future patches will > introduce, it's a neat shortcut to avoid using relative paths. > > Signed-off-by: Erik Skultety > --- > ci/build.sh | 2 ++ > 1 file changed, 2 insertions(+)

Re: [libvirt PATCH 01/33] ci: build.sh: Add variables from .gitlab-ci.yml

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:09PM +0200, Erik Skultety wrote: > These are common variables we wish to use in containerized environments > both in GitLab and locally. Having these defined in a single place > rather than twice is highly preferable. > > Signed-off-by: Erik Skultety > --- >

Re: [libvirt PATCH 04/33] ci: build.sh: Add a wrapper function over meson's setup

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:12PM +0200, Erik Skultety wrote: > The reason for this wrapper is that all job functions introduced in > future patches will refer to this one instead of open-coding the same > 'meson setup' invocation N times. It also prevents 'setup' to be called > multiple times as

Re: [libvirt PATCH 05/33] ci: build.sh: Add a wrapper function executing 'shell' commands

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:13PM +0200, Erik Skultety wrote: > This would normally be not needed at all, but the problem here is the > Shell-in-YAML which GitLab interprets. It outputs every command that > appears as a line in the 'script' segment in a color-coded fashion for > easy

Re: [libvirt PATCH 00/33] ci: Unify the GitLab CI jobs with local executions && adopt lcitool container executions

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:08PM +0200, Erik Skultety wrote: > Technically a v2 of: > https://listman.redhat.com/archives/libvir-list/2023-February/237552.html > > However, the approach here is slightly different and what that series said > about migration to lcitool container executions as a

Re: [libvirt PATCH 4/7] qemuSaveImageStartProcess: make it possible to use without header

2023-08-31 Thread Pavel Hrdina
On Thu, Aug 31, 2023 at 05:43:35PM +0200, Peter Krempa wrote: > On Thu, Aug 31, 2023 at 16:55:03 +0200, Pavel Hrdina wrote: > > When used with internal snapshots there is no header to be used and no > > memory state to be decompressed. > > I didn't yet have a look at the rest, but this made me

Re: Sunset libvirt-snmp?

2023-08-31 Thread Daniel P . Berrangé
On Mon, Aug 21, 2023 at 10:32:42AM +0200, Michal Prívozník wrote: > It's been a while since libvirt-snmp was actively developed. Now it > receives only libvirt-ci related commits. The code compiles with > net-snmp-5.9.3 but the freshly released net-snmp-5.9.4 [1] breaks > compilation [2]. Now,

Re: [libvirt PATCH 6/7] qemu_snapshot: correctly load the saved memory state file

2023-08-31 Thread Peter Krempa
On Thu, Aug 31, 2023 at 16:55:05 +0200, Pavel Hrdina wrote: > Original code assumed that the memory state file is only migration > stream but it has additional metadata stored by libvirt. To correctly > load the memory state file we need to reuse code that is used when > restoring domain from

Re: [libvirt PATCH 0/7] external snapshot revert fixes

2023-08-31 Thread Peter Krempa
On Thu, Aug 31, 2023 at 17:06:35 +0200, Pavel Hrdina wrote: > On Thu, Aug 31, 2023 at 04:54:59PM +0200, Pavel Hrdina wrote: > > This fixes reverting external snapshots to not error out in cases where > > it should work and makes it correctly load the memory state when > > reverting to

Re: [libvirt PATCH 4/7] qemuSaveImageStartProcess: make it possible to use without header

2023-08-31 Thread Peter Krempa
On Thu, Aug 31, 2023 at 16:55:03 +0200, Pavel Hrdina wrote: > When used with internal snapshots there is no header to be used and no > memory state to be decompressed. I didn't yet have a look at the rest, but this made me curious. What are you actually doing with this with internal snapshots?

Re: Sunset libvirt-snmp?

2023-08-31 Thread Daniel P . Berrangé
On Thu, Aug 31, 2023 at 08:09:05AM -0700, Andrea Bolognani wrote: > On Tue, Aug 22, 2023 at 12:44:16PM +0100, Daniel P. Berrangé wrote: > > On Tue, Aug 22, 2023 at 01:26:47PM +0200, Erik Skultety wrote: > > > On Mon, Aug 21, 2023 at 10:32:42AM +0200, Michal Prívozník wrote: > > > > It's been a

Re: [PATCH v2] accel: Remove HAX accelerator

2023-08-31 Thread Alex Bennée
Philippe Mathieu-Daudé writes: > HAX is deprecated since commits 73741fda6c ("MAINTAINERS: Abort > HAXM maintenance") and 90c167a1da ("docs/about/deprecated: Mark > HAXM in QEMU as deprecated"), released in v8.0.0. > > Per the latest HAXM release (v7.8 [*]), the latest QEMU supported > is

Re: Sunset libvirt-snmp?

2023-08-31 Thread Andrea Bolognani
On Tue, Aug 22, 2023 at 12:44:16PM +0100, Daniel P. Berrangé wrote: > On Tue, Aug 22, 2023 at 01:26:47PM +0200, Erik Skultety wrote: > > On Mon, Aug 21, 2023 at 10:32:42AM +0200, Michal Prívozník wrote: > > > It's been a while since libvirt-snmp was actively developed. Now it > > > receives only

Re: [libvirt PATCH 0/7] external snapshot revert fixes

2023-08-31 Thread Pavel Hrdina
On Thu, Aug 31, 2023 at 04:54:59PM +0200, Pavel Hrdina wrote: > This fixes reverting external snapshots to not error out in cases where > it should work and makes it correctly load the memory state when > reverting to snapshot of running VM. Forget to mention that we need this to be in

[libvirt PATCH 2/7] qemuSaveImageStartProcess: allow setting reason for audit log

2023-08-31 Thread Pavel Hrdina
When called by snapshot code we will need to use different reason. Signed-off-by: Pavel Hrdina --- src/qemu/qemu_saveimage.c | 6 -- src/qemu/qemu_saveimage.h | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_saveimage.c b/src/qemu/qemu_saveimage.c index

[libvirt PATCH 0/7] external snapshot revert fixes

2023-08-31 Thread Pavel Hrdina
This fixes reverting external snapshots to not error out in cases where it should work and makes it correctly load the memory state when reverting to snapshot of running VM. Pavel Hrdina (7): qemu_saveimage: extract starting process to qemuSaveImageStartProcess

[libvirt PATCH 5/7] qemu_snapshot: fix reverting external snapshot when not all disks are included

2023-08-31 Thread Pavel Hrdina
We need to skip all disks that have snapshot type other than 'external'. Signed-off-by: Pavel Hrdina --- src/qemu/qemu_snapshot.c | 12 1 file changed, 12 insertions(+) diff --git a/src/qemu/qemu_snapshot.c b/src/qemu/qemu_snapshot.c index d943281e35..ff85d56572 100644 ---

[libvirt PATCH 3/7] qemuSaveImageStartProcess: add snapshot argument

2023-08-31 Thread Pavel Hrdina
When called from snapshot code we will need to pass snapshot object in order to make internal snapshots work correctly. Signed-off-by: Pavel Hrdina --- src/qemu/qemu_saveimage.c | 6 -- src/qemu/qemu_saveimage.h | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git

[libvirt PATCH 6/7] qemu_snapshot: correctly load the saved memory state file

2023-08-31 Thread Pavel Hrdina
Original code assumed that the memory state file is only migration stream but it has additional metadata stored by libvirt. To correctly load the memory state file we need to reuse code that is used when restoring domain from saved image. Signed-off-by: Pavel Hrdina --- src/qemu/qemu_snapshot.c

[libvirt PATCH 7/7] NEWS: document support for reverting external snapshots

2023-08-31 Thread Pavel Hrdina
Signed-off-by: Pavel Hrdina --- NEWS.rst | 8 1 file changed, 8 insertions(+) diff --git a/NEWS.rst b/NEWS.rst index e40c8ac259..a3be76d6cc 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -28,6 +28,14 @@ v9.7.0 (unreleased) 2) pre-binding the variant driver using the ``--driver`` option

[libvirt PATCH 4/7] qemuSaveImageStartProcess: make it possible to use without header

2023-08-31 Thread Pavel Hrdina
When used with internal snapshots there is no header to be used and no memory state to be decompressed. Signed-off-by: Pavel Hrdina --- src/qemu/qemu_saveimage.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_saveimage.c b/src/qemu/qemu_saveimage.c index

[libvirt PATCH 1/7] qemu_saveimage: extract starting process to qemuSaveImageStartProcess

2023-08-31 Thread Pavel Hrdina
Part of qemuSaveImageStartVM() function will be used when reverting external snapshots. To avoid duplicating code and logic extract the shared bits into separate function. Signed-off-by: Pavel Hrdina --- src/qemu/qemu_saveimage.c | 103 ++

Re: [libvirt PATCH 16/33] gitlab-ci.yml: Add 'after_script' stage to prep for artifact collection

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:24PM +0200, Erik Skultety wrote: > This is one of the preparation steps that if not done would otherwise > collide with local container executions where we: > 1) don't collect artifacts > 2) are not limited by GitLab's environment and hence moving build >artifacts

Re: [libvirt PATCH 05/33] ci: build.sh: Add a wrapper function executing 'shell' commands

2023-08-31 Thread Daniel P . Berrangé
On Fri, Aug 25, 2023 at 07:55:13PM +0200, Erik Skultety wrote: > This would normally be not needed at all, but the problem here is the > Shell-in-YAML which GitLab interprets. It outputs every command that > appears as a line in the 'script' segment in a color-coded fashion for > easy

[PULL 01/41] accel: Remove HAX accelerator

2023-08-31 Thread Philippe Mathieu-Daudé
HAX is deprecated since commits 73741fda6c ("MAINTAINERS: Abort HAXM maintenance") and 90c167a1da ("docs/about/deprecated: Mark HAXM in QEMU as deprecated"), released in v8.0.0. Per the latest HAXM release (v7.8 [*]), the latest QEMU supported is v7.2: Note: Up to this release, HAXM supports

Re: [libvirt PATCH] rpm: Recommend libvirt-daemon for with_modular_daemons distros

2023-08-31 Thread Andrea Bolognani
On Thu, Aug 31, 2023 at 10:29:23AM +0100, Daniel P. Berrangé wrote: > On Thu, Aug 31, 2023 at 02:14:49AM -0700, Andrea Bolognani wrote: > > On Thu, Aug 31, 2023 at 09:50:07AM +0100, Daniel P. Berrangé wrote: > > > I'm not seeing a good way to deal with the upgrade problem though. > > > > Yeah. The

Re: [libvirt PATCH] rpm: Recommend libvirt-daemon for with_modular_daemons distros

2023-08-31 Thread Daniel P . Berrangé
On Thu, Aug 31, 2023 at 02:14:49AM -0700, Andrea Bolognani wrote: > On Thu, Aug 31, 2023 at 09:50:07AM +0100, Daniel P. Berrangé wrote: > > I don't much like it to be honest, as I was hoping we would get away > > from having it exist in any new installs, such that people were not > > mis-directed

Re: [libvirt PATCH] rpm: Recommend libvirt-daemon for with_modular_daemons distros

2023-08-31 Thread Erik Skultety
On Thu, Aug 31, 2023 at 09:50:07AM +0100, Daniel P. Berrangé wrote: > On Thu, Aug 31, 2023 at 01:43:44AM -0700, Andrea Bolognani wrote: > > On Wed, Aug 30, 2023 at 07:25:23PM +0200, Erik Skultety wrote: > > > On Wed, Aug 30, 2023 at 06:22:33PM +0200, Andrea Bolognani wrote: > > > > A default

Re: [libvirt PATCH] rpm: Recommend libvirt-daemon for with_modular_daemons distros

2023-08-31 Thread Daniel P . Berrangé
On Thu, Aug 31, 2023 at 11:00:27AM +0200, Erik Skultety wrote: > On Thu, Aug 31, 2023 at 09:50:07AM +0100, Daniel P. Berrangé wrote: > > On Thu, Aug 31, 2023 at 01:43:44AM -0700, Andrea Bolognani wrote: > > > On Wed, Aug 30, 2023 at 07:25:23PM +0200, Erik Skultety wrote: > > > > On Wed, Aug 30,

Re: [libvirt PATCH] rpm: Recommend libvirt-daemon for with_modular_daemons distros

2023-08-31 Thread Andrea Bolognani
On Thu, Aug 31, 2023 at 09:50:07AM +0100, Daniel P. Berrangé wrote: > I don't much like it to be honest, as I was hoping we would get away > from having it exist in any new installs, such that people were not > mis-directed into trying to use it. > > I'm not seeing a good way to deal with the

Re: [libvirt PATCH] rpm: Recommend libvirt-daemon for with_modular_daemons distros

2023-08-31 Thread Daniel P . Berrangé
On Thu, Aug 31, 2023 at 01:43:44AM -0700, Andrea Bolognani wrote: > On Wed, Aug 30, 2023 at 07:25:23PM +0200, Erik Skultety wrote: > > On Wed, Aug 30, 2023 at 06:22:33PM +0200, Andrea Bolognani wrote: > > > A default deployment on modern distros uses modular daemons but > > > switching back to the

  1   2   >