[libvirt PATCH v2 03/10] docs: Provide an article on how to add a custom runner to the project

2022-07-14 Thread Erik Skultety
Since running our functional test suite in GitLab cannot make use of the shared resources it makes sense to document the process of adding own HW to run the custom libvirt executor that powers the integration suite. This article will likely make even more sense in the future with GitLab severely cu

[libvirt PATCH v2 08/10] docs: ci: Add info about the two major types of jobs our CI runs

2022-07-14 Thread Erik Skultety
Signed-off-by: Erik Skultety --- docs/ci.rst | 39 --- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/docs/ci.rst b/docs/ci.rst index d5282c4d25..566bc42db7 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -8,6 +8,9 @@ The libvirt project uses Git

[libvirt PATCH v2 02/10] docs: Provide more information about the TCK test suite

2022-07-14 Thread Erik Skultety
Most importantly, how to get it, how install dependencies and how to run it. Signed-off-by: Erik Skultety Reviewed-by: Daniel P. Berrangé --- docs/testtck.rst | 99 ++-- 1 file changed, 88 insertions(+), 11 deletions(-) diff --git a/docs/testtck.rst

[libvirt PATCH v2 05/10] docs: Replace testsuites article with the new 'testing' article

2022-07-14 Thread Erik Skultety
The new article provides more in-depth information on testing options in libvirt. Signed-off-by: Erik Skultety Reviewed-by: Daniel P. Berrangé --- docs/docs.rst | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/docs.rst b/docs/docs.rst index 9fd8b7c37e..b17d73743a 1

[libvirt PATCH v2 06/10] docs: Drop the testsuites article

2022-07-14 Thread Erik Skultety
The article was replaced with a new one in previous commit, so we don't need this one anymore. Signed-off-by: Erik Skultety Reviewed-by: Daniel P. Berrangé --- docs/meson.build| 1 - docs/testsuites.rst | 37 - 2 files changed, 38 deletions(-) delete mo

[libvirt PATCH v2 07/10] docs: Change the CI headline from "CI Testing" to "CI"

2022-07-14 Thread Erik Skultety
It's not just strategy the master CI article talks (or will talk in the future) about. Signed-off-by: Erik Skultety --- docs/docs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs.rst b/docs/docs.rst index b17d73743a..8ca0afdc1a 100644 --- a/docs/docs.rst +++

[libvirt PATCH v2 00/10] docs: Expand documentation on the CI/testing topic

2022-07-14 Thread Erik Skultety
This series was inspired by QEMU's documentation on CI and focuses on the following: - adds documentation on how to add a custom runner to the libvirt project or its fork, mainly in context of running functional testing - replaces the old 'testsuites' article with a more detailed landing page des

[libvirt PATCH v2 04/10] docs: Provide an article on testing

2022-07-14 Thread Erik Skultety
Currently we don't have much information on how testing is done in libvirt and the little we have is scattered among multiple files. This patch creates a common landing page containing all important bits about testing in libvirt, providing links to respective sections which deserve their own articl

[libvirt PATCH v2 01/10] docs: Move the CI dashboard to its own RST module

2022-07-14 Thread Erik Skultety
The dashboard itself simply takes away focus from everything else that makes sense to have in the CI article, so move it to it's own article and link it from the main CI article. Signed-off-by: Erik Skultety Reviewed-by: Daniel P. Berrangé --- docs/ci-dashboard.rst | 216 +++

Re: [libvirt PATCH v2] docs: add info about factors affecting CPU compatibility

2022-07-12 Thread Michal Prívozník
On 7/12/22 18:27, Daniel P. Berrangé wrote: > While libvirt solves most of the problem of ensuring compatibility, when > there is incompatibility it can be hard for users to track down the > cause. Everything knows to check the physical CPU model, but there are a > surprisingly large number of othe

[libvirt PATCH v2] docs: add info about factors affecting CPU compatibility

2022-07-12 Thread Daniel P . Berrangé
While libvirt solves most of the problem of ensuring compatibility, when there is incompatibility it can be hard for users to track down the cause. Everything knows to check the physical CPU model, but there are a surprisingly large number of other factors influencing compatibility. Signed-off-by:

Re: [libvirt PATCH v2] tools: add virt-qmp-proxy for proxying QMP clients to libvirt QEMU guests

2022-07-01 Thread Michal Prívozník
On 6/20/22 19:19, Daniel P. Berrangé wrote: > Libvirt provides QMP passthrough APIs for the QEMU driver and these are > exposed in virsh. It is not especially pleasant, however, using the raw > QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and > exposes a human friendly interacti

Re: [libvirt PATCH v2] tools: add virt-qmp-proxy for proxying QMP clients to libvirt QEMU guests

2022-06-21 Thread Markus Armbruster
Daniel P. Berrangé writes: > Libvirt provides QMP passthrough APIs for the QEMU driver and these are > exposed in virsh. It is not especially pleasant, however, using the raw > QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and > exposes a human friendly interactive shell. It is

[libvirt PATCH v2] tools: add virt-qmp-proxy for proxying QMP clients to libvirt QEMU guests

2022-06-20 Thread Daniel P . Berrangé
Libvirt provides QMP passthrough APIs for the QEMU driver and these are exposed in virsh. It is not especially pleasant, however, using the raw QMP JSON syntax. QEMU has a tool 'qmp-shell' which can speak QMP and exposes a human friendly interactive shell. It is not possible to use this with libvir

Re: [libvirt PATCH v2 0/7] ci: Drop Fedora 34, add Fedora 36

2022-06-09 Thread Erik Skultety
On Thu, Jun 09, 2022 at 07:51:45AM -0400, Andrea Bolognani wrote: > On Thu, Jun 09, 2022 at 08:11:17AM +0200, Erik Skultety wrote: > > On Tue, Jun 07, 2022 at 01:06:41PM +0200, Erik Skultety wrote: > > > On Tue, Jun 07, 2022 at 02:34:50AM -0700, Andrea Bolognani wrote: > > > > Can I push the remain

Re: [libvirt PATCH v2 0/7] ci: Drop Fedora 34, add Fedora 36

2022-06-09 Thread Andrea Bolognani
On Thu, Jun 09, 2022 at 08:11:17AM +0200, Erik Skultety wrote: > On Tue, Jun 07, 2022 at 01:06:41PM +0200, Erik Skultety wrote: > > On Tue, Jun 07, 2022 at 02:34:50AM -0700, Andrea Bolognani wrote: > > > Can I push the remaining two patches now, or are the issues that made > > > it not possible at

Re: [libvirt PATCH v2 0/7] ci: Drop Fedora 34, add Fedora 36

2022-06-08 Thread Erik Skultety
On Tue, Jun 07, 2022 at 01:06:41PM +0200, Erik Skultety wrote: > On Tue, Jun 07, 2022 at 02:34:50AM -0700, Andrea Bolognani wrote: > > On Thu, May 26, 2022 at 04:24:41PM +0200, Erik Skultety wrote: > > > On Thu, May 26, 2022 at 04:01:45PM +0200, Andrea Bolognani wrote: > > > > Test pipeline: > > >

Re: [libvirt PATCH v2 0/7] ci: Drop Fedora 34, add Fedora 36

2022-06-07 Thread Erik Skultety
On Tue, Jun 07, 2022 at 02:34:50AM -0700, Andrea Bolognani wrote: > On Thu, May 26, 2022 at 04:24:41PM +0200, Erik Skultety wrote: > > On Thu, May 26, 2022 at 04:01:45PM +0200, Andrea Bolognani wrote: > > > Test pipeline: > > > > > > https://gitlab.com/abologna/libvirt/-/pipelines/548848259 > > >

Re: [libvirt PATCH v2 0/7] ci: Drop Fedora 34, add Fedora 36

2022-06-07 Thread Andrea Bolognani
On Thu, May 26, 2022 at 04:24:41PM +0200, Erik Skultety wrote: > On Thu, May 26, 2022 at 04:01:45PM +0200, Andrea Bolognani wrote: > > Test pipeline: > > > > https://gitlab.com/abologna/libvirt/-/pipelines/548848259 > > > > Only patches 1-5 should be pushed until the issues outlined in > > Go ahe

Re: [libvirt PATCH v2 81/81] RFC: qemu: Keep vCPUs paused while migration is in postcopy-paused

2022-06-06 Thread Daniel P . Berrangé
On Mon, Jun 06, 2022 at 10:32:03AM -0400, Peter Xu wrote: > [copy Dave] > > On Mon, Jun 06, 2022 at 12:29:39PM +0100, Daniel P. Berrangé wrote: > > On Wed, Jun 01, 2022 at 02:50:21PM +0200, Jiri Denemark wrote: > > > QEMU keeps guest CPUs running even in postcopy-paused migration state so > > > th

Re: [libvirt PATCH v2 81/81] RFC: qemu: Keep vCPUs paused while migration is in postcopy-paused

2022-06-06 Thread Peter Xu
[copy Dave, for real] On Mon, Jun 06, 2022 at 10:32:03AM -0400, Peter Xu wrote: > [copy Dave] > > On Mon, Jun 06, 2022 at 12:29:39PM +0100, Daniel P. Berrangé wrote: > > On Wed, Jun 01, 2022 at 02:50:21PM +0200, Jiri Denemark wrote: > > > QEMU keeps guest CPUs running even in postcopy-paused migr

Re: [libvirt PATCH v2 81/81] RFC: qemu: Keep vCPUs paused while migration is in postcopy-paused

2022-06-06 Thread Peter Xu
[copy Dave] On Mon, Jun 06, 2022 at 12:29:39PM +0100, Daniel P. Berrangé wrote: > On Wed, Jun 01, 2022 at 02:50:21PM +0200, Jiri Denemark wrote: > > QEMU keeps guest CPUs running even in postcopy-paused migration state so > > that processes that already have all memory pages they need migrated to

Re: [libvirt PATCH v2 81/81] RFC: qemu: Keep vCPUs paused while migration is in postcopy-paused

2022-06-06 Thread Jiri Denemark
On Mon, Jun 06, 2022 at 15:37:45 +0200, Peter Krempa wrote: > On Wed, Jun 01, 2022 at 14:50:21 +0200, Jiri Denemark wrote: > > QEMU keeps guest CPUs running even in postcopy-paused migration state so > > that processes that already have all memory pages they need migrated to > > the destination can

Re: [libvirt PATCH v2 81/81] RFC: qemu: Keep vCPUs paused while migration is in postcopy-paused

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:50:21 +0200, Jiri Denemark wrote: > QEMU keeps guest CPUs running even in postcopy-paused migration state so > that processes that already have all memory pages they need migrated to > the destination can keep running. However, this behavior might bring > unexpected delay

Re: [libvirt PATCH v2 80/81] qemu: Fix VSERPORT_CHANGE event in post-copy migration

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:50:20 +0200, Jiri Denemark wrote: > When a domain has a guest agent channel enabled and the agent is running > in the guest, we will get VSERPORT_CHANGE event on a destination host as > soon as we start vCPUs there. This is not an issue for normal migration, > but post-co

Re: [libvirt PATCH v2 79/81] Introduce VIR_JOB_MIGRATION_SAFE job type

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:50:19 +0200, Jiri Denemark wrote: > This is a special job for operations that need to modify domain state > during an active migration. The modification must not affect any state > that could conflict with the migration code. This is useful mainly for > event handlers tha

Re: [libvirt PATCH v2 76/81] qemu: Implement VIR_DOMAIN_ABORT_JOB_POSTCOPY flag

2022-06-06 Thread Peter Krempa
On Mon, Jun 06, 2022 at 14:24:05 +0200, Peter Krempa wrote: > On Wed, Jun 01, 2022 at 14:50:16 +0200, Jiri Denemark wrote: > > Signed-off-by: Jiri Denemark > > --- > > > > Notes: > > Version 2: > > - dropped DeviceNotFound QMP error handling and replaced it with > > reporting an err

Re: [libvirt PATCH v2 67/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Finish phase

2022-06-06 Thread Peter Krempa
On Fri, Jun 03, 2022 at 14:29:58 +0200, Jiri Denemark wrote: > On Wed, Jun 01, 2022 at 14:50:07 +0200, Jiri Denemark wrote: > > Everything was already done in the normal Finish phase and vCPUs are > > running. We just need to wait for all remaining data to be transferred. > > > > Signed-off-by: Ji

Re: [libvirt PATCH v2 66/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase

2022-06-06 Thread Peter Krempa
On Fri, Jun 03, 2022 at 16:10:07 +0200, Jiri Denemark wrote: > On Wed, Jun 01, 2022 at 14:50:06 +0200, Jiri Denemark wrote: > > The QEMU process is already running, all we need to do is to call > > migrate-recover QMP command. Except for some checks and cookie handling, > > of course. > > > > Sign

Re: [libvirt PATCH v2 66/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase

2022-06-06 Thread Peter Krempa
On Fri, Jun 03, 2022 at 14:27:55 +0200, Jiri Denemark wrote: > On Wed, Jun 01, 2022 at 14:50:06 +0200, Jiri Denemark wrote: > > The QEMU process is already running, all we need to do is to call > > migrate-recover QMP command. Except for some checks and cookie handling, > > of course. > > > > Sign

Re: [libvirt PATCH v2 52/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Begin phase

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:52 +0200, Jiri Denemark wrote: > Mostly we just need to check whether the domain is in a failed post-copy > migration that can be resumed. > > Signed-off-by: Jiri Denemark > --- > > Notes: > Version 2: > - s/return NULL/return false/ > - dropped line bre

Re: [libvirt PATCH v2 45/81] qemu: Make qemuMigrationCheckPhase failure fatal

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:45 +0200, Jiri Denemark wrote: > The check can reveal a serious bug in our migration code and we should > not silently ignore it. > > Signed-off-by: Jiri Denemark > --- > > Notes: > Version 2: > - no change > > src/qemu/qemu_migration.c | 58 ++

Re: [libvirt PATCH v2 34/81] qemu: Handle migration job in qemuMigrationDstFinish

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:34 +0200, Jiri Denemark wrote: > The function which started a migration phase should also finish it by > calling qemuMigrationJobFinish/qemuMigrationJobContinue so that the code > is easier to follow. > > Signed-off-by: Jiri Denemark > --- > > Notes: > Version 2

Re: [libvirt PATCH v2 33/81] qemu: Introduce qemuMigrationDstFinishActive

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:33 +0200, Jiri Denemark wrote: > Refactors qemuMigrationDstFinish by moving some parts to a dedicated > function for easier introduction of postcopy resume code without > duplicating common parts of the Finish phase. The goal is to have the > following call graph: > >

Re: [libvirt PATCH v2 03/81] Introduce VIR_DOMAIN_RUNNING_POSTCOPY_FAILED

2022-06-06 Thread Jiri Denemark
On Mon, Jun 06, 2022 at 14:35:10 +0200, Peter Krempa wrote: > On Mon, Jun 06, 2022 at 14:29:20 +0200, Peter Krempa wrote: > > On Wed, Jun 01, 2022 at 14:49:03 +0200, Jiri Denemark wrote: > > > This new "post-copy failed" reason for the running state will be used on > > > the destination host when p

Re: [libvirt PATCH v2 21/81] qemu: Add support for postcopy-recover QEMU migration state

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:21 +0200, Jiri Denemark wrote: > Signed-off-by: Jiri Denemark > --- > > Notes: > Version 2: > - rebased on top of newly added "qemu: Use switch in > qemuProcessHandleMigrationStatus" > - debug message on a single line > > src/qemu/qemu_migration.c

Re: [libvirt PATCH v2 20/81] qemu: Handle 'postcopy-paused' migration state

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:20 +0200, Jiri Denemark wrote: > When connection breaks during post-copy migration, QEMU enters > 'postcopy-paused' state. We need to handle this state and make the > situation visible to upper layers. > > Signed-off-by: Jiri Denemark > --- > > Notes: > Version

Re: [libvirt PATCH v2 19/81] qemu: Use switch in qemuProcessHandleMigrationStatus

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:19 +0200, Jiri Denemark wrote: > Signed-off-by: Jiri Denemark > --- > > Notes: > Version 2: > - new patch Reviewed-by: Peter Krempa

Re: [libvirt PATCH v2 15/81] qemu: Restore async job start timestamp on reconnect

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:15 +0200, Jiri Denemark wrote: > Jobs that are supposed to remain active even when libvirt daemon > restarts were reported as started at the time the daemon was restarted. > This is not very helpful, we should restore the original timestamp. > > Signed-off-by: Jiri De

Re: [libvirt PATCH v2 04/81] qemu: Keep domain running on dst on failed post-copy migration

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:04 +0200, Jiri Denemark wrote: > There's no need to artificially pause a domain when post-copy fails > from our point of view unless QEMU connection is broken too as migration > may still be progressing well. > > Signed-off-by: Jiri Denemark > --- > > Notes: > V

Re: [libvirt PATCH v2 03/81] Introduce VIR_DOMAIN_RUNNING_POSTCOPY_FAILED

2022-06-06 Thread Peter Krempa
On Mon, Jun 06, 2022 at 14:29:20 +0200, Peter Krempa wrote: > On Wed, Jun 01, 2022 at 14:49:03 +0200, Jiri Denemark wrote: > > This new "post-copy failed" reason for the running state will be used on > > the destination host when post-copy migration fails while the domain is > > already running the

Re: [libvirt PATCH v2 03/81] Introduce VIR_DOMAIN_RUNNING_POSTCOPY_FAILED

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:49:03 +0200, Jiri Denemark wrote: > This new "post-copy failed" reason for the running state will be used on > the destination host when post-copy migration fails while the domain is > already running there. > > Signed-off-by: Jiri Denemark > --- > > Notes: > Versi

Re: [libvirt PATCH v2 78/81] NEWS: Add support for post-copy recovery

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:50:18 +0200, Jiri Denemark wrote: > Signed-off-by: Jiri Denemark > Reviewed-by: Peter Krempa > Reviewed-by: Pavel Hrdina > --- > > Notes: > Version 2: > - s/fecovering/recovering/ > - added `` around flag name and virsh command > > NEWS.rst | 5 + >

Re: [libvirt PATCH v2 76/81] qemu: Implement VIR_DOMAIN_ABORT_JOB_POSTCOPY flag

2022-06-06 Thread Peter Krempa
On Wed, Jun 01, 2022 at 14:50:16 +0200, Jiri Denemark wrote: > Signed-off-by: Jiri Denemark > --- > > Notes: > Version 2: > - dropped DeviceNotFound QMP error handling and replaced it with > reporting an error if the VIR_DOMAIN_ABORT_JOB_POSTCOPY flag is used > for something e

Re: [libvirt PATCH v2 81/81] RFC: qemu: Keep vCPUs paused while migration is in postcopy-paused

2022-06-06 Thread Daniel P . Berrangé
On Wed, Jun 01, 2022 at 02:50:21PM +0200, Jiri Denemark wrote: > QEMU keeps guest CPUs running even in postcopy-paused migration state so > that processes that already have all memory pages they need migrated to > the destination can keep running. However, this behavior might bring > unexpected del

Re: [libvirt PATCH v2 00/81] Add support for post-copy recovery

2022-06-03 Thread Jiri Denemark
On Wed, Jun 01, 2022 at 14:49:00 +0200, Jiri Denemark wrote: > This series implements a new VIR_MIGRATE_POSTCOPY_RESUME flag (virsh > migrate --resume) for recovering from a failed post-copy migration. > > You can also fetch the series from my gitlab fork (the last RFC patch is > missing there): >

Re: [libvirt PATCH v2 66/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase

2022-06-03 Thread Jiri Denemark
On Wed, Jun 01, 2022 at 14:50:06 +0200, Jiri Denemark wrote: > The QEMU process is already running, all we need to do is to call > migrate-recover QMP command. Except for some checks and cookie handling, > of course. > > Signed-off-by: Jiri Denemark > Reviewed-by: Peter Krempa > Reviewed-by: Pav

Re: [libvirt PATCH v2 67/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Finish phase

2022-06-03 Thread Jiri Denemark
On Wed, Jun 01, 2022 at 14:50:07 +0200, Jiri Denemark wrote: > Everything was already done in the normal Finish phase and vCPUs are > running. We just need to wait for all remaining data to be transferred. > > Signed-off-by: Jiri Denemark > Reviewed-by: Peter Krempa > Reviewed-by: Pavel Hrdina

Re: [libvirt PATCH v2 66/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase

2022-06-03 Thread Jiri Denemark
On Wed, Jun 01, 2022 at 14:50:06 +0200, Jiri Denemark wrote: > The QEMU process is already running, all we need to do is to call > migrate-recover QMP command. Except for some checks and cookie handling, > of course. > > Signed-off-by: Jiri Denemark > Reviewed-by: Peter Krempa > Reviewed-by: Pav

Re: [libvirt PATCH v2 60/81] qemu: Use autoptr for mig in qemuMigrationDstPrepareFresh

2022-06-01 Thread Ján Tomko
On a Wednesday in 2022, Jiri Denemark wrote: Signed-off-by: Jiri Denemark --- Notes: Version 2: - replaces [62/80] qemu: Simplify cleanup in qemuMigrationDstPrepareFresh - it's the same patch with different commit message and changes touching origErr removed :-) src/qemu/qem

[libvirt PATCH v2 21/81] qemu: Add support for postcopy-recover QEMU migration state

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark --- Notes: Version 2: - rebased on top of newly added "qemu: Use switch in qemuProcessHandleMigrationStatus" - debug message on a single line src/qemu/qemu_migration.c| 1 + src/qemu/qemu_monitor.c | 2 +- src/qemu/qemu_monitor.h |

[libvirt PATCH v2 39/81] qemu: Ignore missing memory statistics in query-migrate

2022-06-01 Thread Jiri Denemark
We want to use query-migrate QMP command to check the current migration state when reconnecting to active domains, but the reply we get to this command may not contain any statistics at all if called on the destination host. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pav

[libvirt PATCH v2 25/81] qemu: Rename "endjob" label in qemuMigrationDstFinish

2022-06-01 Thread Jiri Denemark
Let's call it "error" so that it's clear the label is only used in failure path. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c | 26 +- 1 file changed, 13 insertions

[libvirt PATCH v2 67/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Finish phase

2022-06-01 Thread Jiri Denemark
Everything was already done in the normal Finish phase and vCPUs are running. We just need to wait for all remaining data to be transferred. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c |

[libvirt PATCH v2 78/81] NEWS: Add support for post-copy recovery

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - s/fecovering/recovering/ - added `` around flag name and virsh command NEWS.rst | 5 + 1 file changed, 5 insertions(+) diff --git a/NEWS.rst b/NEWS.rst index 28ed46e361..9

[libvirt PATCH v2 81/81] RFC: qemu: Keep vCPUs paused while migration is in postcopy-paused

2022-06-01 Thread Jiri Denemark
QEMU keeps guest CPUs running even in postcopy-paused migration state so that processes that already have all memory pages they need migrated to the destination can keep running. However, this behavior might bring unexpected delays in interprocess communication as some processes will be stopped unt

Re: [libvirt PATCH v2 28/81] qemu: Drop obsolete comment in qemuMigrationDstFinish

2022-06-01 Thread Ján Tomko
On a Wednesday in 2022, Jiri Denemark wrote: The comment about QEMU < 0.10.6 has been irrelevant for years. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c | 4 1 file changed, 4 deletions

[libvirt PATCH v2 48/81] qemu: Use QEMU_MIGRATION_PHASE_POSTCOPY_FAILED

2022-06-01 Thread Jiri Denemark
This phase marks a migration protocol as broken in a post-copy phase. Libvirt is no longer actively watching the migration in this phase as the migration API that started the migration failed. This may either happen when post-copy migration really fails (QEMU enters postcopy-paused migration state

[libvirt PATCH v2 80/81] qemu: Fix VSERPORT_CHANGE event in post-copy migration

2022-06-01 Thread Jiri Denemark
When a domain has a guest agent channel enabled and the agent is running in the guest, we will get VSERPORT_CHANGE event on a destination host as soon as we start vCPUs there. This is not an issue for normal migration, but post-copy migration will remain running after we started vCPUs on the destin

[libvirt PATCH v2 74/81] qemu: Implement virDomainAbortJobFlags

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - VIR_DEBUG added src/qemu/qemu_driver.c | 18 -- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index

[libvirt PATCH v2 72/81] qemu: Enable support for VIR_MIGRATE_POSTCOPY_RESUME

2022-06-01 Thread Jiri Denemark
Since all parts of post-copy recovery have been implemented now, it's time to enable the corresponding flag. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.h | 1 + 1 file changed, 1 insertion

[libvirt PATCH v2 66/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase

2022-06-01 Thread Jiri Denemark
The QEMU process is already running, all we need to do is to call migrate-recover QMP command. Except for some checks and cookie handling, of course. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migra

[libvirt PATCH v2 42/81] qemu: Introduce qemuMigrationSrcBeginXML helper

2022-06-01 Thread Jiri Denemark
Turn the final part of Begin phase formatting a domain XML for migration into a reusable helper. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - dropped driver parameter from qemuMigrationSrcBeginXML src/qemu/qemu_migration.c |

[libvirt PATCH v2 51/81] qemu: Don't set VIR_MIGRATE_PAUSED for post-copy resume

2022-06-01 Thread Jiri Denemark
For historical reasons we automatically enabled VIR_MIGRATE_PAUSED flag when a migration was started for a paused domain. However, when resuming failed post-copy migration the domain on the source host will always be paused (as it is already running on the destination host). We must avoid enabling

[libvirt PATCH v2 35/81] qemu: Make final part of migration Confirm phase reusable

2022-06-01 Thread Jiri Denemark
By separating it into a dedicated qemuMigrationSrcComplete function which can be later called in other places. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - fixed indentation src/qemu/qemu_migration.c | 70 +++

[libvirt PATCH v2 71/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for peer-to-peer migration

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c | 53 ++- 1 file changed, 36 insertions(+), 17 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qe

[libvirt PATCH v2 45/81] qemu: Make qemuMigrationCheckPhase failure fatal

2022-06-01 Thread Jiri Denemark
The check can reveal a serious bug in our migration code and we should not silently ignore it. Signed-off-by: Jiri Denemark --- Notes: Version 2: - no change src/qemu/qemu_migration.c | 58 --- 1 file changed, 36 insertions(+), 22 deletions(-) diff

[libvirt PATCH v2 65/81] qemu: Start a migration phase in qemuMigrationAnyConnectionClosed

2022-06-01 Thread Jiri Denemark
Non-postcopy case talks to QEMU monitor and thus needs to create a nested job. Since qemuMigrationAnyConnectionClosed is called in case there's no thread processing a migration API, we need to make the current thread a temporary owner of the migration job to avoid "This thread doesn't seem to be th

[libvirt PATCH v2 50/81] virsh: Add --postcopy-resume option for migrate command

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change docs/manpages/virsh.rst | 9 +++-- tools/virsh-domain.c| 8 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/docs/manpages/virsh.rst b/doc

[libvirt PATCH v2 62/81] qemu: Rename qemuMigrationSrcCleanup

2022-06-01 Thread Jiri Denemark
The function is now called qemuMigrationAnyConnectionClosed to make it clear it is supposed to handle broken connection during migration. It will soon be used on both sides of migration so the "Src" part was changed to "Any" to avoid renaming the function twice in a row. The original *Cleanup name

[libvirt PATCH v2 76/81] qemu: Implement VIR_DOMAIN_ABORT_JOB_POSTCOPY flag

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark --- Notes: Version 2: - dropped DeviceNotFound QMP error handling and replaced it with reporting an error if the VIR_DOMAIN_ABORT_JOB_POSTCOPY flag is used for something else than outgoing post-copy migration src/qemu/qemu_driver.c | 48 +++

[libvirt PATCH v2 55/81] qemu: Add support for 'resume' parameter of migrate QMP command

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c| 8 ++-- src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-) diff

[libvirt PATCH v2 41/81] qemu: Check flags incompatible with offline migration earlier

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - dropped line breaks from error messages src/qemu/qemu_migration.c | 41 ++- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/src/

[libvirt PATCH v2 09/81] qemu: Introduce qemuProcessCleanupMigrationJob

2022-06-01 Thread Jiri Denemark
The function can be used as a callback for qemuDomainCleanupAdd to automatically clean up a migration job when a domain is destroyed. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_process.c | 25 ++

[libvirt PATCH v2 40/81] qemu: Improve post-copy migration handling on reconnect

2022-06-01 Thread Jiri Denemark
When libvirt daemon is restarted during an active post-copy migration, we do not always mark the migration as broken. In this phase libvirt is not really needed for migration to finish successfully. In fact the migration could have even finished while libvirt was not running or it may still be happ

[libvirt PATCH v2 60/81] qemu: Use autoptr for mig in qemuMigrationDstPrepareFresh

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark --- Notes: Version 2: - replaces [62/80] qemu: Simplify cleanup in qemuMigrationDstPrepareFresh - it's the same patch with different commit message and changes touching origErr removed :-) src/qemu/qemu_migration.c | 3 +-- 1 file changed, 1

[libvirt PATCH v2 56/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Perform phase

2022-06-01 Thread Jiri Denemark
It just calls migrate QMP command with resume=true without having to worry about migration capabilities or parameters, storage migration, etc. since everything has already been done in the normal Perform phase. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina ---

[libvirt PATCH v2 47/81] qemu: Do not set job owner in qemuMigrationJobSetPhase

2022-06-01 Thread Jiri Denemark
Both qemuMigrationJobSetPhase and qemuMigrationJobStartPhase were doing the same thing, which made little sense. Let's follow the difference between qemuDomainObjSetJobPhase and qemuDomainObjStartJobPhase and change job owner only in qemuMigrationJobStartPhase. Signed-off-by: Jiri Denemark Review

[libvirt PATCH v2 64/81] qemu: Handle incoming migration in qemuMigrationAnyConnectionClosed

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - part of this patch squashed into the previous one src/qemu/qemu_migration.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.c b/src/q

[libvirt PATCH v2 75/81] Add VIR_DOMAIN_ABORT_JOB_POSTCOPY flag for virDomainAbortJobFlags

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change include/libvirt/libvirt-domain.h | 16 src/libvirt-domain.c | 10 ++ 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/

[libvirt PATCH v2 61/81] qemu: Add support for migrate-recover QMP command

2022-06-01 Thread Jiri Denemark
This command tells QEMU to start listening for an incoming post-copy recovery connection. Just like migrate-incoming is used for starting fresh migration on the destination host. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - ad

[libvirt PATCH v2 79/81] Introduce VIR_JOB_MIGRATION_SAFE job type

2022-06-01 Thread Jiri Denemark
This is a special job for operations that need to modify domain state during an active migration. The modification must not affect any state that could conflict with the migration code. This is useful mainly for event handlers that need to be processed during migration and which could otherwise tim

[libvirt PATCH v2 49/81] Introduce VIR_MIGRATE_POSTCOPY_RESUME flag

2022-06-01 Thread Jiri Denemark
This flag can be used to restart post-copy migration once it failed because of a broken connection. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change include/libvirt/libvirt-domain.h | 5 + src/libvirt-domain.c

[libvirt PATCH v2 52/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Begin phase

2022-06-01 Thread Jiri Denemark
Mostly we just need to check whether the domain is in a failed post-copy migration that can be resumed. Signed-off-by: Jiri Denemark --- Notes: Version 2: - s/return NULL/return false/ - dropped line breaks from error messages - dropped driver argument when calling qemuMigrationS

[libvirt PATCH v2 54/81] qemu: Separate starting migration from qemuMigrationSrcRun

2022-06-01 Thread Jiri Denemark
qemuMigrationSrcRun does a lot of thing before and after telling QEMU to start the migration. Let's make the core reusable by moving it to a new qemuMigrationSrcStart function. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - drop

[libvirt PATCH v2 38/81] qemu: Finish completed unattended migration

2022-06-01 Thread Jiri Denemark
So far migration could only be completed while a migration API was running and waiting for the migration to finish. In case such API could not be called (the connection that initiated the migration is broken) the migration would just be aborted or left in a "don't know what to do" state. But this w

[libvirt PATCH v2 59/81] qemu: Refactor qemuMigrationDstPrepareFresh

2022-06-01 Thread Jiri Denemark
Offline migration jumps over a big part of qemuMigrationDstPrepareFresh. Let's move that part into a new qemuMigrationDstPrepareActive function to make the code easier to follow. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - dr

[libvirt PATCH v2 53/81] qemu: Refactor qemuMigrationSrcPerformPhase

2022-06-01 Thread Jiri Denemark
To make the code flow a bit more sensible. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - fixed "qmeu" typo in the commit message src/qemu/qemu_migration.c | 26 -- 1 file changed, 12 insertions(+), 14

[libvirt PATCH v2 70/81] qemu: Call qemuDomainCleanupAdd from qemuMigrationJobContinue

2022-06-01 Thread Jiri Denemark
Every single call to qemuMigrationJobContinue needs to register a cleanup callback in case the migrating domain dies between phases or when migration is paused due to a failure in postcopy mode. Let's integrate registering the callback in qemuMigrationJobContinue to make sure the current thread do

[libvirt PATCH v2 57/81] qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Confirm phase

2022-06-01 Thread Jiri Denemark
Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - moved "qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob)" to patch 12 (qemu: Keep migration job active after failed post-copy) src/qemu/qemu_migration.c | 48 ++

[libvirt PATCH v2 32/81] qemu: Separate cookie parsing for qemuMigrationDstFinishOffline

2022-06-01 Thread Jiri Denemark
To keep all cookie handling (parsing and formatting) in the same function. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c | 22 -- 1 file changed, 16 insertions(+), 6 del

[libvirt PATCH v2 34/81] qemu: Handle migration job in qemuMigrationDstFinish

2022-06-01 Thread Jiri Denemark
The function which started a migration phase should also finish it by calling qemuMigrationJobFinish/qemuMigrationJobContinue so that the code is easier to follow. Signed-off-by: Jiri Denemark --- Notes: Version 2: - rewritten without goto src/qemu/qemu_migration.c | 30 +++

[libvirt PATCH v2 73/81] Add virDomainAbortJobFlags public API

2022-06-01 Thread Jiri Denemark
The original virDomainAbortJob did not support flags. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change include/libvirt/libvirt-domain.h | 3 +++ src/driver-hypervisor.h | 5 src/libvirt-domain.c

[libvirt PATCH v2 18/81] qemu: Fetch paused migration stats

2022-06-01 Thread Jiri Denemark
Even though a migration is paused, we still want to see the amount of data transferred so far and that the migration is indeed not progressing any further. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - rebased on top of "qemu:

[libvirt PATCH v2 26/81] qemu: Generate migration cookie in Finish phase earlier

2022-06-01 Thread Jiri Denemark
The final part of Finish phase will be refactored into a dedicated function and we don't want to generate the cookie there. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c | 38 ++

[libvirt PATCH v2 27/81] qemu: Make final part of migration Finish phase reusable

2022-06-01 Thread Jiri Denemark
By separating it into a dedicated qemuMigrationDstComplete function which can be later called in other places. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - no change src/qemu/qemu_migration.c | 99 ++-

[libvirt PATCH v2 58/81] qemu: Introduce qemuMigrationDstPrepareFresh

2022-06-01 Thread Jiri Denemark
Moves most of the code from qemuMigrationDstPrepareAny to a new qemuMigrationDstPrepareFresh so that qemuMigrationDstPrepareAny can be shared with post-copy resume. Signed-off-by: Jiri Denemark Reviewed-by: Peter Krempa Reviewed-by: Pavel Hrdina --- Notes: Version 2: - dropped line bre

[libvirt PATCH v2 69/81] qemu: Register qemuProcessCleanupMigrationJob after Begin phase

2022-06-01 Thread Jiri Denemark
The callback will properly cleanup non-p2p migration job in case the migrating domain dies between Begin and Perform while the client which controls the migration is not cooperating (normally the API for the next migration phase would handle this). The same situation can happen even after Prepare

[libvirt PATCH v2 68/81] qemu: Create completed jobData in qemuMigrationSrcComplete

2022-06-01 Thread Jiri Denemark
Normally the structure is created once the source reports completed migration, but with post-copy migration we can get here even after libvirt daemon was restarted. It doesn't make sense to preserve the structure in our status XML as we're going to rewrite almost all of it while refreshing the stat

[libvirt PATCH v2 31/81] qemu: Introduce qemuMigrationDstFinishOffline

2022-06-01 Thread Jiri Denemark
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph: - qemuMigrationDstFinish - qemuMigrationDstFinishOffline

<    1   2   3   4   5   6   7   8   9   10   >