[PULL 19/25] migration: remove the QEMUFileOps 'shut_down' callback

2022-06-23 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé This directly implements the shutdown logic using QIOChannel APIs. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Daniel P. Berrangé Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela Signed-off-by: Dr. David Alan Gilbert --- migration/qemu-file-channel.c

[PULL 20/25] migration: remove the QEMUFileOps 'set_blocking' callback

2022-06-23 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé This directly implements the set_blocking logic using QIOChannel APIs. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Daniel P. Berrangé Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela Signed-off-by: Dr. David Alan Gilbert --- migration/qemu-file-channe

[PULL 22/25] migration: remove the QEMUFileOps 'get_buffer' callback

2022-06-23 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé This directly implements the get_buffer logic using QIOChannel APIs. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Daniel P. Berrangé Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela Signed-off-by: Dr. David Alan Gilbert dgilbert: Fixup len = *-*EIO as

[PULL 25/25] migration: remove the QEMUFileOps abstraction

2022-06-23 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé Now that all QEMUFile callbacks are removed, the entire concept can be deleted. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Daniel P. Berrangé Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela Signed-off-by: Dr. David Alan Gilbert --- migration/channel

[PULL 21/25] migration: remove the QEMUFileOps 'close' callback

2022-06-23 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé This directly implements the close logic using QIOChannel APIs. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Daniel P. Berrangé Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela Signed-off-by: Dr. David Alan Gilbert --- migration/qemu-file-channel.c | 1

[PULL 23/25] migration: remove the QEMUFileOps 'writev_buffer' callback

2022-06-23 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé This directly implements the writev_buffer logic using QIOChannel APIs. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Daniel P. Berrangé Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela Signed-off-by: Dr. David Alan Gilbert --- migration/qemu-file-chann

[PULL 09/29] multifd: Copy pages before compressing them with zlib

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Ilya Leoshkevich zlib_send_prepare() compresses pages of a running VM. zlib does not make any thread-safety guarantees with respect to changing deflate() input concurrently with deflate() [1]. One can observe problems due to this with the IBM zEnterprise Data Compression accelerator capabl

[PULL 11/29] migration: Postcopy preemption preparation on channel creation

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Create a new socket for postcopy to be prepared to send postcopy requested pages via this specific channel, so as to not get blocked by precopy pages. A new thread is also created on dest qemu to receive data from this new channel based on the ram_load_postcopy() routine. The ram

[PULL 01/29] accel/kvm/kvm-all: Refactor per-vcpu dirty ring reaping

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Add a non-required argument 'CPUState' to kvm_dirty_ring_reap so that it can cover single vcpu dirty-ring-reaping scenario. Signed-off-by: Hyman Huang(黄勇) Reviewed-by: Peter Xu Message-Id: Signed-off-by: Dr. David Alan Gilbert --- accel/kvm/kvm-all.c | 23

[PULL 00/29] migration queue

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The following changes since commit da7da9d5e608200ecc0749ff37be246e9cd3314f: Merge tag 'pull-request-2022-07-19' of https://gitlab.com/thuth/qemu into staging (2022-07-19 13:05:06 +0100) are available in the Git repository at: https://gitlab.com/dagrh/qemu.g

[PULL 13/29] migration: Postcopy recover with preempt enabled

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu To allow postcopy recovery, the ram fast load (preempt-only) dest QEMU thread needs similar handling on fault tolerance. When ram_load_postcopy() fails, instead of stopping the thread it halts with a semaphore, preparing to be kicked again when recovery is detected. A mutex is in

[PULL 02/29] cpus: Introduce cpu_list_generation_id

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Introduce cpu_list_generation_id to track cpu list generation so that cpu hotplug/unplug can be detected during measurement of dirty page rate. cpu_list_generation_id could be used to detect changes of cpu list, which is prepared for dirty page rate measurement. Signed-off

[PULL 12/29] migration: Postcopy preemption enablement

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu This patch enables postcopy-preempt feature. It contains two major changes to the migration logic: (1) Postcopy requests are now sent via a different socket from precopy background migration stream, so as to be isolated from very high page request delays. (2) For huge pa

[PULL 10/29] migration: Add postcopy-preempt capability

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Firstly, postcopy already preempts precopy due to the fact that we do unqueue_page() first before looking into dirty bits. However that's not enough, e.g., when there're host huge page enabled, when sending a precopy huge page, a postcopy request needs to wait until the whole huge

[PULL 19/29] migration: Respect postcopy request order in preemption mode

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu With preemption mode on, when we see a postcopy request that was requesting for exactly the page that we have preempted before (so we've partially sent the page already via PRECOPY channel and it got preempted by another postcopy request), currently we drop the request so that afte

[PULL 17/29] migration: Export tls-[creds|hostname|authz] params to cmdline too

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu It's useful for specifying tls credentials all in the cmdline (along with the -object tls-creds-*), especially for debugging purpose. The trick here is we must remember to not free these fields again in the finalize() function of migration object, otherwise it'll cause double-free

[PULL 18/29] migration: Enable TLS for preempt channel

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu This patch is based on the async preempt channel creation. It continues wiring up the new channel with TLS handshake to destionation when enabled. Note that only the src QEMU needs such operation; the dest QEMU does not need any change for TLS support due to the fact that all cha

[PULL 15/29] migration: Add property x-postcopy-preempt-break-huge

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Add a property field that can conditionally disable the "break sending huge page" behavior in postcopy preemption. By default it's enabled. It should only be used for debugging purposes, and we should never remove the "x-" prefix. Reviewed-by: Dr. David Alan Gilbert Reviewed-by

[PULL 16/29] migration: Add helpers to detect TLS capability

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Add migrate_channel_requires_tls() to detect whether the specific channel requires TLS, leveraging the recently introduced migrate_use_tls(). No functional change intended. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220707185513.27421-1-pet...@re

[PULL 04/29] softmmu/dirtylimit: Implement vCPU dirtyrate calculation periodically

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Introduce the third method GLOBAL_DIRTY_LIMIT of dirty tracking for calculate dirtyrate periodly for dirty page rate limit. Add dirtylimit.c to implement dirtyrate calculation periodly, which will be used for dirty page rate limit. Add dirtylimit.h to export util functions

[PULL 14/29] migration: Create the postcopy preempt channel asynchronously

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu This patch allows the postcopy preempt channel to be created asynchronously. The benefit is that when the connection is slow, we won't take the BQL (and potentially block all things like QMP) for a long time without releasing. A function postcopy_preempt_wait_channel() is introdu

[PULL 25/29] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg(). Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted. Fixes:

[PULL 03/29] migration/dirtyrate: Refactor dirty page rate calculation

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) abstract out dirty log change logic into function global_dirty_log_change. abstract out dirty page rate calculation logic via dirty-ring into function vcpu_calculate_dirtyrate. abstract out mathematical dirty page rate calculation into do_calculate_dirtyrate, decouple it f

[PULL 20/29] tests: Move MigrateCommon upper

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu So that it can be used in postcopy tests too soon. Reviewed-by: Daniel P. Berrange Signed-off-by: Peter Xu Message-Id: <20220707185522.27638-1-pet...@redhat.com> Signed-off-by: Dr. David Alan Gilbert --- tests/qtest/migration-test.c | 144 +-- 1

[PULL 08/29] tests: Add dirty page rate limit test

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Add dirty page rate limit test if kernel support dirty ring, The following qmp commands are covered by this test case: "calc-dirty-rate", "query-dirty-rate", "set-vcpu-dirty-limit", "cancel-vcpu-dirty-limit" and "query-vcpu-dirty-limit". Signed-off-by: Hyman Huang(黄勇) Ack

[PULL 22/29] tests: Add postcopy tls recovery migration test

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu It's easy to build this upon the postcopy tls test. Rename the old postcopy recovery test to postcopy/recovery/plain. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220707185527.27747-1-pet...@redhat.com> Signed-off-by: Dr. David Alan Gilbert dgil

[PULL 06/29] softmmu/dirtylimit: Implement virtual CPU throttle

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Setup a negative feedback system when vCPU thread handling KVM_EXIT_DIRTY_RING_FULL exit by introducing throttle_us_per_full field in struct CPUState. Sleep throttle_us_per_full microseconds to throttle vCPU if dirtylimit is in service. Signed-off-by: Hyman Huang(黄勇) Revie

[PULL 26/29] Add dirty-sync-missed-zero-copy migration stat

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras Signed-off-by: Leonardo Bras Acked-by: Markus Armbruster Acked-by: Peter Xu Reviewed-by: Daniel P. Berrangé Message-Id: <2022071122.18951-3-leob...@redhat.com> Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 2 ++ monitor/hmp-cmds.c| 5 + qa

[PULL 07/29] softmmu/dirtylimit: Implement dirty page rate limit

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Implement dirtyrate calculation periodically basing on dirty-ring and throttle virtual CPU until it reachs the quota dirty page rate given by user. Introduce qmp commands "set-vcpu-dirty-limit", "cancel-vcpu-dirty-limit", "query-vcpu-dirty-limit" to enable, disable, query d

[PULL 29/29] migration: Avoid false-positive on non-supported scenarios for zero-copy-send

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras Migration with zero-copy-send currently has it's limitations, as it can't be used with TLS nor any kind of compression. In such scenarios, it should output errors during parameter / capability setting. But currently there are some ways of setting this not-supported scenarios

[PULL 05/29] accel/kvm/kvm-all: Introduce kvm_dirty_ring_size function

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Introduce kvm_dirty_ring_size util function to help calculate dirty ring ful time. Signed-off-by: Hyman Huang(黄勇) Acked-by: Peter Xu Message-Id: Signed-off-by: Dr. David Alan Gilbert --- accel/kvm/kvm-all.c| 5 + accel/stubs/kvm-stub.c | 5 + include/sysem

[PULL 23/29] tests: Add postcopy preempt tests

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Four tests are added for preempt mode: - Postcopy plain - Postcopy recovery - Postcopy tls - Postcopy tls+recovery Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220707185530.27801-1-pet...@redhat.com> Signed-off-by: Dr. David Alan Gilbert

[PULL 24/29] migration: remove unreachable code after reading data

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé The code calls qio_channel_read() in a loop when it reports QIO_CHANNEL_ERR_BLOCK. This code is reported when errno==EAGAIN. As such the later block of code will always hit the 'errno != EAGAIN' condition, making the final 'else' unreachable. Fixes: Coverity CID 1490203

[PULL 27/29] migration/multifd: Report to user when zerocopy not working

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism. After each full dirty-bitmap scan there should be a zero-co

[PULL 21/29] tests: Add postcopy tls migration test

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Peter Xu We just added TLS tests for precopy but not postcopy. Add the corresponding test for vanilla postcopy. Rename the vanilla postcopy to "postcopy/plain" because all postcopy tests will only use unix sockets as channel. Signed-off-by: Peter Xu Message-Id: <20220707185525.27692-1-p

[PULL 28/29] multifd: Document the locking of MultiFD{Send/Recv}Params

2022-07-19 Thread Dr. David Alan Gilbert (git)
From: Juan Quintela Reorder the structures so we can know if the fields are: - Read only - Their own locking (i.e. sems) - Protected by 'mutex' - Only for the multifd channel Signed-off-by: Juan Quintela Message-Id: <20220531104318.7494-2-quint...@redhat.com> Reviewed-by: Dr. David Alan Gilbert

[PATCH] Revert "gitlab: disable accelerated zlib for s390x"

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" This reverts commit 309df6acb29346f89e1ee542b1986f60cab12b87. With Ilya's 'multifd: Copy pages before compressing them with zlib' in the latest migration series, this shouldn't be a problem any more. Suggested-by: Peter Maydell Signed-off-by: Dr. David Alan Gilber

[PULL 04/30] softmmu/dirtylimit: Implement vCPU dirtyrate calculation periodically

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Introduce the third method GLOBAL_DIRTY_LIMIT of dirty tracking for calculate dirtyrate periodly for dirty page rate limit. Add dirtylimit.c to implement dirtyrate calculation periodly, which will be used for dirty page rate limit. Add dirtylimit.h to export util functions

[PULL 01/30] accel/kvm/kvm-all: Refactor per-vcpu dirty ring reaping

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Add a non-required argument 'CPUState' to kvm_dirty_ring_reap so that it can cover single vcpu dirty-ring-reaping scenario. Signed-off-by: Hyman Huang(黄勇) Reviewed-by: Peter Xu Message-Id: Signed-off-by: Dr. David Alan Gilbert --- accel/kvm/kvm-all.c | 23

[PULL 00/30] migration queue

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The following changes since commit 68e26e1e812c8b09313d7929271f6cbd47ef4c07: Merge tag 'pull-la-20220719' of https://gitlab.com/rth7680/qemu into staging (2022-07-19 22:54:43 +0100) are available in the Git repository at: https://gitlab.com/dagrh/qemu.git ta

[PULL 08/30] tests: Add dirty page rate limit test

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Add dirty page rate limit test if kernel support dirty ring, The following qmp commands are covered by this test case: "calc-dirty-rate", "query-dirty-rate", "set-vcpu-dirty-limit", "cancel-vcpu-dirty-limit" and "query-vcpu-dirty-limit". Signed-off-by: Hyman Huang(黄勇) Ack

[PULL 03/30] migration/dirtyrate: Refactor dirty page rate calculation

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) abstract out dirty log change logic into function global_dirty_log_change. abstract out dirty page rate calculation logic via dirty-ring into function vcpu_calculate_dirtyrate. abstract out mathematical dirty page rate calculation into do_calculate_dirtyrate, decouple it f

[PULL 13/30] migration: Postcopy recover with preempt enabled

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu To allow postcopy recovery, the ram fast load (preempt-only) dest QEMU thread needs similar handling on fault tolerance. When ram_load_postcopy() fails, instead of stopping the thread it halts with a semaphore, preparing to be kicked again when recovery is detected. A mutex is in

[PULL 05/30] accel/kvm/kvm-all: Introduce kvm_dirty_ring_size function

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Introduce kvm_dirty_ring_size util function to help calculate dirty ring ful time. Signed-off-by: Hyman Huang(黄勇) Acked-by: Peter Xu Message-Id: Signed-off-by: Dr. David Alan Gilbert --- accel/kvm/kvm-all.c| 5 + accel/stubs/kvm-stub.c | 5 + include/sysem

[PULL 10/30] migration: Add postcopy-preempt capability

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Firstly, postcopy already preempts precopy due to the fact that we do unqueue_page() first before looking into dirty bits. However that's not enough, e.g., when there're host huge page enabled, when sending a precopy huge page, a postcopy request needs to wait until the whole huge

[PULL 02/30] cpus: Introduce cpu_list_generation_id

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Introduce cpu_list_generation_id to track cpu list generation so that cpu hotplug/unplug can be detected during measurement of dirty page rate. cpu_list_generation_id could be used to detect changes of cpu list, which is prepared for dirty page rate measurement. Signed-off

[PULL 12/30] migration: Postcopy preemption enablement

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu This patch enables postcopy-preempt feature. It contains two major changes to the migration logic: (1) Postcopy requests are now sent via a different socket from precopy background migration stream, so as to be isolated from very high page request delays. (2) For huge pa

[PULL 26/30] Add dirty-sync-missed-zero-copy migration stat

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras Signed-off-by: Leonardo Bras Acked-by: Markus Armbruster Acked-by: Peter Xu Reviewed-by: Daniel P. Berrangé Message-Id: <2022071122.18951-3-leob...@redhat.com> Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 2 ++ monitor/hmp-cmds.c| 5 + qa

[PULL 06/30] softmmu/dirtylimit: Implement virtual CPU throttle

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Setup a negative feedback system when vCPU thread handling KVM_EXIT_DIRTY_RING_FULL exit by introducing throttle_us_per_full field in struct CPUState. Sleep throttle_us_per_full microseconds to throttle vCPU if dirtylimit is in service. Signed-off-by: Hyman Huang(黄勇) Revie

[PULL 07/30] softmmu/dirtylimit: Implement dirty page rate limit

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Hyman Huang(黄勇) Implement dirtyrate calculation periodically basing on dirty-ring and throttle virtual CPU until it reachs the quota dirty page rate given by user. Introduce qmp commands "set-vcpu-dirty-limit", "cancel-vcpu-dirty-limit", "query-vcpu-dirty-limit" to enable, disable, query d

[PULL 15/30] migration: Add property x-postcopy-preempt-break-huge

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Add a property field that can conditionally disable the "break sending huge page" behavior in postcopy preemption. By default it's enabled. It should only be used for debugging purposes, and we should never remove the "x-" prefix. Reviewed-by: Dr. David Alan Gilbert Reviewed-by

[PULL 14/30] migration: Create the postcopy preempt channel asynchronously

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu This patch allows the postcopy preempt channel to be created asynchronously. The benefit is that when the connection is slow, we won't take the BQL (and potentially block all things like QMP) for a long time without releasing. A function postcopy_preempt_wait_channel() is introdu

[PULL 19/30] migration: Respect postcopy request order in preemption mode

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu With preemption mode on, when we see a postcopy request that was requesting for exactly the page that we have preempted before (so we've partially sent the page already via PRECOPY channel and it got preempted by another postcopy request), currently we drop the request so that afte

[PULL 11/30] migration: Postcopy preemption preparation on channel creation

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Create a new socket for postcopy to be prepared to send postcopy requested pages via this specific channel, so as to not get blocked by precopy pages. A new thread is also created on dest qemu to receive data from this new channel based on the ram_load_postcopy() routine. The ram

[PULL 09/30] multifd: Copy pages before compressing them with zlib

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Ilya Leoshkevich zlib_send_prepare() compresses pages of a running VM. zlib does not make any thread-safety guarantees with respect to changing deflate() input concurrently with deflate() [1]. One can observe problems due to this with the IBM zEnterprise Data Compression accelerator capabl

[PULL 22/30] tests: Add postcopy tls recovery migration test

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu It's easy to build this upon the postcopy tls test. Rename the old postcopy recovery test to postcopy/recovery/plain. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220707185527.27747-1-pet...@redhat.com> Signed-off-by: Dr. David Alan Gilbert dgil

[PULL 16/30] migration: Add helpers to detect TLS capability

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Add migrate_channel_requires_tls() to detect whether the specific channel requires TLS, leveraging the recently introduced migrate_use_tls(). No functional change intended. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220707185513.27421-1-pet...@re

[PULL 24/30] migration: remove unreachable code after reading data

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Daniel P. Berrangé The code calls qio_channel_read() in a loop when it reports QIO_CHANNEL_ERR_BLOCK. This code is reported when errno==EAGAIN. As such the later block of code will always hit the 'errno != EAGAIN' condition, making the final 'else' unreachable. Fixes: Coverity CID 1490203

[PULL 20/30] tests: Move MigrateCommon upper

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu So that it can be used in postcopy tests too soon. Reviewed-by: Daniel P. Berrange Signed-off-by: Peter Xu Message-Id: <20220707185522.27638-1-pet...@redhat.com> Signed-off-by: Dr. David Alan Gilbert --- tests/qtest/migration-test.c | 144 +-- 1

[PULL 21/30] tests: Add postcopy tls migration test

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu We just added TLS tests for precopy but not postcopy. Add the corresponding test for vanilla postcopy. Rename the vanilla postcopy to "postcopy/plain" because all postcopy tests will only use unix sockets as channel. Signed-off-by: Peter Xu Message-Id: <20220707185525.27692-1-p

[PULL 17/30] migration: Export tls-[creds|hostname|authz] params to cmdline too

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu It's useful for specifying tls credentials all in the cmdline (along with the -object tls-creds-*), especially for debugging purpose. The trick here is we must remember to not free these fields again in the finalize() function of migration object, otherwise it'll cause double-free

[PULL 18/30] migration: Enable TLS for preempt channel

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu This patch is based on the async preempt channel creation. It continues wiring up the new channel with TLS handshake to destionation when enabled. Note that only the src QEMU needs such operation; the dest QEMU does not need any change for TLS support due to the fact that all cha

[PULL 25/30] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg(). Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted. Fixes:

[PULL 23/30] tests: Add postcopy preempt tests

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Four tests are added for preempt mode: - Postcopy plain - Postcopy recovery - Postcopy tls - Postcopy tls+recovery Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220707185530.27801-1-pet...@redhat.com> Signed-off-by: Dr. David Alan Gilbert

[PULL 30/30] Revert "gitlab: disable accelerated zlib for s390x"

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" This reverts commit 309df6acb29346f89e1ee542b1986f60cab12b87. With Ilya's 'multifd: Copy pages before compressing them with zlib' in the latest migration series, this shouldn't be a problem any more. Suggested-by: Peter Maydell Signed-off-by: Dr. David Alan Gilber

[PULL 27/30] migration/multifd: Report to user when zerocopy not working

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism. After each full dirty-bitmap scan there should be a zero-co

[PULL 28/30] multifd: Document the locking of MultiFD{Send/Recv}Params

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Juan Quintela Reorder the structures so we can know if the fields are: - Read only - Their own locking (i.e. sems) - Protected by 'mutex' - Only for the multifd channel Signed-off-by: Juan Quintela Message-Id: <20220531104318.7494-2-quint...@redhat.com> Reviewed-by: Dr. David Alan Gilbert

[PULL 29/30] migration: Avoid false-positive on non-supported scenarios for zero-copy-send

2022-07-20 Thread Dr. David Alan Gilbert (git)
From: Leonardo Bras Migration with zero-copy-send currently has it's limitations, as it can't be used with TLS nor any kind of compression. In such scenarios, it should output errors during parameter / capability setting. But currently there are some ways of setting this not-supported scenarios

[PATCH] migration: Add canary to VMSTATE_END_OF_LIST

2022-01-12 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" We fairly regularly forget VMSTATE_END_OF_LIST markers off descriptions; given that the current check is only for ->name being NULL, sometimes we get unlucky and the code apparently works and no one spots the error. Explicitly add a flag, VMS_END that should be set

[PATCH v2 1/3] ppc: Fix vmstate_pbr403 subsection name

2022-01-13 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The pbr403 subsection is part of the tlb6xx state, so I believe it's name needs to be: .name = "cpu/tlb6xx/pbr403", Signed-off-by: Dr. David Alan Gilbert --- target/ppc/machine.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/target/ppc/

[PATCH v2 3/3] migration: Perform vmsd structure check during tests

2022-01-13 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Perform a check on vmsd structures during test runs in the hope of catching any missing terminators and other simple screwups. Signed-off-by: Dr. David Alan Gilbert --- migration/savevm.c | 39 +++ 1 file changed, 39 insertions

[PATCH v2 0/3] vmsd checks

2022-01-13 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Aftern the patch the other day where I added a missing END_OF_LIST, Peter suggested adding something more robust. Here I: add a check for a flag at the end of the list add a check that's performed in vmstate_register_with_alias_id only within qtest recursiv

[PATCH v2 2/3] migration: Add canary to VMSTATE_END_OF_LIST

2022-01-13 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" We fairly regularly forget VMSTATE_END_OF_LIST markers off descriptions; given that the current check is only for ->name being NULL, sometimes we get unlucky and the code apparently works and no one spots the error. Explicitly add a flag, VMS_END that should be set

[PULL 0/1] virtiofs queue

2022-01-26 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The following changes since commit aeb0ae95b7f18c66158792641cb6ba0cde5789ab: Merge remote-tracking branch 'remotes/jsnow-gitlab/tags/python-pull-request' into staging (2022-01-22 12:03:22 +) are available in the Git repository at: https://gitlab.com/dagr

[PULL 1/1] virtiofsd: Drop membership of all supplementary groups (CVE-2022-0358)

2022-01-26 Thread Dr. David Alan Gilbert (git)
From: Vivek Goyal At the start, drop membership of all supplementary groups. This is not required. If we have membership of "root" supplementary group and when we switch uid/gid using setresuid/setsgid, we still retain membership of existing supplemntary groups. And that can allow some operation

[PATCH v2] Fix %#0 misuses

2022-02-02 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Joe Perches pointed out on lkml [1] that the format special %# (which adds 0x) is included in the character count, i.e.: printf("0: %#08x\n0: %#08x\n", 0xabcdef01,1); gives: 0: 0xabcdef01 0: 0x01 rather than padding to the expected 8 data characters. Replac

[PATCH] Deprecate C virtiofsd

2022-02-09 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" There's a nice new Rust implementation out there; recommend people do new work on that. Signed-off-by: Dr. David Alan Gilbert --- docs/about/deprecated.rst | 14 ++ 1 file changed, 14 insertions(+) diff --git a/docs/about/deprecated.rst b/docs/about/

[PATCH] acpi: Bodge acpi_index migration

2022-04-05 Thread Dr. David Alan Gilbert (git)
binkkNQxjetjk.bin Description: Binary data

[PATCH] acpi: Bodge acpi_index migration

2022-04-06 Thread Dr. David Alan Gilbert (git)
binS1iWwfufXq.bin Description: Binary data

[PATCH] acpi: Bodge acpi_index migration

2022-04-06 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The 'acpi_index' field is a statically configured field, which for some reason is migrated; this never makes much sense because it's command line static. However, on piix4 it's conditional, and the condition/test function ends up having the wrong pointer passed to

[PATCH] migration: Fix operator type

2022-04-06 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Clang spotted an & that should have been an &&; fix it. Reported by: David Binderman / https://gitlab.com/dcb Fixes: 65dacaa04fa ("migration: introduce save_normal_page()") Resolves: https://gitlab.com/qemu-project/qemu/-/issues/963 Signed-off-by: Dr. David Alan Gi

[PATCH] deprecation: x86 default machine types

2022-03-01 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Declare the intent to require a machine type to be specified on x86 system emulation. Signed-off-by: Dr. David Alan Gilbert --- docs/about/deprecated.rst | 8 1 file changed, 8 insertions(+) diff --git a/docs/about/deprecated.rst b/docs/about/deprecated

[PULL 01/18] clock-vmstate: Add missing END_OF_LIST

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add the missing VMSTATE_END_OF_LIST to vmstate_muldiv Fixes: 99abcbc7600 ("clock: Provide builtin multiplier/divider") Signed-off-by: Dr. David Alan Gilbert Message-Id: <2022001934.115028-1-dgilb...@redhat.com> Reviewed-by: Peter Maydell Reviewed-by: Philippe

[PULL 05/18] qapi/monitor: allow VNC display id in set/expire_password

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Stefan Reiter It is possible to specify more than one VNC server on the command line, either with an explicit ID or the auto-generated ones à la "default", "vnc2", "vnc3", ... It is not possible to change the password on one of these extra VNC displays though. Fix this by adding a "display

[PULL 06/18] migration/rdma: set the REUSEADDR option for destination

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Jack Wang We hit following error during testing RDMA transport: in case of migration error, mgmt daemon pick one migration port, incoming rdma:[::]:8089: RDMA ERROR: Error: could not rdma_bind_addr Then try another -incoming rdma:[::]:8103, sometime it worked, sometimes need another try wi

[PULL 10/18] migration: Introduce postcopy channels on dest node

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Postcopy handles huge pages in a special way that currently we can only have one "channel" to transfer the page. It's because when we install pages using UFFDIO_COPY, we need to have the whole huge page ready, it also means we need to have a temp huge page when trying to receive t

[PULL 14/18] migration: Enlarge postcopy recovery to capture !-EIO too

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu We used to have quite a few places making sure -EIO happened and that's the only way to trigger postcopy recovery. That's based on the assumption that we'll only return -EIO for channel issues. It'll work in 99.99% cases but logically that won't cover some corner cases. One examp

[PULL 11/18] migration: Dump ramblock and offset too when non-same-page detected

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu In ram_load_postcopy() we'll try to detect non-same-page case and dump error. This error is very helpful for debugging. Adding ramblock & offset into the error log too. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220301083925.33483-6-pet...@redhat

[PULL 08/18] migration: Finer grained tracepoints for POSTCOPY_LISTEN

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu The enablement of postcopy listening has a few steps, add a few tracepoints to be there ready for some basic measurements for them. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220301083925.33483-3-pet...@redhat.com> Signed-off-by: Dr. David Alan Gi

[PULL 16/18] migration: Add migration_incoming_transport_cleanup()

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Add a helper to cleanup the transport listener. When do it, we should also null-ify the cleanup hook and the data, then it's even safe to call it multiple times. Move the socket_address_list cleanup altogether, because that's a mirror of the listener channels and only for the pur

[PULL 13/18] migration: Move static var in ram_block_from_stream() into global

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Static variable is very unfriendly to threading of ram_block_from_stream(). Move it into MigrationIncomingState. Make the incoming state pointer to be passed over to ram_block_from_stream() on both caller sites. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Messag

[PULL 03/18] monitor/hmp: add support for flag argument with value

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Stefan Reiter Adds support for the "-xs" parameter type, where "-x" denotes a flag name and the "s" suffix indicates that this flag is supposed to take an arbitrary string parameter. These parameters are always optional, the entry in the qdict will be omitted if the flag is not given. Rev

[PULL 12/18] migration: Add postcopy_thread_create()

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Postcopy create threads. A common manner is we init a sem and use it to sync with the thread. Namely, we have fault_thread_sem and listen_thread_sem and they're only used for this. Make it a shared infrastructure so it's easier to create yet another thread. Reviewed-by: Dr. Davi

[PULL 15/18] migration: postcopy_pause_fault_thread() never fails

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu Per the title, remove the return code and simplify the callers as the errors will never be triggered. No functional change intended. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220301083925.33483-12-pet...@redhat.com> Signed-off-by: Dr. David Alan

[PULL 04/18] qapi/monitor: refactor set/expire_password with enums

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Stefan Reiter 'protocol' and 'connected' are better suited as enums than as strings, make use of that. No functional change intended. Suggested-by: Markus Armbruster Reviewed-by: Markus Armbruster Signed-off-by: Stefan Reiter [FE: update "Since: " from 6.2 to 7.0 put 'keep' first i

[PULL 18/18] migration: Remove load_state_old and minimum_version_id_old

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Maydell There are no longer any VMStateDescription structs in the tree which use the load_state_old support for custom handling of incoming migration from very old QEMU. Remove the mechanism entirely. This includes removing one stray useless setting of minimum_version_id_old in a VM

[PULL 17/18] tests: Pass in MigrateStart** into test_migrate_start()

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu test_migrate_start() will release the MigrateStart structure that passed in, however that's not super clear to the caller because after the call returned the pointer can still be referenced by the callers. It can easily be a source of use-after-free. Let's pass in a double pointe

[PULL 00/18] migration queue

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The following changes since commit 64ada298b98a51eb2512607f6e6180cb330c47b1: Merge remote-tracking branch 'remotes/legoater/tags/pull-ppc-20220302' into staging (2022-03-02 12:38:46 +) are available in the Git repository at: https://gitlab.com/dagrh/qemu

[PULL 07/18] migration: Dump sub-cmd name in loadvm_process_command tp

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Peter Xu It'll be easier to read the name rather than index of sub-cmd when debugging. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20220301083925.33483-2-pet...@redhat.com> Signed-off-by: Dr. David Alan Gilbert --- migration/savevm.c | 3 ++- migration/

[PULL 02/18] virtiofsd: Let meson check for statx.stx_mnt_id

2022-03-02 Thread Dr. David Alan Gilbert (git)
From: Hanna Reitz In virtiofsd, we assume that the presence of the STATX_MNT_ID macro implies existence of the statx.stx_mnt_id field. Unfortunately, that is not necessarily the case: glibc has introduced the macro in its commit 88a2cf6c4bab6e94a65e9c0db8813709372e9180, but the statx.stx_mnt_id

<    9   10   11   12   13   14   15   16   17   18   >