Hi, Peter,
On 2023/3/6 δΈε6:05, Peter Xu wrote:
Hi, Chuang,
On Fri, Mar 03, 2023 at 06:56:50PM +0800, Chuang Xu wrote:
Sorry to forget to update the test results in the last patch of v6.
In this version:
- add peter's patch.
- split mr_do_commit() from mr_commit().
- adjust the sanity check in address_space_to_flatview().
- rebase to latest upstream.
- replace 8260 with 8362 as testing host.
- update the latest test results.
Here I list some cases which will trigger do_commit() in
address_space_to_flatview():
I ran qtest cases and used systemtap to trace those do_commit().
1.virtio_load->virtio_init_region_cache
2.virtio_load->virtio_set_features_nocheck
What is this one specifically? I failed to see quickly why this needs to
reference the flatview.
0x560f3bb26510 : memory_region_transaction_do_commit+0x0/0x1c0
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bb26e45 : address_space_get_flatview+0x95/0x100
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bb296b6 : memory_listener_register+0xf6/0x300
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3baec59f : virtio_device_realize+0x12f/0x1a0
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb3b1f : device_set_realized+0x2ff/0x660
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb6ec6 : property_set_bool+0x46/0x70
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb9173 : object_property_set+0x73/0x100
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbbc1ef : object_property_set_qobject+0x2f/0x50
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb93e4 : object_property_set_bool+0x34/0xa0
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3b9830d9 : virtio_pci_realize+0x299/0x4e0
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3b901204 : pci_qdev_realize+0x7e4/0x1090
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb3b1f : device_set_realized+0x2ff/0x660
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb6ec6 : property_set_bool+0x46/0x70
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb9173 : object_property_set+0x73/0x100
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbbc1ef : object_property_set_qobject+0x2f/0x50
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bbb93e4 : object_property_set_bool+0x34/0xa0
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3b99e4a0 : qdev_device_add_from_qdict+0xb00/0xc40
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3bac0c75 : virtio_net_set_features+0x385/0x490
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3baec238 : virtio_set_features_nocheck+0x58/0x70
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x560f3baf1e9e : virtio_load+0x33e/0x820
[/data00/migration/qemu-open/build/qemu-system-x86_64]
3.vapic_post_load
Same confusion to this one..
0x557a57b0e510 : memory_region_transaction_do_commit+0x0/0x1c0
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x557a57b0e9b5 : memory_region_find_rcu+0x2e5/0x310
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x557a57b11165 : memory_region_find+0x55/0xf0
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x557a57a07638 : vapic_prepare+0x58/0x250
[/data00/migration/qemu-open/build/qemu-system-x86_64]
0x557a57a07a7b : vapic_post_load+0x1b/0x80
[/data00/migration/qemu-open/build/qemu-system-x86_64]
4.tcg_commit
This is not enabled if kvm is on, right?
Yes.
I obtained these results from qtests. And some qtests enabled tcg.π
5.ahci_state_post_load
During my test, virtio_init_region_cache() will frequently trigger
do_commit() in address_space_to_flatview(), which will reduce the
optimization effect of v6 compared with v1.
IIU above 1 & 4 could leverage one address_space_to_flatview_rcu() which
can keep the old semantics of address_space_to_flatview() by just assert
rcu read lock and do qatomic_rcu_read() (e.g., tcg_commit() will run again
at last stage of vm load). Not sure how much it'll help.
This can really improve the performance of the existing patch, which is
basically the same as that of v1, but it needs to add
address_space_to_flatview_rcu()
and address_space_get_flatview_rcu(). I have also considered this, but will
others find it confusing? Because there will be to_flatview(), to_flatview_raw()
and to_flatview_rcu()..
You may also want to have a look at the other patch to optimize ioeventfd
commit here; I think that'll also speed up vm load but not sure how much
your series can further do upon:
https://lore.kernel.org/all/20230228142514.2582-1-longpe...@huawei.com/
Thanks,
Here are the latest test results:
test info:
- Host
- Intel(R) Xeon(R) Platinum 8362 CPU
- Mellanox Technologies MT28841
- VM
- 32 CPUs 128GB RAM VM
- 8 16-queue vhost-net device
- 16 4-queue vhost-user-blk device.
time of loading non-iterable vmstate
before 112 ms
long's patch applied 103 ms
my patch applied 44 ms
both applied 39 ms
add as_to_flat_rcu 19 ms
