On Wed, Mar 09, 2022 at 11:43:48AM +0000, Dr. David Alan Gilbert wrote: > * Claudio Fontana (cfont...@suse.de) wrote: > > On 3/7/22 1:28 PM, Dr. David Alan Gilbert wrote: > > > * Claudio Fontana (cfont...@suse.de) wrote: > > >> On 3/7/22 1:20 PM, Daniel P. Berrangé wrote: > > >>> On Mon, Mar 07, 2022 at 01:09:55PM +0100, Claudio Fontana wrote: > > >>>> Got it, this explains it, sorry for the noise on this. > > >>>> > > >>>> I'll continue to investigate the general issue of low throughput with > > >>>> virsh save / qemu savevm . > > >>> > > >>> BTW, consider measuring with the --bypass-cache flag to virsh save. > > >>> This causes libvirt to use a I/O helper that uses O_DIRECT when > > >>> saving the image. This should give more predictable results by > > >>> avoiding the influence of host I/O cache which can be in a differnt > > >>> state of usage each time you measure. It was also intended that > > >>> by avoiding hitting cache, saving the memory image of a large VM > > >>> will not push other useful stuff out of host I/O cache which can > > >>> negatively impact other running VMs. > > >>> > > >>> Also it is possible to configure compression on the libvirt side > > >>> which may be useful if you have spare CPU cycles, but your storage > > >>> is slow. See 'save_image_format' in the /etc/libvirt/qemu.conf > > >>> > > >>> With regards, > > >>> Daniel > > >>> > > >> > > >> Hi Daniel, thanks for these good info, > > >> > > >> regarding slow storage, for these tests I am saving to /dev/null to > > >> avoid having to take storage into account > > >> (and still getting low bandwidth unfortunately) so I guess compression > > >> is out of the question. > > > > > > What type of speeds do you get if you try a migrate to a netcat socket? > > > > much faster apparently, 30 sec savevm vs 7 seconds for migration to a > > netcat socket sent to /dev/null. > > > > nc -l -U /tmp/savevm.socket > > > > virsh suspend centos7 > > Domain centos7 suspended > > > > virsh qemu-monitor-command --cmd '{ "execute": "migrate", "arguments": { > > "uri": "unix:///tmp/savevm.socket" } }' centos7 > > > > virt97:/mnt # virsh qemu-monitor-command --cmd '{ "execute": > > "query-migrate" }' centos7 > > {"return":{"blocked":false,"status":"completed","setup-time":118,"downtime":257,"total-time":7524,"ram":{"total":32213049344,"postcopy-requests":0,"dirty-sync-count":3,"multifd-bytes":0,"pages-per-second":1057530,"page-size":4096,"remaining":0,"mbps":24215.572437483122,"transferred":22417172290,"duplicate":2407520,"dirty-pages-rate":0,"skipped":0,"normal-bytes":22351847424,"normal":5456994}},"id":"libvirt-438"} > > > > virt97:/mnt # virsh qemu-monitor-command --cmd '{ "execute": > > "query-migrate-parameters" }' centos7 > > {"return":{"cpu-throttle-tailslow":false,"xbzrle-cache-size":67108864,"cpu-throttle-initial":20,"announce-max":550,"decompress-threads":2,"compress-threads":8,"compress-level":0,"multifd-channels":8,"multifd-zstd-level":1,"announce-initial":50,"block-incremental":false,"compress-wait-thread":true,"downtime-limit":300,"tls-authz":"","multifd-compression":"none","announce-rounds":5,"announce-step":100,"tls-creds":"","multifd-zlib-level":1,"max-cpu-throttle":99,"max-postcopy-bandwidth":0,"tls-hostname":"","throttle-trigger-threshold":50,"max-bandwidth":9223372036853727232,"x-checkpoint-delay":20000,"cpu-throttle-increment":10},"id":"libvirt-439"} > > > > > > I did also a run with multifd-channels:1 instead of 8, if it matters: > > I suspect you haven't actually got multifd enabled ( check > query-migrate-capabilities ?). > > > > virt97:/mnt # virsh qemu-monitor-command --cmd '{ "execute": > > "query-migrate" }' centos7 > > {"return":{"blocked":false,"status":"completed","setup-time":119,"downtime":260,"total-time":8601,"ram":{"total":32213049344,"postcopy-requests":0,"dirty-sync-count":3,"multifd-bytes":0,"pages-per-second":908820,"page-size":4096,"remaining":0,"mbps":21141.861157274227,"transferred":22415264188,"duplicate":2407986,"dirty-pages-rate":0,"skipped":0,"normal-bytes":22349938688,"normal":5456528}},"id":"libvirt-453"} > > > > virt97:/mnt # virsh qemu-monitor-command --cmd '{ "execute": > > "query-migrate-parameters" }' centos7 > > {"return":{"cpu-throttle-tailslow":false,"xbzrle-cache-size":67108864,"cpu-throttle-initial":20,"announce-max":550,"decompress-threads":2,"compress-threads":8,"compress-level":0,"multifd-channels":1,"multifd-zstd-level":1,"announce-initial":50,"block-incremental":false,"compress-wait-thread":true,"downtime-limit":300,"tls-authz":"","multifd-compression":"none","announce-rounds":5,"announce-step":100,"tls-creds":"","multifd-zlib-level":1,"max-cpu-throttle":99,"max-postcopy-bandwidth":0,"tls-hostname":"","throttle-trigger-threshold":50,"max-bandwidth":9223372036853727232,"x-checkpoint-delay":20000,"cpu-throttle-increment":10},"id":"libvirt-454"} > > > > > > Still we are in the 20 Gbps range, or around 2560 MiB/s, way faster than > > savevm which does around 600 MiB/s when the wind is in its favor.. > > Yeh that's what I'd hope for off a decent CPU; hmm there's not that much > savevm specific is there?
BTW, quick clarification here. IIUC, Claudio says the test is 'virsh save $VMNAME /some/file'. This is *not* running 'savevm' at the QEMU level. So it is a bit misleading refering to it as savevm in the thread here. 'virsh save' is simply wired up to the normal QEMU 'migrate' commands, with libvirt giving QEMU a pre-opened FD, which libvirt processes the other end of to write out to disk. IOW, the performance delta is possibly on libvirt's side rather than QEMU's. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|