Summary on top to be more readable TBH - while I could confirm and no finally see myself what you reported I didn't seen anything of that to be likely in qemu/kvm but instead this is the windows memory management. Qemu/KVM just responds with the allocations for qemu to the pages touched (or freed) by the Guest OS. And using an OS I somewhat understand and can follow the memory management actions it totally behaves as expected.
I'd think you would need to talk to a Windows Expert (or their support) and ask about that behavior. If anything KVM related then maybe the virtio-balloon driver might be affected, but AFAICS it seems it would not "do" this but only might "trigger" the windows behavior that we see in effect here. An alternative might be to report it to qemu mailing [1] list as I'm sure many others have stumbled, wondered and accepted this as windows behavior. If that assumption is true they might have the right pointers to explain it for you. [1]: https://lists.nongnu.org/mailman/listinfo/qemu-discuss What can we see in the detailed log below #1.1 shows that host and guest agree with ~900M of the 12G being used after boot #1.2 increasing the balloon does neither change VSS (max process allocation) nor RSS (actual allocation) #1.2 The guest accessible memory is reduced to 8G as intended #1.3 we see RSS rising to match the real usage (4g plus some overhead). #1.3 we also see that ~3.2G are still totally unused by the guest #1.4.1 we see that the host does not realize it could free some mem again (still 5.8G RSS) #1.4.2 we see the mem is back free again from the guest POV #1.5 as expected inflating the balloon in the guest makes it let go of memory and real usgae on the host (RSS) shrinks again #1.6 We see that the host took a chance to reduce effective allocation a bit (expected) Think of things like de-duplictaion or pruning for page where flags allow it to do so. But from the Guest POV nothing changed at all (expected) Migrating back it even had a bit more RSS again. So far all cases behaved as expected, nothing but the max mem and the baloon size is really static. Qemu does fault-in pages on demand and can only let them go if the guest let's them go (or via deup and such). Now is this different with Windows ? #2.1 right off boot it is more memory hungry than a linux and it seems to always consume ~8G. The installer allocated the same and this is without any workload and seems independent to the sizing. #2.2.1 We see RSS drop as expected as response to the balloon increase #2.2.2 We see that the guest "available" memory was reduced (as expected) The balloon seems to be accounted to the "in use" section of the windows mem stats which is ok, from the guests POV #2.5 Increasing the balloon is pushing down the effective memory allocation on the host as expected #2.6 as soon as the memory in the balloon is let go the guest consumes it again and scales up to 8G I checked the numbers and this does not seem anything that qemu/kvm does it is windows that touches this memory as soon as it is made available by the balloon deflating. #2.7 confirms your statement that after migration it is "small" for a short amount of time. But we have seen that in the Ubuntu guest as well and as I said before discarding pages before migration is smart and therefore (if possible) likely done in several stages of the mirgation. #2.8 Confirms that decreasing the balloon triggers sometihng inside the guest and windows memory management to ramp up the consumotion again It still seems to me that this would be normal behavior. I've spawned a Windows (10 trial as of today) and a Ubuntu (21.04 cloud image) guests. They all start with a sizing of 4vcpus and 12G max¤t memory. A balloon is set up and initially of size 0. #1 Ubuntu guest: #1.1 right after boot #1.1.1 Host POV $ pidstat -p $(pgrep -f 'name guest=h-test') -T ALL -r 5 Average: UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command Average: 64055 8290 - 0.00 0.00 14428168 898880 2.74 qemu-system-x86 Average: 64055 - 8290 0.00 0.00 14428168 898880 2.74 |__qemu-system-x86 Average: 64055 - 8294 0.00 0.00 14428168 898880 2.74 |__qemu-system-x86 Average: 64055 - 8296 0.00 0.00 14428168 898880 2.74 |__IO mon_iothread Average: 64055 - 8297 0.00 0.00 14428168 898880 2.74 |__CPU 0/KVM Average: 64055 - 8298 0.00 0.00 14428168 898880 2.74 |__CPU 1/KVM Average: 64055 - 8299 0.00 0.00 14428168 898880 2.74 |__CPU 2/KVM Average: 64055 - 8300 0.00 0.00 14428168 898880 2.74 |__CPU 3/KVM Average: 64055 - 8301 0.00 0.00 14428168 898880 2.74 |__SPICE Worker Average: 64055 - 8302 0.00 0.00 14428168 898880 2.74 |__vnc_worker $ virsh dominfo h-test | grep mem Max memory: 12582912 KiB Used memory: 12582912 KiB $ virsh dommemstat --domain h-test actual 12582912 swap_in 0 swap_out 0 major_fault 1007 minor_fault 236292 unused 11908600 available 12260044 usable 11847720 last_update 1615284587 disk_caches 174704 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 898880 #1.1.2 Guest POV $ cat /proc/meminfo MemTotal: 12260044 kB MemFree: 11892076 kB MemAvailable: 11839920 kB Buffers: 14308 kB Cached: 169068 kB SwapCached: 0 kB Active: 76920 kB Inactive: 158528 kB Active(anon): 976 kB Inactive(anon): 58200 kB Active(file): 75944 kB Inactive(file): 100328 kB Unevictable: 19368 kB Mlocked: 19368 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 612 kB Writeback: 0 kB AnonPages: 71468 kB Mapped: 59304 kB Shmem: 1040 kB KReclaimable: 26516 kB Slab: 59216 kB SReclaimable: 26516 kB SUnreclaim: 32700 kB KernelStack: 2640 kB PageTables: 1868 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 6130020 kB Committed_AS: 427716 kB VmallocTotal: 34359738367 kB VmallocUsed: 10684 kB VmallocChunk: 0 kB Percpu: 2240 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 89968 kB DirectMap2M: 3055616 kB DirectMap1G: 11534336 kB #1.2 bump the balloon to 4G leaving 8G for the guest $ virsh setmem h-test 8G --current #1.2.1 Host POV $ pidstat -p $(pgrep -f 'name guest=h-test') -T ALL -tr 5 Average: UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command Average: 64055 8290 - 0.00 0.00 14428168 871024 2.66 qemu-system-x86 Average: 64055 - 8290 0.00 0.00 14428168 871024 2.66 |__qemu-system-x86 Average: 64055 - 8294 0.00 0.00 14428168 871024 2.66 |__qemu-system-x86 Average: 64055 - 8296 0.00 0.00 14428168 871024 2.66 |__IO mon_iothread Average: 64055 - 8297 0.00 0.00 14428168 871024 2.66 |__CPU 0/KVM Average: 64055 - 8298 0.00 0.00 14428168 871024 2.66 |__CPU 1/KVM Average: 64055 - 8299 0.00 0.00 14428168 871024 2.66 |__CPU 2/KVM Average: 64055 - 8300 0.00 0.00 14428168 871024 2.66 |__CPU 3/KVM Average: 64055 - 8301 0.00 0.00 14428168 871024 2.66 |__SPICE Worker Average: 64055 - 8302 0.00 0.00 14428168 871024 2.66 |__vnc_worker $ virsh dominfo h-test | grep mem Max memory: 12582912 KiB Used memory: 8636416 KiB $ virsh dommemstat --domain h-test actual 8388608 swap_in 0 swap_out 0 major_fault 1062 minor_fault 275921 unused 7701720 available 8065740 usable 7651916 last_update 1615285262 disk_caches 184908 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 871024 #1.2.2 guest POV $ cat /proc/meminfo MemTotal: 8065740 kB MemFree: 7695008 kB MemAvailable: 7645304 kB Buffers: 14880 kB Cached: 170108 kB SwapCached: 0 kB Active: 83476 kB Inactive: 153616 kB Active(anon): 976 kB Inactive(anon): 58096 kB Active(file): 82500 kB Inactive(file): 95520 kB Unevictable: 19368 kB Mlocked: 19368 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 672 kB Writeback: 0 kB AnonPages: 71480 kB Mapped: 59384 kB Shmem: 1040 kB KReclaimable: 27924 kB Slab: 60604 kB SReclaimable: 27924 kB SUnreclaim: 32680 kB KernelStack: 2624 kB PageTables: 1884 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4032868 kB Committed_AS: 435808 kB VmallocTotal: 34359738367 kB VmallocUsed: 10652 kB VmallocChunk: 0 kB Percpu: 2288 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 89968 kB DirectMap2M: 3055616 kB DirectMap1G: 11534336 kB #1.3 Consuming 4G memory in the guest in the guest do: $ stress-ng --vm 4 --vm-keep --vm-bytes 4g #1.3.1 Host POV Linux 5.4.0-65-generic (h-winvm-1) 03/09/21 _x86_64_ (12 CPU) 10:32:52 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 10:32:57 64055 8290 - 0.00 0.00 14444588 5847016 17.85 qemu-system-x86 10:32:57 64055 - 8290 0.00 0.00 14444588 5847016 17.85 |__qemu-system-x86 10:32:57 64055 - 8294 0.00 0.00 14444588 5847016 17.85 |__qemu-system-x86 10:32:57 64055 - 8296 0.00 0.00 14444588 5847016 17.85 |__IO mon_iothread 10:32:57 64055 - 8297 0.00 0.00 14444588 5847016 17.85 |__CPU 0/KVM 10:32:57 64055 - 8298 0.00 0.00 14444588 5847016 17.85 |__CPU 1/KVM 10:32:57 64055 - 8299 0.00 0.00 14444588 5847016 17.85 |__CPU 2/KVM 10:32:57 64055 - 8300 0.00 0.00 14444588 5847016 17.85 |__CPU 3/KVM 10:32:57 64055 - 8301 0.00 0.00 14444588 5847016 17.85 |__SPICE Worker 10:32:57 64055 - 8302 0.00 0.00 14444588 5847016 17.85 |__vnc_worker 10:32:57 64055 - 9302 0.00 0.00 14444588 5847016 17.85 |__worker 10:32:57 64055 - 9304 0.00 0.00 14444588 5847016 17.85 |__worker 10:32:57 64055 - 9305 0.00 0.00 14444588 5847016 17.85 |__worker 10:32:57 64055 - 9306 0.00 0.00 14444588 5847016 17.85 |__worker $ virsh dominfo h-test | grep mem Max memory: 12582912 KiB Used memory: 8388608 KiB $ virsh dommemstat --domain h-test actual 8388608 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2125212 unused 3220592 available 8065740 usable 3431220 last_update 1615285957 disk_caches 442328 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 5847016 #1.3.2 Guest POV $ cat /proc/meminfo MemTotal: 8065740 kB MemFree: 3219236 kB MemAvailable: 3429856 kB Buffers: 18184 kB Cached: 424124 kB SwapCached: 0 kB Active: 282340 kB Inactive: 4410476 kB Active(anon): 1000 kB Inactive(anon): 4258740 kB Active(file): 281340 kB Inactive(file): 151736 kB Unevictable: 19376 kB Mlocked: 19376 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 148 kB Writeback: 0 kB AnonPages: 4269944 kB Mapped: 69020 kB Shmem: 3416 kB KReclaimable: 38460 kB Slab: 72668 kB SReclaimable: 38460 kB SUnreclaim: 34208 kB KernelStack: 2816 kB PageTables: 10800 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4032868 kB Committed_AS: 4893992 kB VmallocTotal: 34359738367 kB VmallocUsed: 10828 kB VmallocChunk: 0 kB Percpu: 2320 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 98160 kB DirectMap2M: 3047424 kB DirectMap1G: 11534336 kB #1.4 end the mem hogs in the guest #1.4.1 Host POV Average: UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command Average: 64055 8290 - 0.00 0.00 14469176 5848080 17.86 qemu-system-x86 Average: 64055 - 8290 0.00 0.00 14469176 5848080 17.86 |__qemu-system-x86 Average: 64055 - 8294 0.00 0.00 14469176 5848080 17.86 |__qemu-system-x86 Average: 64055 - 8296 0.00 0.00 14469176 5848080 17.86 |__IO mon_iothread Average: 64055 - 8297 0.00 0.00 14469176 5848080 17.86 |__CPU 0/KVM Average: 64055 - 8298 0.00 0.00 14469176 5848080 17.86 |__CPU 1/KVM Average: 64055 - 8299 0.00 0.00 14469176 5848080 17.86 |__CPU 2/KVM Average: 64055 - 8300 0.00 0.00 14469176 5848080 17.86 |__CPU 3/KVM Average: 64055 - 8301 0.00 0.00 14469176 5848080 17.86 |__SPICE Worker Average: 64055 - 8302 0.00 0.00 14469176 5848080 17.86 |__vnc_worker Average: 64055 - 9321 0.00 0.00 14469176 5848080 17.86 |__worker Average: 64055 - 9322 0.00 0.00 14469176 5848080 17.86 |__worker Average: 64055 - 9324 0.00 0.00 14469176 5848080 17.86 |__worker Average: 64055 - 9325 0.00 0.00 14469176 5848080 17.86 |__worker Average: 64055 - 9326 0.00 0.00 14469176 5848080 17.86 |__worker Average: 64055 - 9328 0.00 0.00 14469176 5848080 17.86 |__worker virsh dominfo h-test | grep mem Max memory: 12582912 KiB Used memory: 8388608 KiB virsh dommemstat --domain h-test actual 8388608 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147270 unused 7430348 available 8065740 usable 7641112 last_update 1615286172 disk_caches 440072 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 5848052 #1.4.2 Guest POV $ cat /proc/meminfo MemTotal: 8065740 kB MemFree: 7422912 kB MemAvailable: 7633636 kB Buffers: 18264 kB Cached: 421800 kB SwapCached: 0 kB Active: 282752 kB Inactive: 212672 kB Active(anon): 980 kB Inactive(anon): 61240 kB Active(file): 281772 kB Inactive(file): 151432 kB Unevictable: 19368 kB Mlocked: 19368 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 36 kB Writeback: 0 kB AnonPages: 74804 kB Mapped: 65104 kB Shmem: 1044 kB KReclaimable: 38412 kB Slab: 72396 kB SReclaimable: 38412 kB SUnreclaim: 33984 kB KernelStack: 2688 kB PageTables: 2056 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4032868 kB Committed_AS: 463792 kB VmallocTotal: 34359738367 kB VmallocUsed: 10732 kB VmallocChunk: 0 kB Percpu: 2336 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 100208 kB DirectMap2M: 3045376 kB DirectMap1G: 11534336 kB #1.5 push the guest with the balloon to free some memory again So far everything works as expected, we now have two options to check if we get mem-back The suggested "migrate" guest which should not change anything for the Guest POV and IMHO also not much from the Host as an allocated page is an allocated page. It has to be transferred unless the guests lets go of it. So we can force the guest to let go things via balloon or migrate to see what happens. We know from guest POV that the guest only uses <1G so let us shrink to 2G and then go back up (you have to leave it some time - poll dommemstat). $ virsh setmem h-test 2G --current root@h-winvm-1:~# virsh dommemstat --domain win10 actual 4743168 swap_in 1547780 swap_out 0 major_fault 54563 minor_fault 3146133 unused 5007064 available 12582380 usable 5035708 last_update 1615301180 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 4396436 root@h-winvm-1:~# virsh dommemstat --domain win10 actual 4153344 swap_in 1547780 swap_out 0 major_fault 54563 minor_fault 3146133 unused 5007064 available 12582380 usable 5035708 last_update 1615301180 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 3858796 root@h-winvm-1:~# virsh dommemstat --domain win10 actual 2547712 swap_in 1547852 swap_out 0 major_fault 54566 minor_fault 3150095 unused 1929920 available 12582380 usable 1958564 last_update 1615301185 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 2290888 root@h-winvm-1:~# virsh dommemstat --domain win10 actual 2097152 swap_in 1547852 swap_out 0 major_fault 54566 minor_fault 3150095 unused 1929920 available 12582380 usable 1958564 last_update 1615301185 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1880600 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 6252544 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 5458164 available 6092492 usable 5668960 last_update 1615286457 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 3710968 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 4589568 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 5458164 available 6092492 usable 5668960 last_update 1615286457 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 2045720 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 3619840 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 5458164 available 6092492 usable 5668960 last_update 1615286457 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1122332 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 3095552 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 5458164 available 6092492 usable 5668960 last_update 1615286457 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1122332 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 2562048 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 1795092 available 2429644 usable 2005888 last_update 1615286462 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1038004 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 2097152 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 1795092 available 2429644 usable 2005888 last_update 1615286462 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1037744 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 7746560 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 2966440 available 3603148 usable 3177236 last_update 1615286497 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1037744 root@h-winvm-1:~# virsh dommemstat --domain h-test actual 8388608 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 2966440 available 3603148 usable 3177236 last_update 1615286497 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1037744 So our guest was down to 2G and is back up to 8G again. That should have forced the RSS to reduce, let us check 1.5.1 Host POV $ pidstat -p $(pgrep -f 'name guest=h-test') -T ALL -tr 5 Linux 5.4.0-65-generic (h-winvm-1) 03/09/21 _x86_64_ (12 CPU) 10:42:27 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 10:42:32 64055 8290 - 0.00 0.00 14436392 1037744 3.17 qemu-system-x86 10:42:32 64055 - 8290 0.00 0.00 14436392 1037744 3.17 |__qemu-system-x86 10:42:32 64055 - 8294 0.00 0.00 14436392 1037744 3.17 |__qemu-system-x86 10:42:32 64055 - 8296 0.00 0.00 14436392 1037744 3.17 |__IO mon_iothread 10:42:32 64055 - 8297 0.00 0.00 14436392 1037744 3.17 |__CPU 0/KVM 10:42:32 64055 - 8298 0.00 0.00 14436392 1037744 3.17 |__CPU 1/KVM 10:42:32 64055 - 8299 0.00 0.00 14436392 1037744 3.17 |__CPU 2/KVM 10:42:32 64055 - 8300 0.00 0.00 14436392 1037744 3.17 |__CPU 3/KVM 10:42:32 64055 - 8301 0.00 0.00 14436392 1037744 3.17 |__SPICE Worker 10:42:32 64055 - 8302 0.00 0.00 14436392 1037744 3.17 |__vnc_worker $ virsh dominfo h-test | grep mem Max memory: 12582912 KiB Used memory: 8388608 KiB $ virsh dommemstat --domain h-test actual 8388608 swap_in 0 swap_out 0 major_fault 1388 minor_fault 2147287 unused 7429056 available 8065740 usable 7639852 last_update 1615286542 disk_caches 440080 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1037744 1.5.2 Guest POV $ cat /proc/meminfo MemTotal: 8065740 kB MemFree: 7422808 kB MemAvailable: 7633756 kB Buffers: 18312 kB Cached: 421932 kB SwapCached: 0 kB Active: 282848 kB Inactive: 212704 kB Active(anon): 976 kB Inactive(anon): 61152 kB Active(file): 281872 kB Inactive(file): 151552 kB Unevictable: 19368 kB Mlocked: 19368 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 728 kB Writeback: 0 kB AnonPages: 74676 kB Mapped: 65092 kB Shmem: 1044 kB KReclaimable: 38424 kB Slab: 72236 kB SReclaimable: 38424 kB SUnreclaim: 33812 kB KernelStack: 2704 kB PageTables: 2052 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4032868 kB Committed_AS: 463792 kB VmallocTotal: 34359738367 kB VmallocUsed: 10748 kB VmallocChunk: 0 kB Percpu: 2368 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 102256 kB DirectMap2M: 3043328 kB DirectMap1G: 11534336 kB #1.6 migration I used the guest mem hogs again to get to this from Host POV virsh dommemstat --domain h-test actual 8388608 swap_in 0 swap_out 0 major_fault 1388 minor_fault 3226512 unused 7425512 available 8065740 usable 7636492 last_update 1615286727 disk_caches 440268 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 5239016 So we are back at >=5G real allocation with the guest max size bing 12G and current being 8G. From this let me migrate it: $ virsh migrate --unsafe --live --domain h-test "qemu+ssh://h-winvm-2/system" ... (wait until complete) ... Note: it helps the migration to reduce memory that can be discarded. Therefore it is quite likely that some pages will be gone and that is ok. #1.6.1 (new) Host POV pidstat -p $(pgrep -f 'name guest=h-test') -T ALL -tr Linux 5.4.0-65-generic (h-winvm-2) 03/09/21 _x86_64_ (12 CPU) 10:48:52 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 10:48:52 64055 8760 - 415.84 1.07 13498960 3691104 11.27 qemu-system-x86 10:48:52 64055 - 8760 415.63 0.78 13498960 3691104 11.27 |__qemu-system-x86 10:48:52 64055 - 8764 0.00 0.00 13498960 3691104 11.27 |__qemu-system-x86 10:48:52 64055 - 8766 0.01 0.01 13498960 3691104 11.27 |__IO mon_iothread 10:48:52 64055 - 8767 0.02 0.09 13498960 3691104 11.27 |__CPU 0/KVM 10:48:52 64055 - 8768 0.03 0.08 13498960 3691104 11.27 |__CPU 1/KVM 10:48:52 64055 - 8769 0.01 0.03 13498960 3691104 11.27 |__CPU 2/KVM 10:48:52 64055 - 8770 0.03 0.06 13498960 3691104 11.27 |__CPU 3/KVM 10:48:52 64055 - 8771 0.08 0.01 13498960 3691104 11.27 |__SPICE Worker 10:48:52 64055 - 8772 0.00 0.00 13498960 3691104 11.27 |__vnc_worker virsh dommemstat --domain h-test actual 8388608 swap_in 0 swap_out 0 major_fault 1388 minor_fault 3257482 unused 7428740 available 8065740 usable 7639816 last_update 1615286896 disk_caches 440316 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 3688880 virsh dominfo h-test | grep mem Max memory: 12582912 KiB Used memory: 8388608 KiB #1.6.2 Guest POV $ cat /proc/meminfo MemTotal: 8065740 kB MemFree: 7417436 kB MemAvailable: 7629520 kB Buffers: 18484 kB Cached: 422848 kB SwapCached: 0 kB Active: 283380 kB Inactive: 217308 kB Active(anon): 988 kB Inactive(anon): 65196 kB Active(file): 282392 kB Inactive(file): 152112 kB Unevictable: 19368 kB Mlocked: 19368 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 724 kB Writeback: 0 kB AnonPages: 78724 kB Mapped: 65604 kB Shmem: 1052 kB KReclaimable: 38536 kB Slab: 72572 kB SReclaimable: 38536 kB SUnreclaim: 34036 kB KernelStack: 2720 kB PageTables: 2104 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4032868 kB Committed_AS: 468012 kB VmallocTotal: 34359738367 kB VmallocUsed: 10748 kB VmallocChunk: 0 kB Percpu: 2400 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 102256 kB DirectMap2M: 3043328 kB DirectMap1G: 11534336 kB #2.1 Windows guest after boot <insert a long time here to slowly install windows from a gigantic ISO, ...> Version = win10_20H2 pro trial As disk I used a 25g qcow2 $ sudo qemu-img create -f qcow2 /var/lib/uvtool/libvirt/images/win10.qcow2 25G I installed virtio drivers 0.1.190 (for balloon) but used the default compat device types for windows 10 (no virtio used/needed for those). #2.1.1 Host POV Note: after installing the virtio-balloon driver and setting the poll/update period I got the extended info from that just as I got it from the Ubuntu guest. After starting it is worth to poll on this until it hit a stable plateau. $ virsh dommemstat --domain win10 actual 12582912 swap_in 579012 swap_out 0 major_fault 23501 minor_fault 561159 unused 10976012 available 12582380 usable 10989308 last_update 1615299526 disk_caches 26592 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 8252348 $ virsh dominfo win10 | grep mem Max memory: 12582912 KiB Used memory: 12582912 KiB $ pidstat -p $(pgrep -f 'name guest=win10') -T ALL -tr Linux 5.4.0-65-generic (h-winvm-1) 03/09/21 _x86_64_ (12 CPU) 14:19:51 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 14:19:51 64055 11715 - 157.08 13.56 14713692 8369296 25.55 qemu-system-x86 14:19:51 64055 - 11715 0.61 0.18 14713692 8369296 25.55 |__qemu-system-x86 14:19:51 64055 - 11719 0.00 0.00 14713692 8369296 25.55 |__qemu-system-x86 14:19:51 64055 - 11721 0.00 0.01 14713692 8369296 25.55 |__IO mon_iothread 14:19:51 64055 - 11722 18.72 4.33 14713692 8369296 25.55 |__CPU 0/KVM 14:19:51 64055 - 11723 136.91 8.50 14713692 8369296 25.55 |__CPU 1/KVM 14:19:51 64055 - 11724 0.00 0.00 14713692 8369296 25.55 |__CPU 2/KVM 14:19:51 64055 - 11725 0.00 0.00 14713692 8369296 25.55 |__CPU 3/KVM 14:19:51 64055 - 11726 0.06 0.00 14713692 8369296 25.55 |__SPICE Worker 14:19:51 64055 - 11734 0.04 0.03 14713692 8369296 25.55 |__worker 14:19:51 64055 - 11744 0.04 0.03 14713692 8369296 25.55 |__worker 14:19:51 64055 - 11745 0.05 0.04 14713692 8369296 25.55 |__worker I also see that it breathes a bit, it shifts from 8.3 to 7.8G RSS and back Windows memory management isn't identical to Linux - so that still is quite likely normal. #2.1.1 Guest POV Note: Using resmon to check available/cached/total/installed as well as the split into hw reserved / in-use / modifies /standby / free available/cached/total/installed 10808 872 12287 12288 hw reserved / in-use / modified /standby / free 1 1404 76 796 10011 #2.2 bump the balloon to 4G leaving 8G for the guest $ virsh setmem win10 8G --current #2.2.1 Host POV $ virsh dommemstat --domain win10 actual 8388608 swap_in 1546444 swap_out 0 major_fault 54510 minor_fault 2958516 unused 6819032 available 12582380 usable 6847008 last_update 1615300658 disk_caches 55952 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 5122432 $ virsh dominfo win10 | grep mem Max memory: 12582912 KiB Used memory: 8388608 KiB $ pidstat -p $(pgrep -f 'name guest=win10') -T ALL -tr Linux 5.4.0-65-generic (h-winvm-1) 03/09/21 _x86_64_ (12 CPU) 14:37:47 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 14:37:47 64055 11715 - 150.25 15.83 15085800 5122424 15.64 qemu-system-x86 14:37:47 64055 - 11715 1.00 0.27 15085800 5122424 15.64 |__qemu-system-x86 14:37:47 64055 - 11719 0.00 0.00 15085800 5122424 15.64 |__qemu-system-x86 14:37:47 64055 - 11721 0.00 0.01 15085800 5122424 15.64 |__IO mon_iothread 14:37:47 64055 - 11722 17.93 5.34 15085800 5122424 15.64 |__CPU 0/KVM 14:37:47 64055 - 11723 130.23 9.50 15085800 5122424 15.64 |__CPU 1/KVM 14:37:47 64055 - 11724 0.00 0.00 15085800 5122424 15.64 |__CPU 2/KVM 14:37:47 64055 - 11725 0.00 0.00 15085800 5122424 15.64 |__CPU 3/KVM 14:37:47 64055 - 11726 0.33 0.11 15085800 5122424 15.64 |__SPICE Worker 14:37:47 64055 - 12526 0.00 0.00 15085800 5122424 15.64 |__worker #2.2.2 Guest POV available/cached/total/installed 6635 1213 12287 12288 hw reserved / in-use / modified /standby / free 1 5533 112 1104 5539 #2.3 / 2.4 I don't know how to reproduce 1.3/1.4 in Windows atm But then Windows as-is is mem-hungry enough (~5G RSS) without it to be able to go on as-is with the other steps. #2.5 push the guest with the balloon to free some memory again This might be a but much pressure, but OTOH we see still 5G free reported by windows and only ~1.3G really really used. So we should be able to push it down to 2G as we did with Ubuntu. $ virsh setmem h-test 2G --current #2.5.1 Host POV root@h-winvm-1:~# virsh dommemstat --domain win10 actual 4743168 swap_in 1547780 swap_out 0 major_fault 54563 minor_fault 3146133 unused 5007064 available 12582380 usable 5035708 last_update 1615301180 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 4396436 root@h-winvm-1:~# virsh dommemstat --domain win10 actual 4153344 swap_in 1547780 swap_out 0 major_fault 54563 minor_fault 3146133 unused 5007064 available 12582380 usable 5035708 last_update 1615301180 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 3858796 root@h-winvm-1:~# virsh dommemstat --domain win10 actual 2547712 swap_in 1547852 swap_out 0 major_fault 54566 minor_fault 3150095 unused 1929920 available 12582380 usable 1958564 last_update 1615301185 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 2290888 root@h-winvm-1:~# virsh dommemstat --domain win10 actual 2097152 swap_in 1547852 swap_out 0 major_fault 54566 minor_fault 3150095 unused 1929920 available 12582380 usable 1958564 last_update 1615301185 disk_caches 57288 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1880600 $ virsh dominfo win10 | grep mem Max memory: 12582912 KiB Used memory: 2097152 KiB $pidstat -p $(pgrep -f 'name guest=win10') -T ALL -tr Linux 5.4.0-65-generic (h-winvm-1) 03/09/21 _x86_64_ (12 CPU) 14:48:07 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 14:48:07 64055 11715 - 146.10 17.51 15201120 2037192 6.22 qemu-system-x86 14:48:07 64055 - 11715 0.99 0.26 15201120 2037192 6.22 |__qemu-system-x86 14:48:07 64055 - 11719 0.00 0.00 15201120 2037192 6.22 |__qemu-system-x86 14:48:07 64055 - 11721 0.00 0.01 15201120 2037192 6.22 |__IO mon_iothread 14:48:07 64055 - 11722 17.46 6.05 15201120 2037192 6.22 |__CPU 0/KVM 14:48:07 64055 - 11723 126.58 10.29 15201120 2037192 6.22 |__CPU 1/KVM 14:48:07 64055 - 11724 0.00 0.00 15201120 2037192 6.22 |__CPU 2/KVM 14:48:07 64055 - 11725 0.00 0.00 15201120 2037192 6.22 |__CPU 3/KVM 14:48:07 64055 - 11726 0.33 0.10 15201120 2037192 6.22 |__SPICE Worker 14:48:07 64055 - 12607 0.00 0.02 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12625 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12626 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12628 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12629 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12630 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12631 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12632 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12633 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12634 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12635 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12636 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12637 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12638 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12639 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12640 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12641 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12642 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12643 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12644 0.00 0.01 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12645 0.00 0.00 15201120 2037192 6.22 |__worker 14:48:07 64055 - 12646 0.00 0.00 15201120 2037192 6.22 |__worker #2.5.2 Guest POV available/cached/total/installed 770 699 12287 12288 hw reserved / in-use / modified /standby / free 1 11343 0 913 31 #2.6 Back out 2G -> 8G We have seen that windows seems to grab up to 8G by default due to it's memory management. We can see the same here again. #2.6.1 Host POV $ virsh dommemstat --domain win10 actual 8388608 swap_in 2014816 swap_out 545692 major_fault 72092 minor_fault 4842029 unused 7239508 available 12582380 usable 7262608 last_update 1615301544 disk_caches 46200 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 7959552 $ virsh dominfo win10 | grep mem Max memory: 12582912 KiB Used memory: 8388608 KiB Linux 5.4.0-65-generic (h-winvm-1) 03/09/21 _x86_64_ (12 CPU) 14:52:56 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 14:52:56 64055 11715 - 214.87 17.64 15036144 7927392 24.20 qemu-system-x86 14:52:56 64055 - 11715 0.98 0.28 15036144 7927392 24.20 |__qemu-system-x86 14:52:56 64055 - 11719 0.00 0.00 15036144 7927392 24.20 |__qemu-system-x86 14:52:56 64055 - 11721 0.00 0.01 15036144 7927392 24.20 |__IO mon_iothread 14:52:56 64055 - 11722 36.10 6.14 15036144 7927392 24.20 |__CPU 0/KVM 14:52:56 64055 - 11723 176.58 10.30 15036144 7927392 24.20 |__CPU 1/KVM 14:52:56 64055 - 11724 0.00 0.00 15036144 7927392 24.20 |__CPU 2/KVM 14:52:56 64055 - 11725 0.00 0.00 15036144 7927392 24.20 |__CPU 3/KVM 14:52:56 64055 - 11726 0.45 0.11 15036144 7927392 24.20 |__SPICE Worker 14:52:56 64055 - 13207 0.00 0.00 15036144 7927392 24.20 |__worker #2.6.2 Guest POV available/cached/total/installed 7060 954 12287 12288 hw reserved / in-use / modified /standby / free 1 5201 23 931 6129 #2.7 Migration forward $ virsh migrate --unsafe --live --domain win10 "qemu+ssh://h-winvm-2/system" and back $ virsh migrate --unsafe --live --domain win10 "qemu+ssh://h-winvm-1/system" #2.7.1 Host POV root@h-winvm-2:~# virsh dommemstat --domain win10 actual 8388608 swap_in 2047460 swap_out 545692 major_fault 74019 minor_fault 4954537 unused 7190268 available 12582380 usable 7213584 last_update 1615301818 disk_caches 46632 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 1033240 root@h-winvm-1:~# virsh dommemstat --domain win10 actual 8388608 swap_in 2049312 swap_out 545692 major_fault 74113 minor_fault 5003427 unused 7162700 available 12582380 usable 7186086 last_update 1615301893 disk_caches 46772 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 2185160 pidstat -p $(pgrep -f 'name guest=win10') -T ALL -tr Linux 5.4.0-65-generic (h-winvm-1) 03/09/21 _x86_64_ (12 CPU) 14:58:48 UID TGID TID minflt/s majflt/s VSZ RSS %MEM Command 14:58:48 64055 13341 - 140.92 0.14 13737284 2197548 6.71 qemu-system-x86 14:58:48 64055 - 13341 140.65 0.13 13737284 2197548 6.71 |__qemu-system-x86 14:58:48 64055 - 13345 0.00 0.00 13737284 2197548 6.71 |__qemu-system-x86 14:58:48 64055 - 13347 0.00 0.00 13737284 2197548 6.71 |__IO mon_iothread 14:58:48 64055 - 13348 0.11 0.00 13737284 2197548 6.71 |__CPU 0/KVM 14:58:48 64055 - 13349 0.09 0.00 13737284 2197548 6.71 |__CPU 1/KVM 14:58:48 64055 - 13350 0.00 0.00 13737284 2197548 6.71 |__CPU 2/KVM 14:58:48 64055 - 13351 0.00 0.00 13737284 2197548 6.71 |__CPU 3/KVM 14:58:48 64055 - 13352 0.05 0.00 13737284 2197548 6.71 |__SPICE Worker 14:58:48 64055 - 13360 0.01 0.00 13737284 2197548 6.71 |__worker 14:58:48 64055 - 13362 0.01 0.00 13737284 2197548 6.71 |__worker 14:58:48 64055 - 13370 0.00 0.00 13737284 2197548 6.71 |__worker 14:58:48 64055 - 13371 0.00 0.00 13737284 2197548 6.71 |__worker #2.7.2 Guest POV available/cached/total/installed 7020 954 12287 12288 hw reserved / in-use / modified /standby / free 1 5218 49 929 6094 #2.8 Bump the balloon again This was reported to ramp up the mem consumption again - let me see if I can reproduce. I slightly shrunk the system from 8 to 7G which matches the reported case. $ virsh setmem win10 7G --current ! No increase in host memory consumption here Back up to 8G root@h-winvm-1:~# virsh dommemstat --domain win10 actual 8388608 swap_in 2054668 swap_out 545692 major_fault 74514 minor_fault 5100915 unused 7187536 available 12582380 usable 7210590 last_update 1615302144 disk_caches 46108 hugetlb_pgalloc 0 hugetlb_pgfail 0 rss 3247440 A slight increase to 3.2G ** Changed in: qemu (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1916313 Title: memory allocate issue of windows vm To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1916313/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
