[Bug 1892648] Re: VM eating twice the memory allocated
Top and Free are rather useless for "where does my memory go". They show you "how much is left" which generally should be zero as the system fills up all that is left with caches and it is good that it does so. Therefore Top/Free are really not the tools to look for memory consumption. It always depends how deep you want to go, take a look here and pick whatever suits you. https://share.confex.com/share/123/webprogram/Handout/Session15754/Tools-V6-ShareDesign.pdf So many years and still mostly applies :-) The two recommendations of /proc/meminfo and smem are a good entry point for you I'd think. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1892648 Title: VM eating twice the memory allocated To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1892648] Re: VM eating twice the memory allocated
Hang on. Does the "free" command has a bug too ? I mean, top, htop, free all are showing 72gb of used memory whereas the guest really is using 8gb. It is the first time this happens in two year of service. None of my other servers are behaving this way. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1892648 Title: VM eating twice the memory allocated To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1892648] Re: VM eating twice the memory allocated
top - 18:40:57 up 2 days, 18:34, 1 user, load average: 1,29, 1,20, 1,22 Tasks: 712 total, 1 running, 587 sleeping, 0 stopped, 0 zombie %Cpu(s): 1,9 us, 0,5 sy, 0,0 ni, 97,4 id, 0,1 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem : 82060336 total, 2932672 free, 77069424 used, 2058232 buff/cache KiB Swap: 4194300 total, 4188656 free, 5644 used. 4197740 avail Mem -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1892648 Title: VM eating twice the memory allocated To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1892648] Re: VM eating twice the memory allocated
I checked a 512M guest which has with some overhead ~550M
The majority is the unique guest memory and qemu allocations.
And a slight bit out of shared mappings.
PID User Command Swap USS PSS RSS
65719 libvirt-qemu /usr/bin/qemu-system-x86_64 4.0K 549.5M 550.6M
554.3M
ps_mem here reports 1.1G which is wrong
Private + Shared = RAM used Program
549.3 MiB + 551.6 MiB = 1.1 GiB qemu-system-x86_64
If you look in /proc/smaps you'll find that qemu has
the usual libs and a few unpopulated 64M mappings.
The real majority of memory consumption is in the mapping
of the guest memory.
7fb9b3e0-7fb9d3e0 rw-p 00:00 0
Size: 524288 kB
KernelPageSize:4 kB
MMUPageSize: 4 kB
Rss: 524176 kB
Pss: 524176 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty:524176 kB
Referenced: 519084 kB
Anonymous:524176 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped:0 kB
FilePmdMapped:0 kB
Shared_Hugetlb:0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked:0 kB
THPeligible:1
VmFlags: rd wr mr mw me dc ac dd sd hg mg
There just isn't much more than that:
root@g-newqemu:~# cat /proc/65719/smaps | grep -e '^Rss' | awk 'BEGIN {sum=0}
{sum=sum+$2} END {print sum}'
567464
root@g-newqemu:~# cat /proc/65719/smaps | grep -e '^Pss' | awk 'BEGIN {sum=0}
{sum=sum+$2} END {print sum}'
563837
ps_mem just has a bug in this case,
one can have a shared mapping without shared bytes.
But ps_mem does parse shared and pss to then recalculate it as:
Shared = Pss - Private
That is just wrong, shared is zero.
The calculation above makes it assume twice as much memory as is really
consumed.
As ps_mem isn't in Ubuntu that is at best a bug report at
https://github.com/pixelb/ps_mem/issues
** Changed in: qemu (Ubuntu)
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1892648
Title:
VM eating twice the memory allocated
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1892648] Re: VM eating twice the memory allocated
Will do it tonight as it is critically used. But for now smem does only account for 33.8G of used memory whereas ps_mem sees the whole used memory: # smem -tk 5743 root lxd --logfile /var/snap/lxd0 148.6M 149.4M 152.8M 19111 100107 /usr/sbin/mysqld 732.0K 212.3M 212.5M 214.6M 5236 libvirt-qemu qemu-system-x86_64 -enable- 2.5M30.3G30.6G 32.0G --- 172 26 3.2M31.2G31.7G33.8G # ps_mem 201.5 MiB + 208.9 MiB = 410.5 MiB lxd (5) 212.3 MiB + 212.8 MiB = 425.1 MiB mysqld 145.4 MiB + 417.9 MiB = 563.3 MiB systemd-journald (5) 30.3 GiB + 30.9 GiB = 61.2 GiB qemu-system-x86_64 - 63.5 GiB # free -mh totalusedfree shared buff/cache available Mem:78G 72G3,5G7,3M1,9G4,7G Swap: 4,0G5,0M4,0G -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1892648 Title: VM eating twice the memory allocated To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1892648] Re: VM eating twice the memory allocated
ps_mem is no standard command. Could you please - while the guest is running - get some data where we can be sure which values are reported? example: # Before you start the guest $ sudo smem -tk # Start the guest and let it run into that memory consumption $ sudo smem -tk $ cat /proc/meminfo $ pidstat -p $(pgrep -f 'name guest=ispmanager-master') -T ALL -tr 5 # End the guest (you reported memory is still consumed) $ sudo smem -tk $ cat /proc/meminfo That should be a start to discuss/theorize what is going on ** Changed in: qemu (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1892648 Title: VM eating twice the memory allocated To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1892648] Re: VM eating twice the memory allocated
** Attachment added: "VM definition" https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+attachment/5403944/+files/ispmanager-master.xml -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1892648 Title: VM eating twice the memory allocated To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1892648/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
