I recently found that Qemu is using several hundred megabytes of RSS memory more than older versions such as Qemu 2.2.0. So I started tracing memory allocation and found 2 major reasons for this.
1) We changed the qemu coroutine pool to have a per thread and a global release pool. The choosen poolsize and the changed algorithm could lead to up to 192 free coroutines with just a single iothread. Each of the coroutines in the pool each having 1MB of stack memory. 2) Between Qemu 2.2.0 and 2.3.0 RCU was introduced which lead to delayed freeing of memory. This lead to higher heap allocations which could not effectively be returned to kernel (most likely due to fragmentation). The following series is what I came up with. Beside the coroutine patches I changed some allocations to forcibly use mmap. All these allocations are not repeatly made during runtime so the impact of using mmap should be neglectible. There are still some big malloced allocations left which cannot be easily changed (e.g. the pixman buffers in VNC). So it might an idea to set a lower mmap threshold for malloc since this threshold seems to be in the order of several Megabytes on modern systems. Peter Lieven (15): coroutine-ucontext: mmap stack memory coroutine-ucontext: add a switch to monitor maximum stack size coroutine-ucontext: reduce stack size to 64kB coroutine: add a knob to disable the shared release pool util: add a helper to mmap private anonymous memory exec: use mmap for subpages qapi: use mmap for QmpInputVisitor virtio: use mmap for VirtQueue loader: use mmap for ROMs vmware_svga: use mmap for scratch pad qom: use mmap for bigger Objects util: add a function to realloc mmapped memory exec: use mmap for PhysPageMap->nodes vnc-tight: make the encoding palette static vnc: use mmap for VncState configure | 33 ++++++++++++++++++-- exec.c | 11 ++++--- hw/core/loader.c | 16 +++++----- hw/display/vmware_vga.c | 3 +- hw/virtio/virtio.c | 5 +-- include/qemu/mmap-alloc.h | 7 +++++ include/qom/object.h | 1 + qapi/qmp-input-visitor.c | 5 +-- qom/object.c | 20 ++++++++++-- ui/vnc-enc-tight.c | 21 ++++++------- ui/vnc.c | 5 +-- ui/vnc.h | 1 + util/coroutine-ucontext.c | 66 +++++++++++++++++++++++++++++++++++++-- util/mmap-alloc.c | 27 ++++++++++++++++ util/qemu-coroutine.c | 79 ++++++++++++++++++++++++++--------------------- 15 files changed, 225 insertions(+), 75 deletions(-) -- 1.9.1