Peter,
Greetings from DigitalOcean. We're experiencing the same symptoms
without this patch.
We have, collectively, many gigabytes of un-planned-for RSS being used
per-hypervisor
that we would like to get rid of =).
Without explicitly trying this patch (will do that ASAP), we immediately
noticed that the
192MB mentioned immediately melts away (Yay) when we disabled the
coroutine thread pool explicitly,
with another ~100MB in additional stack usage that would likely also go
away if we
applied the entirety of your patch.
Is there any chance you have revisited this or have a timeline for it?
- Michael
/*
* Michael R. Hines
* Senior Engineer, DigitalOcean.
*/
On 06/28/2016 04:01 AM, Peter Lieven wrote:
I recently found that Qemu is using several hundred megabytes of RSS memory
more than older versions such as Qemu 2.2.0. So I started tracing
memory allocation and found 2 major reasons for this.
1) We changed the qemu coroutine pool to have a per thread and a global release
pool. The choosen poolsize and the changed algorithm could lead to up to
192 free coroutines with just a single iothread. Each of the coroutines
in the pool each having 1MB of stack memory.
2) Between Qemu 2.2.0 and 2.3.0 RCU was introduced which lead to delayed freeing
of memory. This lead to higher heap allocations which could not effectively
be returned to kernel (most likely due to fragmentation).
The following series is what I came up with. Beside the coroutine patches I
changed
some allocations to forcibly use mmap. All these allocations are not repeatly
made
during runtime so the impact of using mmap should be neglectible.
There are still some big malloced allocations left which cannot be easily
changed
(e.g. the pixman buffers in VNC). So it might an idea to set a lower mmap
threshold for
malloc since this threshold seems to be in the order of several Megabytes on
modern systems.
Peter Lieven (15):
coroutine-ucontext: mmap stack memory
coroutine-ucontext: add a switch to monitor maximum stack size
coroutine-ucontext: reduce stack size to 64kB
coroutine: add a knob to disable the shared release pool
util: add a helper to mmap private anonymous memory
exec: use mmap for subpages
qapi: use mmap for QmpInputVisitor
virtio: use mmap for VirtQueue
loader: use mmap for ROMs
vmware_svga: use mmap for scratch pad
qom: use mmap for bigger Objects
util: add a function to realloc mmapped memory
exec: use mmap for PhysPageMap->nodes
vnc-tight: make the encoding palette static
vnc: use mmap for VncState
configure | 33 ++++++++++++++++++--
exec.c | 11 ++++---
hw/core/loader.c | 16 +++++-----
hw/display/vmware_vga.c | 3 +-
hw/virtio/virtio.c | 5 +--
include/qemu/mmap-alloc.h | 7 +++++
include/qom/object.h | 1 +
qapi/qmp-input-visitor.c | 5 +--
qom/object.c | 20 ++++++++++--
ui/vnc-enc-tight.c | 21 ++++++-------
ui/vnc.c | 5 +--
ui/vnc.h | 1 +
util/coroutine-ucontext.c | 66 +++++++++++++++++++++++++++++++++++++--
util/mmap-alloc.c | 27 ++++++++++++++++
util/qemu-coroutine.c | 79 ++++++++++++++++++++++++++---------------------
15 files changed, 225 insertions(+), 75 deletions(-)
