On 2024/7/25 20:50, Dr. David Alan Gilbert wrote:
* Daniel P. Berrang� (berra...@redhat.com) wrote:
On Thu, Jul 25, 2024 at 01:35:21PM +0200, Markus Armbruster wrote:
Guoyi Tu <t...@chinatelecom.cn> writes:
In the test environment, we conducted IO stress tests on all storage disks
within a virtual machine that had five storage devices mounted.During
testing,
we found that the qemu process allocated a large amount of memory (~800MB)
to handle these IO operations.
When the test ended, although qemu called free() to release the allocated
memory, the memory was not actually returned to the operating system, as
observed via the top command.
Upon researching the glibc memory management mechanism, we found that when
small chunks of memory are allocated in user space and then released with
free(), the glibc memory management mechanism does not necessarily return
this memory to the operating system. Instead, it retains the memory until
certain conditions are met for release.
Yes.
Looking at mallopt(3) man page, the M_TRIM_THRESHOLD is said to control
when glibc releases the top of the heap back to the OS. It is said to
default to 128 kb.
I'm curious how we get from that default, to 800 MB of unused memory ?
Is it related to the number of distinct malloc arenas that are in use ?
I wonder which IO mechanism was being used - the 'iothreads' used to sometimes
blow up and start 100s of threads; is that the case here?
No, qemu is not configured with iothread. If what you said is correct,
then in the case where iothread is used, the qemu process might occupy
more memory when storage IO is idle.
I'm curious what malloc_stats() would report before & after malloc_trim
when QEMU is in this situation with lots of wasted memory.
Yes; maybe also trying valgrind's massif:
https://valgrind.org/docs/manual/ms-manual.html
(if it works on Qemu!)
might help say where it's going?
Thank you for your suggestion. I will use this tool to analyze which
part is allocating more memory.
--
Guoyi
Dave
For virtual machines that only have business operations during specific
periods, they remain idle most of the time. However, the qemu process
still occupies a large amount of memory resources, leading to significant
memory resource waste.
Mitigation: the memory free()'s but not returned to the OS can be paged
out.
To address this issue, this patch introduces an API to actively reclaim
idle memory within the qemu process. This API effectively calls
malloc_trim()
to notify glibc to trim free memory. With this api, the management tool
can monitor the virtual machine's state and call this API during idle times
to free up the memory occupied by the virtual machine, thereby allowing more
virtual machines to be provisioned.
How does this affect the test case you described above?
There's an existing use of malloc_trim() in util/rcu.c's
call_rcu_thread(). It's from commit 5a22ab71623:
rcu: reduce more than 7MB heap memory by malloc_trim()
Since there are some issues in memory alloc/free machenism
in glibc for little chunk memory, if Qemu frequently
alloc/free little chunk memory, the glibc doesn't alloc
little chunk memory from free list of glibc and still
allocate from OS, which make the heap size bigger and bigger.
This patch introduce malloc_trim(), which will free heap
memory when there is no rcu call during rcu thread loop.
malloc_trim() can be enabled/disabled by --enable-malloc-trim/
--disable-malloc-trim in the Qemu configure command. The
default malloc_trim() is enabled for libc.
Below are test results from smaps file.
(1)without patch
55f0783e1000-55f07992a000 rw-p 00000000 00:00 0 [heap]
Size: 21796 kB
Rss: 14260 kB
Pss: 14260 kB
(2)with patch
55cc5fadf000-55cc61008000 rw-p 00000000 00:00 0 [heap]
Size: 21668 kB
Rss: 6940 kB
Pss: 6940 kB
Signed-off-by: Yang Zhong <yang.zh...@intel.com>
Message-Id: <1513775806-19779-1-git-send-email-yang.zh...@intel.com>
Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
How would the malloc_trim() you propose interact with this one?
The above usage is automatic, while this proposal requires that
an external mgmt app monitor QEMU and tell it to free memory.
I'm wondering if the latter is really desirable, or whether QEMU
can call this itself when reasonable ?
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
