Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
On Tue, Oct 14, 2014 at 07:28:39PM +0300, Michael S. Tsirkin wrote: On Tue, Oct 14, 2014 at 01:08:15PM +0100, Daniel P. Berrange wrote: On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote: Hi,all I was trying to use hugepage with VM and found that the hugepage not freed when close VM. 1.Before start VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 3072 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 2.Start VM the /proc/meminfo is: AnonHugePages:139264 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 3.Close VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB We can see there are 1024 hugepage leak! I try to found which function used to free hugepage but i'm not sure where the qemu_ram_free is the function to free hugepage. I found that the qemu_ram_free function not call unlink and we know unlink is used to free hugepage(see example of hugepage-mmap.c in kernel source). We can't rely on 'qemu_ram_free' ever executing because we must ensure hugepages are freed upon QEMU crash. It seems we should rely on UNIX filesytstem semantics and simply unlink the memory segment the moment we create it open the FD. That way the kernel will automatically free it when the FD is closed when QEMU process exits. We being libvirt? No, QEMU is responsible for creating the mmap file, so QEMU should take care of unlinking it after opening it. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
On 2014/10/14 20:02, Linhaifeng wrote:
Hi,all
I was trying to use hugepage with VM and found that the hugepage not freed
when close VM.
1.Before start VM the /proc/meminfo is:
AnonHugePages:124928 kB
HugePages_Total:4096
HugePages_Free: 3072
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize: 2048 kB
2.Start VM the /proc/meminfo is:
AnonHugePages:139264 kB
HugePages_Total:4096
HugePages_Free: 2048
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize: 2048 kB
3.Close VM the /proc/meminfo is:
AnonHugePages:124928 kB
HugePages_Total:4096
HugePages_Free: 2048
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize: 2048 kB
We can see there are 1024 hugepage leak!
I try to found which function used to free hugepage but i'm not sure where
the qemu_ram_free is the function to free hugepage.
I found that the qemu_ram_free function not call unlink and we know unlink is
used to free hugepage(see example of hugepage-mmap.c in kernel source).
void qemu_ram_free(ram_addr_t addr)
{
RAMBlock *block;
/* This assumes the iothread lock is taken here too. */
qemu_mutex_lock_ramlist();
QTAILQ_FOREACH(block, ram_list.blocks, next) {
if (addr == block-offset) {
QTAILQ_REMOVE(ram_list.blocks, block, next);
ram_list.mru_block = NULL;
ram_list.version++;
if (block-flags RAM_PREALLOC) {
;
} else if (xen_enabled()) {
xen_invalidate_map_cache_entry(block-host);
#ifndef _WIN32
} else if (block-fd = 0) {
munmap(block-host, block-length);
close(block-fd);
// should we add unlink here to free hugepage?
#endif
} else {
qemu_anon_ram_free(block-host, block-length);
}
g_free(block);
break;
}
}
qemu_mutex_unlock_ramlist();
}
When i run the QEMU with tap backend the hugepage will be freed but not the
vhost-user backend.
Maybe the vhost-user process should close the hugefile.
--
Regards,
Haifeng
Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote: Hi,all I was trying to use hugepage with VM and found that the hugepage not freed when close VM. 1.Before start VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 3072 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 2.Start VM the /proc/meminfo is: AnonHugePages:139264 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 3.Close VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB We can see there are 1024 hugepage leak! I try to found which function used to free hugepage but i'm not sure where the qemu_ram_free is the function to free hugepage. I found that the qemu_ram_free function not call unlink and we know unlink is used to free hugepage(see example of hugepage-mmap.c in kernel source). We can't rely on 'qemu_ram_free' ever executing because we must ensure hugepages are freed upon QEMU crash. It seems we should rely on UNIX filesytstem semantics and simply unlink the memory segment the moment we create it open the FD. That way the kernel will automatically free it when the FD is closed when QEMU process exits. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
On Tue, Oct 14, 2014 at 01:08:15PM +0100, Daniel P. Berrange wrote: On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote: Hi,all I was trying to use hugepage with VM and found that the hugepage not freed when close VM. 1.Before start VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 3072 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 2.Start VM the /proc/meminfo is: AnonHugePages:139264 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 3.Close VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB We can see there are 1024 hugepage leak! I try to found which function used to free hugepage but i'm not sure where the qemu_ram_free is the function to free hugepage. I found that the qemu_ram_free function not call unlink and we know unlink is used to free hugepage(see example of hugepage-mmap.c in kernel source). We can't rely on 'qemu_ram_free' ever executing because we must ensure hugepages are freed upon QEMU crash. It seems we should rely on UNIX filesytstem semantics and simply unlink the memory segment the moment we create it open the FD. That way the kernel will automatically free it when the FD is closed when QEMU process exits. Regards, Daniel We being libvirt? -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [Bug?]When close VM the hugepage not freed
On 2014/10/14 20:08, Daniel P. Berrange wrote: On Tue, Oct 14, 2014 at 08:02:38PM +0800, Linhaifeng wrote: Hi,all I was trying to use hugepage with VM and found that the hugepage not freed when close VM. 1.Before start VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 3072 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 2.Start VM the /proc/meminfo is: AnonHugePages:139264 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB 3.Close VM the /proc/meminfo is: AnonHugePages:124928 kB HugePages_Total:4096 HugePages_Free: 2048 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB We can see there are 1024 hugepage leak! I try to found which function used to free hugepage but i'm not sure where the qemu_ram_free is the function to free hugepage. I found that the qemu_ram_free function not call unlink and we know unlink is used to free hugepage(see example of hugepage-mmap.c in kernel source). We can't rely on 'qemu_ram_free' ever executing because we must ensure hugepages are freed upon QEMU crash. It seems we should rely on UNIX filesytstem semantics and simply unlink the memory segment the moment we create it open the FD. That way the kernel will automatically free it when the FD is closed when QEMU process exits. Regards, Daniel Hi, daniel Thank you for your answer. Does it means libvirt should free the hugepage? QEMU create the hugepage with template file and unlink it before mmap. Do you know why to unlink the hugepage before mmap? When unlink the hugepage before mmap libvirt cannot found the hugepage. How does libvirt to free the hugepage ? Regards, Haifeng
