[Bug 588293] Re: Memory leak in qemu
** Summary changed: - Memory leak + Memory leak in qemu ** Summary changed: - Memory leak in qemu + iovec array memory leak -- iovec array memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Serge: The package from your ppa fixes the problem for me! -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Thanks, David! I resubmitted the source branch for merge into lucid proposed. Will update if/when it gets merged. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
David and Chris, could you please try to reproduce with the packages in ppa:serge-hallyn/virt, which was created with the exact tree which I would like to propose for merge into lucid-proposed? You can try it by doing (as root): cat /etc/apt/sources.list.d/serge-hallyn-virt.list EOF deb http://ppa.launchpad.net/serge-hallyn/virt/ubuntu lucid main EOF apt-get update apt-get upgrade I then ran spew -g --raw --statistics -t -v 5G foo in two guests at the same time, while doing while [ 1 ]; do for p in `pidof kvm`; do grep RSS /proc/$p/status done sleep 10s echo done on the host. Neither guest exceeded its assigned memory size with this package on my system. Please let me know if it fixes the memory leak for you. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
** Branch linked: lp:~serge-hallyn/ubuntu/lucid/qemu-kvm/memleak-fix -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
One of the virtual machines with 2 GB RAM, is already using 3.5 GB RAM here, so the leaking still exists. I hoped that this was the fix... Has anyone already tested qemu 0.12.4? If this didn't fix it, the bug might still exist in the new qemu version. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Dick: I rebuilt 0.12.4 from debian and then maverick to test for my bug (mentioned in comment #14) and it completely fixed the leaking, I was able to leave my IO tests running all day without any sign of leakage. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
I now found the right commit http://git.kernel.org/?p=virt/kvm/qemu- kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960 A virtual Machine running with this for about an hour still uses not more memory than when it was started. Attached a patched version of kvm for amd64 ** Attachment added: Patched kvm http://launchpadlibrarian.net/50674173/kvm_84%2Bdfsg-0ubuntu16%2B0.12.3%2Bnoroms%2B0ubuntu9.1_amd64.deb -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
** Attachment added: Patched qemu-kvm http://launchpadlibrarian.net/50674252/qemu-kvm_0.12.3%2Bnoroms-0ubuntu9.1_amd64.deb -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Chris Jones suggested spew -g --raw --statistics -t -v 5G foo in a guest to reproduce the memory leak, and this worked for me. So now I've 1. confirmed the leak in the stock lucid package 2. built my own package with the commit David suggested applied, and confirmed no memory leak there I've not yet re-installed the package currently in lucid-proposed to confirm that I still see the memory leak there. Assuming I do, then I will push my changes to my bzr tree and re-submit for sponsorship. If I do not, then Ill request further confirmation that the bug still exists in lucid-proposed. David, thanks for pointing to the precise commit id! -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
I cannot reproduce this bug with the qemu-kvm package from lucid- proposed. Can anyone confirm that the bug still exists? David, can you past your /etc/apt/sources.list and output of 'dpkg -l grep qemu', 'uname -a', and tell me what you're running in the guests (both distro/version and workloads)? -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Serge: I ran a test with the (first?) -proposed fix and I was still able to see leaking, assuming I didn't get something wrong with downgrading from 0.12.4 to the -proposed packages. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Thanks, Chris. The I'll propose the branch with the new patch identified by David for merging. FWIW it is at https://code.launchpad.net/~serge-hallyn/ubuntu/lucid/qemu-kvm/memleak- fix -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Thanks for the patch :) I've installed the patched version on two production nodes here. I will have the results in a few days probably. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
I filed bug #591610 which may well be a dupe of this bug. I ran a test overnight (as described in that bug) and I still see the same significant leaking as before -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Dustin: has that package been uploaded? I don't see it yet -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
I'm not familiar with Ubuntu Package Managment but it seems there are some problems to upload the package to lucid https://code.launchpad.net/~serge-hallyn/ubuntu/lucid/qemu-kvm/memleak- fix2/+merge/27644 Hope this gets fixed soon, I have to restart some virtual machines every few days to keep the host running. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
It's still in the queue at: https://edge.launchpad.net/ubuntu/lucid/+queue?queue_state=1queue_text= The bzr branch isn't working, but that's no matter. We just need to get an archive admin sru team member to accept the package for proposed. Dustin -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
ACK from sru team -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Accepted into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Tags added: verification-needed -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
** Branch linked: lp:ubuntu/lucid-proposed/qemu-kvm -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
** Description changed: Binary package hint: qemu-kvm Ubuntu release: 10.04 LTS Package version: 0.12.3+noroms-0ubuntu9 There seems to be a huge memory leak in qemu/kvm 0.12.3. After a while, a virtual machine is using much more memory on the host system, than actually is in use on the guest system. A guest system with only 512 MB assigned, can even use multiple GB's of memory. This bug is also discussed here: http://sourceforge.net/tracker/?func=detailatid=893831aid=2989366group_id=180599 Would it be possible to fix this bug in Ubuntu? It makes KVM a bit useless currently, because running 2 GB of VM's can even cost 16 GB of memory. + + + Impact: A memory leak causes long-running (measured in only a few days) + kvm sessions to invoke the OOM killer. This makes many kvm or cloud + applications impossible. + + The issue is addressed with upstream patch with commit-id + 7eb58a6c556c3880e6712cbf6d24d681261c5095. A back-port of the + patch is attached, and can be seen in + https://code.edge.launchpad.net/~serge-hallyn/ubuntu/lucid/qemu-kvm/memleak-fix2/ + + To reproduce: simply start a kvm instance, ensure that aio is being + used, and let it run. + + Regression potential: the patch simply ensured that already-allocated + memory is freed in an error path which otherwise lets that memory leak. + -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Marking Maverick task fix-released, as this should be in the qemu-kvm 0.12.4 upload that was just pushed to Maverick. ** Also affects: qemu-kvm (Ubuntu Lucid) Importance: Undecided Status: New ** Also affects: qemu-kvm (Ubuntu Maverick) Importance: Medium Assignee: Serge Hallyn (serge-hallyn) Status: Triaged ** Changed in: qemu-kvm (Ubuntu Lucid) Status: New = Triaged ** Changed in: qemu-kvm (Ubuntu Lucid) Importance: Undecided = Medium ** Changed in: qemu-kvm (Ubuntu Lucid) Assignee: (unassigned) = Serge Hallyn (serge-hallyn) ** Changed in: qemu-kvm (Ubuntu Lucid) Milestone: None = lucid-updates ** Changed in: qemu-kvm (Ubuntu Maverick) Status: Triaged = Fix Released -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
I have also sponsored Serge's change: qemu-kvm_0.12.3+noroms-0ubuntu9.1_source.changes This should be available in lucid-proposed eventually. Please test there and let us know if this fixes your problem, and then we can get it promoted to lucid-updates. Thanks. ** Changed in: qemu-kvm (Ubuntu Lucid) Status: Triaged = Fix Committed -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Hi Serge, Assigning this to you. Should be really straightforward to fix. Ping me on IRC, and I'll walk you through it. Thanks! ** Changed in: qemu-kvm (Ubuntu) Status: Incomplete = Triaged ** Changed in: qemu-kvm (Ubuntu) Assignee: (unassigned) = Serge Hallyn (serge-hallyn) -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Bug #591610 is possibly a duplicate. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Same problem here. Also the servers which write much a more affected. As far as I see this commit should fix it. http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=7eb58a6c556c3880e6712cbf6d24d681261c5095 Ubuntu Server Team, can you please test and backport this? -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
The guests are running Debian Lenny with the default 2.6.26 kernel or Ubuntu 10.04 LTS with the default 2.6.32 kernel. They are doing different things, but mostly webserving tasks. The strange thing is, one of the virtual servers is running an Exim/Spamassassin based spam checking gateway. And the memory usage of that specific virtual server grows a lot faster than the others. Data is partly saved on a ramdisk on this virtual machine. So maybe it's related to how much I/O there is on this ramdisk. -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 588293] Re: Memory leak
Could you outline what type of guest are causing the leak? What are the guest doing? ** Changed in: qemu-kvm (Ubuntu) Importance: Undecided = Medium ** Changed in: qemu-kvm (Ubuntu) Status: New = Incomplete -- Memory leak https://bugs.launchpad.net/bugs/588293 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
