Dan, I just reply to the list since I do not want to clutter BZ:
While migrating VMs is easy (and the sampling is already running), can someone tell me the correct polling port to block with iptables? Thanks, On 27.09.2014 13:18, Daniel Helgenberger wrote: > Hallo Dan, > > I just opened a BZ aginst this [1]. Please let me know if I can be of > further assistance. I stopped the cron script for now, so vdsm can 'grow > nicely' (about 91.5 MB per 6hrs) > > Cheers, > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 > > On 24.09.2014 19:20, Dan Kenigsberg wrote: >>> On 01.09.2014 18:08, Dan Kenigsberg wrote: >>>> On Mon, Sep 01, 2014 at 03:30:53PM +0000, Daniel Helgenberger wrote: >>>>> Hello, >>>>> >>>>> in my LAB cluster I run into OOM conditions frequently because of a huge >>>>> VDSM process. The memory stats from my nodes right now: >>>>> >>>>> Node A; running one VM: >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>> 3465 vdsm 0 -20 18.6g 9.8g 8244 S 32.1 50.3 27265:21 vdsm >>>>> >>>>> >>>>> 7439 qemu 20 0 5641m 4.1g 4280 S 22.9 20.9 12737:08 qemu-kvm >>>>> >>>>> >>>>> 2912 root 15 -5 2710m 35m 5968 S 0.0 0.2 0:04.76 >>>>> supervdsmServer >>>>> >>>>> Node B, running 3 VMs including HosedEngine: >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>> 9079 vdsm 0 -20 9.9g 5.0g 7496 S 49.7 43.0 11858:06 vdsm >>>>> >>>>> >>>>> 3347 qemu 20 0 7749m 1.8g 5264 S 4.3 15.8 3:25.71 qemu-kvm >>>>> >>>>> >>>>> 18463 qemu 20 0 3865m 415m 5516 R 1.6 3.5 359:15.24 qemu-kvm >>>>> >>>>> >>>>> 11755 qemu 20 0 3873m 276m 5336 S 80.5 2.3 21639:39 qemu-kvm >>>>> >>>>> Basically VDSM consumes more then all my VMs. >>>>> >>>>> I thought of VDSM as a 'supervisor' process for qemu-kvm? >>>>> >>>>> I attached recend vdsm logs as well as a screen shot. >>>> Thanks for your report. It sounds like >>>> >>>> Bug 1130045 - Very high memory consumption >>>> >>>> which we believe is due to python-ioprocess. >> On Wed, Sep 24, 2014 at 03:39:25PM +0000, Daniel Helgenberger wrote: >>> Hi Dan, >>> >>> just to get this right, you yourself pointed me to the BZ. I was looking >>> at the duplicate, since the metadata tags 3.4. Sorry my lack of >>> knowlage, I really have no idea whatever there is an ioprocess python >>> binding in 3.4 or not - I just see vdsmd resident size growing in 3.4. >>> The top output below was from 3.4.3; I just upgraded to 3.4.4. But >>> clearly vdsmd should not use 10GB RAM? >> I'm sorry to have mislead you. The bug I refered to was indeed due to >> ioprocess, and caused a very dramatic memory leak in 3.5. We have yet >> another memory leak in 3.5.0, when managing gluster blocks >> Bug 1142647 - supervdsm leaks memory when using glusterfs >> >> 3.4.z does not use ioprocess, and do not have that gluster bug, so you >> are seeing completely different and much older. >> >> These leaks are not so easy to debug - but they are important. I'd love >> if you open a BZ about it. Please specify the rate of the leak, when >> does it happen (when a host has VMs? when a host is polled by Engine? On >> nfs? iscsi? >> >> What's `lsof -p pid-of-fat-vdsm`? >> >> I think that Francesco has some debug patches to help nail it down - he >> can provide smarter questions. >> >> Dan. >> >> -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users