[Bug 571093] Re: multipath + libvirtd eats away more memory over time
** Changed in: libvirt (Ubuntu Lucid) Status: New => In Progress ** Changed in: libvirt (Ubuntu Lucid) Importance: Undecided => Medium ** Changed in: libvirt (Ubuntu Lucid) Assignee: (unassigned) => Dustin Kirkland (kirkland) ** Changed in: libvirt (Ubuntu) Status: Confirmed => In Progress ** Changed in: libvirt (Ubuntu) Assignee: (unassigned) => Dustin Kirkland (kirkland) ** Changed in: libvirt (Ubuntu Lucid) Milestone: None => lucid-updates ** Summary changed: - multipath + libvirtd eats away more memory over time + [SRU] multipath + libvirtd eats away more memory over time -- [SRU] multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
Okay, I've created a patch that should fix the issue in two ways, first as observed in my previous comment I suspected that adding a virNodeDeviceDefFree(def) to udevAddOneDevice() would solve the issue, it for the most part did but still left approximately an 8k RES memory leak. Looking at the valgrind data that I had managed to generate (Comment #13) I suspected a second leak and noticed that udevProcessDeviceListEntry() after running udevAddOneDevice() ran udev_device_unref(device); which according to the udev code does: * Drop a reference of a udev device. If the refcount reaches zero, * the resources of the device will be released. Based on the fact that the reference is wiped at the end of udevProcessDeviceListEntry() I have added it to udevEventHandleCallback() which is the other main location of udevAddOneDevice() getting called. This is now resulting in no increase in RES memory (except for natural increases). Patch is attached. ** Patch added: "Rough patch that appears to fix the two main causes." http://launchpadlibrarian.net/49024396/9024-free-memory-for-invalid-devices.patch -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
I've had another look at this bug, reproducing etc, this time through GDB, with the output from valgrind & the debug output in my last two posts, here is what I believe is happening: 1. multipath -v4 gets called (or other appropriate trigger) - multipath detects a device that it believes is new, which triggers a udev add/change event 2. libvirt picks up on the events and runs the code within udevAddOneDevice(device) 3. While running around it seems that one of the if functions around node_device_udev.c:1230 returns non zero and hits the 'goto out' call. 4. As a result virNodeDeviceDefFree(def); does not get called and libvirt still thinks that the device exists. 5. When multipath realizes itself that the device isn't worth anything it triggers a udev remove call. 6. libvirt tries to remove the device but fails early on because it doesn't know what udev is talking about, and again doesn't free memory. For what it's worth, just before the goto out call was made gdb was about to tell me the following: (gdb) p def $4 = (virNodeDeviceDefPtr) 0x1cabd70 (gdb) x/t def 0x1cabd70: (gdb) x/3x def 0x1cabd70: 0x 0x 0x (gdb) x/s def 0x1cabd70: "" I've got a distinct feeling that if virNodeDeviceDefFree(def) was called before the function returned, this memory leak wouldn't have happened. I'm going to give it a test and see what happens. -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
Some more investigation: --technical bits-- (non tech lower down) libvirtd debug logging (attached) shows running multipath -F/multipath -v4 does the following: 14:02:28.570: debug : udevEventHandleCallback:1363 : udev action: 'add' 14:02:28.570: info : udevGetDeviceProperty:111 : udev reports device '252:0' does not have property 'DRIVER' 14:02:28.570: info : udevGetDeviceProperty:111 : udev reports device '252:0' does not have property 'PCI_CLASS' 14:02:28.570: info : udevGetDeviceProperty:111 : udev reports device '252:0' does not have property 'INTERFACE' 14:02:28.570: info : udevGetDeviceType:1082 : Could not determine device type for device with sysfs path '252:0' [snip] 14:02:28.592: debug : udevEventHandleCallback:1363 : udev action: 'add' 14:02:28.592: info : udevGetDeviceProperty:111 : udev reports device 'dm-0' does not have property 'DRIVER' 14:02:28.592: debug : udevGetDeviceProperty:131 : Found property key 'DEVNAME' value '/dev/dm-0' for device with sysname 'dm-0' 14:02:28.592: info : udevGetDeviceProperty:111 : udev reports device 'dm-0' does not have property 'ID_BUS' 14:02:28.592: info : udevGetDeviceProperty:111 : udev reports device 'dm-0' does not have property 'ID_SERIAL' 14:02:28.592: info : udevGetDeviceSysfsAttr:200 : udev reports device 'dm-0' does not have sysfs attr 'device/vendor' 14:02:28.592: info : udevGetDeviceSysfsAttr:200 : udev reports device 'dm-0' does not have sysfs attr 'device/model' 14:02:28.592: info : udevGetDeviceProperty:111 : udev reports device 'dm-0' does not have property 'ID_TYPE' 14:02:28.592: info : udevKludgeStorageType:912 : Could not find definitive storage type for device with sysfs path '/sys/devices/virtual/block/dm-0', trying to guess it 14:02:28.592: info : udevKludgeStorageType:924 : Could not determine storage type for device with sysfs path '/sys/devices/virtual/block/dm-0' [snip] 14:02:28.594: debug : udevEventHandleCallback:1363 : udev action: 'remove' 14:02:28.594: info : udevRemoveOneDevice:1147 : Failed to find device to remove that has udev name '/sys/devices/virtual/bdi/252:0' 14:02:28.598: debug : udevEventHandleCallback:1363 : udev action: 'remove' 14:02:28.598: info : udevRemoveOneDevice:1147 : Failed to find device to remove that has udev name '/sys/devices/virtual/block/dm-0' udevEventHandleCallback(...) will call udevAddOneDevice(device) in turn runs udevGetDeviceProperty(...) indirectly. Assuming (and I'd need to check this) that udevGetDeviceProperty() returns PROPERTY_ERROR then I have a feeling we have found the problem as it does not look to have free'd the memory in the exit path - would have to check, especially if udevRemoveOneDevice also fails. -- non technical bits -- It would be interesting to see if other peoples debug output from libvirt mimics my data. Instructions can be found at http://honk.sigxcpu.org/con/Debugging_libvirt.html In particular instead of libvirtd -d, try: LIBVIRT_DEBUG=1 libvirtd -v I'm going to have a bit more of a play around soon to see if I can work this out a bit more. ** Attachment added: "'LIBVIRT_DEBUG=1 libvirtd -v' output" http://launchpadlibrarian.net/48845025/lp571093-libvirt-verbosedebug -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
At least part of this problem seems to be independent of multipath. I'm seeing libvirtd steadily eating more memory even though I have currently no virtual machines even defined, much less active, and have not had occasion to install multipath. -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
I performed some additional observations, I agree Nigel's statements in #13: disabling the "change" ACTION in /lib/udev/rules.d/95-multipath.rules is just a bandaid, the memory leak is there anyways, just that it takes wy longer to fill in the memory. The problem "solved" by disabling the "change" ACTION in /lib/udev/rules.d/95-multipath.rules seems to be a real bug somewhere in udev or multipath: monitoring with udevadm the number of events gets reduced from "tons per second" to one every 30 seconds in my hardware, which shows all the other events were spurious events generated by a udev/multipath feedback loop... That issue is being tracked in https://bugs.launchpad.net/ubuntu/+source/udev/+bug/578180 and also Debian bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=581791 ** Bug watch added: Debian Bug tracker #581791 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=581791 -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
** Attachment added: "udevadm monitor -e output" http://launchpadlibrarian.net/48809497/udev-events.txt.gz -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
Same memory leak here, running kvm+libvirtd+multipath-tools on nodes with SAN access over QLogic cards However i noticed something which i guess is highly related: udev is constantly "discovering" the 4 multipath devices (sdb/sdc/sdd/sde) and causing a high CPU load of around 3 (8 CPU system). "ps" shows a process tree like this again and again 393 ?Shttps://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
I've done a bit of an experiment and this is what I've found: Running valgrind on libvirtd -v ($ sudo valgrind -v --leak-check=full --show-reachable=yes /usr/sbin/libvirtd -v) produced the output in lp571093-libvirt-val-run1 A quick summary: ==15520== 78,696 (18,048 direct, 60,648 indirect) bytes in 376 blocks are definitely lost in loss record 324 of 325 ==15520==at 0x4C267CC: calloc (vg_replace_malloc.c:467) ==15520==by 0x42CFAD: virAlloc (memory.c:100) ==15520==by 0x47C96D: udevAddOneDevice (node_device_udev.c:1209) ==15520==by 0x47D166: udevDeviceMonitorStartup (node_device_udev.c:1265) ==15520==by 0x54BD1BF: virStateInitialize (in /usr/lib/libvirt.so.0.7.5) ==15520==by 0x41A64B: main (libvirtd.c:3153) ==15520== ==15520== 204,320 (22,736 direct, 181,584 indirect) bytes in 98 blocks are definitely lost in loss record 325 of 325 ==15520==at 0x4C267CC: calloc (vg_replace_malloc.c:467) ==15520==by 0x50508CB: udev_device_new (libudev-device.c:240) ==15520==by 0x505285C: udev_monitor_receive_device (libudev-monitor.c:589) ==15520==by 0x47D367: udevEventHandleCallback (node_device_udev.c:1356) ==15520==by 0x414AF8: virEventRunOnce (event.c:478) ==15520==by 0x4166D8: qemudOneLoop (libvirtd.c:2169) ==15520==by 0x4169B2: qemudRunLoop (libvirtd.c:2278) ==15520==by 0x70669C9: start_thread (pthread_create.c:300) ==15520==by 0x736369C: clone (clone.S:112) ==15520== LEAK SUMMARY: ==15520==definitely lost: 46,856 bytes in 628 blocks ==15520==indirectly lost: 282,769 bytes in 8,181 blocks ==15520== possibly lost: 0 bytes in 0 blocks ==15520==still reachable: 109,152 bytes in 1,303 blocks ==15520== suppressed: 0 bytes in 0 blocks ==15520== I triggered the issue by running: $ sudo multipath -F $ sudo multipath -v4 multiple times to replicate the issue. I'd say that this is definitely a libvirt/udev bug based on the behaviour. I believe that while Nick's suggestion in Comment #9 may avoid the problem it looks like it's really a bandaid and it's possible that other applications will trigger this issue via udev. ** Attachment added: "valgrind -v --leak-check=full --show-reachable=yes /usr/sbin/libvirtd -v output" http://launchpadlibrarian.net/48714083/lp571093-libvirt-val-run1 -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: multipath + libvirtd eats away more memory over time
I've updated the title and marked the bug against the multipath-tools package as well. I'm a little skeptical about that though... The multipath-tools package has only had two minor changes since 10 Mar 2009. Unless something new in libvirt is only now exposing this issue... ** Summary changed: - libvirtd eats away more memory over time + multipath + libvirtd eats away more memory over time ** Changed in: libvirt (Ubuntu) Status: Incomplete => Triaged ** Package changed: libvirt (Ubuntu) => multipath-tools (Ubuntu) ** Also affects: libvirt (Ubuntu) Importance: Undecided Status: New ** Changed in: libvirt (Ubuntu) Status: New => Confirmed ** Changed in: libvirt (Ubuntu) Importance: Undecided => Medium ** Also affects: libvirt (Ubuntu Lucid) Importance: Undecided Status: New ** Also affects: multipath-tools (Ubuntu Lucid) Importance: Undecided Status: New -- multipath + libvirtd eats away more memory over time https://bugs.launchpad.net/bugs/571093 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
