** Description changed: + SRU Justification: + [Impact] + * + + [Test Case] + * + + [Regression Potential] + * + -- + Running a multiple-domain start/destroy loop, libvirtd 1.1.1 restarts occasionally due to assertion failure in libnl. This affects both precise/quantal versions of netcf. Distibution version: Ubuntu 12.04.2 LTS Kernel: 3.5.0-41-generic #64 SMP Mon Dec 9 20:35:04 UTC 2013 x86_64 Libvirt: libvirt_1.1.1-0ubuntu8.2~cloud1 from cloud repository. Qemu-kvm: qemu-kvm_1.0+noroms-0ubuntu14.12 [qemu 1.5 not yet tested] libnl: libnl-3-200_3.2.3-2ubuntu2 Platform: HP ProLiant SL390s G7 x86_64; 2 socket x 6 cores/socket x 2 HT/core; 96GB How to reproduce: 1. Define a number of test domains. E.g., test-nn, nn from 01 .. NN. I use 32 Test domains have 4 vcpu, 8G memory, running an Ubuntu 11.04 image. Network is libvirt default virtual network. Domain mac addresses based on test number nn and dhcp serves fixed IP addresses based on mac address. 2. In a loop: 2a. start the NN domains serially -- waiting for the "virsh start" command to complete before starting next domain. 2b. sleep 20sec -- give domains some time to boot. Test doesn't check that domains have completed booting. 2c. destroy the NN domains "in parallel" -- with "virsh destroy test-$nn &" 2d. sleep 15secs. sleep times are more or less arbitrary. Next pass of starts does usually begin before previous pass of destroys completes, but I've never seen it say that "domain <name> already running" or such. What I expected to happen: start-destroy loop runs indefinitely without error. What happened instead: Eventually, I start seeing errors like: error: Failed to destroy domain lnvtest-31 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor and error: failed to get domain 'lnvtest-12' error: Domain not found: no domain with matching name 'lnvtest-12' Checking the libvirt debug log, I see that libvirtd restarted during this iteration, dumping its internal log buffer in the process. gdb traceback shows: #3 0x00007f845cafb192 in __GI___assert_fail (assertion=0x7f845c4b89bd "0", file=0x7f845c4b8538 "/build/buildd/libnl3-3.2.3/./lib/object.c", line=185, function=0x7f845c4b8668 "nl_object_put") at assert.c:103 #4 0x00007f845c4b4dea in nl_object_put () from /lib/libnl-3.so.200 #5 0x00007f845c4afb92 in nl_cache_remove () from /lib/libnl-3.so.200 #6 0x00007f845c4b4b07 in nl_object_free () from /lib/libnl-3.so.200 #7 0x00007f845c4b4b15 in nl_object_free () from /lib/libnl-3.so.200 #8 0x00007f845c4afb92 in nl_cache_remove () from /lib/libnl-3.so.200 #9 0x00007f845c4afd0b in nl_cache_clear () from /lib/libnl-3.so.200 #10 0x00007f845c4afd3e in nl_cache_free () from /lib/libnl-3.so.200 #11 0x00007f84517f3096 in ?? () from /usr/lib/libnetcf.so.1 #12 0x00007f84517f4220 in ?? () from /usr/lib/libnetcf.so.1 #13 0x00007f84517ef53f in ncf_close () from /usr/lib/libnetcf.so.1 #14 0x00007f8451a0be6f in netcfInterfaceClose (conn=0x7f83f00e7ad0) at /tmp/buildd/libvirt-1.1.1/./src/interface/interface_backend_netcf.c:197 #15 0x00007f845d19d224 in virConnectDispose (obj=0x7f83f00e7ad0) at /tmp/buildd/libvirt-1.1.1/./src/datatypes.c:149 #16 0x00007f845d1246bb in virObjectUnref (anyobj=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/util/virobject.c:262 #17 0x00007f845d1a6e2f in virConnectClose (conn=0x7f83f00e7ad0) at /tmp/buildd/libvirt-1.1.1/./src/libvirt.c:1510 #18 0x00007f845db46581 in remoteClientFreeFunc (data=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./daemon/remote.c:683 #19 0x00007f845d20f362 in virNetServerClientDispose (obj=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/rpc/virnetserverclient.c:911 #20 0x00007f845d1246bb in virObjectUnref (anyobj=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/util/virobject.c:262 #21 0x00007f845d21781d in virNetSocketEventFree (opaque=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./src/rpc/virnetsocket.c:1714 #22 0x00007f845d108b29 in virEventPollCleanupHandles () at /tmp/buildd/libvirt-1.1.1/./src/util/vireventpoll.c:580 #23 0x00007f845d1096e3 in virEventPollRunOnce () at /tmp/buildd/libvirt-1.1.1/./src/util/vireventpoll.c:616 #24 0x00007f845d1086ad in virEventRunDefaultImpl () at /tmp/buildd/libvirt-1.1.1/./src/util/virevent.c:273 #25 0x00007f845d20ecfd in virNetServerRun (srv=0x7f845e818510) at /tmp/buildd/libvirt-1.1.1/./src/rpc/virnetserver.c:1096 #26 0x00007f845db2445e in main (argc=<optimized out>, argv=<optimized out>) at /tmp/buildd/libvirt-1.1.1/./daemon/libvirtd.c BUG() at line 185 in nl_object_put(); reference count going negative: 171 /** 172 * Release a reference from an object 173 * @arg obj object to release reference from 174 */ 175 void nl_object_put(struct nl_object *obj) 176 { 177 if (!obj) 178 return; 179 180 obj->ce_refcnt--; 181 NL_DBG(4, "Returned object reference %p, %d remaining\n", 182 obj, obj->ce_refcnt); 183 184 if (obj->ce_refcnt < 0) 185 BUG(); 186 187 if (obj->ce_refcnt <= 0) 188 nl_object_free(obj); 189 } I hope this is sufficient information.
-- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to netcf in Ubuntu. https://bugs.launchpad.net/bugs/1277157 Title: libvirtd restart due to assertion failure in libnl To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/netcf/+bug/1277157/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs