** Description changed:
SRU Justification:
- [Impact]
- *
+ [Impact]
+ * When starting and stopping large amounts of domains, libvirtd restarts
occasionally due to assertion failure in libnl. This is due to netcf providing
caches in netlink_init (via libnl) which are not thread safe.
[Test Case]
- *
+ * Define a large number of domains (32) that have 4 vCPUS, 8 GB memory. In a
loop start these domains, allow them to boot, and destroy them in parallel.
- [Regression Potential]
- *
+ [Regression Potential]
+ * This is an upstream patch and this code is already present in saucy and
beyond.
+
--
Running a multiple-domain start/destroy loop, libvirtd 1.1.1 restarts
occasionally due to assertion failure in libnl.
This affects both precise/quantal versions of netcf.
Distibution version: Ubuntu 12.04.2 LTS
Kernel: 3.5.0-41-generic #64 SMP Mon Dec 9 20:35:04 UTC 2013 x86_64
Libvirt: libvirt_1.1.1-0ubuntu8.2~cloud1 from cloud repository.
Qemu-kvm: qemu-kvm_1.0+noroms-0ubuntu14.12 [qemu 1.5 not yet tested]
libnl: libnl-3-200_3.2.3-2ubuntu2
Platform: HP ProLiant SL390s G7 x86_64; 2 socket x 6 cores/socket x 2
HT/core; 96GB
How to reproduce:
1. Define a number of test domains. E.g., test-nn, nn from 01 .. NN. I
use 32
Test domains have 4 vcpu, 8G memory, running an Ubuntu 11.04 image.
Network is libvirt default virtual network. Domain mac addresses based
on test number nn and dhcp serves fixed IP addresses based on mac
address.
2. In a loop:
2a. start the NN domains serially -- waiting for the "virsh start" command to
complete before starting next domain.
2b. sleep 20sec -- give domains some time to boot. Test doesn't check that
domains have completed booting.
2c. destroy the NN domains "in parallel" -- with "virsh destroy test-$nn &"
2d. sleep 15secs.
sleep times are more or less arbitrary. Next pass of starts does
usually begin before previous pass of destroys completes, but I've never
seen it say that "domain <name> already running" or such.
What I expected to happen: start-destroy loop runs indefinitely without
error.
What happened instead: Eventually, I start seeing errors like:
error: Failed to destroy domain lnvtest-31
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
and
error: failed to get domain 'lnvtest-12'
error: Domain not found: no domain with matching name 'lnvtest-12'
Checking the libvirt debug log, I see that libvirtd restarted during
this iteration, dumping its internal log buffer in the process.
gdb traceback shows:
#3 0x00007f845cafb192 in __GI___assert_fail (assertion=0x7f845c4b89bd "0",
file=0x7f845c4b8538 "/build/buildd/libnl3-3.2.3/./lib/object.c",
line=185, function=0x7f845c4b8668 "nl_object_put") at assert.c:103
#4 0x00007f845c4b4dea in nl_object_put () from /lib/libnl-3.so.200
#5 0x00007f845c4afb92 in nl_cache_remove () from /lib/libnl-3.so.200
#6 0x00007f845c4b4b07 in nl_object_free () from /lib/libnl-3.so.200
#7 0x00007f845c4b4b15 in nl_object_free () from /lib/libnl-3.so.200
#8 0x00007f845c4afb92 in nl_cache_remove () from /lib/libnl-3.so.200
#9 0x00007f845c4afd0b in nl_cache_clear () from /lib/libnl-3.so.200
#10 0x00007f845c4afd3e in nl_cache_free () from /lib/libnl-3.so.200
#11 0x00007f84517f3096 in ?? () from /usr/lib/libnetcf.so.1
#12 0x00007f84517f4220 in ?? () from /usr/lib/libnetcf.so.1
#13 0x00007f84517ef53f in ncf_close () from /usr/lib/libnetcf.so.1
#14 0x00007f8451a0be6f in netcfInterfaceClose (conn=0x7f83f00e7ad0) at
/tmp/buildd/libvirt-1.1.1/./src/interface/interface_backend_netcf.c:197
#15 0x00007f845d19d224 in virConnectDispose (obj=0x7f83f00e7ad0) at
/tmp/buildd/libvirt-1.1.1/./src/datatypes.c:149
#16 0x00007f845d1246bb in virObjectUnref (anyobj=<optimized out>) at
/tmp/buildd/libvirt-1.1.1/./src/util/virobject.c:262
#17 0x00007f845d1a6e2f in virConnectClose (conn=0x7f83f00e7ad0) at
/tmp/buildd/libvirt-1.1.1/./src/libvirt.c:1510
#18 0x00007f845db46581 in remoteClientFreeFunc (data=<optimized out>) at
/tmp/buildd/libvirt-1.1.1/./daemon/remote.c:683
#19 0x00007f845d20f362 in virNetServerClientDispose (obj=<optimized out>) at
/tmp/buildd/libvirt-1.1.1/./src/rpc/virnetserverclient.c:911
#20 0x00007f845d1246bb in virObjectUnref (anyobj=<optimized out>) at
/tmp/buildd/libvirt-1.1.1/./src/util/virobject.c:262
#21 0x00007f845d21781d in virNetSocketEventFree (opaque=<optimized out>) at
/tmp/buildd/libvirt-1.1.1/./src/rpc/virnetsocket.c:1714
#22 0x00007f845d108b29 in virEventPollCleanupHandles () at
/tmp/buildd/libvirt-1.1.1/./src/util/vireventpoll.c:580
#23 0x00007f845d1096e3 in virEventPollRunOnce () at
/tmp/buildd/libvirt-1.1.1/./src/util/vireventpoll.c:616
#24 0x00007f845d1086ad in virEventRunDefaultImpl () at
/tmp/buildd/libvirt-1.1.1/./src/util/virevent.c:273
#25 0x00007f845d20ecfd in virNetServerRun (srv=0x7f845e818510) at
/tmp/buildd/libvirt-1.1.1/./src/rpc/virnetserver.c:1096
#26 0x00007f845db2445e in main (argc=<optimized out>, argv=<optimized out>)
at /tmp/buildd/libvirt-1.1.1/./daemon/libvirtd.c
BUG() at line 185 in nl_object_put(); reference count going negative:
171 /**
172 * Release a reference from an object
173 * @arg obj object to release reference from
174 */
175 void nl_object_put(struct nl_object *obj)
176 {
177 if (!obj)
178 return;
179
180 obj->ce_refcnt--;
181 NL_DBG(4, "Returned object reference %p, %d remaining\n",
182 obj, obj->ce_refcnt);
183
184 if (obj->ce_refcnt < 0)
185 BUG();
186
187 if (obj->ce_refcnt <= 0)
188 nl_object_free(obj);
189 }
I hope this is sufficient information.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1277157
Title:
libvirtd restart due to assertion failure in libnl
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/netcf/+bug/1277157/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
