[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
This bug was fixed in the package libvirt - 1.2.8-0ubuntu11.3 --- libvirt (1.2.8-0ubuntu11.3) utopic-proposed; urgency=medium * apparmor libvirt-qemu template: allow reading charm-specific ceph config and allow reading under /tmp and /var/tmp (for SRU only) (LP: #1403648) * mutex cgmanager actions (Thanks to Don Bowman for finding the cause) (LP: #1397130) (LP: #1367702) -- Serge Hallyn serge.hal...@ubuntu.com Tue, 06 Jan 2015 10:40:17 -0600 ** Changed in: libvirt (Ubuntu Utopic) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
This bug was fixed in the package libvirt - 1.2.8-0ubuntu11.3 --- libvirt (1.2.8-0ubuntu11.3) utopic-proposed; urgency=medium * apparmor libvirt-qemu template: allow reading charm-specific ceph config and allow reading under /tmp and /var/tmp (for SRU only) (LP: #1403648) * mutex cgmanager actions (Thanks to Don Bowman for finding the cause) (LP: #1397130) (LP: #1367702) -- Serge Hallyn serge.hal...@ubuntu.com Tue, 06 Jan 2015 10:40:17 -0600 ** Changed in: libvirt (Ubuntu Utopic) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
tried 1.2.8-0ubuntu11.3. seems to resolve. ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Hello Don, or anyone else affected, Accepted libvirt into utopic-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libvirt/1.2.8-0ubuntu11.3 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: libvirt (Ubuntu Utopic) Status: Confirmed = Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
** No longer affects: libvirt (Ubuntu Trusty) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
** No longer affects: libvirt (Ubuntu Trusty) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Any change that this patch will make it into Utopic anytime soon? I will get this into utopic-proposed this week. Once accepted it should sit there for about a week before going into utopic-updates (if the fix is verified) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
I added the ppa: $ cat /etc/apt/sources.list.d/serge-hallyn-ubuntu-virt-utopic.list deb http://ppa.launchpad.net/serge-hallyn/virt/ubuntu utopic main whilst the sru process presumably continues. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Any change that this patch will make it into Utopic anytime soon? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: libvirt (Ubuntu Trusty) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: libvirt (Ubuntu Utopic) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
** Changed in: libvirt (Ubuntu) Status: Incomplete = In Progress ** Also affects: libvirt (Ubuntu Utopic) Importance: Undecided Status: New ** Also affects: libvirt (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: libvirt (Ubuntu Trusty) Importance: Undecided = High ** Changed in: libvirt (Ubuntu Utopic) Importance: Undecided = High -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
This bug was fixed in the package libvirt - 1.2.8-0ubuntu18 --- libvirt (1.2.8-0ubuntu18) vivid; urgency=medium * mutex cgmanager actions (Thanks to Don Bowman for finding the cause) (LP: #1397130) (LP: #1367702) -- Serge Hallyn serge.hal...@ubuntu.com Thu, 18 Dec 2014 13:28:03 -0600 ** Changed in: libvirt (Ubuntu) Status: In Progress = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Thanks Don, and thanks very much for nailing down the cause! I'll get a fix out for v today and start the SRU process. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
** Changed in: libvirt (Ubuntu) Status: Incomplete = In Progress ** Also affects: libvirt (Ubuntu Utopic) Importance: Undecided Status: New ** Also affects: libvirt (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: libvirt (Ubuntu Trusty) Importance: Undecided = High ** Changed in: libvirt (Ubuntu Utopic) Importance: Undecided = High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
** Description changed: + + Impact: libvirt hangs + Fix: mutex libvirt's access to cgmanager + Test case: see script in #14 + Regression potential: if this is done wrongly, it could cause a deadlock. No non-cgmanager codepaths are affected. + + reference bug 1367702. As per request, opening new ticket w/ instructions to reproduce. This is on 14.10 server, libvirt-bin 1.2.8-0ubuntu11.1 As per 1367702, this is not using LXC (which u used in your attempt). This is running bare-metal, no container, no hypervisor. Each VM below is started from OpenStack nova-compute (this node is a compute-only node). don@nubo-5:~$ sudo service cgmanager restart cgmanager stop/waiting cgmanager start/running, process 22588 don@nubo-5:~$ virsh list - IdName State + IdName State - 2 instance-15de running - 3 instance-15df running - 4 instance-15e0 running - 5 instance-15e1 running - 6 instance-15e2 running - 7 instance-15e3 running - 8 instance-15e4 running - 9 instance-15e5 running - 10instance-15e6 running - 11instance-15e7 running - 12instance-15e8 running - 13instance-15e9 running - 14instance-15ea running - 15instance-15eb running - 16instance-15ec running - 17instance-15ed running - 18instance-15ee running - 19instance-15ef running - 20instance-15f0 running - 21instance-15f1 running - 22instance-15f2 running - 23instance-15f3 running - 24instance-15f4 running - 25instance-15f5 running - 26instance-15f6 running - 27instance-15f7 running - 28instance-15f8 running - 29instance-15f9 running - 30instance-15fa running - 31instance-15fb running - 32instance-15fc running - 33instance-15fd running - 34instance-15fe running - 35instance-15ff running - 36instance-1600 running + 2 instance-15de running + 3 instance-15df running + 4 instance-15e0 running + 5 instance-15e1 running + 6 instance-15e2 running + 7 instance-15e3 running + 8 instance-15e4 running + 9 instance-15e5 running + 10instance-15e6 running + 11instance-15e7 running + 12instance-15e8 running + 13instance-15e9 running + 14instance-15ea running + 15instance-15eb running + 16instance-15ec running + 17instance-15ed running + 18instance-15ee running + 19instance-15ef running + 20instance-15f0 running + 21instance-15f1 running + 22instance-15f2 running + 23instance-15f3 running + 24instance-15f4 running + 25instance-15f5 running + 26instance-15f6 running + 27instance-15f7 running + 28instance-15f8 running + 29instance-15f9 running + 30instance-15fa running + 31instance-15fb running + 32instance-15fc running + 33instance-15fd running + 34instance-15fe running + 35instance-15ff running + 36instance-1600 running don@nubo-5:~$ sudo service libvirt-bin restart libvirt-bin stop/waiting libvirt-bin start/running, process 22751 don@nubo-5:~$ virsh list error: failed to connect to the hypervisor error: no valid connection error: Cannot recv data: Connection reset by peer If i then run libvirtd manually: root@nubo-5:~# libvirtd -v 2014-11-27 22:38:18.066+: 26422: info : libvirt version: 1.2.8, package: 1.2.8-0ubuntu11.1 2014-11-27 22:38:18.066+: 26422: info : virNetlinkEventServiceStart:521 : starting netlink event service with protocol 0 2014-11-27 22:38:18.066+: 26422: info : virNetlinkEventServiceStart:521 : starting netlink
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
This bug was fixed in the package libvirt - 1.2.8-0ubuntu18 --- libvirt (1.2.8-0ubuntu18) vivid; urgency=medium * mutex cgmanager actions (Thanks to Don Bowman for finding the cause) (LP: #1397130) (LP: #1367702) -- Serge Hallyn serge.hal...@ubuntu.com Thu, 18 Dec 2014 13:28:03 -0600 ** Changed in: libvirt (Ubuntu) Status: In Progress = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
i ran the ppa image on 10 blades for a day in my production. its cycled through ~10K instances. so far so good, it appears to have solved this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
so part of the issue is cgm_* is not thread safe, but is called from various
threads.
e.g. if u start libvirtd w/ several instances already running, it will call
cgm_dbus_connect from each.
there is a pair of global variable:
static NihDBusProxy *cgroup_manager = NULL;
bool cgm_running = false;
in src/util/cgmanager.c, and they cause this issue.
cgm_dbus_connect() overwrites cgroup_manager regardless of the state of
cgm_running (and cgm_running is set @ the end of it anyway).
The use of pthread_once is probably part of the solution. The below
patch partially corrects.
--- libvirt-1.2.8.orig/src/util/cgmanager.c
+++ libvirt-1.2.8/src/util/cgmanager.c
@@ -40,10 +40,12 @@
static NihDBusProxy *cgroup_manager = NULL;
bool cgm_running = false;
+static pthread_once_t cgmanager_is_init = PTHREAD_ONCE_INIT;
+
VIR_LOG_INIT(util.cgmanager);
#define CGMANAGER_DBUS_SOCK unix:path=/sys/fs/cgroup/cgmanager/sock
-bool cgm_dbus_connect(void)
+static bool _cgm_dbus_connect(void)
{
DBusError dbus_error;
DBusConnection *connection;
@@ -83,6 +85,11 @@ bool cgm_dbus_connect(void)
return true;
}
+bool cgm_dbus_connect(void)
+{
+ pthread_once(cgmanager_is_init, _cgm_dbus_connect);
+}
+
void cgm_dbus_disconnect(void)
{
if (cgroup_manager) {
--- libvirt-1.2.8.orig/src/util/cgmanager.h
+++ libvirt-1.2.8/src/util/cgmanager.h
@@ -30,6 +30,7 @@
#include nih/alloc.h
#include nih/error.h
#include nih/string.h
+#include pthread.h
extern bool cgm_running;
Its not clear to me the underlying libs are all thread safe (e.g.
https://bugs.launchpad.net/ubuntu/+source/libnih/+bug/1294200)
there would remain the problem of... what if cgmanager goes away. 1
thread will find it, does it reconnect and fix the others?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1397130
Title:
libvirt-bin crashes / refuses to restart if cgmanager is restarted
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
this patch to libvirt gets it going. ** Patch added: don-cgm.patch https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+attachment/4282322/+files/don-cgm.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Hi Don, indeed the underlying libs (mainly the libnih-dbus ones) are not thread-safe. We've had to serialize access to cgmanager-client in lxc for the same reason. The cgm_dbus_connect() happens once on every cgmanager action, not once at libvirt startup. So doing pthread_once() is not going to do the right thing here IIUC. Rather we should just take a pthread lock at connect and drop it at disconnect. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Thanks for the reproducer script. Note that I had to tweak it to sleep a second before the last virsh list, else it always failed bc libvirt simply hadn't finished starting up yet. I can reproduce the bug with stock utopic. With the libvirt package in ppa:~serge-hallyn/virt, i cannot. Could you please test that package as well? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
I just tried, it did not immediately reproduce, will let it run for a bit. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
The attachment don-cgm.patch seems to be a patch. If it isn't, please remove the patch flag from the attachment, remove the patch tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
the attached script will reproduce this on utopic. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
the attached script will reproduce this on utopic. I did it on a virtual
machine running under kvm, but it will also do it bare-metal.
#!/bin/bash
apt-get -qy install virtinst libvirt-bin
die() {
echo $*
exit 1
}
dir=$(mktemp -d /tmp/repro.XX)
echo rep files in $dir
cd $dir
chgrp libvirtd .
chmod 775 .
[ -f cirros-0.3.3-x86_64-disk.img ] || wget http://download.cirros-
cloud.net/0.3.3/cirros-0.3.3-x86_64-disk.img || die Error fetching
cirros image
ninstances=10
for i in $(seq 0 $ninstances)
do
cp -f cirros-0.3.3-x86_64-disk.img cirros-$i.qcow2
chgrp libvirtd cirros-$i.qcow2
chmod 775 cirros-$i.qcow2
done
for i in $(seq 0 $ninstances)
do
virt-install -n instance-$i -r 256 --cpu host --description instance-$i
--import --disk cirros-$i.qcow2 --os-type=linux --noautoconsole
done
service libvirt-bin restart
echo This command should return instantly with $ninstances in the list
virsh list
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1397130
Title:
libvirt-bin crashes / refuses to restart if cgmanager is restarted
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
script to reproduce (as attachment) ** Attachment added: script to reproduce https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+attachment/4281789/+files/repro -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
Thanks for reporting this bug. I can't seem to reproduce this on my utopic laptop. However, in your example you have 30 instances running. Do you still get this when you have no instances running when restart cgmanager and libvirtd? Can you show the following info both before and after the restart/crash: ps -ef | egrep -e '(cgmanager|cgproxy)' cat /proc/self/mountinfo tree /sys/fs/cgroup ** Changed in: libvirt (Ubuntu) Status: New = Incomplete ** Changed in: libvirt (Ubuntu) Importance: Undecided = High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
attached. it didn't happen the first time [w/ 0 instances running]. i then started 1, and virsh hung [after my earlier restart?] its now dead. this is a different blade, a 2x E5-2640 0 @ 2.50GHz ** Attachment added: ubuntu-virt.txt https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+attachment/4279216/+files/ubuntu-virt.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
463.169884] init: cgmanager main process (510) killed by TERM signal [ 468.885143] show_signal_msg: 1 callbacks suppressed [ 468.885149] libvirtd[44515]: segfault at 0 ip 7fb39575b90f sp 7fb38241b8e0 error 4 in libvirt.so.0.1002.8[7fb3956f+2b3000][ 469.049895] init: libvirt-bin main process (44423) killed by SEGV signal [ 469.049908] init: libvirt-bin main process ended, respawning [ 494.421272] libvirtd[44683]: segfault at 0 ip 7f65efd2e90f sp 7f65dc9ee8e0 error 4 in libvirt.so.0.1002.8[7f65efcc3000+2b3000] [ 494.614372] init: libvirt-bin main process (44584) killed by SEGV signal [ 494.614387] init: libvirt-bin main process ended, respawning [ 494.977478] libvirtd[2476]: segfault at 7f3b0012 ip 7f3b3f811ec7 sp 7f3b2daa1650 error 4 in libc-2.19.so[7f3b3f791000+1ba000] [ 495.125140] init: libvirt-bin main process (2414) killed by SEGV signal [ 495.125153] init: libvirt-bin main process ended, respawning [ 495.604876] init: libvirt-bin main process (2511) killed by ABRT signal [ 495.604888] init: libvirt-bin main process ended, respawning [ 495.946894] libvirtd[2744]: segfault at 0 ip 7f27818c590f sp 7f276fd888e0 error 4 in libvirt.so.0.1002.8[7f278185a000+2b3000] [ 495.946912] libvirtd[2749]: segfault at 7f270012 ip 7f27812f7ec7 sp 7f276d583210 error 4 in libc-2.19.so[7f2781277000+1ba000] [ 496.101145] init: libvirt-bin main process (2647) killed by SEGV signal [ 496.101160] init: libvirt-bin main process ended, respawning [ 496.403354] libvirtd[2911]: segfault at 10 ip 7f55485bd5b6 sp 7f55352ed850 error 4 in libdbus-1.so.3.8.7[7f5548598000+46000] [ 496.524614] init: libvirt-bin main process (2821) killed by SEGV signal [ 496.524628] init: libvirt-bin main process ended, respawning [ 496.993885] init: libvirt-bin main process (2938) killed by ABRT signal [ 496.993910] init: libvirt-bin main process ended, respawning [ 497.363533] libvirtd[3270]: segfault at 0 ip 7fddf3dd690f sp 7fddd3ffe8e0 error 4 in libvirt.so.0.1002.8[7fddf3d6b000+2b3000] [ 497.363551] libvirtd[3266]: segfault at 0 ip 7fddf3dd690f sp 7fdde22998e0 error 4 in libvirt.so.0.1002.8[7fddf3d6b000+2b3000] [ 497.504828] init: libvirt-bin main process (3097) killed by SEGV signal [ 497.504841] init: libvirt-bin main process ended, respawning [ 497.989323] init: libvirt-bin main process (3296) killed by ABRT signal [ 497.989339] init: libvirt-bin main process ended, respawning [ 498.477920] init: libvirt-bin main process (3464) killed by ABRT signal [ 498.477940] init: libvirt-bin main process ended, respawning [ 498.787286] libvirtd[3711]: segfault at 7fd7000a ip 7fd7666cc232 sp 7fd75515e500 error 4 in libc-2.19.so[7fd76664d000+1ba000] [ 498.953864] init: libvirt-bin main process (3624) killed by SEGV signal [ 498.953878] init: libvirt-bin main process ended, respawning [ 499.262920] libvirtd[3910]: segfault at 10 ip 7f34615465b6 sp 7f344affc850 error 4 in libdbus-1.so.3.8.7[7f3461521000+46000] [ 499.409498] init: libvirt-bin main process (3785) killed by SEGV signal [ 499.409522] init: libvirt-bin respawning too fast, stopped -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
the latest repro shows this stack trace if it helps. Linux nubo-9 3.16.0-28-generic #37-Ubuntu SMP Mon Dec 8 17:15:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux root@nubo-9:/var/crash# dpkg -l |grep cgm ii cgmanager 0.33-2 amd64Central cgroup manager daemon ii cgmanager-tests 0.33-2 all Central cgroup manager daemon (tests) ii libcgmanager-dev:amd64 0.33-2 amd64Central cgroup manager daemon (dev) ii libcgmanager0:amd64 0.33-2 amd64Central cgroup manager daemon (client library) root@nubo-9:/var/crash# dpkg -l|grep libvirt ii libvirt-bin 1.2.8-0ubuntu11.1 amd64programs for the libvirt library ii libvirt01.2.8-0ubuntu11.1 amd64library for interfacing with different virtualization systems hi nova-compute-libvirt1:2014.2-0ubuntu1 all OpenStack Compute - compute node libvirt support ii python-libvirt 1.2.8-0ubuntu2 amd64libvirt Python bindings # dpkg -l |grep dbus ii dbus1.8.8-1ubuntu2.1 amd64simple interprocess messaging system (daemon and utilities) ii dbus-1-dbg:amd641.8.8-1ubuntu2.1 amd64simple interprocess messaging system (debug symbols) ii dbus-x111.8.8-1ubuntu2.1 amd64simple interprocess messaging system (X11 deps) ii libdbus-1-3:amd64 1.8.8-1ubuntu2.1 amd64simple interprocess messaging system (library) ii libdbus-1-dev:amd64 1.8.8-1ubuntu2.1 amd64simple interprocess messaging system (development headers) ii libdbus-glib-1-2:amd64 0.102-1 amd64simple interprocess messaging system (GLib-based shared library) ii libnih-dbus-dev:amd64 1.0.3-4ubuntu26 amd64NIH D-Bus Bindings Library (development files) ii libnih-dbus1:amd64 1.0.3-4ubuntu26 amd64NIH D-Bus Bindings Library ii nih-dbus-tool 1.0.3-4ubuntu26 amd64NIH D-Bus Binding Tool ii python3-dbus1.2.0-2build2 amd64simple interprocess messaging system (Python 3 interface) 2014-12-12 05:27:01.331+: 35353: error : cgm_dbus_connect:76 : cgmanager: Error pinging manager: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. process 35331: The last reference on a connection was dropped without closing the connection. This is a bug in an application. See dbus_connection_unref() documentation for details. Most likely, the application was supposed to call dbus_connection_close(), since this is a private connection. 2014-12-12 05:27:01.333+: 35358: error : cgm_dbus_connect:76 : cgmanager: Error pinging manager: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. (null):dbus_error.c:69: Unhandled error from nih_dbus_error_raise: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. Program received signal SIGABRT, Aborted. [Switching to Thread 0x7f342dd40700 (LWP 35356)] 0x7f3440d2fd27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x7f3440d2fd27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x7f3440d31418 in __GI_abort () at abort.c:89 #2 0x7f343eaa1997 in ?? () from /lib/x86_64-linux-gnu/libnih.so.1 #3 0x7f343eaa1be1 in _nih_error_raise_error () from /lib/x86_64-linux-gnu/libnih.so.1 #4 0x7f343ecb46e5 in cgmanager_get_pid_cgroup_sync () from /lib/x86_64-linux-gnu/libcgmanager.so.0 #5 0x7f344134802b in ?? () from /usr/lib/libvirt.so.0 #6 0x7f344133fa3b in ?? () from /usr/lib/libvirt.so.0 #7 0x7f3441340c53 in ?? () from /usr/lib/libvirt.so.0 #8 0x7f3441340ec7 in virCgroupNewDetectMachine () from /usr/lib/libvirt.so.0 #9 0x7f3431a7dec5 in qemuConnectCgroup () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #10
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
another way to reproduce this is to start new instances quickly (e.g.
create back to back), and then restart libvirt-bin.
and sometimes, just start new instances quickly (w/o restart libvirt-
bin)
for example, for host X, if the sleep are commented out, it will
normally crash libvirt if this is run to try and rescue the old
instances. if the sleep is present it will normally not crash.
#!/bin/bash
host=$1
j=0
for i in $(nova list --host $host --all-tenants | awk '/Shutdown/ {print $2}')
do
j=$((j+1))
if [ $j -eq 5 ]
then
echo Wait a bit...
#sleep 30
j=0
fi
echo Rescue $i
nova reboot --hard $i
#sleep 5
done
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1397130
Title:
libvirt-bin crashes / refuses to restart if cgmanager is restarted
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
reproduced with requested steps attached ** Attachment removed: ubuntu-virt.txt https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+attachment/4279216/+files/ubuntu-virt.txt ** Attachment added: ubuntu-virt.txt https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+attachment/4279220/+files/ubuntu-virt.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
[Thread debugging using libthread_db enabled] Using host libthread_db library /lib/x86_64-linux-gnu/libthread_db.so.1. Core was generated by `libvirtd -v'. Program terminated with signal SIGABRT, Aborted. #0 0x7fa266277d27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x7fa266277d27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x7fa266279418 in __GI_abort () at abort.c:89 #2 0x7fa263fde309 in nih_free () from /lib/x86_64-linux-gnu/libnih.so.1 #3 0x7fa26688f862 in cgm_dbus_connect () at /build/buildd/libvirt-1.2.8/./src/util/cgmanager.c:78 #4 0x7fa266887d5c in virCgroupAvailable () at /build/buildd/libvirt-1.2.8/./src/util/vircgroup.c:135 #5 0x7fa256db5e85 in qemuConnectCgroup (driver=optimized out, vm=0x7fa25028d540) at /build/buildd/libvirt-1.2.8/./src/qemu/qemu_cgroup.c:776 #6 0x7fa256dca447 in qemuProcessReconnect (opaque=0x7d3e, opaque@entry=0x7fa25032de10) at /build/buildd/libvirt-1.2.8/./src/qemu/qemu_process.c:3322 #7 0x7fa2668d994e in virThreadHelper (data=optimized out) at /build/buildd/libvirt-1.2.8/./src/util/virthread.c:197 #8 0x7fa26660e0a5 in start_thread (arg=0x7fa245ffb700) at pthread_create.c:309 #9 0x7fa26633b84d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1397130] Re: libvirt-bin crashes / refuses to restart if cgmanager is restarted
(gdb) bt #0 0x7f9fd650ad27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x7f9fd650c418 in __GI_abort () at abort.c:89 #2 0x7f9fd448f538 in cgmanager_get_pid_cgroup_sync () from /lib/x86_64-linux-gnu/libcgmanager.so.0 #3 0x7f9fd6b2302b in cgm_controller_exists (controller=0x7f9fd6ca766b name=systemd) at /build/buildd/libvirt-1.2.8/./src/util/cgmanager.c:296 #4 0x7f9fd6b1aa3b in cg_get_cgroups (group=optimized out) at /build/buildd/libvirt-1.2.8/./src/util/vircgroup.c:374 #5 virCgroupDetectMounts (group=0x12f) at /build/buildd/libvirt-1.2.8/./src/util/vircgroup.c:412 #6 0x7f9fd6b1bc53 in virCgroupDetect (parent=optimized out, path=optimized out, controllers=optimized out, pid=optimized out, group=optimized out) at /build/buildd/libvirt-1.2.8/./src/util/vircgroup.c:729 #7 virCgroupNew (pid=303, path=0x7f9fd6cf4dd8 , parent=0x0, controllers=-1, group=0x7f9fbcff9700) at /build/buildd/libvirt-1.2.8/./src/util/vircgroup.c:1263 #8 0x7f9fd6b1bec7 in virCgroupNewDetectMachine (name=0x7f9fc0279100 instance-15f8, drivername=0x7f9fc70c8ffb qemu, pid=303, partition=0x7f9fc026c1a0 /machine, controllers=-1124100352, group=0x7f9fc027cb50) at /build/buildd/libvirt-1.2.8/./src/util/vircgroup.c:1782 #9 0x7f9fc7048ec5 in qemuConnectCgroup (driver=optimized out, vm=0x7f9fc027e080) at /build/buildd/libvirt-1.2.8/./src/qemu/qemu_cgroup.c:781 #10 0x7f9fc705d447 in qemuProcessReconnect (opaque=0x12f, opaque@entry=0x7f9fc031b890) at /build/buildd/libvirt-1.2.8/./src/qemu/qemu_process.c:3322 #11 0x7f9fd6b6c94e in virThreadHelper (data=optimized out) at /build/buildd/libvirt-1.2.8/./src/util/virthread.c:197 #12 0x7f9fd68a10a5 in start_thread (arg=0x7f9fbcff9700) at pthread_create.c:309 #13 0x7f9fd65ce84d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1397130 Title: libvirt-bin crashes / refuses to restart if cgmanager is restarted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1397130/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
