[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
This bug was fixed in the package libvirt - 0.9.8-2ubuntu17.5 --- libvirt (0.9.8-2ubuntu17.5) precise-proposed; urgency=low * add patch Reduce-udevadm-settle-timeout-to-10-seconds.patch (copied from Debian tree) to fix 3 minute hang during pool-refresh when using LVM backed pools. (LP: #1027987) * add upstream patch command-avoid-double-close-bugs toi avoid a race when starting multiple VMs concurrently. (LP: #1055658) -- Serge HallynWed, 03 Oct 2012 11:48:31 -0500 ** Changed in: libvirt (Ubuntu Precise) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
I have tested the patch on a KVM cluster at my emplyer and based on a day or so of testing under heavy load, it fixes the bug for me. The cluster consists of 7 machines each running up to 8 VM jobs in parallel. Most are varous windows versions but some are Ubuntu Lucid. Each job runs for arround 2 minutes, before being killed and a fresh VM booted in it's place, so we have thousands of KVM invocations per day. Before we saw arround 5% of jobs would fail due to Libvirt errors. (mostly from get_domain_by_name from the perl bindings) This was reduced to 1.5% by making multiple retrys on the API call untill it worked. It now looks like the faliure rate is down to zero with the new version of libvirt. Thank you. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
Verified in precise proposed using the breaklibvirt.sh from bug:961217. breaklibvirt.sh completes 100 rounds of starting and stopping 4 instances without failures after updating to http://launchpad.net/ubuntu/+source/libvirt/0.9.8-2ubuntu17.5 ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
** Branch linked: lp:ubuntu/precise-proposed/libvirt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
Hello Adin, or anyone else affected, Accepted libvirt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libvirt/0.9.8-2ubuntu17.5 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Tags added: precise ** Changed in: libvirt (Ubuntu Precise) Status: Triaged => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
** Description changed: + == + SRU Justification + 1. Impact: when starting multiple VMs simultaneously, many may fail to start. + 2. Development fix: the attached patch was used upstream to fix the race + 3. Stable fix: use the same patch + 4. test case: start many patches imultaneously, make sure all started + 5. Regression potential: there should be none, patch was taken straight from +upstream. + == + Symptoms: When the system is under load and you attempt to start multiple VMs simultaneously via libvirt, many of the VMs fail to start. This happens for us running an OpenStack compute server when VMs are provisioned rapidly (for automated tests, scaling, etc.). We can easily reproduce it by starting 5-10 VMs on a single server simultaneously -- in which case, Libvirt will fail to start about half. Even in more modest scenarios, this is likely to occasionally affect OpenStack compute servers running on precise (i.e. when VMs are started within a few seconds of each other). Diagnosis: I investigated the source of the problem, and it seems that Libvirt has a few double close() problems that have been fixed since 0.9.8. Concurrent tasks in Libvirtd have a decent chance of stepping on each other's toes (incorrectly closing some file descriptor that has been reused for some other purpose). Some of these race conditions have a very small window of opportunity, but one problem seems to be more more common (particularly when the system is under load and a new qemu process might take a while to start). Solution: The problem has since been fixed in upstream Libvirt, and the commit message refers to the RedHat bug here (https://bugzilla.redhat.com/show_bug.cgi?id=823716) .. but I can't see it. I've attached the upstream patch. I've tested it and it fixes the problem and applies cleanly (with a few offsets). It is a small, low- risk patch. I'm submitting this bug because it would be great to have this fix in the LTS release. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
Thanks, Adin - there is already a libvirt package in precise-proposed awaiting verification. I will push this fix as soon as that package is promoted to -updates. ** Also affects: libvirt (Ubuntu Precise) Importance: Undecided Status: New ** Changed in: libvirt (Ubuntu) Status: New => Fix Released ** Changed in: libvirt (Ubuntu Precise) Status: New => Triaged ** Changed in: libvirt (Ubuntu Precise) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
Yep, this is 12.04. This patch is included in libvirt 0.9.13, so I believe 12.10 is already covered. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
Thanks for reporting this bug, and especially for finding the patch fixing the problem! I assume you are seeing this in 12.04 and not in 12.10? (Just making sure) If so I'll go ahead and mark this for SRU -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1055658] Re: Under load, libvirt fails to start VMs concurrently
** Patch added: "Patch (from libvirt upstream) to fix the problem." https://bugs.launchpad.net/bugs/1055658/+attachment/3338597/+files/da831afc.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1055658 Title: Under load, libvirt fails to start VMs concurrently To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1055658/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
