Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
Ihar Hrachyshka writes: > I just saw the plan merged in master: > https://review.openstack.org/#/c/451492/ That's cool! Can we also > backport the change to Ocata and maybe Newton so that we don't hit the > same bug there? The backport is already up: > https://review.openstack.org/#/c/456506/ For Ocata this makes sense, but note that the Newton UCA doesn't include the libvirt-bin package. -- Simon. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
I just saw the plan merged in master: https://review.openstack.org/#/c/451492/ That's cool! Can we also backport the change to Ocata and maybe Newton so that we don't hit the same bug there? The backport is already up: https://review.openstack.org/#/c/456506/ Thanks, Ihar On Mon, Apr 3, 2017 at 4:06 PM, Clark Boylan wrote: > Hello, > > One of the major sets of issues currently affecting gate testing is > Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us > and they happen frequently [0][1][2]. These issues appear to only affect > Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in > #openstack-nova it is clear that Libvirt isn't interested in debugging > such an old version of Libvirt (1.3.1). And while it isn't entirely > clear to me which exact version would be acceptable to them the Ubuntu > Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). > > I have pushed a change to devstack [3] to enable using UCA which pulls > in new Libvirt and mostly seems to work. I think we should consider > switching to UCA as this may fix our Libvirt problems and if it doesn't, > we will be closer to a version of Libvirt that upstream should be > willing to fix. > > This isn't the most straightfoward switch as UCA has a different repo > for each OpenStack release. libvirt-python is sensitve to the underlying > library changing; it is backward compatible but libvirt-python built > against older libvirt won't work against new libvirt. The result is a > libvirt-python wheel built on our wheel mirror does not work with UCA. > On the positive side both the OpenStack puppet modules and OpenStack > Ansible are using UCA with their deployment tooling so this should get > us closer to what people are using in the wild. > > After some thought I think my preferred method of rolling this out would > be to blacklist libvirt-python from our wheel mirror building entirely > and force installs to happen from source so that we are base Xenial and > $UCA_version independent (local testing of these builds show it only > takes a few seconds). Then have specific jobs (like devstack) explicitly > opt into the UCA repo appropriate for them (if any). This last bit is > from feedback from OpenStack Ansible that having the base images be > fairly clean is desirable, but it would also be hard to know which > version of UCA is appropriate for our Xenial images (this likely differs > based on the job). > > Now its entirely possible that newer Libvirt will be worse than current > (old) Libvirt; however, being closer to upstream should make getting > fixes easier. Would be great if those with a better understanding of > Libvirt could chime in on this if I am completely wrong here. > > Finally it is worth noting that we will get newer packages of other > software as well, most notably openvswitch will be version 2.6.1 instead > of 2.5.0. > > Any other thoughts or ideas? > > [0] http://status.openstack.org/elastic-recheck/#1646779 > [1] http://status.openstack.org/elastic-recheck/#1643911 > [2] http://status.openstack.org/elastic-recheck/#1638982 > [3] https://review.openstack.org/451492 > > Thank you, > Clark > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
Hi Clark On Tue, 4 Apr 2017 at 00:08 Clark Boylan wrote: > One of the major sets of issues currently affecting gate testing is > Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us > and they happen frequently [0][1][2]. These issues appear to only affect > Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in > #openstack-nova it is clear that Libvirt isn't interested in debugging > such an old version of Libvirt (1.3.1). And while it isn't entirely > clear to me which exact version would be acceptable to them the Ubuntu > Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). > tl;dr - Christian (cpaelzer) is working towards resolution of the libvirt 1.3.1 issues in Xenial, but right now we're stuck on reproduction of the issues outside of the gate. > I have pushed a change to devstack [3] to enable using UCA which pulls > in new Libvirt and mostly seems to work. I think we should consider > switching to UCA as this may fix our Libvirt problems and if it doesn't, > we will be closer to a version of Libvirt that upstream should be > willing to fix. > > This isn't the most straightfoward switch as UCA has a different repo > for each OpenStack release. libvirt-python is sensitve to the underlying > library changing; it is backward compatible but libvirt-python built > against older libvirt won't work against new libvirt. The result is a > libvirt-python wheel built on our wheel mirror does not work with UCA. > On the positive side both the OpenStack puppet modules and OpenStack > Ansible are using UCA with their deployment tooling so this should get > us closer to what people are using in the wild. > +1 on taking this approach - this also aligned with what we deploy and test with for the OpenStack Charms. Please stick to using the updates pockets from the Ubuntu Cloud Archive (which I can see in the review referenced that you are doing); all UCA testing of packaging and version updates is done in the proposed pockets so this should ensure a better level of stability for the OpenStack gate. [...] > Finally it is worth noting that we will get newer packages of other > software as well, most notably openvswitch will be version 2.6.1 instead > of 2.5.0. Worth noting that in the same way that we update OpenStack packages in the UCA for new minor and patch releases, we also do the same for Open vSwitch patch releases - so the Ocata UCA will get released Open vSwitch versions on the 2.6.x series. The Pike UCA will (probably) get a newer version of Ceph which might be of interest for gate testing as well. Cheers James __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On Tue, 4 Apr 2017 at 14:38 Daniel P. Berrange wrote: > On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote: > > Hello, > > > > One of the major sets of issues currently affecting gate testing is > > Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us > > and they happen frequently [0][1][2]. These issues appear to only affect > > Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in > > #openstack-nova it is clear that Libvirt isn't interested in debugging > > such an old version of Libvirt (1.3.1). And while it isn't entirely > > clear to me which exact version would be acceptable to them the Ubuntu > > Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). > > If going to the libvirt upstream community for help, we'd generally want > to see the latest upstream release being used. Ideally along with > willingness > to test git master if investigating a troublesome issue, but we understand > using git master is not practical for many people. > > If using an old version provided by an OS distro, then we would generally > expect the OS distro maintainers to lead the investigation, and take the > responsibility for reproducing on latest upstream. Upstream libvirt simply > doesn't have bandwidth to do the OS distro maintainers job for them when > using old distro versions. > Agree on the responsibility for maintenance of libvirt in Ubuntu. Christian Ehrhardt (cpaelzer) has been working to help resolve the libvirt 1.3.1 bugs currently impacting the gate and does have updated packages available for testing, but right now we're not able to reproduce the bugs outside of the gate environment so verifying that these actually resolve the underlying issues is proving problematic. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On 04/04/2017 12:14 PM, Daniel P. Berrange wrote: On Tue, Apr 04, 2017 at 09:21:27AM -0700, Clark Boylan wrote: On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote: On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote: I have pushed a change to devstack [3] to enable using UCA which pulls in new Libvirt and mostly seems to work. I think we should consider switching to UCA as this may fix our Libvirt problems and if it doesn't, we will be closer to a version of Libvirt that upstream should be willing to fix. This isn't the most straightfoward switch as UCA has a different repo for each OpenStack release. libvirt-python is sensitve to the underlying library changing; it is backward compatible but libvirt-python built against older libvirt won't work against new libvirt. The result is a libvirt-python wheel built on our wheel mirror does not work with UCA. I'm surprised about that - could you elaborate on what's broken for you. The libvirt.so provides a stable public API, and the standalone python binding only uses public APIs from libvirt.so. IOW you should be able to build libvirt-python against 1.3.0 and then use it against 2.5.0 with no problems. NB, *before* libvirt-python lived on pypi, it used some non-public libvirt.so symbols, so was tied to the exact libvirt.so it was built against. Assuming you're using the pypi version this should no longer apply. The specific issue was "AttributeError: 'module' object has no attribute 'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and traceback at [0]). The libvirt-python module here was built against Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then when running against Libvirt 2.5.0 Nova seems to have detected that newer features should be present that are not reflected in the compiled libvirt-python resulting in the error. This crashed nova compute. Problem was easily corrected by preventing devstack from using our wheel mirror for libvirt-python which resulted in a new installation built against Libvirt 2.5.0. It seems like the API is stable enough for backward compatibility but not forward compatibility. Its also possible that Nova is doing version checking in a buggy way and should be checking what the libvirt-python version is and what it supports? Ok, so yeah your last sentance is the correct interpretation. You've built libvirt-python against libvirt v1.3.1, so it only includes support for constants & methods that exist in that version. The VIR_MIGRATE_POSTCOPY constant was introduced in v1.3.3, so it will not be included in the libvirt-python you built. When checking features Nova calls a libvirt API that returns the version of the libvirtd daemon, which is v2.5.0, and then just blindly assumes libvirt-python has the same version. Unfortunately there is no way for Nova to determine what libvirt version the python binding was built against, so it can't improve its version check in this respect. To deal with this, Nova would two options: - Provide a nova.conf parameter to force it to assume an older libvirt version, thus disabling the features regardless of what libvirtd supports - Make nova check for existance of the python constants / APIs it is trying to use, in addition to checking the libvirt version The first option is pretty trivial to do if needed. The second option would be the more correct approach, but a much bigger maint burden, so I'm not convinced it is worth it. I'm not convinced either are worth it - I think if we just stop building libvirt wheels in our wheel mirror it'll resolve itself. Most places that are actually running OpenStack for real aren't going to be building their install packages blind to what version of the OS they're going to install on - so it's really only a problem with one of our gate optimizations. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On 04/04/2017 12:05 PM, Ihar Hrachyshka wrote: I mentioned that in a meeting today but for greater visibility... This change in how we gate against LTS raises a question whether what we claim in: https://review.openstack.org/#/c/402940/4/reference/project-testing-interface.rst is still valid. The wording of the document is such that suggests we test against LTS, and with the UCA adoption in gate, that will no longer be true. I am not against the change per se, I just want the decision to be thought through and advertised to consumers beyond tight circle of upstream developers. Totally - and that's a good point. I think we may just need to update the wording some. The intent has always been that we develop targetting the software we think is appropriate for OpenStack- but the LTS line and testing are to ensure we don't introduce a dependency that's unreasonably hard to backport on top of an LTS release. An example of that would be growing a dependency on a kernel feature that would be too much to backport - or if we had decided to move to Python3 back in the RHEL6 days (before there was a python3 available) RDO and UCA are both ways that the distro vendors themselves are providing backport software on top of their LTS releases. So if it's in one of those the distro vendor themselves have de-facto communicated that it's not unreasonable to backport that piece of software to run on top of the LTS - since they have already done it. All of that is not spelled out clearly though - so I agree, it would be potentially useful to describe that situation more clearly in the PTI. Ihar On Mon, Apr 3, 2017 at 4:39 PM, Clark Boylan wrote: On Mon, Apr 3, 2017, at 04:33 PM, Ihar Hrachyshka wrote: Are we going to keep some job not using the archive, to make sure we don't break people using packages from LTS? Maybe just periodic/non-voting would be enough to keep it working on older packages. Old stable branches will continue to run against the base LTS release. But considering that this is how users of Ubuntu get newer packages, the deployment projects are using it for newer branches, and we know that the base LTS is broken, I'm not sure how much benefit there would be to keep testing on the base LTS if we make the switch. Its important to note that anyone using base LTS is broken and we know this because of our testing. Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On Tue, Apr 04, 2017 at 09:21:27AM -0700, Clark Boylan wrote: > On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote: > > On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote: > > > I have pushed a change to devstack [3] to enable using UCA which pulls > > > in new Libvirt and mostly seems to work. I think we should consider > > > switching to UCA as this may fix our Libvirt problems and if it doesn't, > > > we will be closer to a version of Libvirt that upstream should be > > > willing to fix. > > > > > > This isn't the most straightfoward switch as UCA has a different repo > > > for each OpenStack release. libvirt-python is sensitve to the underlying > > > library changing; it is backward compatible but libvirt-python built > > > against older libvirt won't work against new libvirt. The result is a > > > libvirt-python wheel built on our wheel mirror does not work with UCA. > > > > I'm surprised about that - could you elaborate on what's broken for you. > > The libvirt.so provides a stable public API, and the standalone python > > binding only uses public APIs from libvirt.so. IOW you should be able > > to build libvirt-python against 1.3.0 and then use it against 2.5.0 with > > no problems. > > > > NB, *before* libvirt-python lived on pypi, it used some non-public > > libvirt.so symbols, so was tied to the exact libvirt.so it was built > > against. Assuming you're using the pypi version this should no longer > > apply. > > The specific issue was "AttributeError: 'module' object has no attribute > 'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and > traceback at [0]). The libvirt-python module here was built against > Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then > when running against Libvirt 2.5.0 Nova seems to have detected that > newer features should be present that are not reflected in the compiled > libvirt-python resulting in the error. This crashed nova compute. > > Problem was easily corrected by preventing devstack from using our wheel > mirror for libvirt-python which resulted in a new installation built > against Libvirt 2.5.0. > > It seems like the API is stable enough for backward compatibility but > not forward compatibility. Its also possible that Nova is doing version > checking in a buggy way and should be checking what the libvirt-python > version is and what it supports? Ok, so yeah your last sentance is the correct interpretation. You've built libvirt-python against libvirt v1.3.1, so it only includes support for constants & methods that exist in that version. The VIR_MIGRATE_POSTCOPY constant was introduced in v1.3.3, so it will not be included in the libvirt-python you built. When checking features Nova calls a libvirt API that returns the version of the libvirtd daemon, which is v2.5.0, and then just blindly assumes libvirt-python has the same version. Unfortunately there is no way for Nova to determine what libvirt version the python binding was built against, so it can't improve its version check in this respect. To deal with this, Nova would two options: - Provide a nova.conf parameter to force it to assume an older libvirt version, thus disabling the features regardless of what libvirtd supports - Make nova check for existance of the python constants / APIs it is trying to use, in addition to checking the libvirt version The first option is pretty trivial to do if needed. The second option would be the more correct approach, but a much bigger maint burden, so I'm not convinced it is worth it. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o-http://search.cpan.org/~danberr/ :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
I mentioned that in a meeting today but for greater visibility... This change in how we gate against LTS raises a question whether what we claim in: https://review.openstack.org/#/c/402940/4/reference/project-testing-interface.rst is still valid. The wording of the document is such that suggests we test against LTS, and with the UCA adoption in gate, that will no longer be true. I am not against the change per se, I just want the decision to be thought through and advertised to consumers beyond tight circle of upstream developers. Ihar On Mon, Apr 3, 2017 at 4:39 PM, Clark Boylan wrote: > On Mon, Apr 3, 2017, at 04:33 PM, Ihar Hrachyshka wrote: >> Are we going to keep some job not using the archive, to make sure we >> don't >> break people using packages from LTS? Maybe just periodic/non-voting >> would >> be enough to keep it working on older packages. > > Old stable branches will continue to run against the base LTS release. > But considering that this is how users of Ubuntu get newer packages, the > deployment projects are using it for newer branches, and we know that > the base LTS is broken, I'm not sure how much benefit there would be to > keep testing on the base LTS if we make the switch. > > Its important to note that anyone using base LTS is broken and we know > this because of our testing. > > Clark > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote: > On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote: > > I have pushed a change to devstack [3] to enable using UCA which pulls > > in new Libvirt and mostly seems to work. I think we should consider > > switching to UCA as this may fix our Libvirt problems and if it doesn't, > > we will be closer to a version of Libvirt that upstream should be > > willing to fix. > > > > This isn't the most straightfoward switch as UCA has a different repo > > for each OpenStack release. libvirt-python is sensitve to the underlying > > library changing; it is backward compatible but libvirt-python built > > against older libvirt won't work against new libvirt. The result is a > > libvirt-python wheel built on our wheel mirror does not work with UCA. > > I'm surprised about that - could you elaborate on what's broken for you. > The libvirt.so provides a stable public API, and the standalone python > binding only uses public APIs from libvirt.so. IOW you should be able > to build libvirt-python against 1.3.0 and then use it against 2.5.0 with > no problems. > > NB, *before* libvirt-python lived on pypi, it used some non-public > libvirt.so symbols, so was tied to the exact libvirt.so it was built > against. Assuming you're using the pypi version this should no longer > apply. The specific issue was "AttributeError: 'module' object has no attribute 'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and traceback at [0]). The libvirt-python module here was built against Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then when running against Libvirt 2.5.0 Nova seems to have detected that newer features should be present that are not reflected in the compiled libvirt-python resulting in the error. This crashed nova compute. Problem was easily corrected by preventing devstack from using our wheel mirror for libvirt-python which resulted in a new installation built against Libvirt 2.5.0. It seems like the API is stable enough for backward compatibility but not forward compatibility. Its also possible that Nova is doing version checking in a buggy way and should be checking what the libvirt-python version is and what it supports? [0] http://logs.openstack.org/92/451492/4/check/gate-devstack-dsvm-updown-ubuntu-xenial/19dca66/logs/screen-n-cpu.txt.gz?level=ERROR#_2017-03-29_22_55_19_038 Hope this helps, Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On 4/4/17, 12:06 AM, "Clark Boylan" wrote: > After some thought I think my preferred method of rolling this out would > be to blacklist libvirt-python from our wheel mirror building entirely > and force installs to happen from source so that we are base Xenial and > $UCA_version independent (local testing of these builds show it only > takes a few seconds). Then have specific jobs (like devstack) explicitly > opt into the UCA repo appropriate for them (if any). This last bit is > from feedback from OpenStack Ansible that having the base images be > fairly clean is desirable, but it would also be hard to know which > version of UCA is appropriate for our Xenial images (this likely differs > based on the job). An approach that could be taken here is to bake the repo config (and install packages from that repo) for the UCA version that’s the earliest version we support for that OS. For example – the earliest version that supports Xenial is Newton, so pre-configure and install packages into the image from UCA Xenial/Newton. Then in any jobs which care about which version of UCA is used the repo config can be changed and the packages updated. This potentially saves a little gate time. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com and delete the original message. Your cooperation is appreciated. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On Tue, Apr 04, 2017 at 07:16:51AM -0400, Sean Dague wrote: > This is definitely a trade off, I know the last time we tried UCA we had > such a high failure rate we had to revert. But, that was a much younger > libvirt that was only just starting to get heavy testing in OpenStack. > So it feels like it's worth a shot. It will at least be interesting to > see if it makes things better. > > The libvirt bump will bring in libvirtd and live migration postcopy for > testability on the Nova side, both of which would be good things. NB, you'd need corresponding QEMU bump too for post-copy, but IIUC the UCA contain that, so it'd be fine. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o-http://search.cpan.org/~danberr/ :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote: > Hello, > > One of the major sets of issues currently affecting gate testing is > Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us > and they happen frequently [0][1][2]. These issues appear to only affect > Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in > #openstack-nova it is clear that Libvirt isn't interested in debugging > such an old version of Libvirt (1.3.1). And while it isn't entirely > clear to me which exact version would be acceptable to them the Ubuntu > Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). If going to the libvirt upstream community for help, we'd generally want to see the latest upstream release being used. Ideally along with willingness to test git master if investigating a troublesome issue, but we understand using git master is not practical for many people. If using an old version provided by an OS distro, then we would generally expect the OS distro maintainers to lead the investigation, and take the responsibility for reproducing on latest upstream. Upstream libvirt simply doesn't have bandwidth to do the OS distro maintainers job for them when using old distro versions. > I have pushed a change to devstack [3] to enable using UCA which pulls > in new Libvirt and mostly seems to work. I think we should consider > switching to UCA as this may fix our Libvirt problems and if it doesn't, > we will be closer to a version of Libvirt that upstream should be > willing to fix. > > This isn't the most straightfoward switch as UCA has a different repo > for each OpenStack release. libvirt-python is sensitve to the underlying > library changing; it is backward compatible but libvirt-python built > against older libvirt won't work against new libvirt. The result is a > libvirt-python wheel built on our wheel mirror does not work with UCA. I'm surprised about that - could you elaborate on what's broken for you. The libvirt.so provides a stable public API, and the standalone python binding only uses public APIs from libvirt.so. IOW you should be able to build libvirt-python against 1.3.0 and then use it against 2.5.0 with no problems. NB, *before* libvirt-python lived on pypi, it used some non-public libvirt.so symbols, so was tied to the exact libvirt.so it was built against. Assuming you're using the pypi version this should no longer apply. > Now its entirely possible that newer Libvirt will be worse than current > (old) Libvirt; however, being closer to upstream should make getting > fixes easier. Would be great if those with a better understanding of > Libvirt could chime in on this if I am completely wrong here. As a general rule your expectation is right - newer libvirt should generally be better. There is always the chance of screwups, but we issue maint releases where needed - the only question mark would be whether UCA pulls in any maint releases. I would like to think that if such a problem happened, openstack would be able to escalate it to a Canonical maintainer to get a maint release / patch into UCA, since presumably any such bug would be important to Canonical customers using OpenStack too. > Finally it is worth noting that we will get newer packages of other > software as well, most notably openvswitch will be version 2.6.1 instead > of 2.5.0. IIUC, you'd get newer QEMU/KVM too, which is arguably just as desirable as getting newer libvirt. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o-http://search.cpan.org/~danberr/ :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On 04/03/2017 06:06 PM, Clark Boylan wrote: Hello, One of the major sets of issues currently affecting gate testing is Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us and they happen frequently [0][1][2]. These issues appear to only affect Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in #openstack-nova it is clear that Libvirt isn't interested in debugging such an old version of Libvirt (1.3.1). And while it isn't entirely clear to me which exact version would be acceptable to them the Ubuntu Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). I have pushed a change to devstack [3] to enable using UCA which pulls in new Libvirt and mostly seems to work. I think we should consider switching to UCA as this may fix our Libvirt problems and if it doesn't, we will be closer to a version of Libvirt that upstream should be willing to fix. This isn't the most straightfoward switch as UCA has a different repo for each OpenStack release. libvirt-python is sensitve to the underlying library changing; it is backward compatible but libvirt-python built against older libvirt won't work against new libvirt. The result is a libvirt-python wheel built on our wheel mirror does not work with UCA. On the positive side both the OpenStack puppet modules and OpenStack Ansible are using UCA with their deployment tooling so this should get us closer to what people are using in the wild. After some thought I think my preferred method of rolling this out would be to blacklist libvirt-python from our wheel mirror building entirely and force installs to happen from source so that we are base Xenial and $UCA_version independent (local testing of these builds show it only takes a few seconds). Then have specific jobs (like devstack) explicitly opt into the UCA repo appropriate for them (if any). This last bit is from feedback from OpenStack Ansible that having the base images be fairly clean is desirable, but it would also be hard to know which version of UCA is appropriate for our Xenial images (this likely differs based on the job). Now its entirely possible that newer Libvirt will be worse than current (old) Libvirt; however, being closer to upstream should make getting fixes easier. Would be great if those with a better understanding of Libvirt could chime in on this if I am completely wrong here. Finally it is worth noting that we will get newer packages of other software as well, most notably openvswitch will be version 2.6.1 instead of 2.5.0. Any other thoughts or ideas? I think it's a great idea. I don't think there is actual value in trying to support ludicrously old libvirt. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
This is definitely a trade off, I know the last time we tried UCA we had such a high failure rate we had to revert. But, that was a much younger libvirt that was only just starting to get heavy testing in OpenStack. So it feels like it's worth a shot. It will at least be interesting to see if it makes things better. The libvirt bump will bring in libvirtd and live migration postcopy for testability on the Nova side, both of which would be good things. -Sean On 04/03/2017 07:06 PM, Clark Boylan wrote: > Hello, > > One of the major sets of issues currently affecting gate testing is > Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us > and they happen frequently [0][1][2]. These issues appear to only affect > Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in > #openstack-nova it is clear that Libvirt isn't interested in debugging > such an old version of Libvirt (1.3.1). And while it isn't entirely > clear to me which exact version would be acceptable to them the Ubuntu > Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). > > I have pushed a change to devstack [3] to enable using UCA which pulls > in new Libvirt and mostly seems to work. I think we should consider > switching to UCA as this may fix our Libvirt problems and if it doesn't, > we will be closer to a version of Libvirt that upstream should be > willing to fix. > > This isn't the most straightfoward switch as UCA has a different repo > for each OpenStack release. libvirt-python is sensitve to the underlying > library changing; it is backward compatible but libvirt-python built > against older libvirt won't work against new libvirt. The result is a > libvirt-python wheel built on our wheel mirror does not work with UCA. > On the positive side both the OpenStack puppet modules and OpenStack > Ansible are using UCA with their deployment tooling so this should get > us closer to what people are using in the wild. > > After some thought I think my preferred method of rolling this out would > be to blacklist libvirt-python from our wheel mirror building entirely > and force installs to happen from source so that we are base Xenial and > $UCA_version independent (local testing of these builds show it only > takes a few seconds). Then have specific jobs (like devstack) explicitly > opt into the UCA repo appropriate for them (if any). This last bit is > from feedback from OpenStack Ansible that having the base images be > fairly clean is desirable, but it would also be hard to know which > version of UCA is appropriate for our Xenial images (this likely differs > based on the job). > > Now its entirely possible that newer Libvirt will be worse than current > (old) Libvirt; however, being closer to upstream should make getting > fixes easier. Would be great if those with a better understanding of > Libvirt could chime in on this if I am completely wrong here. > > Finally it is worth noting that we will get newer packages of other > software as well, most notably openvswitch will be version 2.6.1 instead > of 2.5.0. > > Any other thoughts or ideas? > > [0] http://status.openstack.org/elastic-recheck/#1646779 > [1] http://status.openstack.org/elastic-recheck/#1643911 > [2] http://status.openstack.org/elastic-recheck/#1638982 > [3] https://review.openstack.org/451492 > > Thank you, > Clark > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On 04/04/2017 09:06 AM, Clark Boylan wrote: > I have pushed a change to devstack [3] to enable using UCA which pulls > in new Libvirt and mostly seems to work. I think we should consider > switching to UCA as this may fix our Libvirt problems and if it doesn't, > we will be closer to a version of Libvirt that upstream should be > willing to fix. I'm not 100% sure where this leaves the devstack-plugin-additional-pkg-repos work [1]; although to be honest I've always thought that was a little unspecific. We also have "devstack-plugin-tar-installer" [2] which was working on installing libvirt from upstream tarballs IIRC but looks dead? There was quite some discussion about this in [3] at the time. > Finally it is worth noting that we will get newer packages of other > software as well, most notably openvswitch will be version 2.6.1 instead > of 2.5.0. Just to write down the centos side of things -- we have the RDO repo installed on all centos CI nodes. That brings in the centos virt-sig rebuild of qemu (qemu-kvm-ev) which is definitely required to run trunk nova [4]. We are just using libvirt from standard updates (2.0.0 based). RDO provides openvswitch, which is 2.6.1 too. > Then have specific jobs (like devstack) explicitly opt into the UCA > repo appropriate for them (if any). The idea being, presumably, that when devstack branches, it is essentially pinned to whatever version of UCA it was using at the time for its lifespan. I do wonder if installing the latest version on the base images simplifies things (so that by default, "apt-get install libvirt" in any context gets the UCA version). To handle the above, a devstack branch would have to essentially do the opposite -- override the default to whatever version it is pinned to. More work at branch point, however has the advantage that we don't have to constantly communicate to people they need to "opt-in". -i [1] https://git.openstack.org/cgit/openstack/devstack-plugin-additional-pkg-repos [2] https://git.openstack.org/cgit/openstack/devstack-plugin-tar-installer [3] https://review.openstack.org/#/c/108714/ [4] http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
On Mon, Apr 3, 2017, at 04:33 PM, Ihar Hrachyshka wrote: > Are we going to keep some job not using the archive, to make sure we > don't > break people using packages from LTS? Maybe just periodic/non-voting > would > be enough to keep it working on older packages. Old stable branches will continue to run against the base LTS release. But considering that this is how users of Ubuntu get newer packages, the deployment projects are using it for newer branches, and we know that the base LTS is broken, I'm not sure how much benefit there would be to keep testing on the base LTS if we make the switch. Its important to note that anyone using base LTS is broken and we know this because of our testing. Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
Are we going to keep some job not using the archive, to make sure we don't break people using packages from LTS? Maybe just periodic/non-voting would be enough to keep it working on older packages. On Mon, Apr 3, 2017 at 4:07 PM Clark Boylan wrote: > Hello, > > One of the major sets of issues currently affecting gate testing is > Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us > and they happen frequently [0][1][2]. These issues appear to only affect > Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in > #openstack-nova it is clear that Libvirt isn't interested in debugging > such an old version of Libvirt (1.3.1). And while it isn't entirely > clear to me which exact version would be acceptable to them the Ubuntu > Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). > > I have pushed a change to devstack [3] to enable using UCA which pulls > in new Libvirt and mostly seems to work. I think we should consider > switching to UCA as this may fix our Libvirt problems and if it doesn't, > we will be closer to a version of Libvirt that upstream should be > willing to fix. > > This isn't the most straightfoward switch as UCA has a different repo > for each OpenStack release. libvirt-python is sensitve to the underlying > library changing; it is backward compatible but libvirt-python built > against older libvirt won't work against new libvirt. The result is a > libvirt-python wheel built on our wheel mirror does not work with UCA. > On the positive side both the OpenStack puppet modules and OpenStack > Ansible are using UCA with their deployment tooling so this should get > us closer to what people are using in the wild. > > After some thought I think my preferred method of rolling this out would > be to blacklist libvirt-python from our wheel mirror building entirely > and force installs to happen from source so that we are base Xenial and > $UCA_version independent (local testing of these builds show it only > takes a few seconds). Then have specific jobs (like devstack) explicitly > opt into the UCA repo appropriate for them (if any). This last bit is > from feedback from OpenStack Ansible that having the base images be > fairly clean is desirable, but it would also be hard to know which > version of UCA is appropriate for our Xenial images (this likely differs > based on the job). > > Now its entirely possible that newer Libvirt will be worse than current > (old) Libvirt; however, being closer to upstream should make getting > fixes easier. Would be great if those with a better understanding of > Libvirt could chime in on this if I am completely wrong here. > > Finally it is worth noting that we will get newer packages of other > software as well, most notably openvswitch will be version 2.6.1 instead > of 2.5.0. > > Any other thoughts or ideas? > > [0] http://status.openstack.org/elastic-recheck/#1646779 > [1] http://status.openstack.org/elastic-recheck/#1643911 > [2] http://status.openstack.org/elastic-recheck/#1638982 > [3] https://review.openstack.org/451492 > > Thank you, > Clark > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive
Hello, One of the major sets of issues currently affecting gate testing is Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us and they happen frequently [0][1][2]. These issues appear to only affect Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in #openstack-nova it is clear that Libvirt isn't interested in debugging such an old version of Libvirt (1.3.1). And while it isn't entirely clear to me which exact version would be acceptable to them the Ubuntu Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0). I have pushed a change to devstack [3] to enable using UCA which pulls in new Libvirt and mostly seems to work. I think we should consider switching to UCA as this may fix our Libvirt problems and if it doesn't, we will be closer to a version of Libvirt that upstream should be willing to fix. This isn't the most straightfoward switch as UCA has a different repo for each OpenStack release. libvirt-python is sensitve to the underlying library changing; it is backward compatible but libvirt-python built against older libvirt won't work against new libvirt. The result is a libvirt-python wheel built on our wheel mirror does not work with UCA. On the positive side both the OpenStack puppet modules and OpenStack Ansible are using UCA with their deployment tooling so this should get us closer to what people are using in the wild. After some thought I think my preferred method of rolling this out would be to blacklist libvirt-python from our wheel mirror building entirely and force installs to happen from source so that we are base Xenial and $UCA_version independent (local testing of these builds show it only takes a few seconds). Then have specific jobs (like devstack) explicitly opt into the UCA repo appropriate for them (if any). This last bit is from feedback from OpenStack Ansible that having the base images be fairly clean is desirable, but it would also be hard to know which version of UCA is appropriate for our Xenial images (this likely differs based on the job). Now its entirely possible that newer Libvirt will be worse than current (old) Libvirt; however, being closer to upstream should make getting fixes easier. Would be great if those with a better understanding of Libvirt could chime in on this if I am completely wrong here. Finally it is worth noting that we will get newer packages of other software as well, most notably openvswitch will be version 2.6.1 instead of 2.5.0. Any other thoughts or ideas? [0] http://status.openstack.org/elastic-recheck/#1646779 [1] http://status.openstack.org/elastic-recheck/#1643911 [2] http://status.openstack.org/elastic-recheck/#1638982 [3] https://review.openstack.org/451492 Thank you, Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev