[Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? Obviously this was all made worse by the transient problems we've had with the tests suite infrastructure these past 2 days, but regardless it seems like we have a gap in our merge approval procedures here. IMHO, either SmokeStack needs to be made compulsory, or Jenkins needs to ensure tests are run on suitable hosts like SmokeStack does, or both. Regards, Daniel [1] -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: > Today we face a situation where Nova GIT master fails to pass all > the libvirt test cases. This regression was accidentally introduced > by the following changeset > >https://review.openstack.org/#/c/8778/ > > If you look at the history of that, the first SmokeStack test run > fails with some (presumably) transient errors, and added negative > karma to the change against patchset 2. If it were not for this > transient failure, it should have shown the regression in the > libvirt test case. The libvirt test case in question was one that > is skipped, unless libvirt is actually present on the host running > the tests. SmokeStack had made sure the tests would run on such a > host. > > There were then further patchsets uploaded, and patchset 4 was > approved for merge. Jenkins ran its gate jobs and these all passed > successfully. I am told that Jenkins will actually run the unittests > that are included in Nova, so I would have expected it to see the > flawed libvirt test case, but it didn't. I presume therefore, that > Jenkins is not running on a libvirt enabled host. Kind of - it's sadly more complex than that ... > The end result was that the broken changeset was merged to master, > which in turns means any other developers submitting changes > touching the libvirt area will get broken tests reported that > were not actually their own fault. > > This leaves me with the following questions... > > 1. Why was the recorded failure from SmokeStack not considered > to be a blocker for the merge of the commit by Gerrit or > Jenkins or any of the reviewers ? > > 2. Why did SmokeStack not get re-triggered for the later patch > set revisions, before it was merged ? The answer to 1 and 2 is largely the same - SmokeStack is a community contributed resources and is not managed by the CI team. Dan Prince does a great job with it, but it's not a resource that we have the ability to fix should it start messing up, so we have not granted it the permissions to file blocking votes. The tests that smokestack runs could all be written such that they are run by jenkins. The repos that run the jenkins tests are all in git and managed by openstack's gerrit. If there are testing profiles that it runs that we as a community value and want to see part of the gate, anyone is welcome to port them. > 3. Why did Jenkins not ensure that the tests were run on a libvirt > enabled host ? This is a different, and slightly more complex. We run tests in virtualenvs so that the process used to test the code can be consistently duplicated by all of the developers in the project. This is the reason that we no longer do ubuntu package creation as part of the gate - turns out that's really hard for a developer running on OSX to do locally on their laptop - and if Jenkins reports an blocking error in a patch, we want a developer to be able to reproduce the problem locally so that they can have a chance at fixing it. Problem arise in paradise though. libvirt being one of them. It's not possible to install libvirt into a virtualenv, because it's a swig-based module built as part of the libvirt source itself. One of the solutions to this is to allow the testing virtual environments to use packages installed at the system level. We suggested this a little while ago, but this was rejected by the nova team who valued the benefit of having a restricted test run so that we know we've got all of the depends properly specified. To that end, after chatting with Brian Waldon, I put this up as a possible next try: https://review.openstack.org/#/c/8949/ Which adds an additional testing environment that has system software enabled and also installs additional "optional" things. With that environment, we should be able to run a jenkins gate that tests things with full libvirt, and also tests the mysql upgrade paths, without screwing our fine friends who run OSX. Fundamentally though - we're at a point of trying to have our cake and eat it too. Either we want comprehensive testing of all of the unit tests, or we want to be careful about not making the test environment to hard for a developer to exactly mimic. I'm obviously on the side of having us have gating tests that some devs might not be able to do on their laptops - such as running the libvirt tests properly. We're working on cloud software - worst case scenario if there's an intractable problem, as dev can always spin up an ubuntu image somewhere. > Obviously this was all made worse by the transient problems we've had > with the tests suite infrastructure these past 2 days, but regardless > it seems like we have a gap in our merge approval procedures here. > > IMHO, either SmokeStack needs to be made compulsory, or Jenkins needs > to ensure tests are run on suitable hosts like SmokeStack does, or > both. The second is much more possible and as I've pointed out is in work - but I do think we should develop a clea
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
On Thu, Jun 28, 2012 at 08:13:28AM -0700, Monty Taylor wrote: > On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: > > This leaves me with the following questions... > > > > 1. Why was the recorded failure from SmokeStack not considered > > to be a blocker for the merge of the commit by Gerrit or > > Jenkins or any of the reviewers ? > > > > 2. Why did SmokeStack not get re-triggered for the later patch > > set revisions, before it was merged ? > > The answer to 1 and 2 is largely the same - SmokeStack is a community > contributed resources and is not managed by the CI team. Dan Prince does > a great job with it, but it's not a resource that we have the ability to > fix should it start messing up, so we have not granted it the > permissions to file blocking votes. > > The tests that smokestack runs could all be written such that they are > run by jenkins. The repos that run the jenkins tests are all in git and > managed by openstack's gerrit. If there are testing profiles that it > runs that we as a community value and want to see part of the gate, > anyone is welcome to port them. Ok, this makes sense to me now. I had assumed SmokeStack was a core part of the infrastructure. > > 3. Why did Jenkins not ensure that the tests were run on a libvirt > > enabled host ? > > This is a different, and slightly more complex. We run tests in > virtualenvs so that the process used to test the code can be > consistently duplicated by all of the developers in the project. This is > the reason that we no longer do ubuntu package creation as part of the > gate - turns out that's really hard for a developer running on OSX to do > locally on their laptop - and if Jenkins reports an blocking error in a > patch, we want a developer to be able to reproduce the problem locally > so that they can have a chance at fixing it. > > Problem arise in paradise though. libvirt being one of them. It's not > possible to install libvirt into a virtualenv, because it's a swig-based > module built as part of the libvirt source itself. One of the solutions > to this is to allow the testing virtual environments to use packages > installed at the system level. We suggested this a little while ago, but > this was rejected by the nova team who valued the benefit of having a > restricted test run so that we know we've got all of the depends > properly specified. > > To that end, after chatting with Brian Waldon, I put this up as a > possible next try: > > https://review.openstack.org/#/c/8949/ > > Which adds an additional testing environment that has system software > enabled and also installs additional "optional" things. With that > environment, we should be able to run a jenkins gate that tests things > with full libvirt, and also tests the mysql upgrade paths, without > screwing our fine friends who run OSX. > > Fundamentally though - we're at a point of trying to have our cake and > eat it too. Either we want comprehensive testing of all of the unit > tests, or we want to be careful about not making the test environment to > hard for a developer to exactly mimic. > > I'm obviously on the side of having us have gating tests that some devs > might not be able to do on their laptops - such as running the libvirt > tests properly. We're working on cloud software - worst case scenario if > there's an intractable problem, as dev can always spin up an ubuntu > image somewhere. I think I agree with you, since in practice I believe the additional requirements on developers are not unreasonable in general. Taking the livirt example (though it applies to other examples like MySQL too)... If a developer is submitting changes that touches a part of OpenStack unrelated to the virt drivers, then there's low liklihood that they'll cause libvirt test failures. This means they won't need to have libvirt available themselves in the common case, and thus there's no new onerous requirement placed on them. A similar scenario occurs if they are touching non-libvirt drivers (eg XenAPI, VMWare). Only if they touch the libvirt driver itself, or things that it is derived /from/ (eg the base virt driver code), then are they likely to need to run the tests locally to toubleshoot. In such a case, it is not unreasonable that they be prepared to setup local installs for troubleshooting purposes. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
On 06/28/2012 11:13 AM, Monty Taylor wrote: > Fundamentally though - we're at a point of trying to have our cake and eat it too. Either we want comprehensive testing of all of the unit tests, or we want to be careful about not making the test environment to hard for a developer to exactly mimic. I'm obviously on the side of having us have gating tests that some devs might not be able to do on their laptops - such as running the libvirt tests properly. We're working on cloud software - worst case scenario if there's an intractable problem, as dev can always spin up an ubuntu image somewhere. I'm definitely in agreement here. Something as fundamental as libvirt is to openstack really needs to see testing in the main test environment. I really feel like this is almost the same conversation we had on IRC on Friday about the mysql requirement. There is a deeper level of testing that we can do if we start requiring more of the base OS in the jenkins environment. That takes us out of a place where all the unit tests that run in jenkins can be run in any python environment, but that's ok. More validation here on the gate is worth that. Maybe we need something that's different than the current @skip semantics, some @skip_unless_gate (or similar), because a big piece of this is also confusion about what the gate is actually testing, and what it's skipping. As the person involved in the patch that slipped through, I was as surprised as anyone else that it landed, but had an issue in a real libvirt test case. -Sean -- Sean Dague IBM Linux Technology Center email: sda...@linux.vnet.ibm.com alt-email: slda...@us.ibm.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
On Jun 28, 2012, at 8:13 AM, Monty Taylor wrote: > Which adds an additional testing environment that has system software > enabled and also installs additional "optional" things. With that > environment, we should be able to run a jenkins gate that tests things > with full libvirt, and also tests the mysql upgrade paths, without > screwing our fine friends who run OSX. Just fyi, libvirt is installable on OSX with brew and the tests currently run and pass. Vish___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
On 06/28/2012 12:05 PM, Vishvananda Ishaya wrote: > > On Jun 28, 2012, at 8:13 AM, Monty Taylor wrote: > >> Which adds an additional testing environment that has system software >> enabled and also installs additional "optional" things. With that >> environment, we should be able to run a jenkins gate that tests things >> with full libvirt, and also tests the mysql upgrade paths, without >> screwing our fine friends who run OSX. > > Just fyi, libvirt is installable on OSX with brew and the tests > currently run and pass. Great! As a follow up - the "full" env got merged today. I'll add jenkins jobs to start running it. If you want to give it a spin locally, try: tox -v -efull And it should both install MySQL-python and set up the virtualenv with system-site-packages turned to true, so libvirt tests should run and not skip. In theory. In practice, it's virtualenv, so all bets are off. ;) Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
- Original Message - > From: "Monty Taylor" > To: openstack@lists.launchpad.net > Sent: Thursday, June 28, 2012 11:13:28 AM > Subject: Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers > > > > On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: > > Today we face a situation where Nova GIT master fails to pass all > > the libvirt test cases. This regression was accidentally introduced > > by the following changeset > > > >https://review.openstack.org/#/c/8778/ > > > > If you look at the history of that, the first SmokeStack test run > > fails with some (presumably) transient errors, and added negative > > karma to the change against patchset 2. If it were not for this > > transient failure, it should have shown the regression in the > > libvirt test case. The libvirt test case in question was one that > > is skipped, unless libvirt is actually present on the host running > > the tests. SmokeStack had made sure the tests would run on such a > > host. > > > > There were then further patchsets uploaded, and patchset 4 was > > approved for merge. Jenkins ran its gate jobs and these all passed > > successfully. I am told that Jenkins will actually run the > > unittests > > that are included in Nova, so I would have expected it to see the > > flawed libvirt test case, but it didn't. I presume therefore, that > > Jenkins is not running on a libvirt enabled host. > > Kind of - it's sadly more complex than that ... > > > The end result was that the broken changeset was merged to master, > > which in turns means any other developers submitting changes > > touching the libvirt area will get broken tests reported that > > were not actually their own fault. > > > > This leaves me with the following questions... > > > > 1. Why was the recorded failure from SmokeStack not considered > > to be a blocker for the merge of the commit by Gerrit or > > Jenkins or any of the reviewers ? > > > > 2. Why did SmokeStack not get re-triggered for the later patch > > set revisions, before it was merged ? > > The answer to 1 and 2 is largely the same - SmokeStack is a community > contributed resources and is not managed by the CI team. Dan Prince > does > a great job with it, but it's not a resource that we have the ability > to > fix should it start messing up, so we have not granted it the > permissions to file blocking votes. I would add that if anyone else is interested in collaborating on making SmokeStack better I'm more than happy to give access. Its all open source and has been since Cactus. As is now SmokeStack can can cast a -1 vote and hopefully this is proving to be useful. I'm open to suggestions. > > The tests that smokestack runs could all be written such that they > are > run by jenkins. I actually put in quite a bit of work to maintain an openstack_vpc job on Jenkins post-Cactus. When we talked about gating on this job at the Diablo conference the idea didn't seem to get very far... I kind of saw that as the end of the line for maintaining an openstack_vpc job and eventually it went away. Not sure who deleted it, but anyway. The way I see it there is value in both testing systems. Rather than complaining about why one system exists and/or doesn't port its tests to the other why don't we build on each others strengths. Seeing a green "verified +1" from both Jenkins and SmokeStack on a review should be very encouraging... and if one of the two systems fails it might require further investigation. > The repos that run the jenkins tests are all in git > and > managed by openstack's gerrit. If there are testing profiles that it > runs that we as a community value and want to see part of the gate, > anyone is welcome to port them. > > > 3. Why did Jenkins not ensure that the tests were run on a libvirt > > enabled host ? > > This is a different, and slightly more complex. We run tests in > virtualenvs so that the process used to test the code can be > consistently duplicated by all of the developers in the project. This > is > the reason that we no longer do ubuntu package creation as part of > the > gate - turns out that's really hard for a developer running on OSX to > do > locally on their laptop - and if Jenkins reports an blocking error in > a > patch, we want a developer to be able to reproduce the problem > locally > so that they can have a chance at fixing it. The ability for developers to test things locally is very important. For that matter SmokeStack all started with a project called openstack_vpc, a project
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
On 06/28/2012 01:49 PM, Dan Prince wrote: > > > - Original Message - >> From: "Monty Taylor" To: >> openstack@lists.launchpad.net Sent: Thursday, June 28, 2012 >> 11:13:28 AM Subject: Re: [Openstack] Jenkins vs SmokeStack tests & >> Gerrit merge blockers >> >> >> >> On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: >>> Today we face a situation where Nova GIT master fails to pass >>> all the libvirt test cases. This regression was accidentally >>> introduced by the following changeset >>> >>> https://review.openstack.org/#/c/8778/ >>> >>> If you look at the history of that, the first SmokeStack test >>> run fails with some (presumably) transient errors, and added >>> negative karma to the change against patchset 2. If it were not >>> for this transient failure, it should have shown the regression >>> in the libvirt test case. The libvirt test case in question was >>> one that is skipped, unless libvirt is actually present on the >>> host running the tests. SmokeStack had made sure the tests would >>> run on such a host. >>> >>> There were then further patchsets uploaded, and patchset 4 was >>> approved for merge. Jenkins ran its gate jobs and these all >>> passed successfully. I am told that Jenkins will actually run >>> the unittests that are included in Nova, so I would have expected >>> it to see the flawed libvirt test case, but it didn't. I presume >>> therefore, that Jenkins is not running on a libvirt enabled >>> host. >> >> Kind of - it's sadly more complex than that ... >> >>> The end result was that the broken changeset was merged to >>> master, which in turns means any other developers submitting >>> changes touching the libvirt area will get broken tests reported >>> that were not actually their own fault. >>> >>> This leaves me with the following questions... >>> >>> 1. Why was the recorded failure from SmokeStack not considered to >>> be a blocker for the merge of the commit by Gerrit or Jenkins or >>> any of the reviewers ? >>> >>> 2. Why did SmokeStack not get re-triggered for the later patch >>> set revisions, before it was merged ? >> >> The answer to 1 and 2 is largely the same - SmokeStack is a >> community contributed resources and is not managed by the CI team. >> Dan Prince does a great job with it, but it's not a resource that >> we have the ability to fix should it start messing up, so we have >> not granted it the permissions to file blocking votes. > > I would add that if anyone else is interested in collaborating on > making SmokeStack better I'm more than happy to give access. Its all > open source and has been since Cactus. > > As is now SmokeStack can can cast a -1 vote and hopefully this is > proving to be useful. I'm open to suggestions. I think it's stellar! >> >> The tests that smokestack runs could all be written such that they >> are run by jenkins. > > I actually put in quite a bit of work to maintain an openstack_vpc > job on Jenkins post-Cactus. When we talked about gating on this job > at the Diablo conference the idea didn't seem to get very far... I > kind of saw that as the end of the line for maintaining an > openstack_vpc job and eventually it went away. Not sure who deleted > it, but anyway. > > The way I see it there is value in both testing systems. Rather than > complaining about why one system exists and/or doesn't port its tests > to the other why don't we build on each others strengths. Seeing > a green "verified +1" from both Jenkins and SmokeStack on a review > should be very encouraging... and if one of the two systems fails it > might require further investigation. I completely agree with that. I'm still hoping we'll see more systems from more people so that the set of combinations get larger. I think also there's clearly value in running tests, like how SmokeStack is doing right now, that aren't necessarily part of the gate, but which pro-actively provide useful information to the reviewers. >> The repos that run the jenkins tests are all in git and managed by >> openstack's gerrit. If there are testing profiles that it runs that >> we as a community value and want to see part of the gate, anyone is >> welcome to port them. >> >>> 3. Why did Jenkins not ensure that the tests were run on a >>> libvirt enabled host ? >> >> This is a different, and slight
Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers
On 06/28/2012 11:58 PM, Monty Taylor wrote: On 06/28/2012 01:49 PM, Dan Prince wrote: - Original Message - From: "Monty Taylor" To: openstack@lists.launchpad.net Sent: Thursday, June 28, 2012 11:13:28 AM Subject: Re: [Openstack] Jenkins vs SmokeStack tests & Gerrit merge blockers On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. Kind of - it's sadly more complex than that ... The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? The answer to 1 and 2 is largely the same - SmokeStack is a community contributed resources and is not managed by the CI team. Dan Prince does a great job with it, but it's not a resource that we have the ability to fix should it start messing up, so we have not granted it the permissions to file blocking votes. I would add that if anyone else is interested in collaborating on making SmokeStack better I'm more than happy to give access. Its all open source and has been since Cactus. As is now SmokeStack can can cast a -1 vote and hopefully this is proving to be useful. I'm open to suggestions. I think it's stellar! The tests that smokestack runs could all be written such that they are run by jenkins. I actually put in quite a bit of work to maintain an openstack_vpc job on Jenkins post-Cactus. When we talked about gating on this job at the Diablo conference the idea didn't seem to get very far... I kind of saw that as the end of the line for maintaining an openstack_vpc job and eventually it went away. Not sure who deleted it, but anyway. The way I see it there is value in both testing systems. Rather than complaining about why one system exists and/or doesn't port its tests to the other why don't we build on each others strengths. Seeing a green "verified +1" from both Jenkins and SmokeStack on a review should be very encouraging... and if one of the two systems fails it might require further investigation. I completely agree with that. I'm still hoping we'll see more systems from more people so that the set of combinations get larger. I think also there's clearly value in running tests, like how SmokeStack is doing right now, that aren't necessarily part of the gate, but which pro-actively provide useful information to the reviewers. The repos that run the jenkins tests are all in git and managed by openstack's gerrit. If there are testing profiles that it runs that we as a community value and want to see part of the gate, anyone is welcome to port them. 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? This is a different, and slightly more complex. We run tests in virtualenvs so that the process used to test the code can be consistently duplicated by all of the developers in the project. This is the reason that we no longer do ubuntu package creation as part of the gate - turns out that's really hard for a developer running on OSX to do locally on their laptop - and if Jenkins reports an blocking error in a patch, we want a developer to be able to reproduce the problem locally so that they can have a chance at fixing it. The ability for developers to test things locally is very important. For that matter SmokeStack all started with a project called openstack_vpc, a project to spin up groups of cloud servers installed with the latest OpenStack code. A developer can use a project like openstack_vpc to spin up a se
