Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On Thu, Jun 20, 2013 at 7:31 AM, Dan Smith wrote: > Or, you can run dash.py, which will show you a personalized view of > zuul: > > https://github.com/kk7ds/openstack-gerrit-dashboard > > Running it with a refresh of thirty seconds or so will give you status > of your patches as they make their way through the queues, as well as > early failure warning (in the form of a red line): > > This is really nice! Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
> You can always got out to zuul directly and see patch status in > progress > - http://status.openstack.org/zuul/ Or, you can run dash.py, which will show you a personalized view of zuul: https://github.com/kk7ds/openstack-gerrit-dashboard Running it with a refresh of thirty seconds or so will give you status of your patches as they make their way through the queues, as well as early failure warning (in the form of a red line): http://img196.imageshack.us/img196/881/c9lo.png I run this all the time at the bottom of one of my monitors, and I'm happier as a result :) --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
Good idea. I have the same feeling. +1 Best Regards. -- Shane Martina Kollarova wrote on 2013-06-19: > Jenkins keeps running all the tests, even if the basic pep8 test fails, > and runs all of the (very slow) Tempest Quantum tests, even though > almost all of them are failing. > > I propose that it should fail and stop all of the other tests once there > is a failure in a voting test. For non-voting tests, it should stop only > itself, not the others. > > This would decrease the feedback loop and we wouldn't have to wait for > the non-voting Quantum tests to see that they failed as always. > > The nosetest tool has a parameter '--stop' which will cause it to exit > on first failure. > > Martina ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 06/18/2013 03:14 PM, David Ripton wrote: > On 06/18/2013 12:43 PM, Martina Kollarova wrote: >> Jenkins keeps running all the tests, even if the basic pep8 test fails, >> and runs all of the (very slow) Tempest Quantum tests, even though >> almost all of them are failing. >> >> I propose that it should fail and stop all of the other tests once there >> is a failure in a voting test. For non-voting tests, it should stop only >> itself, not the others. >> >> This would decrease the feedback loop and we wouldn't have to wait for >> the non-voting Quantum tests to see that they failed as always. > > -1 > > In addition to the other objections, we currently get a lot of false > positives (fail, retry, fail, retry, succeed), and it would be harder to > debug these problems if the output was truncated differently each time. > > Is anyone working on fixing the perma-failing Quantum test? When the > Postgres test was perma-failing, one of the infrastructure folks gave us > an ultimatum that if nobody fixed it soon, it would be disabled. (Happy > ending: Mauro fixed it before it got disabled.) That was brought up a little while ago, but we had already spent so much effort to get it working in the first place, none of us had the heart to put in such an ultimatum. But seriously- it might be time for an all-hands-on-deck dogpile to figure out what's up the the quantum gate. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 06/18/2013 12:43 PM, Martina Kollarova wrote: Jenkins keeps running all the tests, even if the basic pep8 test fails, and runs all of the (very slow) Tempest Quantum tests, even though almost all of them are failing. I propose that it should fail and stop all of the other tests once there is a failure in a voting test. For non-voting tests, it should stop only itself, not the others. This would decrease the feedback loop and we wouldn't have to wait for the non-voting Quantum tests to see that they failed as always. -1 In addition to the other objections, we currently get a lot of false positives (fail, retry, fail, retry, succeed), and it would be harder to debug these problems if the output was truncated differently each time. Is anyone working on fixing the perma-failing Quantum test? When the Postgres test was perma-failing, one of the infrastructure folks gave us an ultimatum that if nobody fixed it soon, it would be disabled. (Happy ending: Mauro fixed it before it got disabled.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 06/18/2013 01:17 PM, Jeremy Stanley wrote: > On 2013-06-18 18:43:58 +0200 (+0200), Martina Kollarova wrote: > [...] >> I propose that it should fail and stop all of the other tests once >> there is a failure in a voting test. For non-voting tests, it should >> stop only itself, not the others. > [...] > > We did this for a while (ran pep8 first in the check pipeline, then > ran other jobs if that passed), but it turned out to have a rather > confusing interaction with the requirements validation jobs. > > The long and short of it is that there are roughly two classes of > jobs we want to consider... quick ones which are not very resource > intensive, and more involved tests which are unlikely to succeed > anyway if the quick ones fail. To that end there has been discussion > of adding a "fast-fail" option for certain jobs causing them to > abort others already running and return the result set right away so > that developers can benefit from a more immediate testing feedback > loop. > > I don't think we have a timeline for implementation there, but it's > my recollection that's the direction we resolved to take it. Yeah. What Jeremy said. Actually, fast-fail is one of the reasons that we've been pushing so hard to get people on to testr and thus have the option for the tests to emit subunit output streams. Our current thinking is that we will teach zuul to read subunit streams as they happen, so that fast-fail can report back overall failure and also re-order the gate queue as needed because we know a job will fail - but we can still continue running the jobs. Note, that since this involves us reading a streaming protocol, it doesn't depend on a job failing first. We should be able to report failure for the set of jobs on the first instance of a single test case failing. As of right now, we have not begun development on this feature. I believe we'd be more than happy to point someone in the right direction if they wanted to work on it before we can get to it. Monty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On Tue, Jun 18, 2013 at 9:49 AM, Sean Dague wrote: > On 06/18/2013 12:43 PM, Martina Kollarova wrote: >> >> Jenkins keeps running all the tests, even if the basic pep8 test fails, >> and runs all of the (very slow) Tempest Quantum tests, even though >> almost all of them are failing. >> >> I propose that it should fail and stop all of the other tests once there >> is a failure in a voting test. For non-voting tests, it should stop only >> itself, not the others. >> >> This would decrease the feedback loop and we wouldn't have to wait for >> the non-voting Quantum tests to see that they failed as always. >> >> The nosetest tool has a parameter '--stop' which will cause it to exit >> on first failure. >> We tried something similar where we required all projects to pass pep8 before we ran the other tests (not as flexible as what you describe above but it does shorten the feedback loop when there are syntax/format issues). The problem we ran into there is the one Jaypipes describes where you fix one thing and now there is another thing to fix then another and so on. It ends up being advantageous to get as much information as possible rather than as little as possible. I do think Jim Blair would like to add more generic support for what you describe in Zuul, but that hasn't happened yet. I will let him elaborate. > > You can always got out to zuul directly and see patch status in progress - > http://status.openstack.org/zuul/ This should be helpful. If you want to be proactive you can see failures here before they are reported back to Gerrit. Finally, please run `tox` locally before you push commits to Gerrit. This will catch pep8/pyflakes/flake8 problems (as configured by the project) and run the unit tests which gives you control of that feedback loop before Jenkins tests the change. I think these suggestions come out of tempest development where running `tox` isn't going to be as lightweight and quick, but in general it should help quite a bit. Clark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 2013-06-18 18:43:58 +0200 (+0200), Martina Kollarova wrote: [...] > I propose that it should fail and stop all of the other tests once > there is a failure in a voting test. For non-voting tests, it should > stop only itself, not the others. [...] We did this for a while (ran pep8 first in the check pipeline, then ran other jobs if that passed), but it turned out to have a rather confusing interaction with the requirements validation jobs. The long and short of it is that there are roughly two classes of jobs we want to consider... quick ones which are not very resource intensive, and more involved tests which are unlikely to succeed anyway if the quick ones fail. To that end there has been discussion of adding a "fast-fail" option for certain jobs causing them to abort others already running and return the result set right away so that developers can benefit from a more immediate testing feedback loop. I don't think we have a timeline for implementation there, but it's my recollection that's the direction we resolved to take it. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 06/18/2013 12:43 PM, Martina Kollarova wrote: Jenkins keeps running all the tests, even if the basic pep8 test fails, and runs all of the (very slow) Tempest Quantum tests, even though almost all of them are failing. I propose that it should fail and stop all of the other tests once there is a failure in a voting test. For non-voting tests, it should stop only itself, not the others. This would decrease the feedback loop and we wouldn't have to wait for the non-voting Quantum tests to see that they failed as always. The nosetest tool has a parameter '--stop' which will cause it to exit on first failure. Martina You can always got out to zuul directly and see patch status in progress - http://status.openstack.org/zuul/ -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 06/18/2013 12:43 PM, Martina Kollarova wrote: Jenkins keeps running all the tests, even if the basic pep8 test fails, and runs all of the (very slow) Tempest Quantum tests, even though almost all of them are failing. I propose that it should fail and stop all of the other tests once there is a failure in a voting test. For non-voting tests, it should stop only itself, not the others. This would decrease the feedback loop and we wouldn't have to wait for the non-voting Quantum tests to see that they failed as always. The nosetest tool has a parameter '--stop' which will cause it to exit on first failure. ++ I support this on the check pipeline. It may certainly reduce the time that folks are waiting on checks... The downside, of course, is that the developer may fix only the issue that causes one failure, and then when pushing that next fix, they hit another different failure, and so on. But overall, I think it's probably a good idea. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 06/18/2013 12:49 PM, Jay Pipes wrote: On 06/18/2013 12:43 PM, Martina Kollarova wrote: Jenkins keeps running all the tests, even if the basic pep8 test fails, and runs all of the (very slow) Tempest Quantum tests, even though almost all of them are failing. I propose that it should fail and stop all of the other tests once there is a failure in a voting test. For non-voting tests, it should stop only itself, not the others. This would decrease the feedback loop and we wouldn't have to wait for the non-voting Quantum tests to see that they failed as always. The nosetest tool has a parameter '--stop' which will cause it to exit on first failure. ++ I support this on the check pipeline. It may certainly reduce the time that folks are waiting on checks... First, by the time we get to H2, nose will probably not be running tempest in the gate, so any nose specific options are off the table. Secondly, getting all the results on a fail is typically really useful. Given that this is a non interactive environment, the only way you have to get to the bottom of the issue is to look at the logs. If we don't run some of the tests because we stop early, that really reduces your ability to debug the issue. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Gerrit's Jenkins should stop running tests after first failure
Jenkins keeps running all the tests, even if the basic pep8 test fails, and runs all of the (very slow) Tempest Quantum tests, even though almost all of them are failing. I propose that it should fail and stop all of the other tests once there is a failure in a voting test. For non-voting tests, it should stop only itself, not the others. This would decrease the feedback loop and we wouldn't have to wait for the non-voting Quantum tests to see that they failed as always. The nosetest tool has a parameter '--stop' which will cause it to exit on first failure. Martina -- irc: mkollaro ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
