On Sun 08 Jun 2014 12:57:24 PM CEST, Nir Soffer wrote: > ----- Original Message ----- >> From: "David Caro" <dcaro...@redhat.com> >> To: "Nir Soffer" <nsof...@redhat.com> >> Cc: "Piotr Kliczewski" <piotr.kliczew...@gmail.com>, fsimo...@redhat.com, >> dc...@redhat.com, devel@ovirt.org, "Dan >> Kenigsberg" <dan...@redhat.com> >> Sent: Friday, June 6, 2014 5:16:52 PM >> Subject: Re: [ovirt-devel] local vdsm build fails >> >> On Fri 06 Jun 2014 03:53:41 PM CEST, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "Dan Kenigsberg" <dan...@redhat.com> >>>> To: "Piotr Kliczewski" <piotr.kliczew...@gmail.com>, fsimo...@redhat.com, >>>> nsof...@redhat.com, dc...@redhat.com >>>> Cc: devel@ovirt.org >>>> Sent: Friday, June 6, 2014 12:15:18 PM >>>> Subject: Re: [ovirt-devel] local vdsm build fails >>>> >>>> On Fri, Jun 06, 2014 at 09:19:11AM +0200, Piotr Kliczewski wrote: >>>>> All, >>>>> >>>>> I pulled the latest vdsm from master and noticed that build is failing. >>>>> >>>>> Here is the patch that causes the failuer: >>>>> >>>>> http://gerrit.ovirt.org/#/c/28226 >>>>> >>>>> and looking at jenkins comments I can see that jenkins was failing >>>>> with the same reason: >>>>> >>>>> http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1064/console >>> >>> Nir has already fix that as well. The storage tests were just fine, but >>> a post build script was running cp incorrectly. >>> >>> David pointed that we need a way to distinguish between test errors and >>> failures. >>> He suggested looking up strings in the test output - we should not go >>> there, unless >>> we want to "fix" this many more times in the future. >>> >>> I suggest to use the these rules: >>> >>> - SUCCESS - make check returns 0 >>> - FAILURE - make check returns 1 >>> - ERROR - anything else returned by make check or any other script. >>> >>> I think that make check does work like this, but it should be easy to >>> change. >>> >>> What do you think? >>> >>>> >>>> Thanks for your report. Nir has already fixed this in >>>> http://gerrit.ovirt.org/28426. >>>> >>>> It was introduced in http://gerrit.ovirt.org/#/c/28226/ but missed also >>>> because we have turned PYFLAKES off in unit test jobs. We must turn it on >>>> in >>>> at least one of the tests (or initiate a new jenkins job for `make >>>> check-local`). >>>> >>>> As a quick fix, David has re-enabled PYFLAKES in >>>> http://jenkins.ovirt.org/view/By%20Project/view/vdsm/job/vdsm_master_unit_tests/configure >>>> >>>> Regards, >>>> Dan. >>>> >> >> Perfect for me, but you should know that it will fail also when strange >> things occur, for example, out of memory, of disk space, slave >> disconnected, network error, etc. >> >> If you are willing to treat those (the most common infra failures) as >> devel failures, then no problem on my side, > > I'm not - this is why we should separate test failures from test errors. > >> but I don't want you to >> start ignoring test errors because it's most probably an infra error >> (don't get me wrong, it's totally normal to start ignoring an alarm >> that is not a real problem, as infra members we will try to minimize >> the infra issues, but it's not yet as stable as we'd like it to be). > > This is too late now, people are already ignoring jenkins reports because > of the many false negatives :-)
So the return code is not a good solution then, we have to see if it failed, and if it was due to an infra error or a devel error. I think that it's easier to filter for: * A string that means the tests did ran, probably at the end of the log so if there's a connection failure it will be detected as infra issue. * A string that identified if the test failed or passed And if none of those were found, then an infra failure is supposed. -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Email: dc...@redhat.com Web: www.redhat.com RHT Global #: 82-62605
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
