Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL"): > On Fri, 2015-11-27 at 01:18 -0700, Jan Beulich wrote: > > Neither of these failed in this flight, and there's nothing else blocking > > the push. Why did this not result in a push then? Or in other words > > why do the failures in earlier flights get considered a reason not to > > push? > > @Ian, README.email covers lots of these kinds of patterns, but not this > specific one.
See below for proposed docs patch to explain the general meaning of `fail in X REGR. vs. Y'. > > > build-i386 5 xen-build fail in 65062 REGR. vs. 63449 This is completely explained below, I think. > > > test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 16 > > > guest-localmigrate/x10 fail in 65088 REGR. vs. 63449 As explained below, in 65112 this step did not run because the earlier step `guest-localmigrate' failed: http://logs.test-lab.xenproject.org/osstest/logs/65112/test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html The fact that we have both `guest-localmigrate' and `guest-localmigrate/x10' isn't ideal because it hides from the heisenbug compensator that these are actually the same underlying test. Maybe it is time now to rename `guest-localmigrate/x10' to `guest-localmigrate' and abolish the latter. >From 987dd088192f9f94c59beeddc073cecaad76a24e Mon Sep 17 00:00:00 2001 From: Ian Jackson <ian.jack...@eu.citrix.com> Date: Fri, 27 Nov 2015 11:36:05 +0000 Subject: [OSSTEST PATCH] README.email: Document `fail in 58948 REGR. vs. 63449' Signed-off-by: Ian Jackson <ian.jack...@eu.citrix.com> --- README.email | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/README.email b/README.email index 992a574..40df71a 100644 --- a/README.email +++ b/README.email @@ -71,6 +71,24 @@ history. Here are some examples: detect regressions of this test. Perhaps the test has been recently introduced. + fail in 58948 REGR. vs. 63449 + + The results processor used 58948 (another flight testing the + just-tested version) to convince itself that some other test + failure is intermittent. Look for other references to 58948 in + the report to see which those other test failures are. + + However, in 58948, there were further failures. In particular, + the step being reported here failed, and that failure could not + in turn be justified. + + If this further failure is in a test job, this is usually + because the reported step did not run at all in the most recent + flight, usually because it was blocked by an earlier failure. + (Intermittent build job failures are never considered + justifiable because they prevent other tests from running and + can so conceal bugs.) + fail in 58948 pass in 58965 fail in 58948 like 37628 -- 1.7.10.4 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel