On Wed, Nov 4, 2015 at 10:39 AM, Michael Henretty <[email protected]> wrote:
> Hi Gaia Folk, > > If you've been doing Gaia core work for any length of time, you are > probably aware that we have *many* intermittent Gij test failures on > Treeherder [1]. But the problem is even worse than you may know! You see, > each Gij test is run 5 times within a test chunk (g. Gij4) before it is > marked as failing. Then that chunk itself is retried up to 5 times before > the whole thing is marked as failing. This means that for a test to be > marked as "passing," it only has to run successfully once in *25* times. > I'm not kidding. Our retry logic, especially those inside the test chunk, > make it hard to know which intermittent tests are our worst offenders. This > is bad. > I'm not sure that it is so bad. From my own experience, regressions rarely cause intermittent failures. They mostly pop up as permareds. I think it would make sense to demonstrate that we are, in fact, masking a lot of real broken functionality before making our intermittents noisier for sheriffs. > > My suggestion is to stop doing the retries inside the chunks. That way, > the failures will at least surface on Treeherder, which means we can star > more test, which means we'll have a lot more visibility on the bad > intermittents. Sheriffs will complain a lot, so we have to be ready to act > on these bugs. But the alternative is that we continue to write tests with > a low "raciness" bar which, IMO, have a much lower chance of catching > regressions. The longer we wait, the worse this problem becomes. > > Thoughts? > > Thanks, > Michael > > 1.) > https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure&keywords_type=allwords&list_id=12657856&resolution=---&query_format=advanced&product=Firefox%20OS > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos > >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
