I usually have a look at the blues as well when I look at my pull request runs. And most of the time it's an issue that's not related with the test. I think this account for most of the issues we see, actually. Do we know why this happens ?
See for example Gu13 in [1] (full log is [2]). [1] https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=0d4437d31e8dd8c81d4617d875c7337d858097a4 [2] https://public-artifacts.taskcluster.net/b32i9Hj1T1qurpfDYWyyQA/0/public/logs/live_backing.log Le 04/11/2015 16:39, Michael Henretty a écrit : > Hi Gaia Folk, > > If you've been doing Gaia core work for any length of time, you are > probably aware that we have *many* intermittent Gij test failures on > Treeherder [1]. But the problem is even worse than you may know! You > see, each Gij test is run 5 times within a test chunk (g. Gij4) before > it is marked as failing. Then that chunk itself is retried up to 5 > times before the whole thing is marked as failing. This means that for > a test to be marked as "passing," it only has to run successfully once > in *25* times. I'm not kidding. Our retry logic, especially those > inside the test chunk, make it hard to know which intermittent tests > are our worst offenders. This is bad. > > My suggestion is to stop doing the retries inside the chunks. That > way, the failures will at least surface on Treeherder, which means we > can star more test, which means we'll have a lot more visibility on > the bad intermittents. Sheriffs will complain a lot, so we have to be > ready to act on these bugs. But the alternative is that we continue to > write tests with a low "raciness" bar which, IMO, have a much lower > chance of catching regressions. The longer we wait, the worse this > problem becomes. > > Thoughts? > > Thanks, > Michael > > 1.) > https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure&keywords_type=allwords&list_id=12657856&resolution=---&query_format=advanced&product=Firefox%20OS > > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos
signature.asc
Description: OpenPGP digital signature
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
