On Wed, Nov 4, 2015 at 7:27 PM, Gareth Aye <[email protected]> wrote:
> > I'm not sure that it is so bad. From my own experience, regressions rarely > cause intermittent failures. They mostly pop up as permareds. I think it > would make sense to demonstrate that we are, in fact, masking a lot of real > broken functionality before making our intermittents noisier for sheriffs. > I disagree with this mentality. For one thing, QA files bugs all the time that are themselves intermittent. They even have a template item for it, "Repro rate: XX%". With the current retry count of 25x per test, we simply cannot write an effective Gij test for one of these bugs since the retries will make the test always pass regardless of if the fix was effective. More generally though, all I'm suggesting is that we turn these retries down to something more sane, like maybe 5. And then surface these retries on treeherder as blue runs. Again though, we can't do this unless we make a concerted effort to decrease our overall level of intermittents (and squash some of those Heisenbugs). Otherwise Gij will just wind up hidden again. With Wilfred saying that 2.6 should be a released focused on quality over new features, now would be a great time to make a big testing push.
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
