Thank you for putting this together. It is important. jmaher writes:
> This policy will define an escalation path for when a single test case is > identified to be leaking or failing and is causing enough disruption on the > trees. > Exceptions: > 1) If this test has landed (or been modified) in the last 48 hours, we will > most likely back out the patch with the test > 2) If a test is failing at least 30% of the time, we will file a bug and > disable the test first > I have adjusted the above policy to mention backing out new > tests which are not stable, working to identify a regression in > the code or tests, I see the exception for regressions from test changes, but I didn't notice mention of regressions from code. If a test has started failing intermittently and is failing at least 30% of the time, then I expect it is not difficult to identify the trigger for the regression. It is much more difficult to identify an increase in frequency of failure. Could we make it clear that preferred solution is to back out the cause of the regression? Either something in the opening paragraph or perhaps add the exception: "If a regressing push or series of pushes can be identified then the changesets in those pushes are reverted." We won't always have a single push because other errors preventing tests from running are often not backed out until after several subsequent pushes. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
