On Tue, Mar 07, 2017 at 02:38:38PM -0500, Joel Maher wrote:
On Tue, Mar 7, 2017 at 2:15 PM, Kris Maglione <kmagli...@mozilla.com> wrote:
It would be nice if, rather than disabling the test, we could just
annotate so that it would still run, and show up in Orange Factor, but
wouldn't turn the job orange. And make sure someone is CCed on the bug to
get the daily/weekly nag emails.
I have liked this idea in the past, this falls more into the
|intermittent-if = parameters| category, which seems like one of the better
middle of the road scenarios. We would need slightly better tooling to
ensure we are catching the exact regression vs the same testcase failing.
We still have to define a role for auto classification (already on in-tree)
and this suggestion would overlap heavily with that- I do know we have an
intern starting this summer to work on autoclassification :)
Oh, yeah, it would be great if we could auto-classify based on
the specific failure. It would keeping the test running much
more useful. But failing that, I'd still rather we keep the test
running and accept any failures than disable it entirely. Then
we'd hopefully at least catch if it turns perma-orange or its
frequency increases.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
