For many years we have followed the same process where we file a bug for each failure where the summary is the exact error message (a few exceptions).
This results in a many:one relationship between bugs and testcases as testcases can fail for different reasons. This current process makes it harder to: 1) understand how often the test case is really failing (which indicates noise or need for attention) 2) automatically annotate known failures (sheriffs/try pushes/etc.) The change we will be implementing is to have a single meta bug for each test case (file in mozilla-central). An example is: https://bugzilla.mozilla.org/show_bug.cgi?id=1757074 If there is an investigation to fix the testcase or disable it on a given platform, then we can create a blocking bug on our meta bug for tracking purposes. We will start off small, here are the restrictions we will use: 1) only test failures, not crashes || timeouts || leaks || infra 2) only for xpcshell The plan is to roll this out (in the next 2 weeks) and monitor it for a few weeks before expanding to other test suites. Feel free to ask questions in Matrix: #test (firefox-testing). -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/a3d5ea68-b2c6-4898-8716-189d6acc063dn%40mozilla.org.
