"OrangeFactor Robot" comments in bugs for intermittent test failures now
have additional information:
1. Daily and weekly comments now include the push count and failure rate.
For example:
7 failures in 606 pushes (0.012 failures/push) were associated with this
bug yesterday.
Previously, only the failure count ("7 failures") was reported, sometimes
leading to misconceptions about changes to failure rate (a test failed 3
times yesterday, but 30 times today -- did the test change today...or were
the trees closed yesterday!?).
The push count is still only a very rough approximation of the number of
times the test was run. Sometimes tests are skipped on certain platforms,
skipped on a push to reduce load, or retried/repeated on a single push. In
particular, since SETA regularly skips some test jobs, the number of test
runs per push will vary across test jobs, so the failure rate of one test
relative to another test may not be meaningful.
Use the failure rate to compare changes in the rate of occurrence of one
bug over time; do not use it to compare one bug to another.
2. The weekly comments for the most frequent failures now include the bug's
rank -- its position in that week's top 50 most frequent failures tracked
by OrangeFactor. For example:
This is the #12 most frequent failure this week.
This is an indication of the relative frequency of this bug compared to
other intermittent test failure bugs: The #12 bug had more tracked failures
than the #13 bug. This may be an effective guide for prioritizing work on
intermittent test failure bugs.
3. The weekly comments for bugs with more than 50 failures in the week now
include:
** This failure happened more than 50 times this week! Resolving this bug
is a high priority. **
These high-frequency failures consistently account for a large percentage
of test failures. In the last 7 days, just 36 high-frequency bugs (less
than 4% of the 987 bugs tracked by OrangeFactor) accounted for 4111
failures (about 41% of the 8902 failures tracked by OrangeFactor).
If you can contribute to the resolution of one of these bugs, please make
it a priority: Quick resolution of these bugs can reduce overall test
failure counts dramatically, helping everyone watching treeherder.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
