"OrangeFactor Robot" comments in bugs for intermittent test failures now have additional information:
1. Daily and weekly comments now include the push count and failure rate. For example: 7 failures in 606 pushes (0.012 failures/push) were associated with this bug yesterday. Previously, only the failure count ("7 failures") was reported, sometimes leading to misconceptions about changes to failure rate (a test failed 3 times yesterday, but 30 times today -- did the test change today...or were the trees closed yesterday!?). The push count is still only a very rough approximation of the number of times the test was run. Sometimes tests are skipped on certain platforms, skipped on a push to reduce load, or retried/repeated on a single push. In particular, since SETA regularly skips some test jobs, the number of test runs per push will vary across test jobs, so the failure rate of one test relative to another test may not be meaningful. Use the failure rate to compare changes in the rate of occurrence of one bug over time; do not use it to compare one bug to another. 2. The weekly comments for the most frequent failures now include the bug's rank -- its position in that week's top 50 most frequent failures tracked by OrangeFactor. For example: This is the #12 most frequent failure this week. This is an indication of the relative frequency of this bug compared to other intermittent test failure bugs: The #12 bug had more tracked failures than the #13 bug. This may be an effective guide for prioritizing work on intermittent test failure bugs. 3. The weekly comments for bugs with more than 50 failures in the week now include: ** This failure happened more than 50 times this week! Resolving this bug is a high priority. ** These high-frequency failures consistently account for a large percentage of test failures. In the last 7 days, just 36 high-frequency bugs (less than 4% of the 987 bugs tracked by OrangeFactor) accounted for 4111 failures (about 41% of the 8902 failures tracked by OrangeFactor). If you can contribute to the resolution of one of these bugs, please make it a priority: Quick resolution of these bugs can reduce overall test failure counts dramatically, helping everyone watching treeherder. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform