Hi folks, We have a policy of disabling intermittently failing tests is they fail more than 150 times over 21 days [1] (revised from 200 times over 30 days [2], which was a very similar failure rate just evaluated over a slightly longer period).
When the policy was originally put in place in September 2017, on a typical week we'd have between 800 and 1000 pushes [3], so the policy meant disabling tests if they fail at a rate of roughly 5% (50 failures/week out of 1000 pushes/week). These days, on a typical week we have between 4000 and 5000 pushes [4]. The threshold is the same, so we're now disabling tests if they fail at a rate of roughly 1% (50 failures/week out of 5000 pushes/week). >From an engineering point of view, keeping tests passing at a failure rate of below 1% is a much more significant challenge than keeping them passing at a failure rate of below 5% (since failures that are very infrequent are very time-consuming to reproduce and iterate on). Should we perhaps be revising our disablement threshold to keep pace with the number of pushes per week? Thanks, Botond [1] https://groups.google.com/d/topic/mozilla.dev.platform/346SQCu0NAM/discussion [2] https://groups.google.com/d/topic/mozilla.dev.platform/uJVTekj2l7I/discussion [3] https://bugzilla.mozilla.org/show_bug.cgi?id=1340667#c14 [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1476893#c37 _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
