As the sheriff's know it is frustrating to deal with hundreds of tests that fail on a daily basis, but are intermittent.
When a single test case is identified to be leaking or failing at least 10% of the time, it is time to escalate. Escalation path: 1) Ensure we have a bug on file, with the test author, reviewer, module owner, and any other interested parties, links to logs, etc. 2) We need to needinfo? and expect a response within 2 business days, this should be clear in a comment. 3) In the case we don't get a response, request a needinfo? from the module owner with the expectation of 2 days for a response and getting someone to take action. 4) In the case we go another 2 days with no response from a module owner, we will disable the test. Ideally we will work with the test author to either get the test fixed or disabled depending on available time or difficulty in fixing the test. This is intended to respect the time of the original test authors by not throwing emergencies in their lap, but also strike a balance with keeping the trees manageable. Two exceptions: 1) If a test is failing at least 50% of the time, we will file a bug and disable the test first 2) When we are bringing a new platform online (Android 2.3, b2g, etc.) many tests will need to be disabled prior to getting the tests on tbpl. Please comment if this doesn't make sense. -Joel _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
