This is really great Geoff! Hopefully it can cut down the number of new intermittents we introduce to the tree. Do you know if orangefactor or ActiveData can track the rate of new incoming intermittents? Would be neat to see how much of an impact this tool has on that.
On Mon, Oct 2, 2017 at 1:08 PM Geoffrey Brown <gbr...@mozilla.com> wrote: > The mochitest, reftest, and xpcshell test harnesses now support a > --verify option. For example: > > mach mochitest > docshell/test/test_anchor_scroll_after_document_open.html --verify > > In verify mode, the requested test is run multiple times, in various > "modes", in hopes of quickly finding any intermittent failures. Once > tests are complete, a summary of results is printed: > > ::: > ::: Test verification summary for: > ::: > ::: docshell/test/test_anchor_scroll_after_document_open.html > ::: > ::: 1. Run each test 10 times in one browser. : Pass > ::: 2. Run each test 5 times in a new browser each time. : Pass > ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass > ::: 4. Run each test 5 times in a new browser each time, in chaos mode. : > Pass > ::: > ::: Test verification PASSED > ::: > > There's no flexibility in the number of times the test is run in each > mode and that's by design: I wanted a simple, standard way of checking > "Is this test likely to produce a frequent intermittent failure"? > > Verify mode was developed for the test-verify task (announcement > coming soon!) but it may also be a convenient addition to your local > testing. > > More info at [1]. Bug and enhancement requests welcomed: please file > bugs blocking bug 1357513. > > [1] https://developer.mozilla.org/en-US/docs/Test_Verification > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
