On Monday, October 2, 2017 at 11:12:03 AM UTC-6, Geoffrey Brown wrote: > Today the test-verify test task will start running as a tier 2 job. > Look for the "TV" symbol on treeherder, on linux-64 test platforms. > > TV is intended as an "early warning system" for identifying the > introduction of intermittent test failures. When a mochitest, reftest, > or xpcshell test file is modified on a push, TV runs that particular > test over and over until it fails (orange job, standard failure > messages), or until max iterations are achieved (green job, all's > well), or until TV runs out of time (green job, maybe all's well?). As > a consequence, when a new test is added or a test is modified and an > intermittent failure is introduced, TV will usually be the first job > to fail, and it will fail on the push that modified the test, making > it (usually) simple to identify where the intermittent was introduced. > > In future I hope to run TV on more platforms, apply it to more test > suites, and refine the --verify implementation to find intermittent > failures more efficiently. As a tier 2 task, TV failures will be > starred but will not cause backouts. I hope to move to tier 1 once TV > is proven to be effective. > > More info at [1]. Bug and enhancement requests welcomed: please file > bugs blocking bug 1357513. > > [1] https://developer.mozilla.org/en-US/docs/Test_Verification
Now test-verify runs on all platforms - even Android! And beginning today, there's a new test task, test-verify-wpt, symbol TVw on treeherder, which is just like test-verify, but for web-platform tests (including web-platform reftests and wdspec tests). Both TV and TVw run as tier 2, so bugs are filed for failures, but failures do not result in backouts. These do not run as tier 1 because many tests currently fail verification; I don't want verification to cause a backout when a trivial change triggers verification of a test with an existing fault. Verifying and fixing all our tests is a big project...maybe one day! For now, if your push triggers a verification failure, please have a good look at it and consider fixing the issue right away. TV/TVw failures are generally an indication of a "bad" test that is likely to fail intermittently in the future. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
