This is very cool, Geoff! People have been talking about this idea for a
long, so it is great to see it actually running. I'm glad to see chaos
mode being tested, too.
On 2017-10-02 10:11 AM, Geoffrey Brown wrote:
Today the test-verify test task will start running as a tier 2 job.
Look for the "TV" symbol on treeherder, on linux-64 test platforms.
TV is intended as an "early warning system" for identifying the
introduction of intermittent test failures. When a mochitest, reftest,
or xpcshell test file is modified on a push, TV runs that particular
test over and over until it fails (orange job, standard failure
messages), or until max iterations are achieved (green job, all's
well), or until TV runs out of time (green job, maybe all's well?). As
a consequence, when a new test is added or a test is modified and an
intermittent failure is introduced, TV will usually be the first job
to fail, and it will fail on the push that modified the test, making
it (usually) simple to identify where the intermittent was introduced.
In future I hope to run TV on more platforms, apply it to more test
suites, and refine the --verify implementation to find intermittent
failures more efficiently. As a tier 2 task, TV failures will be
starred but will not cause backouts. I hope to move to tier 1 once TV
is proven to be effective.
More info at [1]. Bug and enhancement requests welcomed: please file
bugs blocking bug 1357513.
[1] https://developer.mozilla.org/en-US/docs/Test_Verification
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
