Re: --verify option added to mochitest, reftest, xpcshell test harnesses

[Geoffrey Brown]

On Tue, Oct 3, 2017 at 1:05 PM, Andrew Halberstadt
 wrote:
> This is really great Geoff! Hopefully it can cut down the number of new
> intermittents we introduce to the tree. Do you know if orangefactor or
> ActiveData can track the rate of new incoming intermittents? Would be neat
> to see how much of an impact this tool has on that.

Thanks Andrew.

I think that data is probably available in ActiveData, but querying
for "new" might be tricky...I wouldn't know where to begin.

Bugzilla might be the way to go: count bugs with keyword
intermittent-failure created today, vs yesterday vs the day before
...?

I suspect those numbers would be dominated by low frequency
intermittent failures, which might not change. In fact I'm actually
hoping to catch only mid- to high-frequency failures with TV, so that
it doesn't end up wasting people's time fixing intermittents that we
would hardly notice otherwise.

 - Geoff
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: --verify option added to mochitest, reftest, xpcshell test harnesses

[Andrew Halberstadt]

This is really great Geoff! Hopefully it can cut down the number of new
intermittents we introduce to the tree. Do you know if orangefactor or
ActiveData can track the rate of new incoming intermittents? Would be neat
to see how much of an impact this tool has on that.

On Mon, Oct 2, 2017 at 1:08 PM Geoffrey Brown  wrote:

> The mochitest, reftest, and xpcshell test harnesses now support a
> --verify option. For example:
>
>   mach mochitest
> docshell/test/test_anchor_scroll_after_document_open.html --verify
>
> In verify mode, the requested test is run multiple times, in various
> "modes", in hopes of quickly finding any intermittent failures. Once
> tests are complete, a summary of results is printed:
>
> :::
> ::: Test verification summary for:
> :::
> ::: docshell/test/test_anchor_scroll_after_document_open.html
> :::
> ::: 1. Run each test 10 times in one browser. : Pass
> ::: 2. Run each test 5 times in a new browser each time. : Pass
> ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
> ::: 4. Run each test 5 times in a new browser each time, in chaos mode. :
> Pass
> :::
> ::: Test verification PASSED
> :::
>
> There's no flexibility in the number of times the test is run in each
> mode and that's by design: I wanted a simple, standard way of checking
> "Is this test likely to produce a frequent intermittent failure"?
>
> Verify mode was developed for the test-verify task (announcement
> coming soon!) but it may also be a convenient addition to your local
> testing.
>
> More info at [1]. Bug and enhancement requests welcomed: please file
> bugs blocking bug 1357513.
>
> [1] https://developer.mozilla.org/en-US/docs/Test_Verification
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: --verify option added to mochitest, reftest, xpcshell test harnesses

[Mike Conley]

This is super cool! Thanks Geoffrey!

On 2017-10-02 1:08 PM, Geoffrey Brown wrote:
> The mochitest, reftest, and xpcshell test harnesses now support a
> --verify option. For example:
> 
>   mach mochitest
> docshell/test/test_anchor_scroll_after_document_open.html --verify
> 
> In verify mode, the requested test is run multiple times, in various
> "modes", in hopes of quickly finding any intermittent failures. Once
> tests are complete, a summary of results is printed:
> 
> :::
> ::: Test verification summary for:
> :::
> ::: docshell/test/test_anchor_scroll_after_document_open.html
> :::
> ::: 1. Run each test 10 times in one browser. : Pass
> ::: 2. Run each test 5 times in a new browser each time. : Pass
> ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
> ::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass
> :::
> ::: Test verification PASSED
> :::
> 
> There's no flexibility in the number of times the test is run in each
> mode and that's by design: I wanted a simple, standard way of checking
> "Is this test likely to produce a frequent intermittent failure"?
> 
> Verify mode was developed for the test-verify task (announcement
> coming soon!) but it may also be a convenient addition to your local
> testing.
> 
> More info at [1]. Bug and enhancement requests welcomed: please file
> bugs blocking bug 1357513.
> 
> [1] https://developer.mozilla.org/en-US/docs/Test_Verification
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

--verify option added to mochitest, reftest, xpcshell test harnesses

[Geoffrey Brown]

The mochitest, reftest, and xpcshell test harnesses now support a
--verify option. For example:

  mach mochitest
docshell/test/test_anchor_scroll_after_document_open.html --verify

In verify mode, the requested test is run multiple times, in various
"modes", in hopes of quickly finding any intermittent failures. Once
tests are complete, a summary of results is printed:

:::
::: Test verification summary for:
:::
::: docshell/test/test_anchor_scroll_after_document_open.html
:::
::: 1. Run each test 10 times in one browser. : Pass
::: 2. Run each test 5 times in a new browser each time. : Pass
::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass
:::
::: Test verification PASSED
:::

There's no flexibility in the number of times the test is run in each
mode and that's by design: I wanted a simple, standard way of checking
"Is this test likely to produce a frequent intermittent failure"?

Verify mode was developed for the test-verify task (announcement
coming soon!) but it may also be a convenient addition to your local
testing.

More info at [1]. Bug and enhancement requests welcomed: please file
bugs blocking bug 1357513.

[1] https://developer.mozilla.org/en-US/docs/Test_Verification
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform