Hey Martin,
This is a good idea, and we've been thinking about approaches like
this. Basically, the idea is to run tests that "(nearly) always pass"
less often. There are currently some tests that fit into this category,
like dom level0,1,2 tests in mochitest-plain, and those are
time-consuming to run. Your idea takes this a step further, by
identifying tests that sometimes fail, correlating those with code
changes, and ensuring those get run.
Both of these require some tooling to implement, so we're experimenting
initially with approaches that we can get nearly for "free", like
running some tests only every other commit, and letting sheriffs trigger
the missing tests in case a failure occurs.
The ultimate solution may blend a bit of both approaches, and will have
to balance implementation cost with the gain we get from the related
reduction in slave load.
Jonathan
On 8/21/2014 10:07 AM, Martin Thomson wrote:
On 20/08/14 17:37, Jonas Sicking wrote:
It would however be really cool if we were able to pull data on which
tests tend to fail in a way that affects all platforms, and which ones
tend to fail on one platform only.
Here's a potential project that might help. For all of the trees
(probably try especially), look at the checkins and for each directory
affected build up a probability of failure for each of the tests.
You would have to find which commits were on m-c at the time of the
run to set the baseline for the checkin; and intermittent failures
would add a certain noise floor.
The basic idea though is that the information would be very simple to
use: For each directory touched in a commit, find all the tests that
cross a certain failure threshold across the assembled dataset and
ensure that those test groups are run.
And this would need to include prerequisites, like builds for the
given runs. You would, of course, include builds as tests.
Setting the threshold might take some tuning, because failure rates
will vary across different test groups. I keep hearing bad things
about certain ones, for instance and build failures are far less
common than test failures on the whole, naturally.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
