On 09/03/17 19:53, Milan Sreckovic wrote:
Not a reply to this message, just continuing the thread.
I'd like to see us run all the intermittently disabled tests once a ...
week, say, or at some non-zero frequency, and automatically re-enable
the tests that magically get better. I have a feeling that some
intermittent failures get fixed without us realizing it (e.g., we reduce
memory usage, so OOMs go away, we speed things up so timeouts stop
triggering) and it would be good to be able to re-enable those tests
that start passing.
It'd make me feel slightly less sad that we're disabling tests that do
their job 90% of the time...
This idea is appealing, but there are some tricky details. Tests may
work fine when run in isolation but either cause problems, or be
problematic, when run in conjunction with related tests. Obviously it's
possible to deal with that situation, but it might be the difference
between "and now run this script that automatically enables all the
tests that were stable over <a few hundred> repetitions" and "the
results of this need careful manual analysis and are likely to result in
tests that flip-flop between enabled and disabled a few times before
people wise up to their peculiar brokenness".
I'm not saying that it's impossible, but I do doubt it's as trivial as
it first appears.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
