On Dec 2, 2009, at 2:43 PM, Julie Parent wrote:
In a recent code review where I was minimizing some test flakiness,
Darin said:
"I wish there was a way to isolate timing-dependent tests separately
from the vast majority of tests that can run at any speed. I'd
prefer to not have tests that pass or fail based on the speed or
load of the computer, but if we do knowingly have them it would be
*so* much better if they were identified somehow."
I agree, and would like to implement something to address this.
Possible ways to mark a test as timing-dependent:
Put tests in a specific directory
Append a suffix to the test name
Add a function call to layoutTestController that is called
explicitly for timing dependent tests
Out of these ways to mark timing-sensitive tests, I prefer a specific
directory. This is a list we should try to drive to 0 over time, so
having a dedicated directory would make it easiest to see how far off
we are from that goal.
From here, the issue becomes how to use the knowledge. Some ideas:
If one of these tests fails, don't turn the bots red, turn some
other color
If one of these tests fails, re-run it. If it passes the second
time, consider it a normal pass
Turn bots red as normal, but with an indicator that the test is
known timing dependent (if we used a suffix on the test name, I
guess this would just be obvious)
Thoughts?
I prefer the following:
- If one of the tests fails a first time, re-run it a second time.
- If it fails the second time, mark the bot red (with an indication
that the test is timing sensitive).
- If the test failed the first time only, mark the bot some color
other than red or green, so we are reminded that this specific test is
flaking out without actually marking the build as broken.
Some theoretically timing-dependent tests may turn out to fail
exceedingly often, or may turn out not to really be timing-dependent
in practice, or may become more flaky at some future point, so it
would be good to have some indicator of that in the buildbot.
Regards,
Maciej
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
