In a recent code review where I was minimizing some test flakiness, Darin said: "I wish there was a way to isolate timing-dependent tests separately from the vast majority of tests that can run at any speed. I'd prefer to not have tests that pass or fail based on the speed or load of the computer, but if we do knowingly have them it would be *so* much better if they were identified somehow."
I agree, and would like to implement something to address this. Possible ways to mark a test as timing-dependent: - Put tests in a specific directory - Append a suffix to the test name - Add a function call to layoutTestController that is called explicitly for timing dependent tests >From here, the issue becomes how to use the knowledge. Some ideas: - If one of these tests fails, don't turn the bots red, turn some other color - If one of these tests fails, re-run it. If it passes the second time, consider it a normal pass - Turn bots red as normal, but with an indicator that the test is known timing dependent (if we used a suffix on the test name, I guess this would just be obvious) Thoughts? Julie
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev