As of yesterday, we now retry any unexpected webkit failures. If they pass the second time around, then we turn the bot orange and list the unexpected flaky tests on the waterfall and at the end of the stdio of run_webkit_test.py. If they fail the second time around we turn the bot red as usual.
Q: Why not retry more than once? A: We don't want to encourage ignoring flakiness. Even with retrying, increased flakiness hurts us. But we have so many tests that fail ~1% of the time that it's worth doing this as a stopgap while we try to fix flakiness. Q: Won't this increase flakiness since people will now just ignore it? A: The layout test dashboard will still show the tests as failing, so they'll still show up as flaky. Hopefully the bot turning orange is enough to encourage people to fix flaky tests. Also, tests that are very flaky will still occasionally turn the bot red as they'll sometimes fail twice in a row, so those really need to be fixed. Q: Can I help fixing flakiness? That sounds hard. A: Actually it turns out to be fairly straightforward to fix much of the flakiness in webkit tests. A lot of it is just fixing obvious race conditions in the test. As proof that it's not that bad, here's a few flaky tests that Julie fixed recently. timeout increase in transition test: http://trac.webkit.org/changeset/51150 remove setTimeout that was extra watchdog: http://trac.webkit.org/changeset/51120 remove setTimeout that was working around old issue: http://trac.webkit.org/changeset/51088 use onload to detect iframe loading: http://trac.webkit.org/changeset/49592 And on the layout test dashboard, you can see they stopped being flaky: http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=LayoutTests%2Ftransitions%2Fzero-duration-with-non-zero-delay-end.html%20LayoutTests%2Ffast%2Fencoding%2Fchar-encoding.html%20%20LayoutTests%2Ffast%2Floader%2Ffile-URL-with-port-number.html -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
