On Nov 20, 2009, at 2:49 PM, Julie Parent wrote:
In Chromium-land, we have been working to eliminate flaky layout
tests. One thing that has helped us gain insight into flaky tests
is the flakiness dashboard. I'd love to see this available for
webkit.org as well. I'm happy to do the work to set up the
dashboard itself, but one thing we'd need is for run-webkit-tests to
output an extra json file, and for the build bots to store this file.
Is a flakiness dashboard something you'd be interested in? Is
storing the extra data file ok (the largest chromium.org one is <
1MB and the webkit.org ones should be much smaller since it is based
on # failing tests)? Anyone who knows Perl interested in adding the
support to run-webkit-tests to output the json (we can provide
details on the format)?
Sounds like a good plan to me.
Julie
Q: Why do we care about flaky layout tests?
A: They cost us time. Specfically, they cause the commit-queue to
reject good patches, lead engineers to spend extra time spent
determining if they caused the tree to go red, mask real regressions
because the tree was already red, put an extra burden on ports
trying to determine if their port is bad or the test is just bad, etc.
Q: How can I help fix flaky tests/not introduce more flaky tests?
A: One big, easily fixable, source of flakiness is setTimeouts.
Whenever possible, use specific events rather than relying on
setTimeouts. Don't use setTimeout to wait for resources to load.
Use onload events instead (iframe, image, body all have onload
events). If there are no events possible, and you need time to
elapse before checking something, provide a little extra wiggle room.
This is something we should watch for when writing new tests and
reviewing patches that include tests. If the test case uses setTimeout
or setInterval, that should be a huge red flag.
Q: Do you have examples of these sorts of easy fixes?
A: Timeout increase: http://trac.webkit.org/changeset/51150. Remove
extra watchdog setTimeout: http://trac.webkit.org/changeset/51120.
Remove unnecessary setTimeout: http://trac.webkit.org/changeset/
51088. Use onload to detect iframe loading: http://trac.webkit.org/changeset/49592
Q. I am still reading, can I see a pretty picture?
A: Sure. Here is the dashboard results clearly showing 2 tests that
got fixed. You can see the flakiness before (to the right) and the
new non-flakiness after (to the left). http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests
=LayoutTests%2Ftransitions%2Fzero-duration-with-non-zero-delay-
end.html%20LayoutTests%2Ffast%2Fencoding%2Fchar-encoding.html
_______________________________________________
webkit-dev mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________
webkit-dev mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
