I wonder if there's interest in the WebKit community to improve granularity (and detail) with which you can document test expectations, like we have done on Chromium:
http://dev.chromium.org/developers/testing/webkit-layout-tests#TOC-Test-Expectations This way, you could monitor known failures with the awesomeness that is Layout Test Flakiness Dashboard: http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html Since we already have done this work on one port, we could definitely extend this to all other ports. :DG< On Wed, Sep 23, 2009 at 11:41 PM, David Levin <le...@google.com> wrote: > If a test is flaky, it doesn't seem good to keep in the mix because it will > turn red for no reason and people will think a check in was bad. > oth, if a test is flaky but has enough logging in it to indicated an > underlying cause, then it may be worth keeping in the mix so that the root > issue may be determined. > imo, a good solution in this case is to improve the test to print out more > information to help track down the underlying bug. > dave > > On Wed, Sep 23, 2009 at 11:23 PM, Maciej Stachowiak <m...@apple.com> wrote: >> >> On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote: >> >> Alexey and I have been discussing if WebKit should add flakey tests to >> Skipped lists: >> https://bugs.webkit.org/show_bug.cgi?id=29322 >> >> Alexey asked that I bring the discussion to a larger audience. >> >> Pros: >> - Buildbots stay green. >> - Red bots/tests means your change caused an error. >> >> Cons: >> - Skipped tests may be more likely to be forgotten (and thus never fixed). >> >> What does WebKit think? Should we skip flakey tests which can't be >> resolved in a timely manner (and instead track issues via bugs) as >> WebKit policy? Or should we reserve the skipped list for other >> conditions (like bugs in the OS)? >> >> Thoughts? >> >> I'm a little concerned about sweeping failures under the carpet, but on >> the other hand the buildbots are much less valuable if they are red too >> much. It's hard to think about this in the abstract. Do you have any >> concrete examples? >> - Maciej >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> > > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev