I agree with Darin. I don't think that this is a good example of where skipping would be useful.
I think more you're identifying that there is a test hierarchy problem here. Chromium really wants to base its tests off of some base "win" implementation, and then "win-apple", "win-chromium", "win-cairo" results could derive from that, similar to how "mac" and "mac-leopard", "mac-tiger", "mac-snowleopard" work. > I think we should skip only tests that endanger the testing strategy because > they are super-slow, crash, or adversely affect other tests in some way. Back to the original topic: I do however see flakey tests as "endangering our testing strategy" because they provide false negatives, and greatly reduce the value of the layout tests and things which run the layout tests, like the buildbots or the commit-bot. I also agree with Darin's earlier comment that WebKit needs something like Chromium's multiple-expected results support so that we can continue to run flakey tests, even if they're flakey instead of having to resort to skipping them. But for now, skipping is the best we have, and I still encourage us to use it when necessary instead of leaving layout tests flakey. :) -eric On Thu, Oct 1, 2009 at 11:47 AM, Darin Adler <da...@apple.com> wrote: > On Oct 1, 2009, at 11:41 AM, Drew Wilson wrote: > >> I don't have an opinion about flakey tests, but flat-out-busted tests >> should get skipped. Any thoughts/objections? > > I object. > > If a test fails on some platforms and succeeds on others, we should have the > success result checked in as the default case, and the failure as an > exception. And we should structure test results and exceptions so that it’s > easy to get the expected failure on the right platforms and success on > others. Your story about a slight inconvenience because a test failed on the > base Windows WebKit but succeeded on the Chromium WebKit does not seem like > a reason to change this! > > Skipping the test does not seem like a good thing to do for the long term > health of the project. It is good to exercise all the other code each test > covers and also to notice when a test result gets even worse or gets better > when a seemingly unrelated change is made. > > I think we should skip only tests that endanger the testing strategy because > they are super-slow, crash, or adversely affect other tests in some way. > > -- Darin > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev