On Oct 1, 2009, at 11:58 AM, Eric Seidel wrote:
I think more you're identifying that there is a test hierarchy
problem here. Chromium really wants to base its tests off of some
base "win" implementation, and then "win-apple", "win-chromium",
"win-cairo" results could derive from that, similar to how "mac" and
"mac-leopard", "mac-tiger", "mac-snowleopard" work.
Something like that would be excellent if this pattern turns up often.
I don’t think we should make the change because of one test, but if it
comes up a lot we definitely should.
Back to the original topic: I do however see flakey tests as
"endangering our testing strategy" because they provide false
negatives, and greatly reduce the value of the layout tests and
things which run the layout tests, like the buildbots or the commit-
bot.
I also agree with Darin's earlier comment that WebKit needs
something like Chromium's multiple-expected results support so that
we can continue to run flakey tests, even if they're flakey instead
of having to resort to skipping them. But for now, skipping is the
best we have, and I still encourage us to use it when necessary
instead of leaving layout tests flakey. :)
I agree on all of this.
Except that the two specific flakey tests we were discussing that got
us started on this discussion were really serious bugs and it was
really good to fix them rather than skipping them. After this
experience, I now do share Alexey’s fear that if we had skipped them
we would not have fixed the regression. Best, if possible, would have
been to notice when they turned from reliable tests to flakey tests
and rolled the change that made them flakey out.
-- Darin
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
