OK, I agree as well - skipping is not a good solution here; I don't think the status quo is perfect, yet probably not imperfect enough to do anything about :) I guess there's just a process wrinkle we need to address on the Chromium side. It's easy to rebaseline a test in Chromium, but less easy to figure out when it's safe to un-rebaseline it. -atw
On Thu, Oct 1, 2009 at 11:57 AM, Eric Seidel <esei...@google.com> wrote: > I agree with Darin. I don't think that this is a good example of > where skipping would be useful. > > I think more you're identifying that there is a test hierarchy problem > here. Chromium really wants to base its tests off of some base "win" > implementation, and then "win-apple", "win-chromium", "win-cairo" > results could derive from that, similar to how "mac" and > "mac-leopard", "mac-tiger", "mac-snowleopard" work. > > > I think we should skip only tests that endanger the testing strategy > because > > they are super-slow, crash, or adversely affect other tests in some way. > > Back to the original topic: I do however see flakey tests as > "endangering our testing strategy" because they provide false > negatives, and greatly reduce the value of the layout tests and things > which run the layout tests, like the buildbots or the commit-bot. > > I also agree with Darin's earlier comment that WebKit needs something > like Chromium's multiple-expected results support so that we can > continue to run flakey tests, even if they're flakey instead of having > to resort to skipping them. But for now, skipping is the best we > have, and I still encourage us to use it when necessary instead of > leaving layout tests flakey. :) > > -eric >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev