On Jun 3, 2010, at 2:30 PM, Ojan Vafai wrote:
> On Wed, Jun 2, 2010 at 6:20 PM, Darin Adler <da...@apple.com> wrote:
> 1) There’s one directory with a pristine copy of the W3C test suite, with
> no WebKit changes.
> 2) If there are some tests that need to be fixed, fixed copies of those
> individual tests would go into another directory.
>
> So we would know that if css3-fixed-up/foo.html exists to skip
> css3-pristine/foo.html?
Not necessarily. We can skip the pristine one if we have to, but generally
speaking I think we should run the pristine one and the fixed up one too.
> 3) The broken tests can be run as-is, and we can land expected results to
> reflect what the broken tests do.
>
> Perhaps I would think differently about some aspect of this if we had
> introduced the “test expectations” concept for platforms other than Chromium.
>
> It's a tradeoff. On the one hand, the test expectations approach lets you
> have a list of failing tests that you drive to 0. On the other hand, checking
> in failing expectations lets you know if you ever unintentionally change the
> the type of failure (e.g. it was failing for one reason and is now failing
> for a different reason due to your patch). If we intend to have a concerted,
> short-term effort to drive the failing tests to 0, then an expectations
> approach seems better.
Maybe we should come up with a system that does both.
> There should be some set of tests that are faster to run that omits the
> slower thorough tests. This was the original goal of “fast” but we have put
> tests outside “fast” more or less at random. Why are “editing” tests outside
> “fast”? Just the whim of the person who added them. Same comment on
> directories like “accessibility”.
>
> I don't like the concept of putting fast tests in a separate top-level
> directory.
Neither do I. I said this was the original concept, but I meant that we should
accomplish this a new way now.
> If we want a way to run just the fast tests, we should just come up with a
> way of annotating tests as slow. Another option is that we could have a
> "slow" subdirectory.
Agreed. This was what I meant to say.
-- Darin
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
