On Wed, Jun 2, 2010 at 6:20 PM, Darin Adler <da...@apple.com> wrote: > > 1) There’s one directory with a pristine copy of the W3C test suite, > with no WebKit changes. > 2) If there are some tests that need to be fixed, fixed copies of those > individual tests would go into another directory. >
So we would know that if css3-fixed-up/foo.html exists to skip css3-pristine/foo.html? 3) The broken tests can be run as-is, and we can land expected results to > reflect what the broken tests do. > > Perhaps I would think differently about some aspect of this if we had > introduced the “test expectations” concept for platforms other than > Chromium. It's a tradeoff. On the one hand, the test expectations approach lets you have a list of failing tests that you drive to 0. On the other hand, checking in failing expectations lets you know if you ever unintentionally change the the type of failure (e.g. it was failing for one reason and is now failing for a different reason due to your patch). If we intend to have a concerted, short-term effort to drive the failing tests to 0, then an expectations approach seems better. > There should be some set of tests that are faster to run that omits the > slower thorough tests. This was the original goal of “fast” but we have put > tests outside “fast” more or less at random. Why are “editing” tests outside > “fast”? Just the whim of the person who added them. Same comment on > directories like “accessibility”. > I don't like the concept of putting fast tests in a separate top-level directory. I want the tests to be grouped by what they're testing. If we want a way to run just the fast tests, we should just come up with a way of annotating tests as slow. Another option is that we could have a "slow" subdirectory. A real-world example, we have some very slow editing tests, but most of them are fast. We could move the slow ones into editing/slow or we could annotate them as slow in an expectations file. Then we add a --fast flag to run-webkit-tests that skips over slow tests. This does mean that we'll have slow subdirectories in many places (or a single file listing the slow tests), but at least tests will all be grouped by what they're testing. The test suites that are imported from elsewhere should go into directories > with clear names such as: > > ImportedTestSuites/W3C-CSS1 > > Rather than names like: > > css1 > This sounds great. It will make understanding the tests much easier. Ojan
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
