WPT has recently passed an RFC <https://github.com/web-platform-tests/rfcs/blob/master/rfcs/reftest_simplification.md> trying to simplify the reftest graph (although it has not been implemented yet), which eliminates a lot of the complexities and concerns.
On Fri, Nov 8, 2019 at 5:07 PM Ryosuke Niwa <[email protected]> wrote: > On Fri, Nov 8, 2019 at 2:01 PM Simon Fraser <[email protected]> > wrote: > > > > I'd like to land a patch to support finding test references via <link > rel="match/mismatch">: > > https://bugs.webkit.org/show_bug.cgi?id=203784 > > > > There has been some discussion about this in the past: > > https://lists.webkit.org/pipermail/webkit-dev/2011-November/018470.html > > > > But I think the benefits outweigh the drawbacks. As that mail states: > > > > *Link element approach* > > Pros: > > > > - Can reuse same ref. file for multiple tests > > > > Still true. > > > > - Can have multiple ref. files for single test > > > > True but no something that we support, and I haven't see any WPT use > this (our importer throws an error if it sees this) > > > > - Information is self-contained in the test file > > > > Still true > > > > - We may get away with test suite build step > > > > It certainly simplifies WPT test import. > > > > Currently importing some CSS suites (e.g. css-backgrounds) results in > broken -expected.html files because copying them breaks references to sub > resources. > > > > (It turns out that we can't convert W3C ref tests to use WebKit > conventions > > due to the first two points.) > > > > We're doing this much more now, and the "multiple references" point is > moot, so I think we can import WPT tests mostly as-is. > > > > Cons: > > > > - Requires us modifying each port's DRT to support this format > > > > No, it just requires webkitpy hacking which I've done in the patch. > > I'm not certain writing a bunch of regular expressions in webkitpy is > a reliable mechanism to find expected results. Another issue I found > back then was that it significantly slowed run-webkit-tests' startup > time because WPT has a workflow to find all tests & their expected > results upfront before any tests could run. > > > > - Adding link elements itself may affect tests (all W3C tests are > > required to have link elements at the moment) > > > > I haven't seen this be an issue. > > Another issue is that if you were to modify a test which happens to be > also used as a reference or a mismatch result (worse) for some other > test, then you may not notice that without inspecting every other test > in existence. > > > - Hard to understand relationship between files. e.g. if we want to > > figure out which tests use ref.html, we must look at all test files > > > > This is true, but I don't really see it being a problem in practice. > > This definitely is an issue. It's possible WPT has improved things but > we've definitely had an experience where tests were used as reference > for other tests, etc... and having to think about this issue every > time I touch test drove me nuts. > > > What I have seen is us importing CSS 2.1 tests that have foo.html and > foo-ref.html, and treating foo-ref.html as a test so generating > foo-expected.txt and foo-ref-expected.txt. That seems worse. > > Seems like we can treat "-ref" as a special suffix like we already do > with support directory and resources directory. > > > So now that WPT is heavily invested in <link rel=> I think we should > follow suite. It will simplify WPT import, and reduced the number of cloned > -expected.html files significantly. > > I really don't want to deal with tests being used as references for > other tests. I'm okay with this approach if we forbid that. > > - R. Niwa > _______________________________________________ > webkit-dev mailing list > [email protected] > https://lists.webkit.org/mailman/listinfo/webkit-dev >
_______________________________________________ webkit-dev mailing list [email protected] https://lists.webkit.org/mailman/listinfo/webkit-dev
