https://github.com/WebKit/WebKit/pull/21912 hopefully serves as good progress
here: this PR is almost largely done by automated tooling.
The first commit is the result of `import-w3c-tests` (the long standing
command), but with no further work (to the expectations or baselines).
It then ran through EWS with the `no-failure-limits` label, before I added a
second commit: the output of `update-test-expectations github-pr`, albeit
limited to only changes it made to the relevant (dom) directory. This is where
all the baselines come from in the PR.
This a roughly a workflow that’s useable in some cases today.
The most obvious limitation is that it only deals with baseline files
(*-expected.txt), and not TestExpectations files (which means for things where
the result is stored there—including reftest failures, debug differences,
timeouts and crashes—it doesn’t work). Of the limitations this causes, reftests
are by far the more apparent and mean this doesn’t work for most CSS tests yet.
The other obvious limitation is that it doesn’t deal with flaky tests, as shown
by the PR failing after the second commit.
The final limitation, hinted at above, is that this doesn’t take any
consideration of the current results on main—so it rebaselines far too much,
depending on how well gardened main is when the PR run.
That said, the first and last of these are very much things we care about
fixing, and we’ll see how problematic the second requiring manual intervention
is.
I’d encourage people importing directories which have few reftests to try using
the scripts mentioned above, and file any bugs they find.
What would be super helpful, to avoid making “turn on automated import” too
painful, is updating directories we have imported that haven’t been updated
recently.
And finally, as an aside, the `import-expectations.json` file now explicitly
skips every top level directory we don’t import: I’d encourage you to take a
look at what’s marked as skip, and if there’s more you think we should be
importing (especially because we have implemented the spec in question!), I’d
encourage you to do so by running `import-w3c-tests` with that directory, which
will also change the relevant line in `import-expectations.json` from “skip” to
“import”.
Happy New Year, (Web)Kittens,
Sam
PS: Apparently I messed up and my original draft with many more links to bugs
didn’t get saved to a server-side draft mailbox, and I’m now on vacation, so
please accept this much less-useful-than-intended email on current WPT import
work! (I’ll try to reply with all the links once I get home in the New Year,
although I will remain on leave for a while beyond.)
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
