I also think we should do 2-way sync-ing of WPT tests, similarly to other browser engines. These tests are extremely useful for interoperability. Every major browser engine is involved.
Google also has a nice dashboard showing the pass rates for each WPT test in each major browser. It makes it easier to identify areas we should improve on. I - for one - would be happy to write tests using testharness.js when it makes sense if it means I can just land them in WebKit with my code change and have them being automagically upstreamed, and run by other browsers as well. Note that I am not advocating we use testharness.js for everything (although I would be OK with this). I propose we make it opt-in. We have a lot of tests that are not necessarily interesting to upstream. However, the ones that are, using testharness.js makes a lot of sense. Chris Dumez > On Feb 5, 2017, at 5:04 PM, youenn fablet <youe...@gmail.com> wrote: > > I too am a big proponent of us moving more and more towards WPT. > As part of the streams and fetch API implementation, most of the related > functional tests have been made as WPT. > The benefits of having tests being improved and updated largely outweighs the > testharness.js initial cost. > Somehow, these tests are becoming part of their related specs. > > The two complaints I heard against testharness.js are: > - They are less easy to debug as test harness.js does not print out messages > for each assert. There might be room for updating testharness to support that > - Async tests are less easy to write. While this is probably true, > testharness has support for promise-based tests which are not verbose. > WPT has also some benefits of its own, like the possibility to run easily the > same tests in worker/window environments, the flexible python-based server... > > One complaint I heard against WPT tests is that they require being run behind > a server. > This is becoming more and more true. > There is still a large portion of tests that could be run locally if we > update the paths to test harness.js/testharnessreport,js. > I am not a big fan of modify any WPT test but maybe we should consider > switching back to modifying these paths at import and export time. > > I would also like to go in a direction where writing WPT tests is the default > for WebKit layout test. > For that to happen, we need an easy way to upstream tests from WebKit to WPT > GitHub. > > I would tend to do the following: > - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout > tests through bugzilla > - At commit queue time, automatically create a PR to WPT GitHub containing > the changes of the WebKit patch > - Ask committer to fix the WPT PR should there be any issue with it > (committer being informed of such issues through bugzilla). > >> Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa <rn...@webkit.org> a écrit : >> Hi all, >> >> Background >> >> Web Platform Tests is a suite of tests written for W3C specifications=, >> and Blink, Edge, Gecko, and WebKit all run them as a part of their >> continuous build system. >> >> Historically, each working group had their own repository for tests, >> but with CSS WG's tests finally getting merged into the Web Platform Tests, >> we will have a single repository with all the tests from W3C. >> >> A few of us attended a meeting to discuss the future of this test suite last >> Monday. >> One of the major topic was that Blink and Gecko have adopted the process to >> write the tests >> using testharness.js in their own test suites, and automatically merge into >> Web Platform Tests >> without having to go through another round of code review for Web Platform >> Tests. >> >> Given we do test-driven development in WebKit, I think WebKit should do the >> same. >> This will benefit other browser vendors to catch their bugs with our tests, >> and we will also benefit >> from other browser vendors "reviewing" our tests against their >> implementation and specifications. >> >> In the long term, I believe this will drastically improve the >> interoperability of the Web, >> and dramatically improve the quality of the tests we run against WebKit. >> >> In fact, there was a general consensus that we should even upstream >> pass-if-not-crash tests >> as well as style and layout invalidation tests to web-platform-tests as they >> tend to be undertested >> right now and almost all engines have bugs in this area. >> >> Problems & Questions >> >> In order to auto-merge tests from WebKit to Web Platform Tests, there are a >> few problems. >> >> 1. We need to start writing more if not all tests using testharness.js >> instead of our own js-test(-pre).js >> >> I've heard of many complaints about testharness.js being too verbose and >> cumbersome to use. >> I agree with all those complaints but I don't think we can convince all >> other browser vendors >> to use our own testing script either since both Blink & Gecko are onboard >> with using testharness.js >> >> At this point, I'd argue that the benefit outweighs the cost of adopting >> testharness.js. >> Also, W3C have dropped many styling requirements for tests so we no longer >> need to have >> link/meta elements, etc... You simply include testharness.js and >> testharness-report.js. >> >> See my recent PR for example. >> >> Do people still object to using testharness.js? If so, what are your reasons? >> >> >> 2. Where should we put upstremable tests? >> >> We need a mechanism to identify & merge tests back from WebKit into Web >> Platform Tests. >> >> One way to do this is to start adding tests directly into >> LayoutTests/imported/w3c/web-platform-tests >> and then automatically identify those tests and upstream them. >> >> Assuming all upstream merges happen in timely happen, this should work fine. >> And it has a benefit that you get to see all related tests in one place. >> >> Another approach is to create another top-level directory like >> LayoutTests/exports/web-platform-tests. >> One downside of this approach is that we need to match the directory >> structure >> in order for any script to upstream tests to appropriate locations. >> >> Do people prefer one approach over another? Or have another idea? >> >> >> 3. What would be the process for upstreaming tests? >> >> One possible approach is to create a PR request for every patch posted on >> Bugzilla >> with tests that can be merged into Web Platform Tests. >> >> Because we don't want to make Web Platform Tests less reliably, >> every PR request to Web Platform Tests go through Travis CI >> which runs the test hundreds of times to make sure it's not flaky. >> >> This Travis CI job (stability check) will then show up as an additional EWS >> bubble. >> >> >> Another approach is to wait until a patch is landed, and automatically >> create a PR >> for tests that can be upstream'ed. This has a downside that the stability >> check >> wouldn't run until the patch is landed, and it's unclear what happens they >> do fail. >> >> Do committers get emails about them? Do the patches then need to be rolled >> back? >> If we don't come up with an appropriate process, we can end up with a lot of >> broken PRs that never get fixed like those failing test expectations :( >> >> >> Again, do you prefer one or the other? Or do you have another proposal? >> >> - R. Niwa >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> https://lists.webkit.org/mailman/listinfo/webkit-dev > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
