Welcome to the second annual update on progress in cross-browser interop
testing through web-platform-tests. This year has seen big improvements
to the platform coverage of wpt, as well as increasing the number of
features that can be tested with wpt, and the visibility of test results
in Gecko and in other browsers. In addition there have been numerous
fixes intended to make the experience of working with wpt better and the
results more reliable. So in no particular order:
== Gecko CI ==
* web-platform-tests are now enabled on CI on all tier-1 platforms i.e.
Desktop Firefox on Linux, Windows and macOS, and mobile Firefox on
Android (using the x86 emulator)
* web-platform-tests reftests run the full set of CSS tests on all
platforms.
* web-platform-tests run with leak checking enabled in debug, and also
run under LSAN. When tests are imported, any existing leaks are
automatically added to the allowed list for those tests.
* wpt wasm tests are running in the jsshell.
== Testability ==
* Substantially increased the scope of the testdriver.js [1] API to
allow writing tests that are not possible using standard DOM APIs. In
particular the following things are now possible:
- Sending trusted key and mouse events
- Sending complex series of trusted pointer and key interactions for
things like in-content drag and drop or pinch zoom
- File upload
Also added the 'bless' API for cases where something trusted is
required but the details are unimportant.
* Implemented support for running tests on HTTP/2 (this is currently
disabled by default due to problems with upstream CI that will be solved
by migrating from Travis).
* Added support for fuzzy matching in reftests, to allow reftests to
pass in the face of small unavoidable differences between test and ref
(this is not yet merged but is expected to be landed by the end of the
year).
* Added multiple top-level domains to wpt to allow cross-site tests.
Changed wpt runner to run tests in a true top level browsing context
(i.e. with null `opener`).
== Results Collection and Viewability ==
* Many fixes and improvements to the wpt.fyi interface to allow
efficient selection and comparison of multiple test runs.
* Set up Taskcluster to run all tests after each upstream commit on
Firefox nightly and Chrome dev on Linux.
* Set up daily runs of stable and weekly runs of beta, also using
Taskcluster.
* Regular, reliable, Edge and Safari Stable/TP runs using a custom
buildbot setup and Sauce Labs for Edge.
== Sync ==
* We deployed a new, faster, sync for mozilla-central that enables us to
sync test changes made in Gecko repositories as soon as they land in our
repositories, creates Bugzilla bugs to track upstream PRs and
corresponding changes to Firefox test results, and allows us to
downstream changes from web-platform-tests much more frequently.
* Servo's sync system was updated to allow continuous syncing of PRs
from servo/servo to web-platform-tests and frequent downstreaming.
* Since launching the new sync we have upstreamed over 500 changes from
Gecko bugs to the web-platform-tests repository.
* Over the same time period we have seen about 1,500 changes synced from
the Chromium repository to upstream web-platform-tests.
== Developer Ergonomics ==
* Stopped using the in-tree MANIFEST.json file and instead downloaded it
on demand. Apart from avoiding merge conflicts, this helps avoid
breaking many pieces of tooling including Phabricator and git-cinnabar.
* Added reftest-analyzer compatible output to the default logger used by
mach wpt.
* Improved multi-global tests by extending .any.js tests to be more
flexible and support more kinds of global scope.
== GitHub Infrastructure ==
* Moved test-stability checks from Travis to Taskcluster for performance
and reliability
* Started running PR checks in Safari using Azure Pipelines.
* Started checking for tests that are close to the timeout value and
likely to become intermittent.
* Continued making more infrastructure Python 3 compatible.
* Running affected tests on Chrome, Firefox and Safari and showing
regressions as PR statuses (currently in beta)
== Meta ==
* Moved the web-platform-tests GitHub repository to its own organisation.
* Started the process of creating a more formal governance structure for
web-platform-tests, forming a provisional core team with membership from
a range of stakeholders.
A couple of the mentioned items are still under review, but will almost
certainly merge before the end of the year :) These improvements have
been the result of a close collaboration between people across the
web-platform-tests community, including the Interop Testing team at
Mozilla, the Ecosystem Infra team at Google and many, many, others. Huge
thanks to everyone involved.
[1] https://web-platform-tests.org/writing-tests/testdriver.html
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
