[ Bcc: public-webapps-testsuite ]
Hi All,
I think it would be helpful if WebApps had a common way to "manage" test
case bugs, in particular:
1. A common way to report a test bug
2. An easy way to determine if a specific test suite has any open bugs.
(For example, if someone goes to the websockets test suite on WPT, it
should be easy to determine if any of the test cases have open bugs.)
Currently, tests bugs are reported in a few different ways:
* Bugzilla - WebApps has a single Testing component in Bugzilla for all
of the specs and it has been used to report a few test case bugs [Bugs].
Using Bugzilla addresses #1, however, a single component makes #2 a bit
tricky although that could be addressed by prefixing titles with the
spec name (e.g. [workers] ...).
* GitHub PRs - some bugs are noted in comments to a PR (and thus making
#2 a bit difficult).
* GitHub Issues - some bugs are reported as a Github Issue but GH's
Issue granularity is for all of [WPT] and not per spec (thus making #2 a
bit tricky, although title prefixing could help).
* The WPT root [WPT] is silent on how to file bugs (although some
sub-resource could address that) and that could be appropriate if we
expect every test suite to be able to customize its bug reporting policy.
If we agree Bugzilla should be used to report all test case bugs: 1)
should we have an agreed way for a test suite in WPT to point to
Bugzilla (although Bugzilla has bug reports for websockets and workers
tests, that link is missing from the test suites); 2) should we continue
to lump all of the tests in a single component or create per test suite
components (e.g. tests-workers, workers-tests, ...).
Comments, proposals, etc. are welcome.
-Thanks, ArtB
[WPT] <https://github.com/w3c/web-platform-tests>
[Bugs]
<https://www.w3.org/Bugs/Public/buglist.cgi?component=Test%20suite&list_id=30503&product=WebAppsWG&resolution=--->
