Thanks for the feedback!
Yes, I agree early and thorough review is needed and my expectation
was/is that those vested in a spec and its test suite would actively
participate in the creation and review of tests, regardless of whether
that function was documented or not. I will add some related text to the
wiki in a day or two. I think the gist of the review and approve process
is that a contributor or test suite maintainer may start a Request for
Review (RfR) on public-webapps-testsuite (p-w-ts) for some set of test
cases and if no issues are raised the maintainer will copy the test
cases to the approved directory. I can imagine some test suites having
multiple RfRs. After a test suite is approved by the p-w-ts community
and the WG considers its spec complete, I think it would then make sense
to have a formal CfC among the entire WG to approve a test suite.
Re various "roles", I agree they should be clarified. Below is a rough
draft and comments are welcome. If someone wants to lead WebApps' test
coordination, please let me know.
Re clear test suite status, below is a rough draft for that section and
comments on that are also welcome, especially pointers to the process
and mechanisms others use to track test case status.
-ArtB
== Roles ==
* Contributor - someone that contributes one or more test cases
* Test Suite Maintainer - the person(s) responsible for maintaining a
specification's test suite, including: updating test cases, maintaining
the test suite's status, etc. Normally, this role is assumed by the
specification's Editor(s).
* Testing Coordinator - the person responsible for coordinating all of
WebApps' testing efforts, including: assuring only approved tests are
put in <code>.../approved/</code> directories, making sure the test
suites' status are current, etc. By default, this role is taken by a WG
Chair, a WebApps '''Team Contact''' or a member of the WG. (In practice,
this role may be shared by more than one person.)
== Test Suite Status ==
To facilitate determining the status of a test suite, a Test Suite
Maintainer is responsible for maintaining the test suite's status. Each
test suite includes a '''Status''' file i.e.
<code>.../tests/Status.html</code> and this file includes at least the
following information:
# Test Suite Maintainer(s): <names ...>
# Approval status: normally, one of: a) no test cases are approved; b)
test cases in the <code>.../approved/</code> directory are approved by
members of WebApps' test group (public-webapps-testsuite); c) the WG has
formally approved all of the test cases in the
<code>.../approved/</code> directory.
On Apr/20/2011 7:10 PM, ext Adrian Bateman wrote:
First, thanks to Art for pulling all this content together. We're looking
forward to a more structured process for testing as various specifications
in the WebApps increase in maturity.
I have a couple of small comments related to the issues Aryeh raised.
Apologies for the lateness of these comments; I spent time sharing this
process with a number of teams at Microsoft before responding.
1. Early approval of tests
We think that waiting for Last Call or Candidate Recommendation before
approving tests loses some of the value of tests in driving ambiguity out
of the specification. The CSS WG found many issues in CSS 2.1 as a
consequence of tests and some of these issues we substantive enough that
the spec went back to Working Draft status. Avoiding this by reviewing test
cases earlier in the process will help to improve spec quality. I think
of this in the following way: a bug filed against the spec requesting a
change represents someone's view that the spec is wrong. On the other hand,
an approved test with consensus of reviewers in the working group helps to
identify more stable sections of the spec. It doesn't mean it can't change
but it does mean the spec has had additional review on the assertions made
in the test and that's useful.
2. Calls for Consensus
When a contributor submits a set of tests, hopefully members of the
public-webapps-testsuite list will review them. When the contributor believes
they have addressed all of the feedback on the tests, we think they should
be able to put the tests forward for a Call for Consensus on approving them.
The WG may choose to issue such a call on a regular schedule for tests that
have had sufficient time for review if the number of submissions makes doing
this "on demand" too cumbersome. This call may just be made to the testsuite
mailing list. A full WG consensus shouldn't be necessary until document
transition.
3. Test suite maintainer
It's not clear from the discussion below what the role of the maintainer is
and whether there's one per spec or one for the group. This seems like largely
an administrative role in ensuring that the appropriate tests are moved into
the approval folder at the correct time and that the status of the test suite
is accurately recorded. Perhaps members of the working group could volunteer
to help with this or maybe the Team could help. Microsoft would be willing
to contribute here too.
Thanks,
Adrian.
On Tuesday, April 19, 2011 9:22 AM, Arthur Barstow wrote:
I agree the need for clear test suite status is implied and should be
explicit. I've added a new requirement for this to [1]. As to how this
requirement is addressed, perhaps we should adopt/re-use some existing
good practice; otherwise perhaps we can use a Status/Readme file in each
.../tests/ directory.
Besides "WG Approval", you see the need for "maintainer approval".
Additionally, you think there will be scenarios where multiple
submissions make it difficult to know which tests have been reviewed
(e.g. by the maintainer). This could be addressed by the maintainer
creating a separate directory (e.g. "reviewed") and the maintainer would
use it to place copies of test cases (e.g. from submissions directories)
she reviewed (and possibly edited and approved). I don't think the
process as currently defined would preclude the maintainer from doing
something like this and if it becomes a common/good practice, we could
document it. In the case of a single contributor, it's probably not
something that would be needed (but wouldn't necessarily be harmful).
-Art Barstow
[1] http://www.w3.org/2008/webapps/wiki/Testing#Testing_Goals
On Apr/17/2011 7:06 PM, ext Aryeh Gregor wrote:
On Wed, Apr 13, 2011 at 10:02 AM, Arthur Barstow<art.bars...@nokia.com>
wrote:
I have updated WebApps' testing process documents to reflect comments
submitted to the initial draft process [1]. As such, this is a Call
for
Consensus to agree to the testing process as described in:
http://www.w3.org/2008/webapps/wiki/Testing
http://www.w3.org/2008/webapps/wiki/Submission
http://www.w3.org/2008/webapps/wiki/Approval
http://www.w3.org/2008/webapps/wiki/Harness
Sorry for the lateness of this review -- I was swamped with work and
didn't find the time to respond earlier. I still have a few issues
with the proposed approval procedure. The way it sounds is that tests
can either be non-approved, or approved. Non-approved tests sound
like (I'm not totally clear on this) they're supposed to live in
per-contributor submission/ directories, without anyone else
necessarily having any say over them. Approved tests, on the other
hand, only exist at LC or later, and can't be changed without Working
Group approval. (Which means what? I'm not sure.)
The problem I've seen with the submission/ vs. approved/ approach in
the HTMLWG is that there's no distinction between tests in submission/
that are useful and correct but just haven't been reviewed by anyone,
and tests that aren't complete or correct yet and aren't really
intended for serious use. We should have a clear test suite that's
usable in practice well before LC, even if not all the tests are
reviewed yet.
So what I'd prefer is that the contents of approved/ be under the
control of the maintainer of the test suite, like the editor controls
the spec. If people are submitting tests, the maintainer should be
allowed to approve them without a CfC, at least while the spec is
still a Working Draft. That way we have a single repository from the
beginning that should include all useful tests, instead of having many
tests of varying quality scattered throughout submission/. Once a
test is in approved/, it should be possible to file bug reports on it,
discuss it on the mailing list, etc. It needs to be in a central
location and the responsibility of the working group, not the
submitter, so that the working group can ensure the test suite's
quality from the earliest possible point.
Then a CfC would be whether the WG is okay with the current contents
of approved/ or whether some of the tests should be un-approved. A
CfC shouldn't be necessary to approve tests to begin with, any more
than it's necessary to make an edit to a spec.
