On 04/21/2011 01:10 AM, Adrian Bateman wrote:
First, thanks to Art for pulling all this content together. We're looking
forward to a more structured process for testing as various specifications
in the WebApps increase in maturity.
I have a couple of small comments related to the issues Aryeh raised.
Apologies for the lateness of these comments; I spent time sharing this
process with a number of teams at Microsoft before responding.
1. Early approval of tests
We think that waiting for Last Call or Candidate Recommendation before
approving tests loses some of the value of tests in driving ambiguity out
of the specification. The CSS WG found many issues in CSS 2.1 as a
consequence of tests and some of these issues we substantive enough that
the spec went back to Working Draft status. Avoiding this by reviewing test
cases earlier in the process will help to improve spec quality. I think
of this in the following way: a bug filed against the spec requesting a
change represents someone's view that the spec is wrong. On the other hand,
an approved test with consensus of reviewers in the working group helps to
identify more stable sections of the spec. It doesn't mean it can't change
but it does mean the spec has had additional review on the assertions made
in the test and that's useful.
I agree that late review is not helpful to anyone.
The way I think test review should work is similar to any other form of
code review:
A user pushes a series of commits to the repository
The user requests review of those commits
Different reviewers comment on the patch
If no problems are found, the tests are considered approved
This encourages early review, makes it obvious what people consider
stable enough to be reviewed, and allows for sharing of not-yet-ready
works in progress without having to have multiple public repositories.
It also eliminates the need for seperate submitted and approved folders
since approved tests are ones where all the related commits have
positive review and there are no open bugs.
As far as I can tell the main problem with adopting this workflow is
that some tooling support is required.
