Travis Leithead wrote:
Ultimately, I believe we need to make sure that all the assertions in
WebIDL have some testing coverage. I started looking at Cameron's
submitted tests today, and they are a blend of tests that could be
covered by idlharness.js and those that we would be unable to
automatically verify using the auto-generated tests.
I think the next step is to map what parts of WebIDL v1 are already
covered by the auto-gen'd tests of idlharness.js, and also which
parts of the spec are covered by Cameron's recently submitted tests
(I see the tests are all marked up, I just need to go through and
cross check.) I'll try to do that while I review Cam's tests, and
also what's in idlharness.js. ETA 2 weeks?
Between the two, if we have coverage for all the essentials (Cam
notes some issues where there aren't two testable
specs/implementations, and we should review those), then we should
try to move on to the next step, which is an "implementation report",
right?
Thanks Travis for looking into the coverage. I think you are right that
with these tests and those covered by idlharness.js, we should be in the
position to start putting together an implementation report. From the
testing I was doing while writing the tests, I don't think we have two
passing implementations of all the tests yet.
Also, I imagine we would want to take only the idlharness.js-generated
tests that correspond to the set of API features we want to rely on.
Does that sound right?
As for the other half of the exit criteria -- whether specifications are
correctly using all of the Web IDL features -- then I think we can base
this on the features we are relying on for the tests. I don't recall
coming across any invalid IDL that I wrote tests against. So I believe
we can state that we have met this criterion, apart from the exceptions
I listed in the notes.txt.
I am not sure where the various IDL parser tools come in to this. There
isn't a conformance class for IDL processors in the spec, and I'm not
sure whether the grammar in the spec being actually parseable is
something that is interesting to demonstrate by having programs that can
do that. At least because we would then need to have some tests for
those programs to show that they are correct.
