On Mon, 04 Jul 2011 11:47:22 +0200, Rich Tibbett <ri...@opera.com> wrote:
RFC2119 'Key words for use in RFCs to Indicate Requirement Levels'
defines the keyword 'SHOULD' as:
"This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course."
Generally, I think we can agree that anything less than MUST or MUST NOT
requirements in a spec are pretty useless when it comes to conformance
testing. We try to write specs to these keywords but other keywords tend
to creep in to most specifications.
Hmm. No, I think a spec that uses these correctly has requirements where
there are legitimate reasons not to do the usual thing, but a strong case
for doing it unless there is a clear reason.
We currently define tests in test suites for SHOULD requirements. A
problem occurs when those tests are used to gauge the overall compliance
of an implementation to the full test suite. An implementation could
theoretically be 100% compliant without needing to pass non-MUST and
non-MUST NOT tests.
I don't think so. RFC 2119 is pretty clear about what must and should
mean. As Bjoern said a numerical score for tests is generally not that
useful. But compliance is doing all the things you must, and in practical
terms that can readily be translated to *at least* passing all the test
that test things the spec says must happen.
Perhaps we should introduce 'bonus' points for SHOULD/SHOULD NOT/MAY and
RECOMMENDED tests and not have them contribute to overall compliance
output, thereby allowing implementations to claim 100% compliance to
MUST/MUST NOT tests. An implementation can then optionally collect any
available, optional bonus points as available from requirements marked
up with other keywords.
Wondering if there is any set W3C thinking on this or a way of including
SHOULD tests in test suites but clearly indicating that they are,
basically, optional and do not count towards the overall compliance
score? I couldn't find anything in [1].
I don't think it's a good idea to do numerical scoring of conformance. Yu
would need to be sure that the things being tested are of equal
importance, or work out how to balance the scores. And the cost of that
already outweighs the benefit by so much it seems pointless to go there.
just my 2 cents
chaals
- Rich
[1] http://www.w3.org/TR/test-methodology/
--
Charles McCathieNevile Opera Software, Standards Group
je parle français -- hablo español -- jeg lærer norsk
http://my.opera.com/chaals Try Opera: http://www.opera.com
