Hi
I'm wondering if we're not making it too hard by staying in the
existing model. If we want to follow the (old) junit model, we
should recognize that it had 3 levels: compilation error, test
failure and test error.
How about:
StepValidationException
raised by a step when the validation fails, e.g. attribute
missing or with an invalid value, previous page missing
or of the wrong type, ...
This is the nearest we could come to a compilation error.
StepFailedException
raised by a step when an explicit check fails.
This would mostly be the case for the verifyXXX steps.
StepExecutionException
for all the other cases, in particular as a wrapper for
exceptions thrown by other code, be it html unit, jaxen,
javascript or groovy.
<not> would in this case only react to StepFailedException.
Further, new steps may be needed to check for http return code,
or socket timeout, something like:
<verifyReturnCode code="404">
<invoke label="dead link" />
</verifyReturnCode>
This would for example
pass if he invoke results in a 404,
throw a SFE if the invoke doesn't result in a 404,
throw a SVE if code is not a 3-digit number,
throw SEE if there is timeout.
Best
dna
On 27 févr. 06, at 12:24, Paul King wrote:
Another case to consider. ScriptStep currently throw SFE if an
assert fails - applies at least to using groovy within scriptStep.
I believe the Groovy step throws a SEE in this circumstance.
Is everyone happy to change groovy step behaviour to SFE?
Rationale: assert is normally like a verification and I guess
you could also throw a SEE in your script (I haven't tried this
in groovy yet??) to get the other behaviour.
Paul.
Paul King wrote:
I am not sure there is a fully deterministic answer but that doesn't
mean we can't try. I'll give two definitions:
[1] (close to Marc's original rules):
StepFailedException: Should indicate that the application under
test (AUT)
worked with no unexpected errors but didn't meet the expectations
outlined
in the test. (Either the AUT has a bug or AUT is working but while
I have
correctly set up the test I have supplied the wrong expectations).
If it is not possible to be sure that this holds but we expect it
holds > 80% of the time then use this exception.
StepExecutionException: All other cases
<not> only applies to StepFailedException as per current behaviour.
[2] Return SFE whenever it might make sense to use <not> to test
that an AUT doesn't do something.
Application of these rules below.
Marc Guillemot wrote:
let's start making it clearer (Paul, your explaination is
interesting but not
deterministic ;-()
What should be thrown when:
- step has invalid parameters
[1] and [2] indicate SEE: Test can't proceed until params fixed,
nothing can be
said about whether test expectations are met or not
- step "context" is not present (ex: no current page for a step
working on a page)
[1] and [2] indicate SEE: usually indicates a bug in the test
- http error code is received
Using [2] we would go with SFE. Using [1] it could be SFE or SEE.
A normal scenario would be we are trying to test a page
and there is an environmental issue - we forget to start the test
server.
This should be SEE. Another scenario is we are trying to ensure
that we get
a permission denied when we visit a protected page without logging
in.
Or perhaps we used to have a link to something on every page but that
link has changed and now we want to check that no page has the
legacy link.
In both these case being able to use <not> would be handy.
We could make this SEE and the workaround for the other scenario
would
be to set ThrowExceptionOnFailingStatusCode to false and check the
content
on the error pages. I guess the other alternative is to have similar
booleans (like ThrowExceptionOnFailingStatusCode) ourselves which
work
at the WebTest level rather than HtmlUnit level. We could set the
flag
to get either SEE or SFE invoked.
- JavaScript error occured
If the JavaScript is caused by a parse error because my JavaScript
is not well formed then I would expect SEE. There possibly could be
some cases where SFE might be useful but I can't think of any.
I would probably always set this one to SEE. If we find a case where
SFE is desirable we could always make a flag which you could set to
get the SFE behaviour. So, [1] and [2] probably say SEE.
- in a verification, the actual value and the expected one differ
[1] and [2] indicate probably always SFE.
Which of these cases should be "catched" by a wrapping <not>?
only SFE
Socket timeout is a bit harder. Using [1] I would probably set it
to SEE.
Most of the time, some part of my environment is probably not set
up if I receive this error but sometimes it means my AUT is running
too slow and needs to be fixed. <not> could be useful here but it
isn't real clear. [2] might suggest SFE but it is a weak case.
I think we need [3]:
Use rule [2] but in cases where it isn't always useful to have SFE
we supply a flag which lets the user decide between SFE and SEE
with the default probably being SEE.
I think that we should make at least a distinction between
invalid parameters
and the other cases (invalid parameters can be compared to a
compilation error
in junit). I don't have any problem to put all the other cases in
the same category.
Like Denis, I think that the invalid parameters case should not
be catched by a
<not>.
Marc.
_______________________________________________
WebTest mailing list
[email protected]
http://lists.canoo.com/mailman/listinfo/webtest
--
Accept without pride, relinquish without a struggle.
-- Marcus Aurelius Antonius, Meditations 8.33
_______________________________________________
WebTest mailing list
[email protected]
http://lists.canoo.com/mailman/listinfo/webtest
