Hi Frank at all,
Frank Schönheit - Sun Microsystems Germany wrote:
Hi Oliver,
Do you have such tests? Those that are able to find more regressions
than they overlook?
Hmm? How do you measure *this*? If they find regressions, that's good.
Every test will overlook some regressions.
Those that run only several hours not weeks like the
current ones?
That's important indeed. If I have to wait several days betwen finishing
my builds and passing the CWS to QA, just "because" of the test, this
would certainly be a serious hurdle.
Why it is a problem to wait 1-3 days before you set a CWS in state
'ready for QA'? Mostly it doesn't hinder you to work on another CWS.
So the time for running the tests is not so important for me.
If is more important, that the tests do not run on machines from the
developers. Only then it will not hinder Sun internally or developers
in the community to work on other projects. We need a tinderbox with a
defined environment to build the install sets, install it, run smoke
tests and the TestTool tests. Then the time for running the tests isn't
important.
Additionally, I'd like to raise the wish for "deterministic" tests. Our
current testtool scripts not always fulfull this - the QA guys using it
all day long will tell you there are tests which sometimes fail, but
succeed the next time you run them. That'd be inacceptable for required
tests.
You are right, that TestTool do not run without problems. But the guys
are working on making more "deterministic".
But this shouldn't be a problem at all. Because the test scripts with
known problems will not used for the test runs in the first steps. Only
test cases with run "deterministic" will be checked in for this tooling.
And, while I am at bashing the testtool :) (no pun intended):
Tests are only useful if you are able to track down the problem with a
reasonable effort. If the outcome of the test is "foo failed", but it
takes you hours to just identify what foo this is, then the test is
useless. Everybody who ever tried finding an error reported by the
testtool knows that this is ... strictly important.
The problem is the debugging in TestTool. But often thrown errors are
easy to understand and easy to reproduce. But you are right, some cases
are more tricky. But often the users of the TestTool do not understand
the error message 'button XYZ is disabled, when doing YXZ => Bug'. :-(
Thorsten
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
