Hi André,
André Schnabel wrote:
Hi Ingrid,
Ingrid Halama schrieb:
Mathias Bauer wrote:
The problem is that the usual test runs obviously don't find the bugs
That is not obvious to me. Too often the mandatory tests haven't been
run. And if tests do not find an important problem, hey then the
tests should be improved.
See my post at d...@qa (
http://qa.openoffice.org/servlets/ReadMsg?list=dev&msgNo=11964 ).
Our toolset for automated tests even reports "all ok" if the testttool
correctly identified a stopper bug. Many other bugs cannot be found by
the testtool (e.g. "visual problems" like issue 99662 ).
I think, the benefits of automated testing are overstimated (or the
costs are underestimated).
Convwatch is a tool that does test visual problems quite nicely already.
It loads a set of documents with a CWS and creates 'screenshots' (
postscript-printouts ). The same is done with the corresponding master
on which the CWS is based upon. Then the 'screenshots' from the CWS are
compared with the 'screenshots' from the master and it is detected
automatically if the CWS would introduce visual changes. With this test
and a good collection of test documents you can test a major part of the
office automatically: 'load and display'. I would like to see a simple
button within each CWS html frontend to start this test and a simple
color indicator that says whether the test has been passed successfully.
that now bite us, most of them have been found by users or testers
*working* with the program. Adding more CWS test runs and so shortening
the time for real-life testing will not help us but make things worse.
I don't agree. Preventing the integration of bugs earlier in the
production phase especially before the integration into the master
trunk would give us much more freedom. Now we always need to react on
show stoppers and react and react and uh then the release time line
is on risk. All that, because the bugs are already in the product. If
you instead detect the bugs before they are integrated into the
product you can keep cool, refuse the bad CWS and thus not the
release is on risk but only the single bad CWS.
The point is *if* you detect the bugs in the CWS. At the moment we
obviuosly do not identify enough critical issues while CWS testing
(even if the mandatory tests are done).
....
I assume you are talking about the VCL-Testtool-tests. But we have more
tests - we have UNO-API-tests, we have performancetests and we have
convwatch. And we have even much more VCL-Tests than those that are
mandatory. So there is much room for improvement even without writing
any new test.
I am missing a stimulation for good behaviour in this plans. There
are people who do the feature design, who do the developing work, who
do the testing, who create the automatic test, who do the
documetnation and after all these people have done their work and
lets assume they have done it good and without show stoppers, after
all this there comes someone else and says, oh no, I do not think
that I want to have this for this release, there are other things
that I want to have more and in the sum I guess that it might be to
much for the next release? Where is the stimulation for good
behaviour here?
The problem is, that we first need to prove that "good behaviour" is
really helpfull and prevents critical bugs in the master. I'm all for
promoting good behaviour - and in most cases I would like to see more
people who follow the rules (like publishing specs correctly at the
specs website).
Well so lets try convwatch as mandatory test for the next release to
prove whether it helps.
But we must be allowed to review our processes and identify the parts
that are not so helpfull.
Fully agreed. In addition it must be allowed to identify parts where
helpful things are missing.
There is none, instead it is a stimulation to push in the changes
quickly into the product and skip careful testing.
If this is some stimulation: thanks for all the chart specs that are
correctly linked at the specs website. These are very helpfull to
speed up testing, get an idea about the new functionality and in many
cases arethe only resource to get our translations correct.
Thanks André! ;-) . It is nice to hear that that part of the work has a
real benefit.
My idea is that the integration of a CWS could be and should be the
gratification and stimulation for "good behavior".
Ingrid
André
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
