Hi Mathias and Martin,
most of you want to find regressions in less than an hour. These tooling
doesn't exists for a complex program like OpenOffice.org. Christoph
wrote that all API-tests will run more than 4 hours. And API testing is
one of the quickest tests which exists.
I want the same as you, but this than we need UNIT tests first for all
implementations. This is the quickest type of test, which could show you
first regressions. Than write API and complex tests for your
implementation, and you are nearly save, that GUI testing with TestTool
will find less errors.
But this doesn't exists in most cases. So we have to live with long
testing phases with TestTool on GUI level.
I do not understand, why you write ever, you want quick results for
your implementation. This is the case. You will get the results quicker,
when the tooling is implemented automatically between development and
QA. If you want to wait every time, until QA rep has the time for your
CWS, it is OK for me. But then you will wait between 5-10 days
currently. And with a higher mount of CWS, which increased in the last
weeks by community CWS, this time will increase.
Thorsten
Mathias Bauer wrote:
Martin Hollmichel wrote:
Do we have some statistics in which areas we have what amount of
regressions ?
For example I would think that regression caused by broken resources
doesn't occur that much any more, are also easy to find by broad
testing. On the other hand I could image that regressions in document
layout do occur much more often and would be reported much more later
than broken resources ?
Exactly this is one of my concerns. The tests that Jogi mentioned are
surely useful to prevent huge disasters but I'm not sure if they are
qualified for testing for the regressions we usually have.
Regression tests IMHO should tackle the areas where we know that
regressions happen more frequently. I hope it's undisputed that we can
only execute a very limited set of tests and so it should be our
interest to use those test that are able to catch the biggest fish possible.
From the Writer's perspective interesting areas could be text formatting
and layout, undo/redo, redlining, numbering and some more. Regressions
in resource files or the complete inability to start an application
rarely happen and the latter IMHO is already covered by the smoketest.
BTW: extending the smoketest would be better than adding another tool
for testing.
I'm also not concerned about regressions in the "main features" of the
applications as usually regressions in these areas are discoverd pretty
fast. I'm concerned about regressions that are not so obvious and
usually are found too late. My impression was that this was what created
the idea to have more regression testing, so we should put our focus on
them.
These regressions are not only crashes, often they are more or less
subtile formatting or functionality differences where I wonder how the
provided tests could discover them.
Tests are software and we have learned that for good software one needs
to understand the requirements first and then select the best design to
implement them. I know that we are not perfect in following that but
IMHO we should at least try.
As my requirement would be to find as much regressions as possible with
as less effort as possible I would like to :
- identify the areas of regressions (we should have some data about it)
- think about what kind of tests are best suited to find them earlier
- try to implement such tests so that they are reliable and as easy and
fast to execute as possible
- help developers to find out fast and easily which test should be
applied when
As obviously noone else is interested in a content oriented discussion I
think I will quit and wait what we can test with the provided tests and
if this is what we need.
Until then my impression is that continuing this discussion will be less
valuable then the waste air our computers create while we are writing
our mails. :-)
'nuff said,
Mathias
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
