(Originally posted to dev and I forgot to cc this list)
Hi Damjan,
On 10/28/20 12:21 AM, Damjan Jovanovic wrote:
On Sat, Oct 24, 2020 at 8:14 PM Carl Marcum <cmar...@apache.org> wrote:
Hi All,
I've been testing builds with the automated BVT and FVT tests lately.
I have a few questions:
1. Is there anything documented about how much coverage these tests
provide vs.functionality?
2. Is there yet a place to list new cases it would good to add test for?
I think there is a lot that could be done in this area to attract new
contributors if we had a place to work from.
Both in documenting and work on some flaky tests that I've run into.
I'm willing to put some effort into this, both organizing and developing.
Hi Carl
Thank you for helping with the tests.
I wrote a good email summarizing the different tests we have some years
ago, please see
https://lists.apache.org/thread.html/bb1851b82ba009d2cefdf5af9997099b6fdfb04bddac3753172f2698%401459253891%40%3Cdev.openoffice.apache.org%3E
Some things changed since then, eg. the smoketest location moved.
There are
a few other emails too, search for them.
I also did some test fixes to bvt/fvt and others at various times. This
year I learned quite a lot about them. For example many tests fail because
they expect features that the .DOC file format cannot provide (such as
strikeout styles unique to .ODT), there were timeouts, registry
modifications had to be enabled to fix a test, confirmation dialogs that
hang the tests, a 2048 byte limit in FreeBSD's "ps" was causing a pid
lookup to fail, java.util.Calendar was being used incorrectly, etc. I
fixed
some of these, but others remain. Understanding why the .DOC tests fail
requires understanding the .DOC file format, so fixing those tests isn't
easy. Look through the Git log, I wrote pretty descriptive commit
messages.
One thing I didn't mention in that email is the thousands of unused tests
we have in main/qadevOOo, but they seem difficult to set up, and use a
custom test framework, not JUnit. They have a complex architecture, with
some code implementing UNO components and some code testing them. There
might even be code missing for some of them. There's a mixture of Java and
StarBasic tests there, with much duplication - were they first written in
one language then semi-ported to another?
Anyway I can't help much at the moment, but good luck and let us know how
it goes.
Damjan
Thanks for the great information and the email link.
I saw some of commits on some of the flaky tests where you added a sleep
before checking the results.
I think there are more of these but I was holding off thinking it would
be low hanging fruit for a new volunteer :)
But I'll take care if it if not.
I liked Patricia's suggestion of a page for this work. I'll try to find
if one was started and if not start one.
I'll look into testing the OOO_SUBSEQUENT_TESTS flag also.
It seems building the bvt/fvt tests using ant require an office to have
been built and use idlc and the uno jars from solver like:
test/smoketestdoc.build.xml imports main/ant.properties which includes
OUTDIR=<my-path-to-source>/openoffice/main/solver/450/unxlngx6.pro
OUTDIR is used in the path to idlc and the build fails if I've cleaned
the office build.
I think it would be good if we could use idlc and uno jars from
whichever office is under test so you don't need a build environment
necessarily but I'm still looking at the implications of that.
Please let me know if you have an opinion on that since you've dug a lot
deeper.
Looking ahead I see new tools like testcontainers [1] you can use in
maven and gradle builds for spinning up and disposing of databases,
servers, and other things for integration tests.
This looks like fertile ground for some work :)
[1] https://www.testcontainers.org/
Thanks,
Carl
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
For additional commands, e-mail: qa-h...@openoffice.apache.org
