Tim Ellison wrote:
Geir Magnusson Jr wrote:
Thanks for bringing this up. This has been on my to-do list to talk about.
inline
Vladimir Strigun wrote:
We already discussed several questions about tests. But I'd like to
open a new topic about regression test suite. Some API bugs have been
fixed (or patch suggested) and I think it will be useful to have
regression tests for all these bugs to avoid them reappearing in the
future.
I think the natural place for the regression tests should be together
with the unit tests. I am not sure, however, if the regression tests
should be marked explicitly or differ from the unit tests in any other
specific way.
What do you think about it? Any preferences?
I think that putting them in a parallel tree is worth considering, just
for the sake of organization, as long as this is transparent to the
tools a developer/user would use to do run the tests. I do think that
"regression test" is a broad term, and we might want to harvest things
out of regression tests for where our unit tests fell short.
Aren't we getting a bit ahead of ourselves here, the existing tests
hardly need organizing (all three of them!) -- but let's go mad and
assume we have lots of tests housed in our repository.
Not really, as there are quite a few in the intel security contribution.
(3? That's pathetic!)
What is the useful distinction for regression tests being kept separate?
I can see that you may distinguish unit and 'system-level' tests just
because of the difference in frameworks etc. required, but why do I care
if the test was written due to a JIRA issue or test-based development or
someone who get's kicks out of writing tests to break the code?
Well, it's been my experience that unit tests generally are fairly
straightforward and isolated, where a regression test used to
demonstrate a bug might be fairly complicated, incorporating more than
the single unit you are trying to test/fix/debug/whatever.
I also believe we should be encourging bug reporters to submit a test
that demonstrates the bug. We'd acquire a good set of tests at that point.
Agreed, a simple failing test with a description of expected results is
the ideal way to report a bug. In particular, people are not expected
to suggest a patch when they raise issues, though they are of course
welcome to do so.
s/welcome/strongly encouraged/ :)
I was going to take a hard look at the build/test framework this
week(end). Might as well start now. Currently, it's JUnit. I'm
wondering if anyone would consider switching to TestNG. I've never used
it, but it's written by a guy I really respect and the point was to
address shortcomings he found in JUnit...
I've never used TestNG, so have no opinion at the moment. What's so
good about it?
I'll report back.
Since we have no current setup for tests, I'll assume some leeway to
take a run at it as there are no existing toes to step on. First task
will be to get your 3 [pathetic! :) ] unit tests running
geir
Regards,
Tim
geir
Thank you,
Vladimir Strigun, Intel Middleware Products Division.
