Re: C/C++ White-Box Unit Testing and Test::More
On 26 Jun 2004, at 12:51, Fergal Daly wrote:
On Fri, Jun 25, 2004 at 10:13:52PM +0100, Adrian Howard wrote:
[snip]
What xUnit gives you is a little bit more infrastructure to make these
sorts of task easier.
That's fair enough but that infrastructure is just extra baggage in
some
cases.
True. The nice thing about Perl's framework is that we can avoid it
when we don't need it.
Although the extra baggage that people are complaining about is often
because of the verboseness of a language's OO code rather than xUnit
itself.
Actually, just after I wrote the email, I realised I had used xUnit
before, in Delphi. With DUnit, testing a single class takes a
phenomenal amount of boilerplate code and I guess that's why I'd
blocked it from my memory :).
I think DUnit would be an example of exactly what I'm talking about.
For example the following DUnit
unit Project1TestCases;
interface
uses
TestFrameWork;
type
TTestCaseFirst = class(TTestCase)
published
procedure TestFirst;
end;
implementation
procedure TTestCaseFirst.TestFirst;
begin
Check(1 + 1 = 2, 'Catastrophic arithmetic failure!');
end;
initialization
TestFramework.RegisterTest(TTestCaseFirst.Suite);
end.
would be written in Test::Class as:
package Project1TestCases;
use base qw( Test::Class );
use Test::More;
sub catastrophic_arithmetic_failure : Test { is 1+1, 2 };
or, since we don't need any xUnit magic here, as plain old:
use Test::More tests = 1;
is 1+1, 2, 'catastrophic arithmetic failure';
This is why I like Perl!
As you say, we already have a good chunk of xUnit style with
Test::Harness, with each .t file corresponding somewhat to a suite
but without the nestability.
You could also compare each .t file to a test method, since the tests
in different .t files tend to be isolated from each other.
I think the baggage only pays for itself when you end up doing a lot of
inheriting between test classes,
For me the baggage pays off as soon as test isolation becomes a factor.
Having setup/teardown to help create test fixtures saves me typing.
YMMV.
Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 01:07:36PM -0400, Andrew Pimlott wrote: You are also circumventing the isolation part of the xUnit model, because you don't get setup/teardown for each test data. Possibly you don't care about that in this case, but if you did, you wouldn't be able to do the above, so I consider this another limitation. I can happily call the setup/teardown methods inside the loop if I need to. They're just normal methods. So to me, the above code is basically Test::More style, just using methods for organization. *That* code yes. But it was your example ... On that note, it seems that for some people the appeal of xUnit-style testing is less the model above (failure == exception, isolation), and more the modularity and reuse you can get by writing your tests with objects. failure = exception, per se, no. However, I can throw exceptions if I want and Test::Class will DTRT, so that's another advantage. isolation, yes. But I don't think you mean the same thing by this as I do. That said, I tend much more towards startup/shutdown that setup/teardown. Moduluarity and reuse: who doesn't love that? :) Tony
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 10:13:52PM +0100, Adrian Howard wrote: On 25 Jun 2004, at 16:51, Fergal Daly wrote: [snip] NB: I haven't used xUnit style testing so I could be completely off the mark but some (not all) of these benefits seem to be available in T::M land. Just so I'm clear - I'm /not/ saying any of this is impossible with T::M and friends. That's obviously silly since you can build an xUnit framework with Test::Builder and friends. What xUnit gives you is a little bit more infrastructure to make these sorts of task easier. That's fair enough but that infrastructure is just extra baggage in some cases. Actually, just after I wrote the email, I realised I had used xUnit before, in Delphi. With DUnit, testing a single class takes a phenomenal amount of boilerplate code and I guess that's why I'd blocked it from my memory :). As you say, we already have a good chunk of xUnit style with Test::Harness, with each .t file corresponding somewhat to a suite but without the nestability. I think the baggage only pays for itself when you end up doing a lot of inheriting between test classes, F
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 02:18:49PM -0500, Andy Lester wrote:
On Fri, Jun 25, 2004 at 04:51:29PM +0100, Fergal Daly ([EMAIL PROTECTED]) wrote:
* I never have to type repetitive tests like
isa_ok Foo-new(), 'Foo'
again because it's handled by a base class that all my test classes
inherit from.
Repetition is good. I feel very strongly that you should be checking
your constructor results in every single test, and checked against
literals, not variables.
my $foo = My::Foo-new();
isa_ok( $foo, 'My::Foo' );
# and then use it.
#
# Later on...
my $foo = My::Foo-new( bar = 14, bat = \$wango );
isa_ok( $foo, 'My::Foo' );
The more checks you have, the better. Sure, the first isa_ok
technically covers the constructor, but why not check after EVERY
constructor? The 2nd example is really an entirely different test.
@_ solves that.
sub constructor_ok
{
my $class = shift;
isa_ok($class-new(@_), $class);
}
I don't think xUnit style makes it any easier to run the same test with many
different inputs,
F
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 07:13:08PM -0400, Andrew Pimlott wrote: But (I thought) the idea was that every test needs the same setup. If they're all in one method, they won't get that. How's that? Also, if you add lots of tests in a single method, (again as I understand) they will stop after the first failure, which is not ideal if the rest of the tests can still run. That's a feature. If they can still run, move them to a different method. You can group dependant tests into one method and have the method return or die at any point... As above, they won't all get the setup/teardown. I really don't understand what you're getting at here. Tony
Re: C/C++ White-Box Unit Testing and Test::More
On 24 Jun 2004, at 20:19, Andrew Pimlott wrote:
On Thu, Jun 24, 2004 at 05:08:44PM +0100, Adrian Howard wrote:
Where xUnit wins for me are in the normal places where OO is useful
(abstraction, reuse, revealing intention, etc.).
Since you've thought about this, and obviously don't believe it's OO
so
it's better, I'd be interested in seeing an example if you have one in
mind.
Off the top of my head.
* I never have to type repetitive tests like
isa_ok Foo-new(), 'Foo'
again because it's handled by a base class that all my test classes
inherit from.
* I can create units of testing that can be reused multiple times. If I
have an Iterator interface I can write a test suite for it once and
reuse it any class that implements the Iterator interface.
* I have conception level available higher than individual tests (in
T::M land) or asserts (in xUnit land). I can say something like:
sub addition_is_commutative : Test {
is 10 + 5, 15;
is 5 + 10, 15;
};
and talk about addition_is_commutative test as a concept separate from
the tests/assertions that implement it. I can easily move test methods
around as I refactor without having to worry about it breaking some
other part of the test suite.
* The setup/teardown methods provide an infrastructure for creating
test fixtures and isolating tests, which can often save typing and
speed everything up considerably.
* Need to check that a class invariant still holds after each test?
Chuck it in a teardown method.
Cheers,
Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 07:35:26AM +0100, Tony Bowden wrote:
On Thu, Jun 24, 2004 at 07:13:08PM -0400, Andrew Pimlott wrote:
But (I thought) the idea was that every test needs the same setup. If
they're all in one method, they won't get that.
How's that?
I thought the isolation principle that people were talking about is
that before every test, a setup method is called, and after every test
a teardown is called, automatically by the test harness. This
seems to require one method == one test.
Also, if you add lots of tests in a single method, (again as I understand)
they will stop after the first failure, which is not ideal if the rest of
the tests can still run.
That's a feature. If they can still run, move them to a different
method.
I was responding to your suggestion to put all the tests in one method
if they are just parametrized by data. How do you suggest writing the
equivalent of
foreach (@test_data) {
is(my_func($_-{input}), $_-{output}, $_-{test_name});
}
in xUnit style, such that the first failure does not cause the rest not
to run?
Andrew
Re: C/C++ White-Box Unit Testing and Test::More
On 24 Jun 2004, at 21:41, Ovid wrote: [snip] I also like the thought of inheriting tests, but I know not everyone is fond of this idea. There was a moderately interesting discussion about this on Perlmonks: http://www.perlmonks.org/index.pl?node_id=294571 [snip] Yeah, I meant to contribute to that but never got the spare tuits. I tend to use them when I have an abstract interface that several different classes are implementing. Seems a bad idea to either ignore testing functionality that's being changed or waste time reimplementing the basically the same code. Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 11:10:19AM -0400, Andrew Pimlott wrote:
I thought the isolation principle that people were talking about is
that before every test, a setup method is called, and after every test
a teardown is called, automatically by the test harness. This
seems to require one method == one test.
That depends on what you class as a test for this principle. A test
(as in a test method) can contain lots of tests (as in Test::More
function calls). There's no problem with having lots of 'asserts' in one
test method.
I was responding to your suggestion to put all the tests in one method
if they are just parametrized by data. How do you suggest writing the
equivalent of
foreach (@test_data) {
is(my_func($_-{input}), $_-{output}, $_-{test_name});
}
in xUnit style, such that the first failure does not cause the rest not
to run?
erm,
sub test_my_func : Test(no_plan) {
my $self = shift;
my @test_data = get_test_data_from_somewhere();
$self-num_tests(scalar @test_data);
foreach (@test_data) {
is(my_func($_-{input}), $_-{output}, $_-{test_name});
}
}
What am I missing?
Tony
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 04:05:09PM +0100, Adrian Howard wrote:
On 24 Jun 2004, at 20:19, Andrew Pimlott wrote:
On Thu, Jun 24, 2004 at 05:08:44PM +0100, Adrian Howard wrote:
Where xUnit wins for me are in the normal places where OO is useful
(abstraction, reuse, revealing intention, etc.).
Since you've thought about this, and obviously don't believe it's OO
so
it's better, I'd be interested in seeing an example if you have one in
mind.
NB: I haven't used xUnit style testing so I could be completely off the mark
but some (not all) of these benefits seem to be available in T::M land.
Off the top of my head.
* I never have to type repetitive tests like
isa_ok Foo-new(), 'Foo'
again because it's handled by a base class that all my test classes
inherit from.
sub constructor_ok
{
my $class = shift;
isa_ok $class-new, $class;
}
* I can create units of testing that can be reused multiple times. If I
have an Iterator interface I can write a test suite for it once and
reuse it any class that implements the Iterator interface.
What's stopping you doing this in T::M,
sub test_iterator
{
my $iterator = shift;
# test various things about $iterator.
}
* I have conception level available higher than individual tests (in
T::M land) or asserts (in xUnit land). I can say something like:
sub addition_is_commutative : Test {
is 10 + 5, 15;
is 5 + 10, 15;
};
and talk about addition_is_commutative test as a concept separate from
the tests/assertions that implement it. I can easily move test methods
around as I refactor without having to worry about it breaking some
other part of the test suite.
I don't get this. What is the difference between having this as a method vs
as a sub?
* The setup/teardown methods provide an infrastructure for creating
test fixtures and isolating tests, which can often save typing and
speed everything up considerably.
* Need to check that a class invariant still holds after each test?
Chuck it in a teardown method.
The T::M land you could put your setup and teardown in modules and call them
before and after. Then if they're named consistently you could automate that
at which point, you'd almost have xUnit. So xUnit seems to win here for sure,
F
Re: C/C++ White-Box Unit Testing and Test::More
On 25 Jun 2004, at 16:10, Andrew Pimlott wrote: [snip] I thought the isolation principle that people were talking about is that before every test, a setup method is called, and after every test a teardown is called, automatically by the test harness. This seems to require one method == one test. It doesn't. I think there's a confusion of vocabulary. In the Perl world 'test' refers to something like 'is' and 'ok' from Test::More and friends. We're interested in the number of successful tests. In the xUnit world these are called assertions, not a tests, and in general we're /not/ concerned with the number of assertions that succeed. In the xUnit world a /test/ is a method with one or more assertions that checks one particular bit of behaviour in the code. A failed test is a method where one assertion failed. A passed test is a method where all assertions succeed. From this perspective it makes sense to abort after the first assertion has failed - since the 'test' has failed. Think of it like logical '' short circuiting, or maybe like SKIP blocks (there's no point doing the other assertions because the thing we're testing has already failed). Test isolation is the idea each test (not assertion) can run independently from every other. This means when we have a failure we can quickly focus in on exactly what caused the problem. We can be confidant that four test failures indicates four separate problems, not a cascade of failure within the test suite itself. There is a school of thought that one-assertion-per-test is a good goal to aim for, but not everybody agrees. For some discussion see: - http://www.testdriven.com/modules/newbb/viewtopic.php? viewmode=flattopic_id=363forum=6 - http://www.artima.com/weblogs/viewpost.jsp?thread=35578 Hopefully this makes some vague sort of sense. Cheers, Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 04:41:15PM +0100, Tony Bowden wrote:
On Fri, Jun 25, 2004 at 11:10:19AM -0400, Andrew Pimlott wrote:
I was responding to your suggestion to put all the tests in one method
if they are just parametrized by data. How do you suggest writing the
equivalent of
foreach (@test_data) {
is(my_func($_-{input}), $_-{output}, $_-{test_name});
}
in xUnit style, such that the first failure does not cause the rest not
to run?
erm,
sub test_my_func : Test(no_plan) {
my $self = shift;
my @test_data = get_test_data_from_somewhere();
$self-num_tests(scalar @test_data);
foreach (@test_data) {
is(my_func($_-{input}), $_-{output}, $_-{test_name});
}
}
Now I'm confused too. None of the Test::Unit examples I've seen use
is, they use some form of assert. Is this the Test::More is, or
something else? Doesn't an xUnit-style test need to throw an exception
to signal failure? Does your is do that? If so, no test data after
the first failure will be tried. If not, how does this work?
If this style is described somewhere and I just missed it, point me the
way. My Test::Unit doesn't use the : Test attribute, so maybe I have
old or wrong code.
Andrew
Re: C/C++ White-Box Unit Testing and Test::More
--- Andrew Pimlott [EMAIL PROTECTED] wrote: Now I'm confused too. None of the Test::Unit examples I've seen use is, they use some form of assert. You were looking at Test::Class code, not Test::Unit code. Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
Re: C/C++ White-Box Unit Testing and Test::More
Ok, now that I understand what library you're using ...
On Fri, Jun 25, 2004 at 04:41:15PM +0100, Tony Bowden wrote:
On Fri, Jun 25, 2004 at 11:10:19AM -0400, Andrew Pimlott wrote:
I was responding to your suggestion to put all the tests in one method
if they are just parametrized by data. How do you suggest writing the
equivalent of
foreach (@test_data) {
is(my_func($_-{input}), $_-{output}, $_-{test_name});
}
in xUnit style, such that the first failure does not cause the rest not
to run?
erm,
sub test_my_func : Test(no_plan) {
my $self = shift;
my @test_data = get_test_data_from_somewhere();
$self-num_tests(scalar @test_data);
foreach (@test_data) {
is(my_func($_-{input}), $_-{output}, $_-{test_name});
}
}
By not using the failure == exception part of the xUnit model, you avoid
the issue I was getting at. Obviously, the above would stop after the
first test data that provokes a failure if you were using that model,
which is why I consider it limiting.
You are also circumventing the isolation part of the xUnit model,
because you don't get setup/teardown for each test data. Possibly you
don't care about that in this case, but if you did, you wouldn't be able
to do the above, so I consider this another limitation.
So to me, the above code is basically Test::More style, just using
methods for organization. On that note, it seems that for some people
the appeal of xUnit-style testing is less the model above (failure ==
exception, isolation), and more the modularity and reuse you can get by
writing your tests with objects. On this point there's not much to
argue about--if that helps you write better tests, great!
Which leaves me wondering if anyone defends Test::Unit per se. :-)
Andrew
Re: C/C++ White-Box Unit Testing and Test::More
On 24 Jun 2004, at 21:10, Tony Bowden wrote: On Thu, Jun 24, 2004 at 02:59:30PM -0400, Andrew Pimlott wrote: I see this more as a limitation than a feature. It seems to mean that - You need to use the same setup/teardown for all your tests. Those that need different things aren't testing the same thing and should move to a different class. Yup. This misunderstanding seems to be a common one. Novice xUnit users often think that there should be a single test class for every class being tested. Sometimes this can work; when you don't need text fixtures or where a single set of test fixtures can cover all of a classes functionality. However many situations require multiple classes each with their own set of fixtures and test behaviour. Cheers, Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 04:51:29PM +0100, Fergal Daly ([EMAIL PROTECTED]) wrote: * I never have to type repetitive tests like isa_ok Foo-new(), 'Foo' again because it's handled by a base class that all my test classes inherit from. Repetition is good. I feel very strongly that you should be checking your constructor results in every single test, and checked against literals, not variables. my $foo = My::Foo-new(); isa_ok( $foo, 'My::Foo' ); # and then use it. # # Later on... my $foo = My::Foo-new( bar = 14, bat = \$wango ); isa_ok( $foo, 'My::Foo' ); The more checks you have, the better. Sure, the first isa_ok technically covers the constructor, but why not check after EVERY constructor? The 2nd example is really an entirely different test. Tests are all about quantity. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 02:18:49PM -0500, Andy Lester wrote: Tests are all about quantity. I always thought that tests were about malice: I bet the programmer didn't think of this... What happens if I just do this... Mmm, I wonder if it covers this corner case? Eat pathological data and die! Surrender feeble routine. You are no match for the weaponry of my tests! Then again, I do like having lots of tests. And parameterising tests so that I can feed the same basic test lots of different permutations of the data. [To attempt to send the most variations of malice, in the hope that one of them knocks out the routine and finds a bug] Nicholas Clark
Re: C/C++ White-Box Unit Testing and Test::More
On 25 Jun 2004, at 20:18, Andy Lester wrote: Repetition is good. I feel very strongly that you should be checking your constructor results in every single test, and checked against literals, not variables. I'm not complaining about repetitive tests, and I agree with what you said about testing constructor results if it's something that can reasonably fail. I'm complaining about /typing/ repetitive tests. Why the heck should I have to type the same code in twice if I can get the computer to do it for me. Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On 24 Jun 2004, at 19:59, Andrew Pimlott wrote:
[snip]
- You don't have much control (correct me if I'm wrong) about the order
of tests, or the relationship between tests, eg you can't say if
this
test fails, skip these others. This is straightforward in
Test::More's simple procedural style.
[snip]
This is probably due to the same test/assertion confusion - but the
whole /point/ of xUnit is that the test order shouldn't matter :-)
Test isolation == good!
If I need isolation, why can't I just ask for it directly?
with_test_setup {
run_tests();
}
You can.
But then the test writer/reader has to code/understand the
with_test_setup() code for each test script, and the structure of each
of those routines is pretty much the same.
So you might decide to generalise it and have a standard mechanism for
you to plug in your start up code.
Then you discover that some of your tests need to clean up after
themselves so you add another slot to run after the tests.
Then you might find that you often have common sets of tests that are
run in different situations, so it would be nice if you could package
them up and use them as a unit. So you add some infrastructure to
support it.
Well done - you've implemented an xUnit framework!
(or rather provided the few elements of an xUnit framework not already
supplied by Test::Harness, Test::Builder and friends).
These sorts of requirements are exactly what made me build Test::Class
- adding the bits of xUnit not already in the Perl testing framework.
I didn't start out to write an xUnit framework. What I did was refactor
several large and slow test suites and the very xUnit-ish Test::Class
just fell out.
I don't really think of xUnit as a competitor to Perl's normal testing
infrastructure - it's more of a superset. The fact that it's so easy to
layer the missing bits of xUnit on top of Perl's standard mechanisms
shows that.
[snip]
Even better would be to put Test::Builder in skip mode, where it
skips
automatically whenever a test fails:
skip_mode {
ok(something);
is(this, that);
}
sub skip_mode () {
my $test_sub = shift;
my $old_ok = \Test::Builder::ok;
my $test_passed=1;
local $Test::Builder::Level = $Test::Builder::Level + 1;
no warnings;
local *Test::Builder::ok = sub {
die unless $test_passed = $old_ok-(@_);
};
eval { $test_sub-() };
die $@ if $@ $test_passed;
};
:-)
[snip]
Every time I hear about xUnit, I figure there must be something other
than setup and teardown in its favor. If that's all there is, I'm
not
sold.
You have to remember that xUnit isn't just setup/teardown routines.
It's all the rest too - assertions, a test runner, test suites, OO
reuse framework, etc. Most of which we already have with Test::Harness,
Test:Builder and friends.
When people speak about the advantages of xUnit, they're mostly talking
about the advantages of having a standard testing infrastructure.
Cheers,
Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On 25 Jun 2004, at 16:51, Fergal Daly wrote:
[snip]
NB: I haven't used xUnit style testing so I could be completely off
the mark
but some (not all) of these benefits seem to be available in T::M land.
Just so I'm clear - I'm /not/ saying any of this is impossible with
T::M and friends. That's obviously silly since you can build an xUnit
framework with Test::Builder and friends.
What xUnit gives you is a little bit more infrastructure to make these
sorts of task easier.
Off the top of my head.
* I never have to type repetitive tests like
isa_ok Foo-new(), 'Foo'
again because it's handled by a base class that all my test classes
inherit from.
sub constructor_ok
{
my $class = shift;
isa_ok $class-new, $class;
}
But you still have to call constructor_ok().
I can still all my common tests in a base test class and just by
inheriting from it get them all run automagically.
* I can create units of testing that can be reused multiple times. If
I
have an Iterator interface I can write a test suite for it once and
reuse it any class that implements the Iterator interface.
What's stopping you doing this in T::M,
sub test_iterator
{
my $iterator = shift;
# test various things about $iterator.
}
Nothing. But xUnit supplies a little extra bit of magic to mark 'test'
subroutines so they can be automatically found and run with no effort
on your part.
* I have conception level available higher than individual tests (in
T::M land) or asserts (in xUnit land). I can say something like:
sub addition_is_commutative : Test {
is 10 + 5, 15;
is 5 + 10, 15;
};
and talk about addition_is_commutative test as a concept separate from
the tests/assertions that implement it. I can easily move test methods
around as I refactor without having to worry about it breaking some
other part of the test suite.
I don't get this. What is the difference between having this as a
method vs as a sub?
By having it as a test method it gets automatically picked up and run
by the test environment. I can move the test method to another class
and, without altering another line, it will be automatically picked up
and run by the new class.
* The setup/teardown methods provide an infrastructure for creating
test fixtures and isolating tests, which can often save typing and
speed everything up considerably.
* Need to check that a class invariant still holds after each test?
Chuck it in a teardown method.
The T::M land you could put your setup and teardown in modules and
call them before and after. Then if they're named consistently you
could automate that at which point, you'd almost have xUnit. So xUnit
seems to win here for sure,
Exactly! This is all stuff that Test::Builder based modules can do -
xUnit just sprinkles some convenience fairy dust over everything to
make doing it easier :-)
The /nice/ thing about Perl's infrastructure is that we can abandon the
extra infrastructure when we don't need it - making simpler test
scripts more compact and easier to understand.
Cheers,
Adrian
Re: C/C++ White-Box Unit Testing and Test::More
Andrew Pimlott [EMAIL PROTECTED] writes: On Wed, Jun 09, 2004 at 08:18:30AM -0700, Ovid wrote: As for porting a Test::More style framework, I tried doing that with Python and was actually doing well with it, but I was shot down pretty quickly. Any specific reasons why (is the discussion archived)? Is there any site that compares the styles? The xUnit style framework does a much better job of enforcing test isolation than Test::More does (but you have to remember that what Test::More thinks of as a test, xUnit thinks of as an assertion to be used *in* a test).
Re: C/C++ White-Box Unit Testing and Test::More
On 24 Jun 2004, at 07:09, Piers Cawley wrote:
[snip]
The xUnit style framework does a much better job of enforcing test
isolation than Test::More does (but you have to remember that what
Test::More thinks of as a test, xUnit thinks of as an assertion to be
used *in* a test).
To be fair to Test::More and friends xUnit doesn't /enforce/ test
isolation any more than Test::More prevents it. Writing isolated tests
with Test::More is trivial, just do something like:
sub make_fixture {
return ( Cash-new(10), Cash-new(20) );
};
isa_ok( $_, 'Cash' ) foreach make_fixture();
{
my ($ten, $twenty) = make_fixture();
is_deeply $ten + $twenty, Cash-new(30);
};
... etc ...
I had a mild rant about this on the TDD list a few months back. You can
write isolated tests in a procedural style quite easily. You can also
easily write tightly-coupled tests in an xUnit style. It's all reliant
on developer discipline. xUnit provides some infrastructure that helps,
but it doesn't enforce it. Developers do that.
(Apologies for rant. Consider it a symptom of the number of ghastly
xUnit test classes that I've seen with 100 line test methods and no
setup methods.)
Where xUnit wins for me are in the normal places where OO is useful
(abstraction, reuse, revealing intention, etc.). Where xUnit loses are
the times when you don't need it all the extra infrastructure and it
just becomes overhead that gets in the way of understanding the test
suite.
Where the Perl testing framework wins for me:
- it gives me the flexibility to do both procedural and xUnit styles as
I see fit
- it also provides SKIP and TODO tests, which I've not come across
elsewhere. TODO test in particular I find useful for tracking technical
debt
- Test::Harness has a nice ASCII protocol that I can use to feed
non-Perl stuff into the testing framework
Anyway enough rambling ;-)
Adrian
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 07:09:40AM +0100, Piers Cawley wrote:
The xUnit style framework does a much better job of enforcing test
isolation than Test::More does
I see this more as a limitation than a feature. It seems to mean that
- You need to use the same setup/teardown for all your tests.
- You're restricted to one test == one method, so you can't paramatrize
your tests by data (eg, second example below).
- You don't have much control (correct me if I'm wrong) about the order
of tests, or the relationship between tests, eg you can't say if this
test fails, skip these others. This is straightforward in
Test::More's simple procedural style.
If I need isolation, why can't I just ask for it directly?
with_test_setup {
run_tests();
}
and even
foreach (@my_test_data) {
with_test_setup {
run_tests($_);
}
}
(but you have to remember that what
Test::More thinks of as a test, xUnit thinks of as an assertion to be
used *in* a test).
This is the one point of actual difference, but I think we can take care
of it. For starters,
SKIP: {
ok(something) || skip
is(this, that) || skip;
}
Even better would be to put Test::Builder in skip mode, where it skips
automatically whenever a test fails:
skip_mode {
ok(something);
is(this, that);
}
This can't play nice with test counts, but that's a brain-dead idea
anyway.
Every time I hear about xUnit, I figure there must be something other
than setup and teardown in its favor. If that's all there is, I'm not
sold.
Andrew
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 05:08:44PM +0100, Adrian Howard wrote: Where xUnit wins for me are in the normal places where OO is useful (abstraction, reuse, revealing intention, etc.). Since you've thought about this, and obviously don't believe it's OO so it's better, I'd be interested in seeing an example if you have one in mind. Andrew
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, 2004-06-24 at 11:59, Andrew Pimlott wrote: Every time I hear about xUnit, I figure there must be something other than setup and teardown in its favor. If that's all there is, I'm not sold. It's the best option for languages that enforce a nominally pure OO style. (During the tech review of the XP Pocket Guide, Dave Thomas pointed out that many of the XP techniques come from the way Smalltalk forced Ward and Kent to work. xUnit qualifies there.) -- c
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 02:59:30PM -0400, Andrew Pimlott wrote: I see this more as a limitation than a feature. It seems to mean that - You need to use the same setup/teardown for all your tests. Those that need different things aren't testing the same thing and should move to a different class. - You're restricted to one test == one method, so you can't paramatrize your tests by data (eg, second example below). I'm not sure what you mean here. You can have lots of tests in a single method. And subclasses can add their own extra tests to the parent's tests. - You don't have much control (correct me if I'm wrong) about the order of tests, or the relationship between tests, eg you can't say if this test fails, skip these others. This is straightforward in Test::More's simple procedural style. You can group dependant tests into one method and have the method return or die at any point... Every time I hear about xUnit, I figure there must be something other than setup and teardown in its favor. If that's all there is, I'm not sold. The big gain for me with Test::Class is inheritable tests. Subclasses can ensure they still pass all their parent's tests, as well as all of their own, without me having to copy all the tests, or set up a really clumsy testing environment. And of course you get to refactor the test suite quite nicely too so that it's really obvious what each test is doing. If I'm not testing something with a lot of inheritance, then Test::Class loses about 90% of its attraction. Tony
Re: C/C++ White-Box Unit Testing and Test::More
--- Tony Bowden [EMAIL PROTECTED] wrote: The big gain for me with Test::Class is inheritable tests. Subclasses can ensure they still pass all their parent's tests, as well as all of their own, without me having to copy all the tests, or set up a really clumsy testing environment. And of course you get to refactor the test suite quite nicely too so that it's really obvious what each test is doing. I also like the thought of inheriting tests, but I know not everyone is fond of this idea. There was a moderately interesting discussion about this on Perlmonks: http://www.perlmonks.org/index.pl?node_id=294571 Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 09:10:09PM +0100, Tony Bowden wrote: On Thu, Jun 24, 2004 at 02:59:30PM -0400, Andrew Pimlott wrote: I see this more as a limitation than a feature. It seems to mean that - You need to use the same setup/teardown for all your tests. Those that need different things aren't testing the same thing and should move to a different class. What about running the same tests with different sample data (so different setup/teardown)? I suppose you could create a sub-class for each data set, but that seems like busy-work. (Unless you can create all those sub-classes automatically from the data) - You're restricted to one test == one method, so you can't paramatrize your tests by data (eg, second example below). I'm not sure what you mean here. You can have lots of tests in a single method. And subclasses can add their own extra tests to the parent's tests. But (I thought) the idea was that every test needs the same setup. If they're all in one method, they won't get that. Also, if you add lots of tests in a single method, (again as I understand) they will stop after the first failure, which is not ideal if the rest of the tests can still run. - You don't have much control (correct me if I'm wrong) about the order of tests, or the relationship between tests, eg you can't say if this test fails, skip these others. This is straightforward in Test::More's simple procedural style. You can group dependant tests into one method and have the method return or die at any point... As above, they won't all get the setup/teardown. The big gain for me with Test::Class is inheritable tests. I see. Andrew
Re: C/C++ White-Box Unit Testing and Test::More
Ovid wrote: For white box testing C code, I just use assert(). assert() is ok, but ok() is better. :-) I will prolly roll my own custom ok() macro, so instead of: assert(x==y); I can write: ok(x==y, test that x equals y); Writing a lot of tests, I want to be able to easily label each test. (I think I will need an ok() macro rather than a function to get __FILE__ and __LINE__ also for failing tests). /-\ Find local movie times and trailers on Yahoo! Movies. http://au.movies.yahoo.com
Re: C/C++ White-Box Unit Testing and Test::More
Perhaps this will be useful... http://boost.org/libs/test/doc/index.html. -Avram On Jun 8, 2004, at 8:18 PM, Andrew Savige wrote: I am currently using Test::More for my Perl white-box unit tests. I also need to write some C/C++ white-box unit tests and would like to use something similar in spirit to Test::More. Smalltalk's SUnit-style framework has been ported to many languages (JUnit, cppunit, Test::Unit, Test::Class, ...). Has anyone ported a Test::More style framework to C/C++ or Java? How do Perl5/Parrot white-box unit test their C code? /-\ Find local movie times and trailers on Yahoo! Movies. http://au.movies.yahoo.com
Re: C/C++ White-Box Unit Testing and Test::More
--- Andrew Savige [EMAIL PROTECTED] wrote: Has anyone ported a Test::More style framework to C/C++ or Java? How do Perl5/Parrot white-box unit test their C code? For white box testing C code, I just use assert(). As for porting a Test::More style framework, I tried doing that with Python and was actually doing well with it, but I was shot down pretty quickly. xunit style testing, though it has limitations, is very powerful and people are quick to scoff at alternatives (at least from what I've experienced.) Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
Re: C/C++ White-Box Unit Testing and Test::More
On Tue, 2004-06-08 at 20:18, Andrew Savige wrote: I am currently using Test::More for my Perl white-box unit tests. I also need to write some C/C++ white-box unit tests and would like to use something similar in spirit to Test::More. What if you used Inline::C *and* Test::More? If they're white box tests for well-factored code, would it work? -- c
