Re: C/C++ White-Box Unit Testing and Test::More
Andrew Pimlott [EMAIL PROTECTED] writes: On Wed, Jun 09, 2004 at 08:18:30AM -0700, Ovid wrote: As for porting a Test::More style framework, I tried doing that with Python and was actually doing well with it, but I was shot down pretty quickly. Any specific reasons why (is the discussion archived)? Is there any site that compares the styles? The xUnit style framework does a much better job of enforcing test isolation than Test::More does (but you have to remember that what Test::More thinks of as a test, xUnit thinks of as an assertion to be used *in* a test).
xUnit vs. Test::More
--- Piers Cawley [EMAIL PROTECTED] wrote: The xUnit style framework does a much better job of enforcing test isolation than Test::More does (but you have to remember that what Test::More thinks of as a test, xUnit thinks of as an assertion to be used *in* a test). After working with xUnit style testing for almost a year, I definitely agree with this assessment. However, I think each style has its merits. With xUnit frameworks, each test will set up its data in the database, run through its asserts and then rollback the data. The setup and teardown methods ensure that each test is isolated from all other tests and you thus don't have accidental dependencies. The also impose a considerable overhead. At my current position, we've managed to get our test suite (xUnit style) running fast enough that it takes about 50 minutes to run. At my previous job, a comparable size test suite using Test::More took less than 5 minutes to run (I judge comparable sized by comparing Test::More tests to xUnit asserts.) Between the two, the 50 minute tests suite is probably more robust, but taking 10 times as long to run means that we've been bitten a few times by programmers who are in a hurry and don't run the full suite for a change that can't break anything. The other concern I've had with our style of xUnit testing is that we're testing behavior, but not the actual data. With Test::More, we tested against a copy of the live database (when possible -- but this definitely caused some issues) and we sometimes caught data problems that xUnit style testing seems less likely to catch. The reason for this is quite simple: when you setup the data, you're setting up your *idea* of what the data should be. This might not match what your data actually is. Am I missing something fundamental? The long test times and the lack of real data testing are two weaknesses that I would like to correct in our current system, but I see how beneficial xUnit testing is, so I'm concerned about introducing fixes that weaken our testing strategy. How do others deal with this? Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
Devel::Cover bug
Hi,
I've run into Can't call method add_statement on an undefined value
running Devel::Cover. Apologies if this was reported before, but the
list archive is not searchable. I am using perl 5.8.4 and Devel::Cover 0.46.
To reproduce the bug, run
/opt/perl/bin/perl -MDevel::Cover -MFooBar -e FooBar-new-test_foo
The files FooBar.pm and CodeRef.pm are attached.
The bug occurs while calling $sub in CodeRef::to_string. It is probably
related to using B::Deparse, but I was not able to minimize the code
further and still reproduce the error.
Simon
--
Simon (Vsevolod ILyushchenko) [EMAIL PROTECTED]
http://www.simonf.com
Terrorism is a tactic and so to declare war on terrorism
is equivalent to Roosevelt's declaring war on blitzkrieg.
Zbigniew Brzezinski, U.S. national security advisor, 1977-81
package FooBar;
sub new {
my $proto = shift;
my $self = {};
bless $self, $proto;
return $self;
}
use CodeRef;
my $sub = sub {
my $str1 = shift;
};
my $assertion = CodeRef-new($sub);
*{FooBar::test_foo} =
sub {
my $self = shift;
$assertion-do_assertion(@_);
};
1;
package CodeRef;
use strict;
sub new {
my $class = shift;
my $code = shift;
bless \$code = $class;
}
sub do_assertion {
my $self = shift;
$self-to_string(aaa);
}
sub to_string {
my $self = shift;
require B::Deparse;
my $deparser ||= B::Deparse-new(-p);
$deparser-coderef2text($$self);
}
1;
Re: xUnit vs. Test::More
The other concern I've had with our style of xUnit testing is that we're testing behavior, but not the actual data. With Test::More, we tested against a copy of the live database (when possible -- but this definitely caused some issues) and we sometimes caught data problems that xUnit style testing seems less likely to catch. The reason for this is quite simple: when you setup the data, you're setting up your *idea* of what the data should be. This might not match what your data actually is. I take the approach that these are fundamentally two different things. first, as a developer you need to code against what your idea of the data is, taking the known data gives expected results approach to your tests. a good example is a subroutine that uses a regex to parse the data - the best you can do while developing the routine is to make sure your regex handles the conditions of some sample data (which you may in fact be making up in your head). once that is done, you can bang against the routine with real data and see how it holds up. if you find that you get a condition that you didn't think about before, you now have two tests you can use - the real data that caused the error, and some minimal data extracted from the real data that isolates the problem which can be added to your developer tests. this is what I have been doing lately. call them whatever you like, as I'm sure that the XP people have some fancy nomenclature for it, but the idea is to separate developer-level (used while coding) from API-level tests (real-world API usage) and use both in your testing process. the former is what I use for coverage purposes, tracing through the logical branches in as isolated a context as possible. with the latter I try to tie in live (test) databases, servers, and so on, relying on that to fill in the gaps that are exposed in an isolated test environment. HTH --Geoff
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 07:09:40AM +0100, Piers Cawley wrote:
The xUnit style framework does a much better job of enforcing test
isolation than Test::More does
I see this more as a limitation than a feature. It seems to mean that
- You need to use the same setup/teardown for all your tests.
- You're restricted to one test == one method, so you can't paramatrize
your tests by data (eg, second example below).
- You don't have much control (correct me if I'm wrong) about the order
of tests, or the relationship between tests, eg you can't say if this
test fails, skip these others. This is straightforward in
Test::More's simple procedural style.
If I need isolation, why can't I just ask for it directly?
with_test_setup {
run_tests();
}
and even
foreach (@my_test_data) {
with_test_setup {
run_tests($_);
}
}
(but you have to remember that what
Test::More thinks of as a test, xUnit thinks of as an assertion to be
used *in* a test).
This is the one point of actual difference, but I think we can take care
of it. For starters,
SKIP: {
ok(something) || skip
is(this, that) || skip;
}
Even better would be to put Test::Builder in skip mode, where it skips
automatically whenever a test fails:
skip_mode {
ok(something);
is(this, that);
}
This can't play nice with test counts, but that's a brain-dead idea
anyway.
Every time I hear about xUnit, I figure there must be something other
than setup and teardown in its favor. If that's all there is, I'm not
sold.
Andrew
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 05:08:44PM +0100, Adrian Howard wrote: Where xUnit wins for me are in the normal places where OO is useful (abstraction, reuse, revealing intention, etc.). Since you've thought about this, and obviously don't believe it's OO so it's better, I'd be interested in seeing an example if you have one in mind. Andrew
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, 2004-06-24 at 11:59, Andrew Pimlott wrote: Every time I hear about xUnit, I figure there must be something other than setup and teardown in its favor. If that's all there is, I'm not sold. It's the best option for languages that enforce a nominally pure OO style. (During the tech review of the XP Pocket Guide, Dave Thomas pointed out that many of the XP techniques come from the way Smalltalk forced Ward and Kent to work. xUnit qualifies there.) -- c
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 02:59:30PM -0400, Andrew Pimlott wrote: I see this more as a limitation than a feature. It seems to mean that - You need to use the same setup/teardown for all your tests. Those that need different things aren't testing the same thing and should move to a different class. - You're restricted to one test == one method, so you can't paramatrize your tests by data (eg, second example below). I'm not sure what you mean here. You can have lots of tests in a single method. And subclasses can add their own extra tests to the parent's tests. - You don't have much control (correct me if I'm wrong) about the order of tests, or the relationship between tests, eg you can't say if this test fails, skip these others. This is straightforward in Test::More's simple procedural style. You can group dependant tests into one method and have the method return or die at any point... Every time I hear about xUnit, I figure there must be something other than setup and teardown in its favor. If that's all there is, I'm not sold. The big gain for me with Test::Class is inheritable tests. Subclasses can ensure they still pass all their parent's tests, as well as all of their own, without me having to copy all the tests, or set up a really clumsy testing environment. And of course you get to refactor the test suite quite nicely too so that it's really obvious what each test is doing. If I'm not testing something with a lot of inheritance, then Test::Class loses about 90% of its attraction. Tony
Re: C/C++ White-Box Unit Testing and Test::More
--- Tony Bowden [EMAIL PROTECTED] wrote: The big gain for me with Test::Class is inheritable tests. Subclasses can ensure they still pass all their parent's tests, as well as all of their own, without me having to copy all the tests, or set up a really clumsy testing environment. And of course you get to refactor the test suite quite nicely too so that it's really obvious what each test is doing. I also like the thought of inheriting tests, but I know not everyone is fond of this idea. There was a moderately interesting discussion about this on Perlmonks: http://www.perlmonks.org/index.pl?node_id=294571 Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
Re: C/C++ White-Box Unit Testing and Test::More
On Thu, Jun 24, 2004 at 09:10:09PM +0100, Tony Bowden wrote: On Thu, Jun 24, 2004 at 02:59:30PM -0400, Andrew Pimlott wrote: I see this more as a limitation than a feature. It seems to mean that - You need to use the same setup/teardown for all your tests. Those that need different things aren't testing the same thing and should move to a different class. What about running the same tests with different sample data (so different setup/teardown)? I suppose you could create a sub-class for each data set, but that seems like busy-work. (Unless you can create all those sub-classes automatically from the data) - You're restricted to one test == one method, so you can't paramatrize your tests by data (eg, second example below). I'm not sure what you mean here. You can have lots of tests in a single method. And subclasses can add their own extra tests to the parent's tests. But (I thought) the idea was that every test needs the same setup. If they're all in one method, they won't get that. Also, if you add lots of tests in a single method, (again as I understand) they will stop after the first failure, which is not ideal if the rest of the tests can still run. - You don't have much control (correct me if I'm wrong) about the order of tests, or the relationship between tests, eg you can't say if this test fails, skip these others. This is straightforward in Test::More's simple procedural style. You can group dependant tests into one method and have the method return or die at any point... As above, they won't all get the setup/teardown. The big gain for me with Test::Class is inheritable tests. I see. Andrew
