Re: Pondering Test::Depend
On Sunday, June 9, 2002, at 02:59 am, chromatic wrote:
On Saturday 08 June 2002 17:32, Adrian Howard wrote:
I eventually just bit the bullet and started writing more functional
tests. This (of course) had the usual affect of writing more tests ---
it made development faster.
What would one of these functional tests look like?
I was probably unclear in my previous message. By function tests I
meant a separate script running tests with real objects, not a different
kind of test calls in an existing *.t script.
Make sense?
I usually end up with a
few tests per function with names similar to:
- save() should croak() without an 'id' parameter
- ... and should return false if serialization fails
- ... or true if it succeeds
I'll probably also have several other tests that don't exercise save()'s
effective interface. They're not so important for dependency tracking,
so
I'll ignore them for now.
My current thinking is that marking the interface tests as special is
just
about the only way to track them reliably:
$foo-{_store} = $mock;
$mock-set_series( 'serialize', 0, 1 );
eval { $foo-save() };
dlike( @$, qr/No id provided!/, 'save() should croak()...' );
my $result = $foo-save( 77 );
dok( ! $result, '... and should return false...' );
dok( $foo-save( 88 ), '... or true...' );
... where dlike() and dok() are Test::Depend wrappers around
Test::More's
like() and ok().
Test::Depend will save the names and results away and compare them at
the end
of the test suite's run.
There, that's my handwaving magic in a nutshell. I'm not thrilled with
the
dopey names, but haven't a better idea at the moment.
I still think that it'll help less than you think ;-) but, how about
something like:
ok(whatever, 'a non-interface test');
track {
eval { $foo-save() };
like( @$, qr/No id provided!/, 'save() should croak()...' );
my $result = $foo-save( 77 );
ok( ! $result, '... and should return false...' );
ok( $foo-save( 88 ), '... or true...' );
};
ok(whatever, 'another non-interface test');
Since AFAIK every Test::Builder based test boils down to
Test::Builder::ok you could implement like this:
use Test::Builder;
use Hook::LexWrap;
my $Test = Test::Builder-new;
sub track () {
my $sub = shift;
# code this evil may encourage young Mr Schwern to
# implement the details method in Test::Builder :-)
my $temp_tracking = wrap 'Test::Builder::ok', post = sub {
my ($ok, $name) = @_[1..2];
my $test = $Test-current_test;
# you would want to stick it in a DB rather than...
$Test-diag(
tracking test $test in $0 named '$name' which
. ($ok ? 'passed' : 'failed')
);
};
eval $sub;
};
This would have the advantage of not having to overwrite all present
future test functions.
Cheers,
Adrian
--
Adrian Howard [EMAIL PROTECTED]
phone: 01929 550720 fax: 0870 131 3033 www.quietstars.com
Re: Pondering Test::Depend
On Sat, Jun 08, 2002 at 10:45:50AM -0700, chromatic wrote: Taking an average test suite as an example, we could have 'foo.t' and 'bar.t', testing Foo.pm and Bar.pm respectively. Bar depends on Foo, and bar.t mocks a couple of methods of Foo. If bar.t uses Test::Depend, it'll pick up several things: - the module being tested (assumed from a use_ok() or require_ok() call) - the module being mocked (assumed from an entry in %INC) If there's no dependency information, it'll use the current version of Foo.pm, the last modified timestamp of the test file, and the test status of foo.t (if it also uses Test::Depend) to keep track of any changes to Foo.pm. If this dependency information changes, it'll fail a test (or maybe warn) because there's a potential interface change that Bar.pm may need to know. It looks interesting up to this point. Basically, everytime Bar.pm is touched, edited or upgraded, the test will fail. Every doc patch, typo fix and minor upgrade. This will produce orders of magnitude more false negatives than real failures, which will sap the credibility of the test causing it to be ignored. If Perl had strong interfaces, like Java, this could be done a bit more intellegently. But it doesn't. In reality, if Bar.pm changes in an incompatible way Foo.pm's own tests should fail and that should cover you. Like I said earlier, I'm aiming for an 80% solution. There are some assumptions and some fragilities here, but it represents a great improvement over the current system, in my mind. -- This sig file temporarily out of order.
Re: Pondering Test::Depend
On Saturday 08 June 2002 11:02, Michael G Schwern wrote: If this dependency information changes, it'll fail a test (or maybe warn) because there's a potential interface change that Bar.pm may need to know. It looks interesting up to this point. Basically, everytime Bar.pm is touched, edited or upgraded, the test will fail. Every doc patch, typo fix and minor upgrade. This will produce orders of magnitude more false negatives than real failures, which will sap the credibility of the test causing it to be ignored. If these cause a version number bump, yes, that'll be a problem. I was unclear, though. My plan is to use the test for the unit *providing* the dependency as the most accurate sort of information. The heuristic might be, in order of ascending importance: - version number changes in the dependency module may mark a change - timestamp changes of the test file probably mark a change - a change in the number of tests (expected/passing/failed/skipped) marks a change That gives me three levels of certainty. The important thing is that it uses the depending module's tests to be more sure if something has changed. In reality, if Bar.pm changes in an incompatible way Foo.pm's own tests should fail and that should cover you. Not in a strict isolation testing environment, where I don't want to use the real Bar.pm when testing Foo.pm. Maybe I'm being too strict about my isolation, but it's had good results for me so far. -- c
Re: Pondering Test::Depend
On Saturday 08 June 2002 11:39, Michael G Schwern wrote: It gives you three levels of uncertainy. If Bar.pm is being actively developed, the tests and version number will be changing constantly and the dependency check will also be constantly failing, causing it to be ignored, or only given a cursory glance. If Bar.pm is relatively static, ie. from CPAN, failures will be rare but when it does fail, it will still be false orders of magnitude more often than it will be true. Since the developer will have to go in and check if the API has been broken *by hand* they will rapidly tire of it and begin ignoring it, or only giving it a glance. Realistically, the best you could expect is for the developer to look at the module's change log and see if there's any flashing THE API HAS CHANGED lights. I see where you're coming from. It sounds like there are three possibilities for a dependency: 1) It's under active development, so it will change fairly frequently. 2) It's supplied by a third party, so it should be treated as a black box and should be upgraded only after careful human testing. 3) It's in maintenance, so changes are relatively few. I agree with you about #1. Unless Test::Depend is amazingly brilliant about identifying the appropriate bits-of-a-unit level dependencies, it'll produce too many false positives. That's counterproductive, and worse than useless. I don't really care about #2, because a good development process ought to manage it. I'm trying to solve #3 right now. It's intended to say, Hey, you changed Foo.pm. You (might|should|need to) check Bar.pm, Baz.pm, and Fluxx.pm to see if they need similar changes. I have some ideas as to how to approach #1, but it'll require more human intervention and it's more difficult. It looks like it'll have more benefits, though. Not in a strict isolation testing environment, where I don't want to use the real Bar.pm when testing Foo.pm. I presume you're refering to things like IO::Socket and friends that are frequently mocked? Exactly. There are at least three somewhat overlapping classes of dependencies I usually mock: things I don't want to communicate with the outside world, things that mutate important data (data sinks and data sources), and things that are hard to control for testing purposes (sockets, timeouts, deaths, and other exceptional conditions). -- c
Re: Pondering Test::Depend
I'm a big fan of testing units in isolation. It's easier, faster, and tends to more comprehensive and detailed tests. The problem is, if the unit being tested has a legitimate dependency on another unit in the suite, isolation testing cannot identify interface changes. I've been vaguely reading the Schwernomatic commentary, and most of it doesn't interest me directly. I'm not enough of a module-writin' guy to see how it's gonna help me at all. Here are my two concerns: * Please don't make dependencies on *.t as the filenames involved. I'm personally using Test::Harness to run tests on other files as well. * Dependencies should be optionally based on something other than just the timestamp of a file. For instance, I suspect I'd like to be able to make the date dependency on SELECT MAX(DateUpdated) FROM Foo. xoxo, Andy -- 'Andy Lester[EMAIL PROTECTED] Programmer/author petdance.com Daddy parsley.org/quinn Jk'=~/.+/s;print((split//,$) [unpack'C*',n2]3%+\34.'%.'^%4+!o.'])
Re: Pondering Test::Depend
On Sat, 8 Jun 2002, Andy Lester wrote:
* Dependencies should be optionally based on something other than just
the timestamp of a file. For instance, I suspect I'd like to be able to
make the date dependency on SELECT MAX(DateUpdated) FROM Foo.
Hmmm, for example a hash of the code (a la inline) - possible sans pod
Mark.
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t-Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t-Tgoto(cm,$_,$y). $w;select$k,$k,$k,.03}$y+=2}
Re: Pondering Test::Depend
On Saturday 08 June 2002 17:32, Adrian Howard wrote:
I found that, once you have a moderately complex system, it's hard to
determine whether changes you are being warned about are going to be an
issue (beyond simple things like method renaming). I spent too much time
looking at the code, and usually ended up writing a functional test to
make sure that I wasn't missing something.
I eventually just bit the bullet and started writing more functional
tests. This (of course) had the usual affect of writing more tests ---
it made development faster.
What would one of these functional tests look like? I usually end up with a
few tests per function with names similar to:
- save() should croak() without an 'id' parameter
- ... and should return false if serialization fails
- ... or true if it succeeds
I'll probably also have several other tests that don't exercise save()'s
effective interface. They're not so important for dependency tracking, so
I'll ignore them for now.
My current thinking is that marking the interface tests as special is just
about the only way to track them reliably:
$foo-{_store} = $mock;
$mock-set_series( 'serialize', 0, 1 );
eval { $foo-save() };
dlike( $, qr/No id provided!/, 'save() should croak()...' );
my $result = $foo-save( 77 );
dok( ! $result, '... and should return false...' );
dok( $foo-save( 88 ), '... or true...' );
where dlike() and dok() are Test::Depend wrappers around Test::More's
like() and ok().
Test::Depend will save the names and results away and compare them at the end
of the test suite's run.
There, that's my handwaving magic in a nutshell. I'm not thrilled with the
dopey names, but haven't a better idea at the moment.
-- c
