On Saturday, June 8, 2002, at 07:12 pm, chromatic wrote:
> On Saturday 08 June 2002 11:02, Michael G Schwern wrote:
[snip]
>> In reality, if Bar.pm changes in an incompatible way Foo.pm's own tests
>> should fail and that should cover you.
>
> Not in a strict isolation testing environment, where I don't want to
> use the
> real Bar.pm when testing Foo.pm.
>
> Maybe I'm being too strict about my isolation, but it's had good
> results for
> me so far.
[This message has became a bit of a ramble... apologies in advance :-) ]
I played with something a bit like the proposed Test::Depend, but I
abandoned it after a while since it didn't help me in the way I had
expected. Don't know if this will help the discussion but what I did was
roughly as follows...
- Have all of my mock objects stick their class, the method names
they were
called with, and $0 into a database. So you have a bunch of rows like:
MockFoo open Bar/t/bar.t
MockFoo close Bar/t/bar.t
....
- Manually maintain a mapping from MockFoo to the real class hierarchy
that implemented Foo (this could probably have been automated if I
could
have been bothered to think about it :-)
- Have a test for each module that compared an MD5 hash of each
method in the module (got by extracting 'sub method_name { .... }'
string from the .pm)
to the hash from the last run, and throw out some warnings on the tests
that used the mocked version of the same method if it had changed (or
vanished).
My hope was that this would help me in refactoring code. I wouldn't have
to keep track of what modules use what methods, so I could be more
confidant in making changes.
This fell down in three ways. Two minor, one major.
#1: Minor problem - Didn't work for AUTOLOAD methods, generated
methods,
etc.
#2: Minor problem - False negatives when changes were not in
method_name,
but something other private method of Foo that method_name called.
These were not really issues for me, probably because of my personal
coding style, so I didn't get a lot of false negative/positives.
Test::Depend will cope better with these problems since it looks at the
test results of the module as a whole. This should give an excellent
indication of semantic change (at the expense of it being module, rather
than method, specific).
Test::Depend also involves less work in implementing the unit tests
since you don't have to mess with the mocks.
However, it doesn't help with:
#3: Major problem - You have to check the warnings manually!
I found that, once you have a moderately complex system, it's hard to
determine whether changes you are being warned about are going to be an
issue (beyond simple things like method renaming). I spent too much time
looking at the code, and usually ended up writing a functional test to
make sure that I wasn't missing something.
I eventually just bit the bullet and started writing more functional
tests. This (of course) had the usual affect of writing more tests ---
it made development faster.
Worry about making a non-compatible change to a class? Not me. I just do
% make test >& functional1.txt
make the change, then repeat
% make test >&! functional2.txt ; diff functional1.txt functional2.txt
until the only difference I see is the time the tests took.
While this involves some repetition of the unit tests it does help track
things like changing APIs. If you change Foo in some non-compatible way
you get the following little pattern:
- Tests show that Bar's functional tests fail, but unit tests succeed.
- Update MockFoo so that Bar's unit tests fail the same way
- Fix Bar.pm so that all tests pass.
I don't run the functional tests as often as the unit tests so I don't
lose a lot of speed. It doesn't involve a lot of duplicate code since I
keep most of my tests in classes that allow me to swap mock/live objects
by subclassing and/or configuration options.
Once I started writing decent functional tests, my hack to track changed
methods suddenly ceased being useful. I think you may find the same
thing occurring with Test::Depend - it will just encourage you to write
more functional tests ;-)
Cheers,
Adrian
--
Adrian Howard <[EMAIL PROTECTED]>
phone: 01929 550720 fax: 0870 131 3033 www.quietstars.com
