On Sat, Jun 08, 2002 at 11:12:40AM -0700, chromatic wrote: > > > If this dependency information changes, it'll fail a test (or maybe warn) > > > because there's a potential interface change that Bar.pm may need to > > > know. > > > It looks interesting up to this point. Basically, everytime Bar.pm is > > touched, edited or upgraded, the test will fail. Every doc patch, typo fix > > and minor upgrade. This will produce orders of magnitude more false > > negatives than real failures, which will sap the credibility of the test > > causing it to be ignored. > > If these cause a version number bump, yes, that'll be a problem. I was > unclear, though. My plan is to use the test for the unit *providing* the > dependency as the most accurate sort of information. The heuristic might be, > in order of ascending importance: > > - version number changes in the dependency module may mark a change > - timestamp changes of the test file probably mark a change > - a change in the number of tests (expected/passing/failed/skipped) marks a > change > > That gives me three levels of certainty. The important thing is that it uses > the depending module's tests to be more sure if something has changed.
It gives you three levels of uncertainy. If Bar.pm is being actively developed, the tests and version number will be changing constantly and the dependency check will also be constantly "failing", causing it to be ignored, or only given a cursory glance. If Bar.pm is relatively static, ie. from CPAN, failures will be rare but when it does fail, it will still be false orders of magnitude more often than it will be true. Since the developer will have to go in and check if the API has been broken *by hand* they will rapidly tire of it and begin ignoring it, or only giving it a glance. Realistically, the best you could expect is for the developer to look at the module's change log and see if there's any flashing "THE API HAS CHANGED" lights. > > In reality, if Bar.pm changes in an incompatible way Foo.pm's own tests > > should fail and that should cover you. > > Not in a strict isolation testing environment, where I don't want to use the > real Bar.pm when testing Foo.pm. I presume you're refering to things like IO::Socket and friends that are frequently mocked? > Maybe I'm being too strict about my isolation, but it's had good results for > me so far. -- This sig file temporarily out of order.
