I wouldn't use real-world projects for any kind of automated testing - they're moving targets.
That said, real-world projects are almost required to come up with testing requirements, or at least any *additional* testing requirements. If the toolchain breaks on a real-world project and the type of break can be tracked down, then that failure ought to be simulated with a much smaller mock project. Bear in mind that *any* testing is better than none in most cases, though it requires a bit of discipline to keep up, so any initial cracks at testing just have to be indicative, not exhaustive. Generating failures on purpose can really help. I don't know how analogous build processing is to general algorithms, mind you, but I often find that you end up with questions to which the answers are ill-defined: What is the default behaviour if one is not specified? What if a localization is missing? What if the debug and release flags are not consistent? A lot of edge behaviors fail due to this kind of ambiguity. Putting your foot down, saying "it has to be X or ELSE!", and putting together a test case where it is NOT X makes sure that the ambiguity does not get through the tool chain. It's a little better for tool chain substitutions as well, since it's a lot easier in many cases to slap together the "known path" without considering all of the unknown ones. Keeping things as black box as possible is generally good. Using white box knowledge about what might cause something nasty is great, but try to keep it black box, e.g. do not inspect the tool's own intermediate work files or mine its error text (unless there are somehow error code standards). As a simplistic example in the regular coding world, one of my old coworkers was putting together some marshallers for various types. He put one together for the Date/Time type - I looked at his code and realized that it wasn't locale independent, so I threw together a mountain of tests that exercised the marshaller by switching cultures, set the date separator to $, etc. 'Bastardly', but effective. ... Also, on another note: I can well do the reply-all, but that "reply-to-harmful" link - jesus, what a whiner :) Small listservs like this where we all trust one another are all well and good, but any larger or more open listserv should really be posing itself as the sole sender. On top of simple reply-to action, peoples' accounts are anonymous and filtering is a lot simpler. Old listservs of that style used to be really stupid about sending everybody the 'Blabla is out for the holidays' messages, but those haven't been an issue for quite some time now. </soapbox> :) ----- Original Message ----- From: Garrett Serack <[email protected]> Date: Wednesday, April 21, 2010 10:38 am Subject: RE: [Coapp-developers] Thinking abotu testing CoApp's tools. To: Ritchie Annand <[email protected]>, "[email protected]" <[email protected]> > [recycling back to the list ... make sure you use reply-all :) ] > > For Category A, each tool has a fairly constrained output-for > testing them individually, might we just as well setup an > automatic run against a bunch of real-world projects? I'm > just not sure there is an automated way to validate that the > results are what you wanted (although, one could argue that if > you actually got out the packages you wanted, they are probably > *what* you wanted.) > > >> Testing is also as much about trying to break things as anything > > Ah, yes. That is a good way to look at it. Hmmm. Actually, that > goes a long way to thinking about testing; not so much the > validation of what works, as an appropriate halt to bad inputs. > > G > > Garrett Serack | Open Source Software Developer | Microsoft > CorporationI don't make the software you use; I make the > software you use better on Windows.
_______________________________________________ Mailing list: https://launchpad.net/~coapp-developers Post to : [email protected] Unsubscribe : https://launchpad.net/~coapp-developers More help : https://help.launchpad.net/ListHelp
