Hans-Christoph Steiner wrote: > On Dec 21, 2007, at 10:30 AM, IOhannes m zmölnig wrote: > >> Mathieu Bouchard wrote: >>> On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote: >>>> What we really need is regression testing, I think that would >>>> make it >>>> easier to convince Miller to accept patches, and would be a great >>>> tool for finding bugs. Matju started that project, any word on that >>>> Matju? >>> No, I haven't done much progress. The problem with that thing is >>> that no-one wants to do it, not even me. It's hard to consider it >>> a priority as much as it should be. >> >> at least i am using "kind-of" regression tests in zexy and iemlib. >> the "suite" is rather simplistic, but works well for my purposes. >> >> see zexy/tests for how it works > > Any interest in making this a general purpose testing framework? I
have you had a look at it? > would like to come up with a suite that is run every night > automatically, like the nightly builds. It sounds very similar to > what I was thinking. > > If we just come up with a standard set of inlets, outlets, and a > standard output value, well, i think i ahve documented this in the README. since i am using the same framework on several projects (not just the 2 mentioned), i would prefer to keep it rather fixed like it is. writing tests is not that much fun for me. right now there are more than 300 tests in iemmatrix... > then it will be easy to write a script to run > thru them all and report on them. Sounds like you might already have > that working to some degree. have you had a look at it? i am using my suites for >2years now. > > I would like to make a set that tests for the strict aliasing bugs so > I can safely use gcc optimizations. This means testing tilde objects. while there are no explicit tests for ~-objects in any of my suites (i think), they are thought of. > I think it might make sense to start a section in CVS for this called > 'tests', then we can start gathering tests there into subfolders > organized by library. Another option would be a standard folder in > each library called 'tests' like you have. why would you keep them apart? > > That's my two bits, but we really should be doing a lot more of this. i agree. and despite of saying how great my regression-test-suite is, in reality it is not. and i am using several different versions of it out there [1] my advice: - keep the tests as simple as possible, but not simpler one should be able to write a test fairly easily; i think shortcuts are most likely not needed most of the times (e.g. iirc, matju introduced several shortcuts in his unity-test frameworks to test for mathematically important concepts (like assosiativity of operations); i think it is way simpler to just program 3 tests manually instead) - as for results i have found it sufficient to have the following return-states: SUCCESS, FAIL, WAIT i found it makes writing tests simpler if all of them implicitely fail if they do not respond with a return-state _immediately_ (this might make the FAIL-state unnecessary; but again it makes tests simpler if you can provide an explicit FAIL) if the test takes some time (e.g. for signal-objects or timing objects), i return the WAIT state which halts the testing-framework until the currently tested object returns a value. i usually use "1" for success, "0" for fail, "-1" for wait, the rest is fail too. - write tests to test the framework - allow fail-tests which have to return FAIL-state in order to success - handle crash-tests separately - allow "manual" tests (not "scriptable" (as in "as fast as possible")) - i found that tests-patches should not have arguments (especially: default-arguments, like "$0"), as this sometimes makes testing of special features unneccessarily complicated. - ... fmga.sdr IOhannes [1] the most current incarnation of my framework is currently used in http://ftm.cvs.sourceforge.net/ftm/ftm/externals/pd/regression/ _______________________________________________ PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list