Jeremy White <[EMAIL PROTECTED]> writes: > The normal behaviour of the tests is that they are > oblidged to provide an 0 exit code on success; I think that > is an excellent standard. > However, having the diff feature allows us to more > rapdily adapt existing programs to become tests. > Since it's done (and it's trivial code), I don't see > the harm in leaving in the feature. We can hide it/discourage > in in the (as yet unwritten) doco if you like.
The problem is that it's hard to support both types of tests from the makefile, so it would force tests without output to generate dummy output and reference files. I think it's better to make the default be no output and have tests that need output diffs do the extra work. > The entire samples directory that I submitted consisted > of 2,705 bytes. For all CVS checkouts and 'normal' Wine use, > there will be an insignificant penalty. "Normal" Wine use should definitely include running regression tests. I don't want to be the only one to run them. > For example, say we have 70 Winelib/C tests that are big > and bloated and we want to fix that. We could define a > new make target, say a '.test.so'. We could modify winetest > to be able to load a .test.so file and call main() with > the right parameters and suddenly all of our bloat is gone. > We'd just need a rule to build a .test.so file from > a .test.c file, and then we'd need to tweak our foo.test > file to add a new command 'so_test=foo.test.so' instead > of 'invoke=foo' and we're done. We could probably even > eliminate the .spec file (only for console apps, obviously). But without your .test file it's even easier to do, all you have to do is to hack the makefile. Plus you can get the dependencies right (how can make guess what file your .test is referencing?) > And another thing, what if I have an app that creates a > graphical image and then uses Gimp or some other scripting > process to take screen shots and then compares them to a known > reference? In a makefile only solution, it'd be a pain > to add a target to call some fancy script to perform > a regression test. This case is already handled by > the .test file. You can do that from a Perl test too, but I don't think this should be encouraged. Tests should not depend on external programs any more than strictly necessary, otherwise people will soon need a complete Linux distribution on their Windows machine to run our tests. -- Alexandre Julliard [EMAIL PROTECTED]
