> > On Wed, 24 Jan 2007 03:02:19 +0100, Marcin Dalecki <[EMAIL PROTECTED]> said: > > That's largely because individual tests in the test suite are too > long, which in turn is because the tests are testing code at a > per-binary granularity: you have to run all of gcc, or all of one > of the programs invoked by gcc, to do a single test. (Is that true? > Please correct me if I'm wrong.)
No, they are almost all 1 or 2 functions long, nothing more than 20 lines. The larger testcases are really testing the library parts rather than GCC itself. We have more than 40,000 testcases in just the C testsuite. 15,000 in the Fortran testsuite. 14,000 in the C++ testsuite. 1,000 in the objective-C testsuite. You really cannot do unit testing for parsing cases, sorry to say that. So we are testing many different parts of the compiler at once. > I'm not going to argue against having a good end-to-end test suite > around, but it would be quite doable, over the course of a couple of > years, to move to a model where a commit required about 10 minutes of > testing (including all the unit tests and a few smoke end-to-end > tests), and you had separate, automated runs of nightly end-to-end > tests that caught problems that slipped through the unit tests. (And, > of course, whenever the nightly tests detected problems, you'd update > the unit tests accordingly.) I doubt that because unit testing still will not catch parser/semantic issues where at least a thrid of the regressions are being filed. > I don't think it's a pure human resources issue, but I don't think > it's a purely technical issue, either, if for no other reason than > that people are involved. I think it is a polticial issue rather than either a human reasource issue and technical issue. Remember most of us have day jobs not directly working on the FSF GCC. > Here are numbers to back up my unit test timing claim; these are all > run on a computer that cost lest than a thousand dollars a year ago. > > A C++ example, which is probably closest to your situation: That is just for a simple C++ code. Our unit testing will be over something like a million times larger than most unit testing which case unit testing falls down. Right now we do black box testing which is actually a good way to test the whole thing. Unit testing still does not test the interactions between each part, only the units. So you still don't know if one corner case you can't think of right until you run into it. This is what most of the GCC testsuite is full of, corner testcases. -- PInski
