On 8/18/2013 9:33 AM, David Nadlinger wrote:
Having a system that regularly, automatically runs the test suites of several larger, well-known D projects with the results being readily available to the DMD/druntime/Phobos teams would certainly help. But it's also not ideal, since if a project starts to fail, the exact nature of the issue (regression in DMD or bug in the project, and if the former, a minimal test case) can often be hard to track down for somebody not already familiar with the code base.
That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them?
The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse.
(And the other problem, of course, is the test suite is designed to be runnable fairly quickly. Compiling some other large project and running its test suite can make the autotester much less useful when the turnaround time increases.)
Putting large projects into the autotester has the implication that development and support of those projects has been ceded to the core dev team, i.e. who is responsible for it has been badly blurred.
