On Sunday, 18 August 2013 at 14:52:04 UTC, Andrej Mitrovic wrote:
Normally the project maintainers would do this
themselves, but it's easy to run out of time or just to forget
to test
things, and then it's too late (well we have fixup DMD releases
now so it's not too bad).
The big problem with this right now is that quite frequently, you
run the tests and discover one regression in the beta, file it,
fix it (or wait for it to get fixed), then run the tests again,
discover that they still don't pass, etc.
This is not only an annoying and time-intensive job for the
maintainer of the project (as during beta you have to pretty much
always be on your toes for a new version to test lest Walter
decide to make the final release), but this also increases beta
duration.
One obvious reaction to this (as a project maintainer) would be
to continuously track Git master and report regressions as they
arise. However, this is also not always practical, as quite
often, there is a regression/backwards-incompatible change early
on in the development process that is not fixed until much later,
so that multiple issues can still pile up unnoticed.
Having a system that regularly, automatically runs the test
suites of several larger, well-known D projects with the results
being readily available to the DMD/druntime/Phobos teams would
certainly help. But it's also not ideal, since if a project
starts to fail, the exact nature of the issue (regression in DMD
or bug in the project, and if the former, a minimal test case)
can often be hard to track down for somebody not already familiar
with the code base.
David
