On Mon, Nov 11, 2013 at 9:38 AM, Andi Kleen <a...@firstfloor.org> wrote: > Jeff Law <l...@redhat.com> writes: > >> Thoughts or comments? > > If noone tests java completely then it will quickly bitrot won't it? > > So ideally some bot would still regularly build/test it. > If you don't do that you could as well just remove the code. > > The underlying problem seems to be the requirement for each > contributor to run all the tests themselves. Over time the gcc > test suite has become longer and longer, so it becomes a bigger > burden. But then they still don't test everything, because > of optional components (e.g. not everyone has Ada installed). > Also some tests are just unreliable and do not give good results. > > Perhaps it would make more sense to just split the required tests : > > - quick tests everyone is expected to run before commit (bootstrap + > test suite <10/20/30min?) > That could be some subset of what runs today. Maybe bootstrap > and most of the C/C++ test suite. > - This should be only tests that always pass, not the usual noise that > is there today (e.g. not that guality random generator) > - longer tests that run in centralized bots > - bots automatically bisect regressions > - clear policies that if someone causes a bot regression > they are on the hook to fix it in a given time, or the patch gets > reverted. > - Get rid of all tests that are not reliable. > - With bots it would be also quite feasible to have even > longer tests, that run hours. For example you could add > more slow static code analysis steps. > - With bots you can also have nice central dash boards that give the current > state, instead of the current "hunt random message on gcc-testresults"
Yes. I've been wanting to implement something along these lines for a long time. The basic idea was to use coverage to create a subset of the full testsuite. Anything that covers N% of the coverage of the full testsuite. N would be chosen to guarantee a turnaround time of less than, say, 20 minutes for a full bootstrap+test. We'd use that as the required testing process for everyday development. Bots would run the full testsuite. Diego.
