On 21/09/17 14:11, Mark Wielaard wrote: > Hi, > > First let me say I am also a fan of buildbot. I use it for a couple of > projects and it is really flexible, low on resources, easy to add new > builders/workers and easily extensible if you like python. > > On Thu, 2017-09-21 at 07:18 +0200, Markus Trippelsdorf wrote: >> And it has the basic problem of all automatic testing: that in the >> long run everyone simply ignores it. >> The same thing would happen with the proposed new buildbot. It would >> use still more resources on the already overused machines without >> producing useful results. > > But this is a real concern and will happen if you are too eager testing > all the things all the time. So I would recommend to start as small as > possible. Pick a target that builds as fast as possible. Once you go > over 30 minutes of build/test time it really is already too long. Both > on machine resources and human attention span. And pick a subset of the > testsuite that should be zero-FAIL. Only then will people really take > notice when the build turns from green-to-red. Otherwise people will > always have an excuse "well, those tests aren't really reliable, it > could be something else". And then only once you have a stable > small/fast builder that reliably warns committers that their commit > broke something extend it to new targets/setups/tests as long as you > can keep the false warnings as close to zero as possible. >
Thanks. This is an interesting idea, however it might not be an easy exercise to choose a subset of the tests for each compiled language that PASS, are quick to run and representative. It would be interesting to hear from some of the main developers which of the tests would be better to run. -- Paulo Matos
