Satish Balay <ba...@mcs.anl.gov> writes: > On Sun, 12 Nov 2017, Smith, Barry F. wrote: > >> If the testing/fixing would be faster if we bought more machines then >> decide what machines we need and we order them now. Buying machines is far >> better than wasting even more people time (which is much more expensive). > > Its wading through the numerous logs and and deciding on how to debug > the breakages that's the limiting factor. [the more builds we add the > more logs are generated]. And there is a basic breakage - that gets > replicated in all the 40 logs [sometimes multiple times] > > [so I usually end up punting looking at the logs :(]. Yeah we have > some scripts that checks and sends blame e-mail. I'll have to check if > its still working or broken. [but it doesn't cover all breakages] > > And them I'm usually using brute-force bisection to find the change > that might have triggered [i.e not really debugging] - to identify who > the breakage belongs to. > > Perhaps new hardware might help - but its not clear by how > much. Improvements to the build tools might also help [again have to > figure out how to improve without breaking things..]
I would expect we'll need more capacity if we're automatically testing branches before merging to 'next'. We might consider using a cloud provider like EC2 instead of managing our own hardware far that since the demand will be dynamic -- whenever people update pull requests. If we get basic testing before merging, the only breakage in 'next' should be the more interesting interactions. Most days should be clean.