On Sun, Nov 12, 2017 at 11:45 AM, Jed Brown <j...@jedbrown.org> wrote:
> Satish Balay <ba...@mcs.anl.gov> writes: > > > On Sun, 12 Nov 2017, Smith, Barry F. wrote: > > > >> If the testing/fixing would be faster if we bought more machines > then decide what machines we need and we order them now. Buying machines is > far better than wasting even more people time (which is much more > expensive). > > > > Its wading through the numerous logs and and deciding on how to debug > > the breakages that's the limiting factor. [the more builds we add the > > more logs are generated]. And there is a basic breakage - that gets > > replicated in all the 40 logs [sometimes multiple times] > > > > [so I usually end up punting looking at the logs :(]. Yeah we have > > some scripts that checks and sends blame e-mail. I'll have to check if > > its still working or broken. [but it doesn't cover all breakages] > > > > And them I'm usually using brute-force bisection to find the change > > that might have triggered [i.e not really debugging] - to identify who > > the breakage belongs to. > > > > Perhaps new hardware might help - but its not clear by how > > much. Improvements to the build tools might also help [again have to > > figure out how to improve without breaking things..] > > I would expect we'll need more capacity if we're automatically testing > branches before merging to 'next'. We might consider using a cloud > provider like EC2 instead of managing our own hardware far that since > the demand will be dynamic -- whenever people update pull requests. > > If we get basic testing before merging, the only breakage in 'next' > should be the more interesting interactions. Most days should be clean. > Have we tried histogramming test times? It would be nice to know how much cumulative time it takes to run 37%, 67%, 95%, etc. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>