Cool beans. I'll take a crack at removing some of the ick around displaying along with adding some command line arguments for controlling output.
For long term storage I think it'd be better to store the raw XML files rather than the formatted output which should be an easy addition. On Thu, Dec 12, 2019 at 3:28 PM Joan Touzet <woh...@apache.org> wrote: > > +1. I think this is useful information to collect on every CI run and > store alongside the run, though keeping in mind we only keep the last > 7-10 runs in Jenkins nowadays. > > If we need it longer, we should stash the output with the log stasher. > This could allow graphing if someone wants to build a neato D3 thing > that hits the couchdb-vm2 instance for stats. > > On 2019-12-12 3:51 p.m., Paul Davis wrote: > > Hey all, > > > > I was poking around at Jenkins the other day trying to get a good idea > > of how much time we're spending in various parts of the build. It > > occurred to me that one good way to at least investigate our eunit > > test suite is to parse all of the generated surefire reports. I spent > > an hour or so yesterday throwing a python script together to see what > > that might look like. > > > > I've posted an example output and the script itself here: > > > > https://gist.github.com/davisp/7064c1ef0dc94a99c739729b97fef10e > > > > So far it shows some aggregate timing information along with the top > > few results for total suite time, total setup/teardown time, and > > longest individual test times. Not included in the output, but any > > test failures or skipped tests are also included in the output which > > has the nice benefit of showing exactly which test or tests failed > > during a run. > > > > So far a few interesting things jump out. First, we spend more time in > > setup/teardown fixtures than we do running actual tests which I found > > to be rather surprising. Not so surprising is that our property tests > > are taking the longest for each run. > > > > I thought it might be interesting to include a version of this script > > (with perhaps a somewhat more succinct output) at the end of every > > eunit run (or perhaps failed eunit run) to list out the errors for > > easier debugging of failed CI runs. And then beyond that, having it in > > the source tree would allow devs to poke around and trying to speed up > > some of the slower tests. > > > > Another thought was to take a look at moving the property based tests > > to a nightly build and then actually increasing their run times to > > cover more of the state space for each of those tests. That way we'd > > be doing a bit more of the normal property based testing where its > > more about long soaks to find edge cases rather than acceptance > > testing for a particular change. > > > > So far I've just thrown this together. If there's enough of a > > consensus that we'd like to see something of this nature I can work a > > bit more on improving the script to have something that's useful both > > locally for reducing the test suite times and also for spotting errors > > that failed a CI run. > > > > Paul > >
