+1. I think this is useful information to collect on every CI run and
store alongside the run, though keeping in mind we only keep the last
7-10 runs in Jenkins nowadays.
If we need it longer, we should stash the output with the log stasher.
This could allow graphing if someone wants to build a neato D3 thing
that hits the couchdb-vm2 instance for stats.
On 2019-12-12 3:51 p.m., Paul Davis wrote:
Hey all,
I was poking around at Jenkins the other day trying to get a good idea
of how much time we're spending in various parts of the build. It
occurred to me that one good way to at least investigate our eunit
test suite is to parse all of the generated surefire reports. I spent
an hour or so yesterday throwing a python script together to see what
that might look like.
I've posted an example output and the script itself here:
https://gist.github.com/davisp/7064c1ef0dc94a99c739729b97fef10e
So far it shows some aggregate timing information along with the top
few results for total suite time, total setup/teardown time, and
longest individual test times. Not included in the output, but any
test failures or skipped tests are also included in the output which
has the nice benefit of showing exactly which test or tests failed
during a run.
So far a few interesting things jump out. First, we spend more time in
setup/teardown fixtures than we do running actual tests which I found
to be rather surprising. Not so surprising is that our property tests
are taking the longest for each run.
I thought it might be interesting to include a version of this script
(with perhaps a somewhat more succinct output) at the end of every
eunit run (or perhaps failed eunit run) to list out the errors for
easier debugging of failed CI runs. And then beyond that, having it in
the source tree would allow devs to poke around and trying to speed up
some of the slower tests.
Another thought was to take a look at moving the property based tests
to a nightly build and then actually increasing their run times to
cover more of the state space for each of those tests. That way we'd
be doing a bit more of the normal property based testing where its
more about long soaks to find edge cases rather than acceptance
testing for a particular change.
So far I've just thrown this together. If there's enough of a
consensus that we'd like to see something of this nature I can work a
bit more on improving the script to have something that's useful both
locally for reducing the test suite times and also for spotting errors
that failed a CI run.
Paul
