Hey all, I was poking around at Jenkins the other day trying to get a good idea of how much time we're spending in various parts of the build. It occurred to me that one good way to at least investigate our eunit test suite is to parse all of the generated surefire reports. I spent an hour or so yesterday throwing a python script together to see what that might look like.
I've posted an example output and the script itself here: https://gist.github.com/davisp/7064c1ef0dc94a99c739729b97fef10e So far it shows some aggregate timing information along with the top few results for total suite time, total setup/teardown time, and longest individual test times. Not included in the output, but any test failures or skipped tests are also included in the output which has the nice benefit of showing exactly which test or tests failed during a run. So far a few interesting things jump out. First, we spend more time in setup/teardown fixtures than we do running actual tests which I found to be rather surprising. Not so surprising is that our property tests are taking the longest for each run. I thought it might be interesting to include a version of this script (with perhaps a somewhat more succinct output) at the end of every eunit run (or perhaps failed eunit run) to list out the errors for easier debugging of failed CI runs. And then beyond that, having it in the source tree would allow devs to poke around and trying to speed up some of the slower tests. Another thought was to take a look at moving the property based tests to a nightly build and then actually increasing their run times to cover more of the state space for each of those tests. That way we'd be doing a bit more of the normal property based testing where its more about long soaks to find edge cases rather than acceptance testing for a particular change. So far I've just thrown this together. If there's enough of a consensus that we'd like to see something of this nature I can work a bit more on improving the script to have something that's useful both locally for reducing the test suite times and also for spotting errors that failed a CI run. Paul
