Hi devs,
Our current Test coverage strategy is to fail the build when new code added
results in a coverage lower than the threshold for the module, using jacoco.
This has 2 limitations causing our global TPC to go down from time to time (see
https://markmail.org/message/hqumkdiz7jm76ya6 ).
Thus I’d like to propose the following addition to our strategy:
* We already have a jenkins pipeline to automatically compute the full TPC
using Clover. See
http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HUsingClover2BJenkins
* Make it run more often (it’s currently executed once per month, see
http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to execute.
Thus we could run it once per week or even once per day during the night.
* Add some groovy logic in the pipeline to perform an analysis after the Clover
report has been generated. Perform 2 checks by comparing the previous report
with the one that just executed:
** Find new packages introduced that have a TPC < the average computed TPC
** Find packages and/or files having a TPC lower than the previous TPC
** Find removed packages that had a TPC higher than the average computed TPC
* Save a report in the directory for the Clover report at
http://ci.xwiki.org/view/Tools/job/Clover/
* For all failures, send an email to notificati...@xwiki.org with details and a
link to the saved report
* Ideally, and if we can do it, call the github API to find the authors of
commits for those packages and add them in the report. Examples of APIs we
could use:
**
https://api.github.com/repos/xwiki/xwiki-platform/commits?since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z
(there’s a path parameter that could be used to filter but I don’t think it’ll
work
**
https://github.com/xwiki/xwiki-platform/compare/master@%7B2018-05-07%7D...master@%7B2018-05-10%7D
(the committers/authors need to be extracted from the HTML which is a bit
fragile)
* Add a step in the Release process to ensure that the global TPC has not been
lowered. This would be a way to ensure we pay attention to that and fix it when
we lower it. We would need to tune this to find something that helps keep the
TPC increasing while not putting too much pressure at the same time on the
release date. It doesn’t have to be in the release process but we need some
checkpoint to make sure we look at it and that all devs fix the tests when they
lower the global TPC, or at the very least that an analysis is done in case
where it’s hard to keep the TPC (for example, removing existing code that had a
lot of tests will lower the TPC ;)).
WDYT?
As a dev, would you be willing to pay attention to not lower the global TPC and
work to fix it when it happens?
Thanks
-Vincent
