> On 14 May 2018, at 12:28, Marius Dumitru Florea > <[email protected]> wrote: > > On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[email protected]> wrote: > >> Hi devs, >> >> Our current Test coverage strategy is to fail the build when new code >> added results in a coverage lower than the threshold for the module, using >> jacoco. >> >> This has 2 limitations causing our global TPC to go down from time to time >> (see https://markmail.org/message/hqumkdiz7jm76ya6 ). >> >> Thus I’d like to propose the following addition to our strategy: >> >> * We already have a jenkins pipeline to automatically compute the full TPC >> using Clover. See http://dev.xwiki.org/xwiki/bin/view/Community/Testing# >> HUsingClover2BJenkins >> * Make it run more often (it’s currently executed once per month, see >> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to >> execute. Thus we could run it once per week or even once per day during the >> night. >> * Add some groovy logic in the pipeline to perform an analysis after the >> Clover report has been generated. Perform 2 checks by comparing the >> previous report with the one that just executed: >> > > >> ** Find new packages introduced that have a TPC < the average computed TPC >> > > What is the average computed TPC currently?
It’s about 70%, see for example: http://maven.xwiki.org/site/clover/20180511/clover-commons+rendering+platform-20180511-0147/dashboard.html > > >> ** Find packages and/or files having a TPC lower than the previous TPC >> ** Find removed packages that had a TPC higher than the average computed >> TPC >> > > I would add: > > ** find the packages that have the TPC higher than what is declared in the > pom (because we don't always update the TPC value declared in the pom when > we refactor the code or when we add new tests) Yes I agree. We should do that but not in the pipeline for the global coverage. We should have another pipeline for this and update the pom.xml files in it. > >> * Save a report in the directory for the Clover report at >> http://ci.xwiki.org/view/Tools/job/Clover/ >> * For all failures, send an email to [email protected] with details >> and a link to the saved report >> * Ideally, and if we can do it, call the github API to find the authors of >> commits for those packages and add them in the report. Examples of APIs we >> could use: >> ** https://api.github.com/repos/xwiki/xwiki-platform/commits? >> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path >> parameter that could be used to filter but I don’t think it’ll work >> ** https://github.com/xwiki/xwiki-platform/compare/master@ >> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need >> to be extracted from the HTML which is a bit fragile) >> * Add a step in the Release process to ensure that the global TPC has not >> been lowered. This would be a way to ensure we pay attention to that and >> fix it when we lower it. We would need to tune this to find something that >> helps keep the TPC increasing while not putting too much pressure at the >> same time on the release date. It doesn’t have to be in the release process >> but we need some checkpoint to make sure we look at it and that all devs >> fix the tests when they lower the global TPC, or at the very least that an >> analysis is done in case where it’s hard to keep the TPC (for example, >> removing existing code that had a lot of tests will lower the TPC ;)). >> >> WDYT? >> >> As a dev, would you be willing to pay attention to not lower the global >> TPC and work to fix it when it happens? @Marius: what about this question? Would you be ok with it? :) Thanks -Vincent >> >> Thanks >> -Vincent
