On Mon, May 14, 2018 at 2:17 PM, Vincent Massol <[email protected]> wrote:
> > > > On 14 May 2018, at 12:28, Marius Dumitru Florea < > [email protected]> wrote: > > > > On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[email protected]> > wrote: > > > >> Hi devs, > >> > >> Our current Test coverage strategy is to fail the build when new code > >> added results in a coverage lower than the threshold for the module, > using > >> jacoco. > >> > >> This has 2 limitations causing our global TPC to go down from time to > time > >> (see https://markmail.org/message/hqumkdiz7jm76ya6 ). > >> > >> Thus I’d like to propose the following addition to our strategy: > >> > >> * We already have a jenkins pipeline to automatically compute the full > TPC > >> using Clover. See http://dev.xwiki.org/xwiki/ > bin/view/Community/Testing# > >> HUsingClover2BJenkins > >> * Make it run more often (it’s currently executed once per month, see > >> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to > >> execute. Thus we could run it once per week or even once per day during > the > >> night. > >> * Add some groovy logic in the pipeline to perform an analysis after the > >> Clover report has been generated. Perform 2 checks by comparing the > >> previous report with the one that just executed: > >> > > > > > >> ** Find new packages introduced that have a TPC < the average computed > TPC > >> > > > > What is the average computed TPC currently? > > It’s about 70%, see for example: > http://maven.xwiki.org/site/clover/20180511/clover- > commons+rendering+platform-20180511-0147/dashboard.html > > Requiring 70% TPC for new packages is a nice goal but I find it hard to achieve in practice. > > > > > >> ** Find packages and/or files having a TPC lower than the previous TPC > >> ** Find removed packages that had a TPC higher than the average computed > >> TPC > >> > > > > I would add: > > > > ** find the packages that have the TPC higher than what is declared in > the > > pom (because we don't always update the TPC value declared in the pom > when > > we refactor the code or when we add new tests) > > Yes I agree. We should do that but not in the pipeline for the global > coverage. We should have another pipeline for this and update the pom.xml > files in it. > > > > >> * Save a report in the directory for the Clover report at > >> http://ci.xwiki.org/view/Tools/job/Clover/ > >> * For all failures, send an email to [email protected] with > details > >> and a link to the saved report > >> * Ideally, and if we can do it, call the github API to find the authors > of > >> commits for those packages and add them in the report. Examples of APIs > we > >> could use: > >> ** https://api.github.com/repos/xwiki/xwiki-platform/commits? > >> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path > >> parameter that could be used to filter but I don’t think it’ll work > >> ** https://github.com/xwiki/xwiki-platform/compare/master@ > >> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need > >> to be extracted from the HTML which is a bit fragile) > >> * Add a step in the Release process to ensure that the global TPC has > not > >> been lowered. This would be a way to ensure we pay attention to that and > >> fix it when we lower it. We would need to tune this to find something > that > >> helps keep the TPC increasing while not putting too much pressure at the > >> same time on the release date. It doesn’t have to be in the release > process > >> but we need some checkpoint to make sure we look at it and that all devs > >> fix the tests when they lower the global TPC, or at the very least that > an > >> analysis is done in case where it’s hard to keep the TPC (for example, > >> removing existing code that had a lot of tests will lower the TPC ;)). > >> > >> WDYT? > >> > >> As a dev, would you be willing to pay attention to not lower the global > >> TPC and work to fix it when it happens? > > @Marius: what about this question? Would you be ok with it? :) > Sure. > > Thanks > -Vincent > > >> > >> Thanks > >> -Vincent > >
