On Mon, May 14, 2018 at 2:41 PM, Vincent Massol <[email protected]> wrote:
> > > > On 14 May 2018, at 13:30, Marius Dumitru Florea < > [email protected]> wrote: > > > > On Mon, May 14, 2018 at 2:17 PM, Vincent Massol <[email protected]> > wrote: > > > >> > >> > >>> On 14 May 2018, at 12:28, Marius Dumitru Florea < > >> [email protected]> wrote: > >>> > >>> On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <[email protected]> > >> wrote: > >>> > >>>> Hi devs, > >>>> > >>>> Our current Test coverage strategy is to fail the build when new code > >>>> added results in a coverage lower than the threshold for the module, > >> using > >>>> jacoco. > >>>> > >>>> This has 2 limitations causing our global TPC to go down from time to > >> time > >>>> (see https://markmail.org/message/hqumkdiz7jm76ya6 ). > >>>> > >>>> Thus I’d like to propose the following addition to our strategy: > >>>> > >>>> * We already have a jenkins pipeline to automatically compute the full > >> TPC > >>>> using Clover. See http://dev.xwiki.org/xwiki/ > >> bin/view/Community/Testing# > >>>> HUsingClover2BJenkins > >>>> * Make it run more often (it’s currently executed once per month, see > >>>> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to > >>>> execute. Thus we could run it once per week or even once per day > during > >> the > >>>> night. > >>>> * Add some groovy logic in the pipeline to perform an analysis after > the > >>>> Clover report has been generated. Perform 2 checks by comparing the > >>>> previous report with the one that just executed: > >>>> > >>> > >>> > >>>> ** Find new packages introduced that have a TPC < the average computed > >> TPC > >>>> > >>> > >>> What is the average computed TPC currently? > >> > >> It’s about 70%, see for example: > >> http://maven.xwiki.org/site/clover/20180511/clover- > >> commons+rendering+platform-20180511-0147/dashboard.html > >> > >> > > Requiring 70% TPC for new packages is a nice goal but I find it hard to > > achieve in practice. > > Two points on this: > > * We should have above 80-90%+ in practice for new modules, not 70%. If we > have 70% or less we can be sure you did a bad job on quality and that > you’re impacting the overall quality of XWiki. > *should* but I don't think it happens in practice, for various reasons, mainly due to limited time. Did you check the TPC of the last 10 new packages? Is it above 70%? > * For some modules, it makes less sense or it’s harder to have unit tests > and integration tests are better. We’re not talking about 70% of unit test > coverage but 70%+ coverage of overall testing (unit, integration, > functional). > > Now we’ll need to put in place to see it in action and verify if there are > cases where this can be hard to achieve. But I’d prefer that we consider > those cases as exceptional and handle them in an ad-hoc manner. > > > > >>> > >>>> ** Find packages and/or files having a TPC lower than the previous TPC > >>>> ** Find removed packages that had a TPC higher than the average > computed > >>>> TPC > >>>> > >>> > >>> I would add: > >>> > >>> ** find the packages that have the TPC higher than what is declared in > >> the > >>> pom (because we don't always update the TPC value declared in the pom > >> when > >>> we refactor the code or when we add new tests) > >> > >> Yes I agree. We should do that but not in the pipeline for the global > >> coverage. We should have another pipeline for this and update the > pom.xml > >> files in it. > >> > >>> > >>>> * Save a report in the directory for the Clover report at > >>>> http://ci.xwiki.org/view/Tools/job/Clover/ > >>>> * For all failures, send an email to [email protected] with > >> details > >>>> and a link to the saved report > >>>> * Ideally, and if we can do it, call the github API to find the > authors > >> of > >>>> commits for those packages and add them in the report. Examples of > APIs > >> we > >>>> could use: > >>>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits? > >>>> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path > >>>> parameter that could be used to filter but I don’t think it’ll work > >>>> ** https://github.com/xwiki/xwiki-platform/compare/master@ > >>>> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors > need > >>>> to be extracted from the HTML which is a bit fragile) > >>>> * Add a step in the Release process to ensure that the global TPC has > >> not > >>>> been lowered. This would be a way to ensure we pay attention to that > and > >>>> fix it when we lower it. We would need to tune this to find something > >> that > >>>> helps keep the TPC increasing while not putting too much pressure at > the > >>>> same time on the release date. It doesn’t have to be in the release > >> process > >>>> but we need some checkpoint to make sure we look at it and that all > devs > >>>> fix the tests when they lower the global TPC, or at the very least > that > >> an > >>>> analysis is done in case where it’s hard to keep the TPC (for example, > >>>> removing existing code that had a lot of tests will lower the TPC ;)). > >>>> > >>>> WDYT? > >>>> > >>>> As a dev, would you be willing to pay attention to not lower the > global > >>>> TPC and work to fix it when it happens? > >> > >> @Marius: what about this question? Would you be ok with it? :) > >> > > > > Sure. > > ok cool > > Thanks > -Vincent > > > > > > >> > >> Thanks > >> -Vincent > >> > >>>> > >>>> Thanks > >>>> -Vincent > >
