+1 Thanks, Marius
On Thu, Mar 15, 2018 at 12:49 PM, Ecaterina Moraru (Valica) < [email protected]> wrote: > +1 > > Thanks, > Caty > > On Thu, Mar 15, 2018 at 12:48 PM, Eduard Moraru <[email protected]> > wrote: > > > Sounds interesting, > > +1. > > > > Thanks, > > Eduard > > > > On Thu, Mar 15, 2018 at 12:26 PM, Thomas Mortagne < > > [email protected] > > > wrote: > > > > > +1 > > > > > > On Thu, Mar 15, 2018 at 9:30 AM, Vincent Massol <[email protected]> > > > wrote: > > > > Hi devs, > > > > > > > > As part of the STAMP research project, we’ve developed a new tool > > > (Descartes, based on Pitest) to measure the quality of tests. It > > generates > > > a mutation score for your tests, defining how good the tests are. > > Technical > > > Descartes performs some extreme mutations on the code under test (e.g. > > > remove content of void methods, return true for methods returning a > > > boolean, etc - See https://github.com/STAMP-project/pitest-descartes). > > If > > > the test continues to pass then it means it’s not killing the mutant > and > > > thus its mutation score decreases. > > > > > > > > So in short: > > > > * Jacoco/Clover: measure how much of the code is tested > > > > * Pitest/Descartes: measure how good the tests are > > > > > > > > Both provide a percentage value. > > > > > > > > I’m proposing to compute the current mutation scores for > xwiki-commons > > > and xwiki-rendering and fail the build when new code is added that > reduce > > > the mutation score threshold (exactly the same as our jacoco threshold > > and > > > strategy). > > > > > > > > I consider this is an experiment to push the limit of software > > > engineering a bit further. I don’t know how well it’ll work or not. I > > > propose to do the work and test this for over 2-3 months and see how > well > > > it works or not. At that time we can then decide whether it works or > not > > > (i.e whether the gains it brings are more important than the problems > it > > > causes). > > > > > > > > Here’s my +1 to try this out. > > > > > > > > Some links: > > > > * pitest: http://pitest.org/ > > > > * descartes: https://github.com/STAMP-project/pitest-descartes > > > > * http://massol.myxwiki.org/xwiki/bin/view/Blog/ > ControllingTestQuality > > > > * http://massol.myxwiki.org/xwiki/bin/view/Blog/ > > MutationTestingDescartes > > > > > > > > If you’re curious, you can see a screenshot of a mutation score > report > > > at http://massol.myxwiki.org/xwiki/bin/download/Blog/ > > > MutationTestingDescartes/report.png > > > > > > > > Please cast your votes. > > > > > > > > Thanks > > > > -Vincent > > > > > > > > > > > > -- > > > Thomas Mortagne > > > > > >
