Hi, > On 17 Oct 2018, at 11:20, Vincent Massol <vinc...@massol.net> wrote: > > Hi, > > [snip] > >> Process to run DSpot: >> 1) Pick a module. Measure coverage and mutation score (or take the value >> there already if they’re in the pom.xml). Same as for Descartes testing. >> 2) Run DSpot on the module, see >> https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot for >> explanations > > One important detail that I had missed. We need to run Dspot with > “—descartes” on the command line so that it uses Descartes for computing the > mutation score for mutations and only keep tests that increase the mutation > score as reported by Descartes.
So actually, after speaking with Benjamin, I’ve realized a few things: * By default DSpot runs with the PIT selector (PitMutantScoreSelector) which is configured to use the default PIT mutations. This is why we need to run with the PIT selector but configured to use the Descartes mutation, and this is done by specifying --descartes. * Now this will optimize the generation of new tests for their increased mutation score. Right now we got 0% all the time on our tests (see https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2hm0jgCKUw/edit#gid=2061481816) and it’s because we didn’t use --descartes. We need to try again or run on new modules with --descartes and see what it gives us. It’s possible it’ll generate even less tests… * For the coverage part, there are 2 other selectors that can be used with DSpot to generate tests that all increase the coverage: ** "--test-criterion JacocoCoverageSelector": uses jacoco and keep tests that increase the instruction coverage ** "--test-criterion CloverCoverageSelector”: uses openclover and keep tests that increase the branch coverage So we need to test with the various selectors and see what we get. If we want to get the best values, we should use --descartes for K03 and either jacoco or clover selector for K01. Now we need to see what tests we get. Thanks -Vincent > >> 3) If DSpot has generated tests, add them to XWiki’s source code in >> src/test/dspot and add the following to the pom of that module: >> >> <build> >> <plugins> >> <!-- Add test source root for executing DSpot-generated tests --> >> <plugin> >> <groupId>org.codehaus.mojo</groupId> >> <artifactId>build-helper-maven-plugin</artifactId> >> </plugin> >> </plugins> >> </build> >> >> Example: >> https://github.com/xwiki/xwiki-commons/tree/244ee07976c691c335b7f54c48e6308004ba3d82/xwiki-commons-core/xwiki-commons-crypto/xwiki-commons-crypto-cipher >> >> Note: The generated tests sometimes need to be modified a bit to pass. >> Personally I’ve only committed tests that were passing and I reported issues >> for those that were not passing. >> >> 4) File the various reports: >> a) https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki >> both for success and failures >> b) >> https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2hm0jgCKUw/edit#gid=2061481816 >> c) for failures, file a github issue at >> https://github.com/STAMP-project/dspot/issues and link to the place on >> https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki >> where we put the failing result. >> >> Note: The reason we need to report failures too is because DSpot fails a lot >> so we need to show what we have tested >> >> Thanks >> -Vincent >> > > [snip] > > Thanks > -Vincent > >
