> On 17 Oct 2018, at 15:54, Vincent Massol <vinc...@massol.net> wrote:
>
> Hi,
>
>> On 17 Oct 2018, at 11:20, Vincent Massol <vinc...@massol.net> wrote:
>>
>> Hi,
>>
>> [snip]
>>
>>> Process to run DSpot:
>>> 1) Pick a module. Measure coverage and mutation score (or take the value
>>> there already if they’re in the pom.xml). Same as for Descartes testing.
>>> 2) Run DSpot on the module, see
>>> https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot for
>>> explanations
>>
>> One important detail that I had missed. We need to run Dspot with
>> “—descartes” on the command line so that it uses Descartes for computing the
>> mutation score for mutations and only keep tests that increase the mutation
>> score as reported by Descartes.
>
> So actually, after speaking with Benjamin, I’ve realized a few things:
>
> * By default DSpot runs with the PIT selector (PitMutantScoreSelector) which
> is configured to use the default PIT mutations. This is why we need to run
> with the PIT selector but configured to use the Descartes mutation, and this
> is done by specifying --descartes.
> * Now this will optimize the generation of new tests for their increased
> mutation score. Right now we got 0% all the time on our tests (see
> https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2hm0jgCKUw/edit#gid=2061481816)
> and it’s because we didn’t use --descartes. We need to try again or run on
> new modules with --descartes and see what it gives us. It’s possible it’ll
> generate even less tests…
> * For the coverage part, there are 2 other selectors that can be used with
> DSpot to generate tests that all increase the coverage:
> ** "--test-criterion JacocoCoverageSelector": uses jacoco and keep tests that
> increase the instruction coverage
> ** "--test-criterion CloverCoverageSelector”: uses openclover and keep tests
> that increase the branch coverage
>
> So we need to test with the various selectors and see what we get.
I’ve retested on xwiki-commons-component-default:
1) With —descartes: failure, see
https://github.com/STAMP-project/dspot/issues/584
2) With jacoco selector: failure, see
https://github.com/STAMP-project/dspot/issues/586. I’ve manually fixed the
tests and remove those that didn’t pass. I got only +0.18% jacoco coverage
increase and -2% descartes mutation score… That’s the problem, we would need a
selector that optimizes for both. I’ve created
https://github.com/STAMP-project/dspot/issues/587
3) With clover selector: no tests generated! Opened
https://github.com/STAMP-project/dspot/issues/588
So my recommendation is to wait for
https://github.com/STAMP-project/dspot/issues/584 to be fixed and then to use
—descartes for our measures FTM.
Thanks
-Vincent
PS: Command lines used for reference:
- java -jar
/Users/vmassol/dev/dspot/dspot/target/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar
--path-to-properties dspot.properties --descartes --verbose
--generate-new-test-class --with-comment
- java -jar
/Users/vmassol/dev/dspot/dspot/target/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar
--path-to-properties dspot.properties --test-criterion JacocoCoverageSelector
--verbose --generate-new-test-class --with-comment
- java -jar
/Users/vmassol/dev/dspot/dspot/target/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar
--path-to-properties dspot.properties --test-criterion CloverCoverageSelector
--verbose --generate-new-test-class --with-comment
>
> If we want to get the best values, we should use --descartes for K03 and
> either jacoco or clover selector for K01. Now we need to see what tests we
> get.
>
> Thanks
> -Vincent
>
>>
>>> 3) If DSpot has generated tests, add them to XWiki’s source code in
>>> src/test/dspot and add the following to the pom of that module:
>>>
>>> <build>
>>> <plugins>
>>> <!-- Add test source root for executing DSpot-generated tests -->
>>> <plugin>
>>> <groupId>org.codehaus.mojo</groupId>
>>> <artifactId>build-helper-maven-plugin</artifactId>
>>> </plugin>
>>> </plugins>
>>> </build>
>>>
>>> Example:
>>> https://github.com/xwiki/xwiki-commons/tree/244ee07976c691c335b7f54c48e6308004ba3d82/xwiki-commons-core/xwiki-commons-crypto/xwiki-commons-crypto-cipher
>>>
>>> Note: The generated tests sometimes need to be modified a bit to pass.
>>> Personally I’ve only committed tests that were passing and I reported
>>> issues for those that were not passing.
>>>
>>> 4) File the various reports:
>>> a) https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki
>>> both for success and failures
>>> b)
>>> https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2hm0jgCKUw/edit#gid=2061481816
>>> c) for failures, file a github issue at
>>> https://github.com/STAMP-project/dspot/issues and link to the place on
>>> https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki
>>> where we put the failing result.
>>>
>>> Note: The reason we need to report failures too is because DSpot fails a
>>> lot so we need to show what we have tested
>>>
>>> Thanks
>>> -Vincent
>>>
>>
>> [snip]
>>
>> Thanks
>> -Vincent
