> On 29 Aug 2018, at 11:20, Vincent Massol <vinc...@massol.net> wrote:
[snip]
> Objectives/KPIs/Metrics for STAMP
> ===========================
>
> The STAMP project has defined 9 KPIs that all partners (and thus XWiki) need
> to work on:
>
> 1) K01: Increase test coverage
> * Global increase by reducing by 40% the non-covered code. For XWiki since
> we’re at about 70%, this means reaching about 80% before the end of STAMP
> (ie. before end of 2019)
> * Increase the coverage contributions of each tool developed by STAMP.
>
> Strategy:
> * Primary goal:
> ** Increase coverage by executing Descartes and improving our tests. This is
> http://markmail.org/message/ejmdkf3hx7drkj52
> ** Don’t do anything with DSpot. I’ll do that part. Note that the goal is to
> write a Jenkins pipeline to automatically execute DSpot from time to time and
> commit the generated tests in a separate test source and have our build
> execute both src/test/java and this new test source.
Contrary to what was proposed initially, it would be nice to run DSpot too.
FTR a good command line to use for DSpot is:
java -jar <path>/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar
--path-to-properties dspot.properties --verbose --generate-new-test-class
--with-comment
The --generate-new-test-class tells DSpot to generate in its output dir only
the new tests added and not include existing tests.
The --with-comment tells DSpot to keep the comments and thus the license header
too
I did a session today and committed the results in
https://github.com/STAMP-project/dspot-usecases-output/commit/113726c0aac3af3df30334d14115d89227eaebdc
What I did:
* For each module tested with DSpot create a folder in
https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki
* For cases where DSpot could generate some tests, commit them and modify the
pom.xml so that they are executed
* Note: tests need to have their license headers adjusted so that they don’t
fail the build
* Computed coverage + mutation scores before and after and reported in the
README.md in each folder
Thanks
-Vincent
> ** Don’t do anything with TestContainers FTM since I need to finish a first
> working version. I may need help in the future to implement docker images for
> more configurations (on Oracle, in a cluster, with LibreOffice, with an
> external SOLR server, etc).
> ** For EvoCrash: We’ll count contributions of EvoCrash to coverage in K08.
> * Secondary goal:
> ** Increase our global TPC as mentioned above by fixing the modules in red.
>
> 2) K02: Reduce flaky tests.
> * Objective: reduce the number of flaky tests by 20%
>
> Strategy:
> * Record flaky tests in jira
> * Fix the max number of them
>
> 3) K03: Better test quality
> * Objective: increase mutation score by 20%
>
> Strategy:
> * Same strategy as K01.
[snip]
Thanks
-Vincent
