So if I sum up, we're moving the scope of global coverage from each
modules to the whole project, right?
+1 for getting email/build failing only if the global TPC decrease
Now I assume in case of decrease, we can get the whole report with info
about which modules had a change in their TPC, right?
Simon
On 11/02/2019 14:48, Thomas Mortagne wrote:
+1
On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol <[email protected]> wrote:
Hi devs,
After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and after
analyzing several modules (I analyzed about 4-5 in total), I think we should
improve a bit the strategy to make it usable and applicable.
Lessons learnt
============
* It takes a lot of time to analyze a single global tpc drop (every time it
takes me around 2 hours)
* In general the results of the analysis are not that great. There are often
“good enough” reasons for the drop. It’s often a lack of unit tests and code
that is exercised by functional tests but the path has changed for various
reasons.
* I find the ratio of time to spend on the analysis vs result to be too low.
* In the end what’s important is that our global TPC continues to grow
New Strategy
===========
* We run the Clover Jenkins pipeline every night (between 11PM-8AM)
* The pipeline sends an email whenever the new report has its global TPC going
down when compared with the baseline (vs one or more modules had their TPC
lowered in TAKE 2)
* The baseline report is the report generated just after each XS release. This
means that we keep the same baseline during a XS release
** Technically it means that the pipeline will update the latest.txt file
(which contains the clover report timestamp) when it notices a version change
* We add a step in the Release Plan Template to have the report passing before
we can release.
* The RM is in charge of a release from day 1 to the release day (already the
case), and is also in charge of making sure that the global coverage job
failures get addressed before the release day so that we’re ready on the
release day.
* Implementation detail: don’t send a failure email when there are failing
tests in the build, to avoid false positives
WDYT?
Thanks
-Vincent
--
Simon Urli
Software Engineer at XWiki SAS
[email protected]
More about us at http://www.xwiki.com
