Re: [Development] Code coverage statistics online

[Sébastien Fricker]

Bill,
this is corrected now: 
http://download.froglogic.com/public/qt5-squishcoco-report/QtBase/source_543.html#line467
Regards
Am 08.10.2012 um 12:26 schrieb Sébastien Fricker :

> Thanks Bill,
> this is effectively not correct, the lines should be marks as executed.
> Let me have a deeper look on it.
> 
> 
> On 08.10.2012 12:19, bill.k...@nokia.com wrote:
>> Off-the-mailing-list…
>>  
>> “QString QSqlDriver::sqlStatement(StatementType type, const QString 
>> &tableName,” function,
>>  
>> I think the marking is wrong.
>> It says: “never executed (the execution status of this line is deducted):  
>> s.reserve(128);” for example, but it shows that by logic, that should be 
>> executed, as the first case of the switch statement below is…
>>  
>> evaluated: i < rec.count()
>> TRUE
>> FALSE
>> yes
>> Evaluation Count:128
>> yes
>> Evaluation Count:64
>>  
>> Now, this may just be optimisation performed by the compiler, but it could 
>> also be erroneous. If it’s the first, I’d probably turn off for testing 
>> (there’s debug dumping functions that aren’t showing as being hit either).
>> I really do think it’s the first, but I’d like it to be checked out in case 
>> it’s actually a bug or corner case.
>>  
>> Thanks,
>> Bill.
>> --
>>  
>> Bill King
>> Test Engineer
>> Nokia, Guidance Team
>>  
>> From: development-bounces+bill.king=nokia@qt-project.org 
>> [mailto:development-bounces+bill.king=nokia@qt-project.org] On Behalf Of 
>> ext Sébastien Fricker
>> Sent: Sunday, 7 October 2012 9:08 AM
>> To: development@qt-project.org
>> Subject: Re: [Development] Code coverage statistics online
>>  
>> One possible request: entry marking in the source reports, ie:
>> QString QMimeType::filterString() const
>> From: 
>> http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html
>>  
>> Was the test entered but the if lines not tested?
>>  
>> Another example: void QMimeTypePrivate::clear()
>>  
>> It would be more understandable if the entry point was highlighted, instead 
>> of the exit point of the block.
>>  
>> MfG,
>> Bill.
>> This is now available.
> 

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[bill.king]

Much more instantly-readable, thankyou very much :)

--

Bill King
Test Engineer
Nokia, Guidance Team

From: development-bounces+bill.king=nokia@qt-project.org 
[mailto:development-bounces+bill.king=nokia@qt-project.org] On Behalf Of 
ext Sébastien Fricker
Sent: Sunday, 7 October 2012 9:08 AM
To: development@qt-project.org
Subject: Re: [Development] Code coverage statistics online


One possible request: entry marking in the source reports, ie:

QString QMimeType::filterString() const

From: 
http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html



Was the test entered but the if lines not tested?



Another example: void QMimeTypePrivate::clear()



It would be more understandable if the entry point was highlighted, instead of 
the exit point of the block.



MfG,

Bill.
This is now available.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[Sébastien Fricker]

> One possible request: entry marking in the source reports, ie:
> QString QMimeType::filterString() const
> From: 
> http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html
> 
> Was the test entered but the if lines not tested?
> 
> Another example: void QMimeTypePrivate::clear()
> 
> It would be more understandable if the entry point was highlighted, instead 
> of the exit point of the block.
> 
> MfG,
> Bill.
This is now available.___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[Sébastien Fricker]

Hi,
providing a code coverage analysis of Qt is a living project and so we would be 
happy of any feedback of the Qt developers.

Issues and suggestions can be addressed to the Squish Coco forum:
http://www.froglogic.com/forum/viewforum.php?f=10

We found that it is better to not put issues on metrics in the general Qt 
developer list, 
and so we prefer that people use a dedicated forum. 
Sébastien___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[bill.king]

Awesome work guys :)

One possible request: entry marking in the source reports, ie:
QString QMimeType::filterString() const
From: 
http://download.froglogic.com/public/qt5-squishcoco-report/libQtWidgets/source_147.html

Was the test entered but the if lines not tested?

Another example: void QMimeTypePrivate::clear()

It would be more understandable if the entry point was highlighted, instead of 
the exit point of the block.

MfG,
Bill.

--

Bill King
Test Engineer
Nokia, Guidance Team

From: development-bounces+bill.king=nokia@qt-project.org 
[mailto:development-bounces+bill.king=nokia@qt-project.org] On Behalf Of 
ext Sébastien Fricker
Sent: Thursday, 27 September 2012 10:15 AM
To: development@qt-project.org
Subject: Re: [Development] Code coverage statistics online


> Overall the coverage was less than I expected, so I started reading the

> source files list starting from lowest coverage and found some surprising

>results, e.g. qmimedata.cpp 0% coverage, qsettings.cpp nearly 0%

> coverage although I know there are tests for these.

> In the execution list at

> http://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.html<http://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.html>

> indeed it seems that tst_QSettings, tst_QMimeData are missing from the

> list :(  I suppose there are others missing too. Any idea what might

> have caused them to be excluded from the results?



If the tests are not present it can be:

 1) the unit tests are not compiling.

 2) the unit tests are not instrumented (should not be the case if part of 
QtBase)

 3) make check does not execute the unit test.

 4) results are not collected



Let me have a look on it.



Rohan, this analysis is the first one provided, and I you know better than 
everybody that

maintaining the test suite is not an easy job.

That's why it is important that if something is not correct that the community 
report it.

This is the only way to get accurate measurements.



Sébastien
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[Sébastien Fricker]

> Would you include add-ons in the reports too?
Actually we start with QtBase. Of course in the future, 
the goal is to provide an analysis on the whole Qt source.


> Anyway to get run output logs from failed tests?
Good suggestion, will be present in the next daily coverage report.

Sébastien

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[Lorn Potter]

On 27/09/12 04:06, Harri Porten wrote:
> Hi!
>
> We started publishing code coverage results of Qt 5 unit test runs as
> produced by our tool Squish Coco:
>
> http://download.froglogic.com/public/qt5-squishcoco-report/ (results)
> 
> http://download.froglogic.com/public/effectively-testing-qt5-using-squishcoco/
>  (setup and example)
>
>>From that link you can browse to the coverage data of libraries, functions
> and source files.
>
> We chose a decision/coverage analysis method. In comparison to line
> coverage this is a bit harder to grasp and gives different numbers but for
> a developer it's the more useful. I plan to follow up with a study of a
> small number of exemplaric cases accompanied with suggested patches to
> tests and maybe code. And to make things easier to understand in the
> mostly-static HTML output we'll also improve the coloring of instrumented
> branches soon.
>
> Provided that the cron job works the results will be updated daily. If
> anyone has an idea how this could be integrated more tightly with the
> automated Qt project test results please let us know. Caroline Chao was
> already using a predecessor of the tool for the same purpose within
> the Nokia build infrastructure btw.
>
> Feedback of any kind is welcome.

Would you include add-ons in the reports too?

Anyway to get run output logs from failed tests?



-- 
Lorn Potter
Senior Software Engineer, QtSensors/QtSensorGestures/QtSystemInfo


___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[Sébastien Fricker]

> Overall the coverage was less than I expected, so I started reading the
> source files list starting from lowest coverage and found some surprising
>results, e.g. qmimedata.cpp 0% coverage, qsettings.cpp nearly 0%
> coverage although I know there are tests for these.
> In the execution list at
> http://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.html
>  
> 
> indeed it seems that tst_QSettings, tst_QMimeData are missing from the
> list :(  I suppose there are others missing too. Any idea what might
> have caused them to be excluded from the results?

If the tests are not present it can be:
 1) the unit tests are not compiling.
 2) the unit tests are not instrumented (should not be the case if part of 
QtBase)
 3) make check does not execute the unit test.
 4) results are not collected

Let me have a look on it.

Rohan, this analysis is the first one provided, and I you know better than 
everybody that
maintaining the test suite is not an easy job.
That's why it is important that if something is not correct that the community 
report it. 
This is the only way to get accurate measurements.

Sébastien

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[Rohan McGovern]

Harri Porten said:
> Hi!
> 
> We started publishing code coverage results of Qt 5 unit test runs as 
> produced by our tool Squish Coco:
> 
>http://download.froglogic.com/public/qt5-squishcoco-report/ (results)
>
> http://download.froglogic.com/public/effectively-testing-qt5-using-squishcoco/
>  (setup and example)
> 
> >From that link you can browse to the coverage data of libraries, functions 
> and source files.
> 
> We chose a decision/coverage analysis method. In comparison to line 
> coverage this is a bit harder to grasp and gives different numbers but for 
> a developer it's the more useful. I plan to follow up with a study of a 
> small number of exemplaric cases accompanied with suggested patches to 
> tests and maybe code. And to make things easier to understand in the 
> mostly-static HTML output we'll also improve the coloring of instrumented 
> branches soon.
> 
> Provided that the cron job works the results will be updated daily. If 
> anyone has an idea how this could be integrated more tightly with the 
> automated Qt project test results please let us know. Caroline Chao was 
> already using a predecessor of the tool for the same purpose within 
> the Nokia build infrastructure btw.
> 
> Feedback of any kind is welcome.

Overall the coverage was less than I expected, so I started reading the
source files list starting from lowest coverage and found some surprising
results, e.g. qmimedata.cpp 0% coverage, qsettings.cpp nearly 0%
coverage although I know there are tests for these.

In the execution list at
http://download.froglogic.com/public/qt5-squishcoco-report/libQtCore.so.5.0.0.html
indeed it seems that tst_QSettings, tst_QMimeData are missing from the
list :(  I suppose there are others missing too. Any idea what might
have caused them to be excluded from the results?
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] Code coverage statistics online

[Jedrzej Nowacki]

On Wednesday 26. September 2012 20.06.21 Harri Porten wrote:
> Feedback of any kind is welcome.

Nice! 

 Jędrek
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development