The difficulty with integration testing is that the services under test run in processes separated from the test one(s).
There is not obvious / existing mechanism to collect coverage data in this case. Several cycles back used to be a backdoor built into nova to enable coverage data collection during integration testing, but it has been removed long ago. andrea On Thu, Sep 29, 2016 at 12:12 PM Assaf Muller <as...@redhat.com> wrote: > On Thu, Sep 29, 2016 at 5:27 AM, milanisko k <vetri...@gmail.com> wrote: > > > > út 27. 9. 2016 v 20:12 odesílatel Assaf Muller <as...@redhat.com> napsal: > > On Tue, Sep 27, 2016 at 2:05 PM, Assaf Muller <as...@redhat.com> wrote: > > > > On Tue, Sep 27, 2016 at 12:18 PM, Timur Nurlygayanov < > tnurlygaya...@mirantis.com> wrote: > > Hi milan, > > we have measured the test coverage for OpenStack components with > coverage.py tool [1]. It is very easy tool and it allows measure the > coverage by lines of code and etc. (several metrics are available). > > [1] https://coverage.readthedocs.io/en/coverage-4.2/ > > > coverage also supports aggregating results from multiple runs, so you can > measure results from combinations such as: > > > > 1) Unit tests > 2) Functional tests > 3) Integration tests > 4) 1 + 2 > 5) 1 + 2 + 3 > > To my eyes 3 and 4 make the most sense. Unit and functional tests are > supposed to give you low level coverage, keeping in mind that 'functional > tests' is an overloaded term and actually means something else in every > community. Integration tests aren't about code coverage, they're about user > facing flows, so it'd be interesting to measure coverage > from integration tests, > > > Sorry, replace integration with unit + functional. > > > then comparing coverage coming from integration tests, and getting the set > difference between the two: That's the area that needs more unit and > functional tests. > > > To reiterate: > > Run coverage from integration tests, let this be c > Run coverage from unit and functional tests, let this be c' > > Let diff = c \ c' > > 'diff' is where you're missing unit and functional tests coverage. > > > Assaf, the tool I linked is a monkey-patched coverage.py but the collector > stores the stats in Redis --- gives the same accumulative collecting. > Is there any interest/effort to collect coverage stats from selected jobs > in CI, no matter the tool used? > > > Some projects already collect coverage stats on their post-merge queue: > > http://logs.openstack.org/61/61af70a734b99e61e751cfb494ddc93a85eec394/post/nova-coverage-db-ubuntu-xenial/55210aa/ > > It's invoked with 'tox -e cover' which you define in your project's > tox.ini file, I imagine most projects if not all have it set up to gather > coverage from a unit tests run. > > > > > > > > > > On Tue, Sep 27, 2016 at 1:06 PM, Jordan Pittier < > jordan.pitt...@scality.com> wrote: > > Hi, > > On Tue, Sep 27, 2016 at 11:43 AM, milanisko k <vetri...@gmail.com> wrote: > > Dear Stackers, > I'd like to gather some overview on the $Sub: is there some infrastructure > in place to gather such stats? Are there any groups interested in it? Any > plans to establish such infrastructure? > > I am working on such a tool with mixed results so far. Here's my approach > taking let's say Nova as an example: > > 1) Print all the routes known to nova (available as a python-routes > object: nova.api.openstack.compute.APIRouterV21()) > 2) "Normalize" the Nova routes > 3) Take the logs produced by Tempest during a tempest run (in > logs/tempest.txt.gz). Grep for what looks like a Nova URL (based on port > 8774) > 4) "Normalize" the tested-by-tempest Nova routes. > 5) Compare the two sets of routes > 6) ???? > 7) Profit !! > > So the hard part is obviously the normalizing of the URLs. I am currently > using a tons of regex.... :) That's not fun. > > I'll let you guys know if I have something to show. > > I think there's real interest on the topic (it comes up every year or so), > but no definitive answer/tool. > > Cheers, > Jordan > > > > > <https://www.scality.com/backup/?utm_source=signatures&utm_medium=email&utm_campaign=backup2016> > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > > Timur, > Senior QA Manager > OpenStack Projects > Mirantis Inc > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
