Hi,
you may remember the last post I sent in January where I proposed the
way of how tests for XML-RPC will be organized. Well, things changed and
since I am going to finish my bachelor thesis I would like to present
you the result.
After a lot of discussions with my supervisors and hours of work I have
managed to connect the code coverage tools. The reason why there is code
coverage is that documentation for both APIs does not exist. Frontend
API comes just with specification of all calls, but it does not mention
any behavior. The test requirements therefore come from the source code.
The "project" consists of two modules -- the tests module and coverage
module.
The tests module contains BeakerLib-based tests. There is also a
BeakerLib plugin which utilizes common actions. These tests basically
call some XML-RPC procedure, assert the results, and check whether there
was an addition to the server log (httpd's error_log or tomcat's
catalina.out based on type of tested API -- frontend/backend). This
module works standalone and does not depend on anything from coverage
module.
The coverage module extends the tests module by measuring the code
coverage (branch coverage, actually). It does need to be installed and
handled by a superuser. Installation instruments the rhn.jar using
Cobertura and updates the mod_wsgi/mod_python configuration to use
different handlers. These handlers just start and stop the collecting of
coverage data using coverage.py and wrap the real handlers. After
installation it is possible to see which parts of code was executed
using bundled scripts. There might be a problem with superuser
privileges (Cobertura flushes the coverage data into file only on
tomcat's shutdown and therefore the tomcat must be restarted before
evaluation of coverage can start) but since it is reasonable to use it
only against development installation, it should not be a problem to run
few scripts as a superuser.
The example of output (just tests module with one failing test out of two):
tests-module$ ./runtests.sh
FAIL of /auth/login: See /tmp/beakerlib-XR30U3q/journal.txt
Passed 1 out of 2 tests
The example of output (the coverage module calling the same tests as above):
coverage-module$ sudo ./runtests.sh
Coverage data file were successfully reset
FAIL of /auth/login: See /tmp/beakerlib-eFh18KQ/journal.txt
Passed 1 out of 2 tests
Branch coverage: Java 1%, Python 3%
If you are interested in the HTML coverage reports, see these:
* Coverage.py HTML report (~ 1.5 MB):
http://vlki.cz/python_html_report.tar.gz
* Cobertura HTML report (~ 5 MB): http://vlki.cz/java_html_report.tar.gz
Everything was tested using Spacewalk 1.2/Fedora 14.
Well... Do you like it? Do you find it useful?
Thanks for your thoughts,
Jan
_______________________________________________
Spacewalk-devel mailing list
Spacewalk-devel@redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-devel
