Hi all,
Thought I would provide some detail on an fwknop development effort around
test coverage. The fwknop test suite (test/test-fwknop.pl) has done a
pretty good job of validating functionality for a while now, but a lot more
tests are being added to try and get to 100% test coverage on a function,
line, and branch basis. It turns out that achieving "100%" coverage for
all three of these is extremely hard, and that is a general statement for
all software - not just fwknop.
Cutting to the chase, here are the current test coverage results as measure
by the "gcov" tool with these .html views generated by "lcov":
http://www.cipherdyne.org/fwknop/lcov-results/
So, in terms of functions, the test suite does fairly well with slightly
over 99% of all functions covered. Line coverage is less at about 85% and
branch coverage is even harder at about 75%. While this may not sound very
good, it turns out that the usage of defensive checks (good for security)
inflate these numbers. That is, fwknop is careful to validate all sorts of
things at various stages, and sometimes these checks are redundant making
later checks harder to trigger than earlier ones. Another example is that
all functions that deal with heap allocated memory have NULL checks as one
would expect, but in practice it is hard to trigger a NULL result unless
there is extreme memory pressure and this would affect test suite
operations at the same time (and worse). In some cases, it would be
possible to account for these problems by building in special test coverage
support into the fwknop code base, and some of this has already started.
For example, the new python fuzzer (see test/spa_fuzzer.py) leverages
special #define's to assist with test coverage. This approach could be
extended to do things like maintain a global counter that is incremented
with any malloc() (and friends) calls, and then force a NULL result
whenever this counter hits any of a set of values during normal fwknop
operations.
All of the above is run time testing, and is complementary to using static
analyzers like Coverity. The goal is to maintain an extremely high
standard for the fwknop code base.
Thanks,
--Mike
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos. Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
Fwknop-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fwknop-discuss
