Hi Mike,
This is excellent stuff. More open source projects should strive to do this!
Thanks for your efforts.
// Sebastien
> On 4 May 2014, at 02:57 pm, Michael Rash <[email protected]> wrote:
>
>
> Hi all,
>
> Thought I would provide some detail on an fwknop development effort around
> test coverage. The fwknop test suite (test/test-fwknop.pl) has done a pretty
> good job of validating functionality for a while now, but a lot more tests
> are being added to try and get to 100% test coverage on a function, line, and
> branch basis. It turns out that achieving "100%" coverage for all three of
> these is extremely hard, and that is a general statement for all software -
> not just fwknop.
>
> Cutting to the chase, here are the current test coverage results as measure
> by the "gcov" tool with these .html views generated by "lcov":
>
> http://www.cipherdyne.org/fwknop/lcov-results/
>
> So, in terms of functions, the test suite does fairly well with slightly over
> 99% of all functions covered. Line coverage is less at about 85% and branch
> coverage is even harder at about 75%. While this may not sound very good, it
> turns out that the usage of defensive checks (good for security) inflate
> these numbers. That is, fwknop is careful to validate all sorts of things at
> various stages, and sometimes these checks are redundant making later checks
> harder to trigger than earlier ones. Another example is that all functions
> that deal with heap allocated memory have NULL checks as one would expect,
> but in practice it is hard to trigger a NULL result unless there is extreme
> memory pressure and this would affect test suite operations at the same time
> (and worse). In some cases, it would be possible to account for these
> problems by building in special test coverage support into the fwknop code
> base, and some of this has already started. For example, the new python
> fuzzer (see test/spa_fuzzer.py) leverages special #define's to assist with
> test coverage. This approach could be extended to do things like maintain a
> global counter that is incremented with any malloc() (and friends) calls, and
> then force a NULL result whenever this counter hits any of a set of values
> during normal fwknop operations.
>
> All of the above is run time testing, and is complementary to using static
> analyzers like Coverity. The goal is to maintain an extremely high standard
> for the fwknop code base.
>
> Thanks,
>
> --Mike
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos. Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Fwknop-discuss mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/fwknop-discuss
------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Fwknop-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fwknop-discuss
