James,

Not dumb questions: an unfortunate situation. I do tool bakeoffs for clients a 
fair amount. I'm responsible for the rules Cigital initially sold to Fortify. I 
also attempt to work closely with companies like Coverity and understand deeply 
the underpinnings of that tool's engine. I've a fair amount of experience with 
Klocwork, unfortunately less with Ounce.

I understand the situation like this: technical guys at each of these companies 
are all great guys, smart, and understand their tool's capabilities and focus. 
They accurately describe what their tool does and don't misrepresent it.

On the other hand, I've experienced competition bashing in the sales process as 
I've helped companies with tool selections and bake offs. I see NO value in 
this. As I said in a previous post to this list: the tools differ both 
macroscopically in terms of approach and microscopically in terms rule 
implementation. Please see my previous post about bake-offs and such if you'd 
like more information on how to disambiguate tool capabilities objectively.

No blanket statement about quality or security fits any vendors' tool; ANY 
vendor. Ignore this level of commentary by the vendors.*(1)

No boolean answer exists to your question, let me give you some of my 
experiences:


 *   Fortify's SCA possesses far-and-away the largest rule set, covering both 
topics people consider purely security and those that may-or-may-not create 
opportunity for exploit (often when combined with other factors) which one may 
call quality rules. My impression is that SCA can be effectively used by 
Security Folk, QA Folk, or developers with a mind to improve the quality or 
security of their code. Recent inclusion of Findbugs bolsters SCA's 
capabilities to give code quality commentary.


 *   Coverity's Prevent often gets pigeon-holed as "a quality tool", but does 
an exceptional job of tracking down memory issues in C, C++. Skilled security 
consultants will tell you that failing to fix Prevent's results in your code 
will result in various memory-based command injection opportunities (BO, format 
string, write-anywhere's, etc.). It also effectively targets time-and-state 
issues, as well as other classes of bug. Prevent can effectively be used by 
Security Folk and Developers (or your rare hardcore QA person) to improve code 
quality and squelch opportunity for exploit.


 *   Klocwork's tool targets rule spaces similar to Fortify, but possesses 
less. Often pegged as a quality tool (as well), do find its UI (more than its 
engine) possess helpful features that only a QA professional would enjoy. This 
includes its defect density calculation, "reverse engineering" capabilities, 
and its reporting/time-series style. Klocwork can be effectively used by a 
Security guy to find security bugs, but I believe Fortify and Ounce have 
widened the rules gap in recent years.

Tackling your other questions in rapid succession:

There is no difference, technically, between the ability to scan for quality or 
security. However, each engine focuses on parsing and keeping track of only 
that state which provides meaningful data to their rules. You can imagine that 
Fortify carries a fair amount of information about where data came from and 
what functions may be dangerous and can therefore support new security rules 
easily. They don't carry around information to aggregate defect density readily 
like K7 can. Does this make one intrinsically better than the other for quality 
or security? Perhaps having worked on static analysis tools I'm cranky but I 
say, "No." If the market clearly mandated something specifically, all the 
vendors would augment their engine to support it. Some would be in a better 
position to offer it than others.

When I talk to vendors about COBOL and similar support they shudder. I think 
this space represents a huge opportunity for the first vendor to support it, 
but as a commercial organization, I wouldn't hold your breath on near-term 
support.

I could answer how these tools support new languages, but that doesn't seem 
like public domain knowledge. I'll let the vendors tackle that 'un.

----
John Steven
Technical Director; Principal, Software Security Group
Direct: (703) 404-5726 Cell: (703) 727-4034
Key fingerprint = 4772 F7F3 1019 4668 62AD  94B0 AE7F EEF4 62D5 F908

Blog: http://www.cigital.com/justiceleague
Papers: http://www.cigital.com/papers/jsteven

http://www.cigital.com
Software Confidence. Achieved.


*(1) I'm also explicitly dodging the quality vs. security debate here. Having 
read/posted to this list for the last 7 years, that semi-annual topic has been 
flogged more than your average dead equine.


________________________________
From: "McGovern, James F (HTSC, IT)" <[EMAIL PROTECTED]>

Most recently, we have met with a variety of vendors including but not
limited to: Coverity, Ounce Labs, Fortify, Klocwork, HP and so on. In
the conversation they all used interesting phrases to describe they
classify their competitors value proposition. At some level, this has
managed to confuse me and I would love if someone could provide a way to
think about this in a more boolean way.

- So when a vendor says that they are focused on quality and not
security, and vice versa what exactly does this mean? I don't have a
great mental model of something that is a security concern that isn't a
predictor of quality. Likewise, in terms of quality, other than
producing metrics on things such as depth of inheritance, cyclomatic
complexity, etc wouldn't bad numbers here at least be a predictor of a
bad design and therefore warrant deeper inspection from a security
perspective?

- Even if the rationale is more about people focus rather than tool
capability, is there anything else that would prevent one tool from
serving both purposes?

- Is it reasonable to expect that all of the vendors in this space will
have the ability to support COBOL, Ruby and Smalltalk sometime next year
so that customers don't have to specifically request it?

- Do the underlying compilers need to do something different since
languages such as COBOL aren't object-oriented which would make analysis
a bit different?

_______________________________________________
Secure Coding mailing list (SC-L) SC-L@securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________

Reply via email to