Steve, I definitely agree that not using the tools were a big limitation -- especially because the web interface wasn't as interactive and powerful as tool GUIs.
But for me, we had a hard time with using a consistent and actually, meaningful scoring: - What is a false-positive? - How important is this particular finding? This was to me one of the most important limitations since eventually we had most of the traces from the different tools. As Chris said, most of these problems should be addressed in the next SATE, and I hope many tool vendors will be in again :) Romain > -----Original Message----- > From: sc-l-boun...@securecoding.org [mailto:sc-l-boun...@securecoding.org] On > Behalf Of > Steven M. Christey > Sent: Wednesday, August 05, 2009 1:24 PM > To: Chris Wysopal > Cc: Secure Coding > Subject: Re: [SC-L] IBM Acquires Ounce Labs, Inc. > > > On Tue, 4 Aug 2009, Chris Wysopal wrote: > > > As a group of security practitioners it is amazing to me that we don't > > have more quantifiable testing and tools/services are just dismissed > > with anecdotal data. I am glad NIST SATE '09 will soon be underway and, > > at least for static analysis tools, we will have unbiased independent > > testing. I am hoping for a big improvement over last year. I especially > > like the category they are using for some flaws found as "valid but > > insignificant". Clearly they are improving based on feedback from SATE > > '08. > > By the way, I don't recall anybody mentioning this to SC-L before, but the > SATE 2008 writeup and raw data are available: > > http://samate.nist.gov/index.php/SATE.html > > In the NIST pub we cover a lot of lessons learned, especially in my paper. > >From the raw data you can see the complexities in doing this kind of > large-scale comparison. In my opinion, our biggest limitation was not > using live tools. > > - Steve > _______________________________________________ > Secure Coding mailing list (SC-L) SC-L@securecoding.org > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l > List charter available at - http://www.securecoding.org/list/charter.php > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) > as a free, non-commercial service to the software security community. > _______________________________________________ _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
