Great stuff Nash. To re-iterate one important statement: Many orgs today will *only* respond to a working exploit. (I'm not sure what the sample (%clue) of orgs I see is vs. Cigital's client, but...)
Pen-test vs. code review, black-box, white-box, whatever: There is absolutely no difference at the end of the day in terms of *verification* between finding SQL injection or susceptibility to XSS attacks via black box attacks or fully white human eyeball source code review. Commission, Omission, transmission, no difference. It's verification. The pen test *can* make an acceptable 'punch card'. Depends on how the punch card results are written: + Does the punch card simply list the "top 10 XSS'able parameters"? + Or does it provide a finding covering both/either: - design Omission *or* - implementation Commission/failure to "encode output appropriately for the User Agent(s) specified"? The rabbit hole can, of course, go far more usefully deeper in terms of problem resolution with source code review. I think that's the worst sentence I've ever written. Today, maybe for the last <= 1.5 years, I get "but what library should I use to accomplish this with [insert_framework]?" instead of "uh, what's output encoding?" or "input validation is too hard/slow". Assertion Falsification can be a tail-chaser. That's why you add in business context. I think you'll find that many good pen testers sit down with the business and help them define security goals for the application, e.g.- "Rob must never get Sally's report" or "Sally's report must remain sacred at all points, in transit, storage, access, etc." Commonly this is achieved through threat modeling, which turns pen testing into a verification mechanism. Ultimately the folks on this list are pretty smart and I'd wonder why/if this dialogue is needed, except that a recent discussion opened my eyes a bit to approaches I thought were doornail dead. A fairly large consultancy with a practice focused on "application security" contacted me earlier this year and in the course of discussing their approach to appsec I asked, "but when do you talk to the business and when do you work with the developers?" and their response was "What?" After repeating the question I got told "Oh, no, you won't talk to those folks or get to see their documentation; we're security professionals, not developers". So evidently there is still a market for high-dollar, completely blind pen tests of apps with zero business context and no understanding of architecture, dev/design goals, etc. Hmmm. That's what I'm guessing Gary means, and surely that sun is slowly setting. -ae p.s. - Nash, when I first read your post, I thought p2 started with "Pen tests are highly addictive". Then I re-read. > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Nash > Sent: Thursday, July 13, 2006 9:18 AM > To: Gary McGraw > Cc: Secure Coding Mailing List > Subject: Re: [SC-L] ddj: beyond the badnessometer > > On Thu, Jul 13, 2006 at 07:56:16AM -0400, Gary McGraw wrote: > > > > Is penetration testing good or bad? > > http://ddj.com/dept/security/189500001 > > > Test coverage is an issue that penetration testers have to deal with, > without a doubt. Pen-tests can never test every possible attack > vector, which means that pen-tests can not always falsify a security > assertion. > > Ok. But... > > First, pen-testers are highly active. The really good ones spend alot > of time in the hacker community keeping up with the latest attack > types, methods, and tools. Hence, the relevance of the test coverage > you get from a skilled pen-tester is actually quite good. In addition, > the tests run are similar to real attacks you're likely to see in the > wild. Also, pen-testsing is often intelligent, focused, and highly > motivated. After all, how would you like to have to go back to your > customer with a blank report? And, the recommendations you get can be > quite good because pen-testers tend to think about the entire > deployment environment, instead of just the code. So, they can help > you use technologies you already have to fix problems instead of > having to write lots and lots of new code to fix them. All of these > make pen-testing a valuable exercise for software environments to go > through. > > Second, every software application in deployment has an associated > level of acceptable risk. In many cases, the level of acceptable risk > is low enough that penetration testing provides all the verificaton > capabilities needed. In some cases, the level of acceptable risk is > really low and even pen-testing is overkill. I do mostly code review > work these days, but I find that pen-testing has more general > applicability to my customers. There are exceptions, but not that > many. > > Third, pen-tests also have real business advantages that don't > directly address risk mitigation. Pen-test reports are typically more > "down to earth." That is, they can be read more easily and the attacks > can be demonstrated more easily to business leaders, executives, and > other stakeholders. In my experience, recommendations from both > pen-tests and code reviews are commonly ignored. But, a good pen-test > gets the executive blood flowing in a way that code-oriented security > evaluations just don't. > > Fourth, assertion falsification isn't always what you're after. Being > able to falsify the statement, "this app is secure enough," is a > common objective, but it's not really that useful for most businesses. > What exactly is secure enough? How do you define it? How do you > measure it? How much accuracy do you need? How do you get more > accuracy, if you want it? How much do you trust your expert's opinion? > > Sometimes, it's better to simply demonstrate a positive assertion, > such as: > > - This application is not subject to known, automatic attacks. > - This application demonstrates the same security profile in all > supported deployment environments. > - This application demonstrates different security profiles, > depending upon the deployment environment. > - The latest MS patch does not affect the testable security > profile of this application. > > These are all assertions that pen-testing is arguably pretty good for > demonstrating. In some cases it might even be better than code > anlaysis--e.g., the effects of new environments or upgrades to > low-level libraries, virtual machines, operating systems. > > Finally, my freind Sam pointed out that only during some kind of > pen-testing can you really identify what the actual attack surface of > an application looks like in its final deployment environment. This is > especially relevant in today's world because applications are now made > as much through integration of existing, off-the-shelf components as > through new development. A "new" application might only have a few > thousand lines of original code, but might be resting on top of a > software stack that has millions. Whether it's J2EE, .NET, or LAMP, > all those environments are only really practical to test using some > form of pen-test. > > Every security assessment methodology has its limits. Pen-testing has > limited falsification capabilities. Code review, various kinds of > code analysis, unit testing, whatever else. These methods can all have > practical financial limitations and information accesibility > problems. > > Of course, all these are good approaches and a wise security manager > will employ as wide a variety of assessment methods as he can afford > so that they compliment each other. But, affordability is a real > concern for most busineses and pen-testing is pretty affordable. > > In the end, no assessment methodology produces results that are as > good as having a skilled Security Developer on your team during the > application design stage. Getting a security architecture in place > that matches your risk tolerance and functional requirements is the > single best way to prevent intrusions, bar none. > > > nash e. foster > Stratum Security, LLC > > > -- > > "the lyf so short, the craft so long to lerne." > - Geoffrey Chaucer > > _______________________________________________ > Secure Coding mailing list (SC-L) > SC-L@securecoding.org > List information, subscriptions, etc - > http://krvw.com/mailman/listinfo/sc-l > List charter available at - > http://www.securecoding.org/list/charter.php _______________________________________________ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php
