I think something significant has been excluded from this thread on BSIMM, and that's the role of SSF. If I am to understand correctly, SSF implementation was the experiment, and BSIMM was the resultant model for assessing (measuring) the maturity of software security programs being developed with SSF. Is that a fair read?
In the end, the same problem exists here: development of a model based on data does not a validated model make. By way of Bayesian thinking, you can definitely create a viable model based on a small sampling (FAIR is based on this principle). However, you still have to go back and /independently/ validate that model by exercising it outside of the original data set. Does BSIMM appear to be a viable model for assessing the maturity of SSF-based software security programs? Sure. Has BSIMM been independently validated using a similar data set? No. Is BSIMM inherently reliant on SSF? Methinks so. Does BSIMM make universal sense for measuring *any* software security program? Unclear, but is questionable. Is SSF a de fact standard for building a software security program? Unclear. Anyway... fun debate! I fully appreciate the new work here, and I think it holds promise. The release of the initial draft is probably cause for a small celebration by the creators. However, until the model has been validated with independent data and has been shown to be generically applicable, I think there is still reason to hold back the really good bubbly. ;) cheers, -ben Brian Chess wrote: > Ben, Pravir, > Skepticism is healthy, but recognize that: > > 1) These criticisms would not be possible if we hadn’t explained how > BSIMM was created and offered up the origins of the data. If we simply > said “here, we think this will probably work”, we could have created a > much more elegant model, and it would have been easy to brush all of the > “how did you validate it?” questions under the rug with lines like > “Trust us, we’re the experts!” > > 2) We published our data set. If you don’t like the model we created, > you can use our data to create your own. If you don’t think we came at > this the right way, you can conduct a better experiment, publish your > data, and demonstrate that ours is inferior. I hope that happens. It’s > how progress occurs in many other disciplines. > > So then, is software security a solved problem? Of course not! There’s > plenty left to be done, and the landscape will be different next year. > We will have new dilemmas and we’ll be working under tighter > tolerances. We will need a constant stream of new and unproven ideas to > try out and report back on. So BSIMM isn’t game over, but in moving > from “no supporting evidence” to “based on the data”, we’ve raised the bar. > > Ben, thanks for the DNS digging. > > Brian > > On 3/11/09 1:32 PM, "Benjamin Tomhave" <[email protected]> > wrote: > > I think the celebration is a bit premature, for many of the reasons > Pravir just covered. I think that perhaps the problem we're having here > is that you've not really tested your results, nor have you iterated > through a 2nd time to reevaluate the working theory. If you were > approaching this scientifically, I think the process would look like > this (http://en.wikipedia.org/wiki/Scientific_method): > 1. Use your experience: Consider the problem and try to make sense > of it. Look for previous explanations. If this is a new problem to you, > then move to step 2. > 2. Form a conjecture: When nothing else is yet known, try to state > an explanation, to someone else, or to your notebook. > 3. Deduce a prediction from that explanation: If you assume 2 is > true, what consequences follow? > 4. Test : Look for the opposite of each consequence in order to > disprove 2. It is a logical error to seek 3 directly as proof of 2. This > error is called affirming the consequent. > > I think what your missing, then, is at least step 4 as well as > reiterating through the process again (and possibly Step 3). It's a bit > abstract, perhaps, to rigidly apply the scientific method to this > program, but I think it's instructive to consider how you might do this. > > BTW, your citation of the xkcd strip on causation vs correlation is > actually instructive here. You've developed a model based on correlation > without demonstrating causation at all. Not to get abstruse, but I don't > see that your model is properly supported or validated. In the end, > ironically, it seems to come down more to expert theories than empirical > evidence. A handful of experts studied 9 organizations and correlated > "highly successful" with "110 practices", but without properly defining > success, without generalizing the model to all types of organizations > (or without defining the scope), and without testing/validating the > model. > > The good news is that you can now test the model. The bad news is that > you ("you" being the collective behind BSI-MM) probably should have > tested the model first before jumping straight to fanfare and hoopla. :) > > In the end, I'm sure that BSI-MM will be a fine model, though the > questions will then be "can I implement it?" and "will it have > sustainable value on its own?" If the value of the model rests on > Cigital and Fortify pushing it into organizations by force (much as the > Big N, ISACA, and ITGI have tried to do with CObIT and valIT), then I > submit that it will encounter problems. It needs to be able to stand on > its own, properly validated, with inherent value through logical > implementation. > > Which perhaps begs a question: is BSI-MM intended as an implementation > model to achieve better security in software development, or is it a > measurement tool for evaluating the current security maturity of > software development? A maturity model is typically used to measure > maturity, which means that someone has to then come along and provide > guidance on how to implement a program that can reach an optimal > measurement. (and mayhaps this would be a good time to get together with > Pravir to see if there's a way that you could both have winning game > plans) > > BTW, when you get to the point of defining success, I would suggest > looking at FAIR (since they lean toward quantitative vs qualitative risk > assessment based on Bayesian statistics) as well as looking at the > concept of "risk resiliency" advocated, in particular, by BT. fwiw. > > Anyway... > > On whether the site is up or not, I think DNS is hosed for the domain... > I tried it from three locations (separate regions, separate providers) > and got the same results: > > $ host bsi-mm.com > Host bsi-mm.com not found: 3(NXDOMAIN) > > $ host bsi-mm.com > ;; connection timed out; no servers could be reached > > freeproxy.us also times out... > > cheers, > > -ben > > Brian Chess wrote: > > Ben! Thank you! When you talk about sample size, it gives me hope > that > > we’re on the right track. We can either: > > > > 1) Use ideas that “experts” theorize will work > > or > > 2) Gather empirical evidence to judge one idea against another. > > > > We in the security crowd often try to hide behind the need for secrecy, > > and that’s pushed us toward relying almost entirely on people who have > > nothing but rhetoric and personal reputation to stand on. BSIMM pretty > > well shows that, in 2009, we can do better. It’s a big step forward to > > collect data and then argue about what it means. I know it’s already > > made the rounds, but let’s have some XKCD to celebrate: > > http://xkcd.com/552/ > > > > I think your question about defining success is an important one. We > > were loose about it in this first round, and I hope it’s something we > > can tighten up in our follow-on work. Here’s my thinking as of today: > > software security is not the goal, it’s one of the many things an > > organization needs to do in order to meet it’s objectives. We need to > > look at how a software security initiative (or lack thereof) > effects the > > organization’s ability to meet it’s objectives. This is going to be > > messy, but it’s either that or go back to making stuff up. > > > > BTW, I checked the BSIMM web site after I read your message. It worked > > for me. Try this? > > http://www.downforeveryoneorjustme.com/bsi-mm.com > > > > Brian > > > > On 3/11/09 10:48 AM, "Benjamin Tomhave" > <[email protected]> > > wrote: > > > > I think it's an interesting leap of faith. Statistically > speaking, 9 is > > a very small sample size. Thus, the proposed model will be viewed > > skeptically until it is validated with a much larger and more > diverse > > sample. Putting it another way, there's no way I can take this to a > > small or medium sized org and have them see immediate > relevance, because > > their first reaction is going to be "those are 9 large orgs > with lots of > > resources - we don't have that luxury." > > > > You quoted "we can say with confidence that these activities are > > commonly found in highly successful programs" - how do you define a > > "highly successful program"? What's the rule or metric? Is this > a rule > > or metric that can be genericized easily to all development teams? > > > > My concern is exactly what you speculate about... organizations are > > going to look at this and either try to tackle everything (and > fail) or > > decide there's too much to tackle (and quit). In my experience > working > > with maturity models as a consultant, very few people actually > > understand the concept. Folks are far more tuned-in to a PCI-like > > prescriptive method. Ironically, the PCI folks say the same > thing you > > did - that it's not meant to be prescriptive, that it's > supposed to be > > based on risk management practices - yet look how it's used. > > > > Maybe you've addressed this, but it doesn't sound like it. I'd > perhaps > > be better educated here if the web site wasn't down... ;) > > > > -ben > > > > Sammy Migues wrote: > > > Hi Pravir, > > > > > > Thanks for clarifying what you're positing. I'm not sure how > we could > > > have been more clear in the BSIMM text accompanying the > exposition of > > > the collective activities about the need to take this > information and > > > work it into your own culture (i.e., do "risk management"). > As a few > > > examples: > > > > > > p. 3: "BSIMM is meant as a guide for building and evolving a > software > > > security initiative. As you will see when you familiarize > yourself > > > with the BSIMM activities, instilling software security into an > > > organization takes careful planning and always involves broad > > > organizational change. By clearly noting objectives and goals > and by > > > tracking practices with metrics tailored to your own > initiative, you > > > can methodically build software security in to your > organization’s > > > software development practices." > > > > > > p. 47: "Choosing which of the 110 BSIMM activities to adopt > and in > > > what order can be a challenge. We suggest creating a software > > > security initiative strategy and plan by focusing on goals and > > > objectives first and letting the activities select themselves. > > > Creating a timeline for rollout is often very useful. Of course > > > learning from experience is also a good strategy." > > > > > > p. 47: "Of the 110 possible activities in BSIMM, there are ten > > > activities that all of the nine programs we studied carry > out. Though > > > we can’t directly conclude that these ten activities are > necessary > > > for all software security initiatives, we can say with confidence > > > that these activities are commonly found in highly successful > > > programs. This suggests that if you are working on an > initiative of > > > your own, you should consider these ten activities particularly > > > carefully (not to mention the other 100)." > > > > > > p. 48: "The chart below shows how many of the nine > organizations we > > > studied have adopted various activities. Though you can use > this as a > > > rough “weighting” of activities by prevalence, a software > security > > > initiative plan is best approached through goals and objectives." > > > > > > Your words (...BSIMM fails...) imply (to me) that you posit > > > organizations attempting to use the collected wisdom in BSIMM > will, > > > inexplicably, look at it and say "Okay, we have to do all 110 of > > > these things exactly as written, so let's get started" > without regard > > > to their local need. This is as opposed to, say, looking at > it and > > > thinking "Here's what nine companies have spent dozens of > > > person-decades and millions of dollars learning about what works; > > > let's see what we can glean from that." Uhmmmm, okay. > > > > > > Yes, previous models exist. Although it may have come up in > > > conversation, we did not ask any of the nine something like "What > > > model did you start with back in the beginning?" because it > simply > > > isn't relevant to what we're trying to accomplish > (documenting what > > > successful organizations are doing), just as "could" and "should" > > > aren't relevant. We asked "What *are* you doing now?" and > documented > > > it so others could learn from it. > > > > > > --Sammy. > > > > > > -----Original Message----- From: Pravir Chandra > > > [mailto:[email protected]] Sent: Wednesday, March 11, 2009 > 4:00 AM To: > > > Sammy Migues; [email protected]; > [email protected] > > > Subject: Re: [SC-L] Positive impact of an SSG > > > > > > Yes, I don't think its exclusive to your BSIMM interviews > that you > > > found when people put controls into place, they saw improvement. > > > That's what I (and I'm sure many other consultanting firms) > have been > > > doing for years based upon previous models (CLASP, MS SDL, etc.). > > > Nothing to do with BSIMM per se (actually, most of what DTCC > started > > > doing was based on CLASP), just that they added controls > 'early into > > > software development lifecycle' and saw benefit, which isn't > > > surprising. > > > > > > That being said, the important part we're missing as 'software > > > security guys' isn't the specification of all the possible things > > > that an organization *could* do, but rather what a given > organization > > > *should* do based on good business decisions around risk > management. > > > And that's the crux of what BSIMM fails to do. By basing the > current > > > maturity model on the collected practices of 9 massive firms that > > > spend the most on that problem, anyone (aside from firms in a > similar > > > situation to your 9) that attempts to apply it to their > organization > > > effectively throws risk management decisions out the window and > > > commits to a much more costly solution than they could have > created > > > based on the knowledge of their own business needs since all the > > > practices are based solely on the behaviors of the select few > firms > > > you interviewed. I'm not discounting the validity of the > empirical > > > data, I'm just positting that it isn't scientifically valid for > > > solving the problem at hand. > > > > > > I'm interested to hear what you learn when you get to the > small and > > > medium sized businesses as well as firms using agile development > > > models (something I particularly considered and accounted for > with > > > SAMM). > > > > > > Regardless of whether we agree on the percentage of orgs for > which a > > > dedicated SSG isn't cost effective, I'm sure we can agree that > > > affording 'someone in charge of success' doesn't equate to a > > > dedicated SSG. There's a myriad of ways that can be > accomplished in > > > any organizational structure. > > > > > > Thanks! > > > > > > p. > > > > > > ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~ ~~~~~ ~~~ ~~ ~ > Pravir > > > Chandra chandra<at>list<dot>org PGP: CE60 > > > 0E10 9207 7290 06EB 5107 4032 63FC 338E 16E4 ~ ~~ ~~~ ~~~~~ > > > ~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ > > > > > > -----Original Message----- From: Sammy Migues > <[email protected]> > > > > > > Date: Tue, 10 Mar 2009 23:15:39 To: > > > [email protected]<sc-l <[email protected]<sc-l> > > <[email protected]<sc-l> > <[email protected]<sc-l>>@securecoding.org> Subject: Re: [SC-L] > > > Positive impact of an SSG > > > > > > > > > Hi Pravir, > > > > > > Yes, I agree completely: the data gathered in the BSIMM > interviews > > > seem to indicate that "the controls over all" led to what the > > > interviewees saw as improvements in their capability to produce > > > secure software. > > > > > > In the nine companies interviewed, those controls (BSIMM > activities, > > > I think) sprang from well established SSGs -- that is, a specific > > > person or persons with the responsibility for ensuring lots (110, > > > collectively) of activities actually get done. > > > > > > The BSIMM data to date from specific large organizations indicate > > > that a little under 100:1 is the average ratio for dev/QA to SSG > > > size. It'll be interesting to see how this changes when we get to > > > interviewing smaller organizations and we see if and how they're > > > actually getting it done. > > > > > > Personally, I don't believe I agree with your guess that 95% of > > > organizations building code can't afford an SSG. I believe every > > > organization that wants to succeed can afford to have someone in > > > charge of success, but that's just my opinion and isn't > relevant to > > > BSIMM. > > > > > > Cheers, > > > > > > --Sammy. > > > > > > > > > -----Original Message----- From: Pravir Chandra > > > [mailto:[email protected]] Sent: Tuesday, March 10, 2009 6:31 > PM To: > > > Sammy Migues Cc: [email protected] Subject: Re: [SC-L] > Positive > > > impact of an SSG > > > > > > Hey Sammy. > > > > > > How does that pertain to a software security group (SSG) per > se? The > > > details below seem to indicate that it was the controls over > all that > > > lead to the positive impact. > > > > > > My main point is that supporting an SSG isn't cost effective > for 95% > > > of the organizations out there that are building code. That's > why in > > > SAMM, we didn't mandate the structure of the organization and > instead > > > concentrated on the functions fulfilled by security guys > (regardless > > > of their placement in the org). > > > > > > p. > > > > > > On Tue, Mar 10, 2009 at 7:48 AM, Sammy Migues > <[email protected]> > > > wrote: > > >> Hi all, > > >> > > >> I've received some private questions about the 110 activities in > > >> BSIMM (bsi-mm.com). Since we built the model directly from > the data > > >> gathered, each activity is actually being done in one of the > nine > > >> organizations interviewed. The question is whether there's any > > >> evidence the activities are actually effective as opposed to > simply > > >> being done. > > >> > > >> Since we can't publish any private data, I'd like to point > folks at > > >> this recent article in Information Security Magazine. Jim Routh, > > >> CISO of DTCC (one of the nine organizations interviewed), is > quoted > > >> as follows relative to the impact of software security group > > >> activities: > > >> > > >> > > > http://searchsecurity.techtarget.com/magazineFeature/0,296894,sid14_gci1346974,00.html > > >> > > >> > > >> "One of Routh's big wins is inserting security controls > early into > > >> software development lifecycle at the DTCC. Vulnerabilities are > > >> weeded out well before they appear in functional code that > ends up > > >> in production and that has resulted in close to $2 million in > > >> productivity gains on a base of $150 million spend for > development, > > >> Routh says. > > >> > > >> "Those gains are exclusively the result of having mature and > > >> effective controls within our system and software development > > >> lifecycle," Routh says. This is a three-year-old initiative that > > >> educates and certifies developers in all DTCC environments in > > >> security. Developers are also provided with the necessary > > >> code-scanning tools and consulting and services help to keep > > >> production code close to pristine." > > >> > > >> --Sammy. > > >> > > >> Sammy Migues Principal, Technology 703.404.5830 - > > >> http://www.cigital.com Software confidence. Achieved. > > >> [email protected] > > >> > > >> > > >> > > >> _______________________________________________ Secure Coding > > >> mailing list (SC-L) [email protected] List information, > > >> subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List > > >> charter available at - > http://www.securecoding.org/list/charter.php > > >> SC-L is hosted and moderated by KRvW Associates, LLC > > >> (http://www.KRvW.com) as a free, non-commercial service to the > > >> software security community. > > >> _______________________________________________ > > >> > > > > > > > > > > > > > -- > > Benjamin Tomhave, MS, CISSP > > [email protected] > > LI: http://www.linkedin.com/in/btomhave > > Blog: http://www.secureconsulting.net/ > > Photos: http://photos.secureconsulting.net/ > > Web: http://falcon.secureconsulting.net/ > > > > [ Random Quote: ] > > "Don't you wish there were a knob on the TV to turn up the > intelligence? > > There's one marked 'Brightness,' but it doesn't work." > > Gallagher > > _______________________________________________ > > Secure Coding mailing list (SC-L) [email protected] > > List information, subscriptions, etc - > > http://krvw.com/mailman/listinfo/sc-l > > List charter available at - > http://www.securecoding.org/list/charter.php > > SC-L is hosted and moderated by KRvW Associates, LLC > > (http://www.KRvW.com) > > as a free, non-commercial service to the software security > community. > > _______________________________________________ > > > > -- > Benjamin Tomhave, MS, CISSP > [email protected] > LI: http://www.linkedin.com/in/btomhave > Blog: http://www.secureconsulting.net/ > Photos: http://photos.secureconsulting.net/ > Web: http://falcon.secureconsulting.net/ > > [ Random Quote: ] > "Why don't they make the whole plane out of that black box stuff." > Steven Wright > > > ------------------------------------------------------------------------ > > _______________________________________________ > Secure Coding mailing list (SC-L) [email protected] > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l > List charter available at - http://www.securecoding.org/list/charter.php > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) > as a free, non-commercial service to the software security community. > _______________________________________________ -- Benjamin Tomhave, MS, CISSP [email protected] LI: http://www.linkedin.com/in/btomhave Blog: http://www.secureconsulting.net/ Photos: http://photos.secureconsulting.net/ Web: http://falcon.secureconsulting.net/ [ Random Quote: ] Osborn's Law: "Variables won't; constants aren't." http://globalnerdy.com/2007/07/18/laws-of-software-development/ _______________________________________________ Secure Coding mailing list (SC-L) [email protected] List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
