Christian, (Stephen)
I’ll confess I’ve only skimmed the discussion but it looks productive. The
questions posed are good ones. I’ll try to provide a few clarifications from
“inside” the BSIMM study that may be helpful in pushing the discussion along:
1) Survey structure/technique attributes BSIMM activities by first seeking
certain confirmations. If interview-based confirmation doesn't provide
confidence, the subject is asked for documentation.
2) The BSIMM document indicates who is interviewed, but it’s not an
exhaustive list. Where confirmation is necessary Dev/Config
Management, architects, and others make the list.
3) Surveying claims to (and in practice) stops short of concrete attestation
across the board.
4) BSIMM survey targets have included organizations, business units, and
in rare cases, smaller scopes.
5) At the organization-level, survey confirmation includes facilities to
differentiate “one group does it”, “this is done by most”, and “the
organization governs mandatory activity conduct”.
6) An organization does/should not get credit for “one group does it”.
7) More qualified BSIMM interviewers exist than Sammy et al. More are
minted as the study grows in size. There isn’t a written certification and
a pin, but there is an involved apprenticeship. And, Sammy runs cross-
checking of the grading process to make sure that interviewers remain
convergent in grading criteria.
Addressing another question raised by the email chain below: just because the
organization does the activity—as a rule—doesn’t mean that every team does it.
Non-complaince may be a reason. Another (better) reason may be that the
organization takes a “risk-based approach” to the activity. In other words, an
organization may choose to do more mature architecture analysis activities on
only a subset of applications—those that are higher risk. This is what BSIMM
activities Strategy and Metrics (SM) Level 3 are about.
Hopefully that helps a bit.
-jOHN
----
John Steven
iCTO, Cigital
+1,703-727-4034 | @M1splacedsoul
https://google.com/+JohnStevenCigital
On Jan 7, 2014, at 8:07 PM, Christian Heinrich <[email protected]>
wrote:
> Stephen,
>
> On Sat, Jan 4, 2014 at 8:12 PM, Stephen de Vries
> <[email protected]> wrote:
>> Leaving the definition of agile aside for the moment, doesn’t the fact that
>> the BSIMM measures
>> organisation wide activities but not individual dev teams mean that we could
>> be drawing inaccurate
>> conclusions from the data? E.g. if an organisation says it is doing Arch
>> reviews, code reviews and
>> sec testing, it doesn’t necessarily mean that every team is doing all of
>> those activities, so it may give
>> the BSIMM reader a false impression of the use of those activities in the
>> real world.
>>
>> In addition to knowing which activities are practiced organisation wide, it
>> would also be valuable to
>> know which activities work well on a per-team or per-project basis.
>
> My reading of the "Roles" section of BSIMM-V.pdf is that the people
> interviewed for the BSIMM sample are:
> 1. Executive Leadership (or CISO, VP of Risk, CSO, etc)
> 2. Everyone else within the Software Security Group (SSG)
>
> What you are asking to be included is what is referred to as the
> "Satellite" within BSIMM-V.pdf and I believe this may also require the
> inclusion of http://cmmiinstitute.com/cmmi-solutions/cmmi-for-development/
> too (why not :) ).
>
> The issue with this is that it would invalidate the statistics from
> the prior five BSIMM releases due to the inclusion of new questions
> and in additional these new statistics were not gathered over time
> either hence the improvements measured over time within BSIMM would be
> invalid too due tot he new dataset.
>
> Furthermore, Gary, Sammy and Brian have limited time to interview all
> 67 BSIMM participating firms.
>
> However, I would be interested to know the "BSIMM Advisory Board" i.e.
> http://bsimm.com/community/ view on this is and if it would be
> possible to undertake this additional sampling within their own BSIMM
> participating firm to determine if there is additional value would be
> gained for BSIMM? However, I suspect that an objective measurement
> would be too hard to quantify due to internal politics of each BSIMM
> participating firm but I could be wrong.
_______________________________________________
Secure Coding mailing list (SC-L) [email protected]
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
_______________________________________________
