All, I'm the editor of the Top 25 list. Thanks to Ken and others on SC-L
who provided some amazing feedback before its publication. I hope we were
able to address most of your concerns and am sorry that we couldn't
address all of them.
Note that MITRE's site for the Top 25 is more technically detailed and has
lots of supporting documents than the SANS site, which is really a
jumping-off point. See http://cwe.mitre.org/top25/. Also, a process
document and changelog is on that site.
Here are some topics that arose during the construction of the Top 25. I
thought these might make some interesting points of debate or discussion
on this list:
1) The inclusion of output encoding (CWE-116) in conjunction with
input validation (CWE-20) generated a lot of mixed reviews. Part of it
seems to come down to different ways of looking at the same problem.
For example, is SQL injection strictly an input validation
vulnerability, or output sanitization/validation/encoding or whatever
you want to call it? In a database, the name "O'Reilly" may pass your
input validation step, but you need to properly quote it before sending
messages to the database. And the actual database platform itself has
no domain model to "validate" whether the incoming query is consistent
with business logic. My personal thinking, which seems reflected by
many web application people, is that many injection issues are related
to encoding at their core, and the role of input validation is more
defense-in-depth (WITH RESPECT TO INJECTION ONLY). Yet smart people
insist that it's still input validation, even when presented with the
example I gave. So So what's the perspective difference that's causing
the disconnect?
2) Countless mitigations were suggested by contributors on top of some
of the ones already in the CWE entries (admittedly some of them
weak). Fortunately, we had time (for some definition of "time"
that sometimes excluded a personal life) to update many of the core
CWE entries. Many mitigations had limitations, either in terms of
impacts on usability, whether they could be applied at all in some
circumstances, or if they were sufficiently effective. The variety
of customized mitigations is staggering, which to me suggests that
more framework/methodology definition is needed.
3) Contributors advocated selecting list items based on how often the
weakness appears in software (prevalence) and how severe the
consequences are when the weakness leads to a vulnerable condition
(severity). Many people advocated using real-world data to make
the determination for prevalence. Problem: there's no real-world
data available! CVE vulnerability data is insufficient - they
concentrate on the vulnerability side ("XSS") instead of the
weakness side ("e.g. use of the wrong encoding at the wrong time").
If people have real-world, weakness-focused data, then they aren't
telling.
4) Some questions with respect to the assignment of severity scores
led me to attempt to build a threat model and to try to more formally
define other supporting fields like ease of detection, in light of the
"skilled, determined attacker." I don't think this model was
sufficiently vetted, and I'm sure people will have concerns with how
it's been defined (including "your threat model is really just talking
about a threat agent.") HOWEVER, I don't know of any other prioritized
list that has tried to define some threat model to help with
prioritization. I would love to see this kind of investigation
continue in other efforts. (An acronym called CWSS comes to mind...)
5) The threat model, as roughly implied by how most people were
"voting" for which items to include on the Top 25, treated availability
as slightly less important than integrity and confidentiality. Thus
CWE-400 (Resource Consumption) had the dubious distinction of being
number 26. (CWE-400 and other also-rans are in the "On the Cusp"
section of MITRE's Top 25 site). Clearly, availability may be more
important in some environments e.g. critical infrastructure or
e-commerce. The unsurprising implications are that no single threat
model will work for different types of organizations when composing a
general list like the Top 25. Thus it has a sort of fudge factor that
helps make it generally applicable to organizations with varying threat
environments, within some degree of tolerance. It seems like a
fundamental problem with a list of that sort.
6) Many expert reviewers will notice the varying abstraction levels
and overlapping concepts in the list. There are various explanations
for this as summarized in the change logs and FAQ. My main point in
bringing this up was, a lot of people want things to be at a fixed
level of abstraction and mutually exclusive, but I don't think it's
feasible given our current understanding of secu
