Just read this. What happened to your blog, http://secreview.blogspot.com?
On 8/11/09, secreview <secrev...@hushmail.com> wrote: > We received 22 requests from different people to perform a review of > Redspin! Their website can be found at http://www.redspin.com. We > haven’t done a review of anyone in quite a while, the last review that > we did was for Pivot Point Security who got an A (we still recommend > them). We apologize for this long delay but we have been very busy > traveling (yes we still have jobs doing consulting work sometimes). > > As you can see from the comments that we received in other posts we > have a lot of catch up work to do, but to be honest we are not sure > that we will be able to do it. This review might be our final and last > review depending on how much more travel we have. (We have lives, some > of us have families, and we can’t keep doing this for free even though > we feel that this is a great service). > > We did a lot of research on Redspin and we managed to get a copy of two > reports that they did for two different customers. We won’t share those > reports with you because that would be unethical, don’t ask. > > Redspin claims that it is a “pure penetration testing firm”. What they > mean by “pure penetration testing” is that they do not resell third > party software or hardware. They also say that “don't find problems on > your network so that [they] can make more money; [their] penetration > testing services reveal vulnerabilities, [that] will help you become > more secure.” > > We verified their claim with our own research. Redspin will not try to > sell you software or hardware… but they might try to sell you software > as a service. (see their www.jetmetric.com website). > > Redspin takes it a step further and is brutally honest about their > methodology for delivering penetration-testing services. They openly > admit that their services rely on automated vulnerability scanners > (Nessus) and are enhanced by manual testing. In fact, Redspin says that > automated scanners “can miss about 40% of the security risk so they > alone do not adequately assess risk. Furthermore, about half of the > findings from a vulnerability scan are false positives”. > > Any security company that relies on automated scanners can weed out > false positives, but doing that doesn’t really increase the depth and > accuracy of testing. A false positive, also known as an error of the > first kind, or a Type I Error, is the rejection of a null hypothesis > when it is in fact true. In more simple terms, this is the error of > observing a difference when in fact there isn’t one. Identifying false > positives is fairly easily done, as it only requires inspecting the > results produced by a scanner. > > But what about False Negatives? A False Negative, also called a Type II > Error, or an error of the second kind, is the error of failing to > reject a null hypothesis when it is in fact not true. More simply, a > False Negative is the error of failing to observe a difference when in > truth there is one. So, if an automated vulnerability scanner tests a > vulnerable service (a known vulnerability) but the scanner doesn’t > detect the vulnerability then the vulnerability is excluded from the > report. If this is the case then Redspin’s methodology will break down > because there will be no result in the report for Redspin to manually > test. That vulnerability will fly under the Redspin radar but might not > be missed by a hacker. So how many vulnerabilities does Redspin miss? > It’s a question worth asking. > > Redspin does say that “vulnerability scanning is not suitable on its > own as a complete or billable service offering, it does provides some > value in the early reconnaissance phase of a more comprehensive > External Network Security Assessment”. They have a typo in that > sentence, but other than that, they are right. Vulnerability scanning > does have a position in the industry and is a huge time saver, > especially when testing large numbers of systems. Just don’t rely on > one vulnerability scanner like Redspin does, use two or more like the > OSSTMM proposes. > > Redspin says “manual analysis is at the heart of all of [their] > assessments which not only gives you confidence that you have a > complete view of your security risk, but provides tailored reporting > and recommendations enabling simple work-arounds and cost-effective > mitigation strategies for most security issues.” Based on our research > Redspin’s “manual analysis” isn’t what we expected it to be. It is not > based on vulnerability research and is strictly based on the inspection > and verification of scanner output. > > What we can say is that their “manual analysis” doesn’t produce the > highest quality reports that ever we’ve seen, but it does produce > reports that are higher than average quality. The Redspin reports have > very few, if any, False Positives but will contain more False Negatives > than a report that is centered on solid (vulnerability) research. > > One thing that Redspin does that we really don’t like is to ask their > customers to lower their defenses before they do testing. That’s right, > they ask their customers to white list their scanner’s IP addresses so > that the customer’s Intrusion Prevention System doesn’t block the > scanner. We verified this during 3 different interviews on three > different dates. We even talked to one Redspin customer to confirm it, > and they did. We think that a security testing company should be able > to test around a customer Intrusion Prevention System. If they can’t > then that really brings their capabilities into question. > > We feel this way because Intrusion Prevention Systems are a part of the > networks defenses and they should be tested. Disabling them for a > security test prevents them from being tested. If they aren’t tested > then how does one know how effective they are? It just doesn’t’ make > sense. On top of that, the test won’t properly reflect the actual > security level of the network being tested. > > Something that Redspin claims is that they’ve done is “ground breaking > security research”. We’ve searched high and low for this “ground > breaking security research” but haven’t found it anywhere, so we’re not > sure what they are talking about. When looking at the research page on > their website we see white papers that might make good blog entries, > but we don’t see any “ground breaking security research”. > > When we’re told that a company does “ground breaking security research” > we expect to see things like them finding security bugs in critical > systems, or publishing professional security advisories, and maybe even > publishing proof of concept code. Redspin doesn’t do any of that. The > only thing that we were able to find was an “Ultr@ VNC 1.0.1 viwer PoC” > (and what’s the point of that?). > > In conclusion, Redspin’s services are slightly better than average. > Their manual testing isn’t true manual testing at all; it’s the > inspection of output from scanners and the elimination of false > positives. We don’t like the fact that Redspin asks its customers to > disable their IPS before being tested, and Redspin doesn’t seem to have > any Vulnerability Research capability. > > Its not all bad, Redspin is very honest about their methodology, they > are focused on quality, and they are passionate about what they do. > We’d recommend Redspin to people with testing requirements that do not > require extreme depth and that can afford some False Negatives. By no > means is Redspin a company that we’d suggest you stay clear of, but > they’re certainly not the best in the industry. > > As normal, if there are any issues with this review and its > truthfulness please let us know and please provide proof. We will make > changes if we need to and we strive to be as honest and fair as we can > be. Thanks for reading! > > Score Card (Click to Enlarge) > > > > > -- > Posted By secreview to Professional IT Security Providers - Exposed at > 8/10/2009 08:51:00 PM -- -- Gichuki John Ndirangu, C.E.H , C.P.T.P, O.S.C.P I.T Security Analyst and Penetration Tester infosig...@inbox.com {FORUM}http://lists.my.co.ke/pipermail/security/ http://nspkenya.blogspot.com/ http://chuksjonia.blogspot.com/ _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/