Unfortunately, you seem to be be ignoring what I wrote and talking about something else.
On 13/11/2018 14:31, Ryan Sleevi wrote: > I suppose I had unreasonably hoped it would be self-evident, particularly > for someone who claims to follow the issues, to understand how directly > that issue was related. Unfortunately, whether for intent or otherwise, it > appears not. > This discussion has been overfilled with discussions about the meanings of words, weather or not something is under one rule or another etc. etc., making it very hard to get a clear overview of the primary issues. Furthermore the start of the thread was off-list. Also neither I, nor some other participants have access to the audit reports etc. in CCADB. This basic combination of noise and missing data is why I asked for a one-stop overview of your complaints against TUVIT, similar to the lists compiled for previous situations with multiple complaints against one party. > While I do not believe nor agree with your approach to framing the issues, > I do hope you can agree that both through the bug - which itself is an > amalgamation of and reference to several bugs - that during the prior two > audit cycles, T-Systems contained a substantial amount of misissuance that > were undetected by TUVIT and that shared the same root cause: > misconfiguration and misapplication of the relevant rules, both in terms of > ASN.1 and in terms of normative requirement. > "Misconfiguration and misapplication of the relevant rules..." is so broad as to describe the majority of CA failures without giving any useful specifics to assess the situation. It's like saying someone's crime was to "violate and break the relevant laws" (which would apply to anything from jaywalking to mass murder). It would also be useful to quantify the word substantial: Of all the certificates issued by the audited CA organization, how large a percentage suffered from each flaw, how many from none. This is a key number when assessing if statistical sampling by the auditor should have caught an issue. It is also a key number when assessing the level of incompetence of the CA (but the CA is not the subject of this thread). Bug #1391074 contained counts of affected certificates, but not how many certificates did not have the listed issues. So it is not clear if those issues were likely to be spotted by 3% sampling. T4. One valid complaint is that TUVIT apparently (according to what I read in this thread) didn't check the mailing list and bugzilla to find out about the Bug #1391074 issue and include them in a relevant audit statement (as this was already public information, the confidentiality excuse doesn't apply). T5. Another valid complaint is that TUVIT apparently did not see the corrective measures described in bug #1391074, which should have been in the CA's technical audit logs. T6. Then there is the valid complain that the administrative paper trail related to Bug #1391074 should have resulted in at least some event observed by TUVIT as part of their ETSI/ISO monitoring during the audit period, and that this should have caused TUVIT to look for the rest of the issue and include it in the audit or audit followup reports. Issue U1 (Qc-statement misencoding) apparently affected all certificates from one issuing CA, and should thus have been caught by sampling by the auditor. The auditor has (according to earlier posts) admitted that the bug was present in the sampled certificates from that issuing CA, but that this was overlooked because that particular extension was not one they had specific experience looking at. Once the problem was pointed out the auditor looked at the previously collected evidence and confirmed the problem by checking that detail from first principles (similar to software developer hand-executing a function with pen and paper to confirm a bug). T7. So this IS a valid complaint against TUVIT: That the auditor didn't check the Qc-statement encoding from first principles during the original audit, and thus failed to spot the problem. T8. Then there is the issue that TUVIT didn't issue an official qualification of the CA's status once the issue was both known and public. S9. There is also the issue if TUVIT should have issued an official qualification of the CA's status if they knew before the U1 issue became public. This is subject to the debate about the confidentiality provisions in the ISO audit standard versus the expectations of Mozilla and Chrome. S10. Then there is the issue if TUVIT was right or wrong in accepting a slow phase out of the certificates affected by U1. This involves both the principal issue if there should be zero tolerance for incorrect certificates, the practical issue of how much harm this specific standards validation can cause, and how much time should be allowed for an orderly replacement process. Multiple months seems a quite long time. 1 day quite a short time. > If you are attempting to excuse such misissuance, rather than address it, > one would take a similar tact as you are here; suggesting, for example, > that it was T-Systems rather than TUVIT that did the misissuance or by > suggesting that the incidence was low to be insignificant. I am not excusing the misissuance, merely trying to catalog it as part of the evidence to assess the level of diligence or failure by TUVIT. It is of cause the purpose of any audit scheme to check for the absence of irregularities, and to report if any were found. But it is quite rare for the audit to essentially redo every piece of administrative work done by the audited company. Thus there is often the question of why an irregularity (in this case incorrect CA issuances) did not make it into the public audited accounts. And thus if the auditor was (also) at fault, which is the topic of this thread. > I was careful > not to try to muddy the conversation through an indictment on T-Systems, to > avoid diluting the conversation, and because they’ve already provided > several enumerations of the issues and that doing so again, as you’ve done, > does not add value. For assessment of TUVIT, this cataloging of known T-System mistakes versus what was in the audit reports and audit statements is a key measure of the quality of the auditing. > However, it should be readily apparent from both the > bug discussion and the list of issues a common pattern of misconfiguring > relevant profiles and failing to ensure they comply with the relevant > requirements. Of cause. But it is not clear from the documents you linked to (the discussion message and but #1391074) that all, or even most, of the issues were specifically profile misconfigurations rather than a litany of other mistakes. This in turn ties in to the question how many of those mistakes should have been obvious to the auditors versus being buried too deep to be spotted. The debate in bug #1391074 about the template used for ECDSA certificates is a good example. According to the bug, the ECDSA certificate profile/template was correct, but some piece of software mishandled approved ECDSA certificate requests and used the RSA certificate profile/template, for at least some of the issued certificates. An incorrect ECDSA profile/template saying to set the KeyEncryption bit should have been spotted by a configuration audit and review (by TUVIT). But code bugs are notoriously harder to spot. > > In the context of ETSI, each of these configuration changes - particularly > once qualified - undergoes some review; whether after the fact > (pre-qualification) or prior to such change. This is why it is interesting to look at each issue to determine if it was subject to such review by or notification to TUVIT. > Similarly, misissuance > involves a degree of notification to the CAB. Only once known (e.g. around the time of the bug reports). Because I don't think you expect a CA to notify the auditor that it is about to misissue, and then proceed to actually misissue instead of stopping itself. > As such, it is entirely > reasonable to expect a degree of supervision, as that is the value of the > certification scheme. All of this information would have been available at > the time of configuring qualified certificates, including the pattern of > issues existing when configuring profiles and templates. Should have been available doesn't mean it was available. There is always a limit to the depth of audits, and thus we need to assess if TUVIT was being sloppy, complacent, complicit or just unlucky. > > As such, we functionally see two issues; the inadequate supervision that > resulted in the first batch of misissuance, which can be attempted to be > argued away by suggesting it was some small volume that sampling would not > have caught (despite the inconsistencies of that argument with the > criteria), Due to the lack of a consistent list of issues, I have not seen specific statements for each of the Bug #1391074 issues if that one should have been spotted by the auditor before it was found by the community, and why. For example were the 3 incorrect ECDSA certificates 100%, 1% or 0.01% of the issued certificates from that particular issuing CA? And how many of the certificates from that issuing CA should have been sampled under the criteria? This is of cause in addition to the question if or why not TUVIT was subsequently fully aware of the bug report and the reports uploaded to bugzilla by the CA. > and inadequate supervision leading to this current issue, > despite having all of that previous information available as context during > the review and despite their being 100% misissuance rate. For the current issue (U1), there is also a major question of how hard or easy to spot that particular that encoding error was if not looking specifically for that particular error. This applies both when "supervising" the creation of whatever configuration, template or code contained the bug and when auditing samples of issued certificates. One could always take the extreme position that a supervisor has no responsibility for the acts of the supervised, or the equally extreme position that a supervisor is fully responsible for every act of the supervised. > Both of these > share a clear commonality of inadequate supervision, a key role played over > the past several years. Only if you define away all specifics and reduce the words to meaninglessness. > > Audits understandably and obviously do not prevent a CA from making a > change tomorrow that undermines the past audits; there is no guarantee they > won’t start actively misissuing once the auditor has left the building. It > is, however, meant to provide assurance regarding the present (and past) > configuration. When a CA like T-Systems does misissue, whether this or > previous incidents, it is entirely reasonable to ask “Was this > configuration something the auditor previously reviewed, and did they catch > it?” and, in the case of ETSI, “was this a change the auditor approved in > relation to ongoing certification?” Indeed, but the question has to be asked specifically, not just as a hypothetical generality. For example did TUVIT review the ASN.1 module used by T-Systems to encode the qc-statements in the incorrect certificates? Did TUVIT compare it to the right edition of the upstream standard? Did subtleties of the ASN.1 semantics and conventions obscure the difference? > > The qcStatements demonstrates a failure of the latter, the bug demonstrates > a failure of the former, both speak to the process of review and the > qualifications of the reviewer. > Indeed, none of these mistakes should, ideally, have gotten past a perfect auditor. The question is how far from ideal TUVIT was. > If you don’t agree with the large swath of undetected past misissuance > being a concern, it would be helpful if you could explain why it isn’t > concerning. It is concerning, the question is how large this swath really was. Which requires actually looking at it, not just making blanket statements that it was "large" or "small". > For example, do you believe that these requirements > (collectively, for any of these issues) were not covered by existing > criteria? They were all clearly covered by the criteria that mistakes should not happen and auditing should detect mistakes. Most, if not all, were also covered by more specific criteria. > Do you believe that sufficient documentation of TUVIT’s > methodology exists so as to explain why such failure to detect may be seen > as reasonable? I have yet to see sufficient documentation either way, and the thread has been a lot of grandstanding on all sides. > Do you believe that ETSI does not require consideration by > auditors prior to operational and configuration changes? I am not a trained ETSI auditor, and don't know for certain. You have certainly said ETSI does require consideration before deliberate operational and configuration changes. On the other hand I doubt the auditor is supposed to consider every software patch to the level of a full code review. > In short, do you > disagree that, when presented with CA misissuance, such as by T-Systems, > that it is both relevant and appropriate to question why the auditor failed > to detect and/or prevent such misissuance? > It is relevant to ask, but it takes a considerable level of certainty before starting formal proceedings to disqualify an auditor due to the failings of a single audit subject. In comparison, E&Y was involved in auditing multiple bad CAs and RAs by the time some E&Y branches where disqualified by Mozilla. In the world of technical review and testing, TUV SUD is a major player, reviewing the safety of many technical products far outside Germany, they rank in the same league as UL in the US. > I am not arguing that an audit be a guarantee against misissuance; for > example, a statistical sample will be just that, a sample, and stuff can > reasonably slip through. I am, however, advocating that it’s both > appropriate and necessary to question whether sampling was even done, and > how it was constructed (e.g. CA selects the samples and sizes vs auditor), > and what was reviewed, in order to ascertain whether or not it was > “reasonable” to have missed something. > In the case of T-Systems past > misissuances, the collective sum - especially with respect to things like > misconfigured templates - raises legitimate concerns about TUVITs approach > and methodology, and those concerns are each themselves distinct issues > with TUVIT for every misissuance “type” by T-Systems. > Of all the T-Systems mistakes listed so far, only U1, and maybe U11 are clearly cases of bad templates. Many others were mistakes at other parts of the issuing process. Many in the area of technically (not semantically) validating the data inserted into templates. For example, allowing absolute form DNS names when the RFCs require root-relative DNS names with the root period omitted. No amount of checking and reviewing the templates could have found that the values being inserted would be wrong. These are still bad of cause, but they involve other parts of the audit. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
