On Mon, Oct 30, 2017 at 3:50 PM, Kathleen Wilson via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > How do we get all auditors to start meeting our audit statement > requirements? > > Why haven't all included CAs communicated these requirements to their > auditors? > > Why am I seeing so many audit statements (particularly ETSI audit > statements) that do not meet our requirements? > > I will greatly appreciate thoughtful and constructive ideas on this. > > Thanks, > Kathleen >
Kathleen, Thanks for raising this matter! I think it's an important highlighting of the very different approaches to auditing employed by WebTrust and ETSI, and the underlying reliability and assurance of those audits. Below is my attempt to summarize my understanding so far: - ETSI EN 319 411-1 specifies generally applicable policy and security requirements for trust service providers (TSPs) - including website certificates. - The purpose of EN 319 411-1 is to provide a framework for assessment of a TSP, but does not specify how that assessment is to be caried out (c.f. Section 1 of EN 319 411-1) - EN 319 411-1 mentions EN 319 403 for guidance on such assessments - EN 319 403 provides the framework for conformity assessment bodies (CABs) to evaluate TSPs. It's based on/extends ISO/IEC 17065 specific to TSPs. - ISO/IEC 17065 is incorporated due to Regulation (EC) No 765/2008 to ensure consistency in CABs in evaluating TSPs As noted within 319 403 (Introduction), several other documents are incorporated as well - ISO/IEC 17021 (common requirements for conformity assessment bodies evaluating management systems) and ISO/IEC 27006 (common requirements for CABs evaluating information security management systems), for example. If we put the layer cake together and simplify: * ISO/IEC 17065 - Common requirements for conformity assessment bodies looking at products/services (e.g. "What should all auditors do") * ISO/IEC 17021 - Common requirements for conformity assessment bodies looking at management systems * ISO/IEC 27006 - Common requirements for conformity assessment bodies looking at information security management systems * EN 319 403 - Common requirements for conformity assessment bodies evaluating TSPs (e.g. "What makes an auditor qualified to be a CA auditor") * EN 319 411-1 - Common requirements on the TSP for websites (combined with EN 319 401, which 319 411-1 incorporates-and-builds-on) In trying to understand why the reports are what they are, we need to look in particular at 17021 and 17065, and the framework they use for both audit engagements and reporting. 17065 describes a certification scheme. Reproducing a paragraph from the introduction: """ Certification of products, processes or services is a means of providing assurance that they comply with specified requirements in standards and other normative documents. Some product, process or service certification schemes may include initial testing or inspection and assessment of its suppliers' quality management systems, followed by surveillance that takes into account the quality management system and the testing or inspection of samples from the production and the open market. Other schemes rely on initial testing and surveillance testing, while still others comprise type testing only. """ 319 411-1 certification describes a system that is based on an initial testing or inspection, along with periodic surveillance. Importantly, within 17065 and 17021, the way of ensuring continued compliance is done based on contracts and reporting - that is, the client is responsible for reporting changes to their conformity assessment body, and the CAB may determine to revoke the certification or indicate corrective actions should be taken (see ISO/IEC 17065:2012(E) 4.1.2.2, ISO/IEC 17021:2006(E) 8.6.3). ISO/IEC 17065:2012(E), Section 7.7 describes the general reporting: the name of the CAB, the date the certification is granted, the name and address of the client, the scope of the certification, the validity period/expiration date of the certificate, and anything else required by the certification scheme (319 411-1). Within the WebTrust scheme, the reports are based on either US (AICPA) Standards of AT101, Canadian (CPA Canada) Standards of Section 5025, or IFAC's ISAE3000 standards. What is important and notable about these is that they are reports over information, with a scope and norm (e.g. WebTrust for CAs) applied. This is why there's a consistent period of time - information is collected and the auditor evaluates, on the basis of that information, the level of assurance that is being met. I'm still working to have a better understanding here (and this is all in a personal capacity), but my conclusion is this: - "ETSI" audits reflect an engagement at a particular point in time, where a series of system controls are evaluated, and the result is a certificate indicating the process and products comply with the relevant criteria (319 411-1). This is similar to a "Point in Time Readiness Assessment". After the certification is granted, the TSP is responsible for notifying the CAB of changes (in the future), and the CAB evaluates whether those changes should cause it to revoke certification or institute some corrective action (which may not be noted in the certification). An ETSI audit is thus meant to indicate a CA "is" or "will be" conforming. - "WebTrust" audits reflect an engagement looking over the past, by the CA providing documentation and evidence that their policies were consistent with the requirements. That is, a WebTrust audit is meant to assess whether or not a CA is being accurate when it says it "was" conforming, but makes no guarantees about whether or not a CA "is" conforming, nor whether or not the CA disclosed all relevant information. The challenge is in determining whether this is a correct understanding of the respective differences, and then determining whether or not the certification-based approach provides the desired or sufficient level of assurance - in particular, its 'forward-looking' approach with self-reporting, rather than the WebTrust-style assurance engagement that is more 'backward-looking' without future promises. _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
