Re: Auditing of CA facilities in lockdown because of an environmental disaster/pandemic

[Jeff Ward via dev-security-policy]

On Saturday, March 7, 2020 at 8:24:57 AM UTC-6, Ryan Sleevi wrote:
> On Fri, Mar 6, 2020 at 9:03 PM jwardcpa--- via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
> 
> > Great follow on questions Ryan.  As far as the detailed report, whether
> > the end product is in the current form, or in the detailed version, the
> > lead auditor is taking full responsibility and does not make mention of the
> > other auditor in both the opinion, and the detailed section of the controls
> > tested for a detailed report. That being said, nothing prohibits a CA from
> > creating a Bug to draw attention to the fact and explain the auditors
> > obtained the assistance of another firm to complete the scope of the
> > testing.  If WebTrust did allow a carve out approach, there would be more
> > flexibility to allow the reference to another firm, but since it is
> > inclusive and the lead auditor takes full responsibility, that is not an
> > option.
> >
> 
> Good to know! I don't think this poses any intrinsic problems, since as you
> note, the lead auditor is taking full responsibility, but it's helpful to
> know what disclosures, if any, would arise in such a situation.
> 
> 
> > This example demonstrates the firm was able to complete the scope of the
> > audit testing on July 20th.  It is up to the auditor's judgment as to how
> > far the opinion can be dual dated/extended.  Once too much time passes,
> > this option is no longer viable.
> >
> 
> Right, you addressed the scenario of a single report, subsequently updated.
> I was actually contemplating two full reports with two full engagements.
> That is, the first engagement and report may be qualified, due to the lack
> of the datacenter. My question was whether it's possible to engage the
> auditor in a second full engagement, this time considering all the
> facilities, for the original time period in question.
> 
> Think of this as a variation for what we see some CAs do, which is scope
> their annual reports into two or more reports, one of which may be
> qualified. That is, they may have a Jan - July report which is qualified,
> and a July - Dec report which is unqualified. However, those are
> non-overlapping date periods. I was wondering if, again, using our March to
> March scenario, that it's conceivable a report is delivered in April that
> is qualified, access to the facility is restored in July, and the auditor
> (either the original firm or a new firm) conducts a full audit of the
> original March-to-March period. In effect, conducting a second audit.
> 
> I'm trying to tease out if there are limitations on the original firm
> performing that work (e.g. because they'd previously been engaged in an
> audit of that period), as well as whether there are limitations as to how
> far back one can go. For example, could a CA engage an auditor, today, for
> a Jan 1, 2018 to Jan 1, 2019 period? What if the engagement was for a
> October 1, 2018 to October 1, 2019 period (e.g. 6 months ago)? I can
> understand the difficulty of obtaining an audit today for, say, the period
> 2014-01-01 to 2015-01-01, but I'm wondering what options might exist for
> examination of those remaining facilities after-the-fact.
> 
> My worst case scenario is that it is determined to not be possible after
> some period of time (e.g. 6 months) to obtain such originally-expected
> assurances. In those cases, I think the honest and pragmatic answer may
> involve discussions of removal of trust in that root, and so I want to make
> sure to explore alternatives and options before having to start such
> discussions.

I hate giving an "it depends" answer, but that's where I'll start.  In the case 
where a report is issued with a qualified opinion due to the inability to visit 
a CA's data center, as an example, and issued primarily to comply with the 90 
day rule, and the data center subsequently becomes available, it is possible 
for the auditor to perform necessary procedures and collect relevant 
documentation / artifacts subsequent to the original audit period to allow them 
to issue an unqualified opinion on the original audit period that was 
previously qualified.  Kind of a mouthful.  It depends if this will work on the 
documentation and artifacts that can be obtained during that original period.  
Collecting evidence on those controls outside of the period is problematic as 
it is not part of the audit period.  That is not to say that the documentation 
could be obtained from the original audit period and providing the necessary 
comfort the auditor needed in their previously issued qualified report.
   In short, the auditor is looking for evidence after the period that the 
controls in fact were operating effectively during the period.  Some 
documentation lends itself easily to make this determination, but I can see 
challenges.  Reviewing logs and video surveillance are two means that come to 
mind that would be part of the auditor's assessment to see if that 

Re: Welcome Ben Wilson to Mozilla!

[Jeff Ward via dev-security-policy]

On Monday, April 13, 2020 at 12:07:40 PM UTC-5, Kathleen Wilson wrote:
> All,
> 
> I am pleased to announce that Ben Wilson has joined Mozilla as a CA 
> Program Manager!
> 
> Ben has worked in PKI security, compliance, and policy since 1998. 
> Previously, he worked at DigiCert in various roles, including VP of PKI 
> Operations, VP of Compliance, and Chair of DigiCert’s Policy Authority 
> team to develop and communicate policies and practices for internal and 
> external stakeholders. Ben has also been very involved in the CA/Browser 
> Forum. He is a former Chair of the CA/Browser Forum and of the American 
> Bar Association’s Information Security Committee. Over the years, Ben 
> has also participated in this mozilla.dev.security.policy forum.
> 
> Here are some of the things that Ben will be responsible for:
> + Steps 3 through 9 of Mozilla’s root inclusion process[1], which 
> includes the detailed CP/CPS reviews[2] and the public discussion phase[3]
> + CA Incident Bugs[4]
> + Updates to Mozilla’s Root Store Policy[5] and the Common CCADB 
> Policy[6], including prioritizing potential changes[7] and leading 
> discussions to determine the actual changes
> + Represent Mozilla in the CA/Browser Forum, along with Wayne
> 
> I have added Ben to the Policy_Participants wiki page[8], and updated 
> Wayne's entry.
> 
> Welcome, Ben!
> 
> Thanks,
> Kathleen
> 
> [1] https://wiki.mozilla.org/CA/Application_Process
> [2] https://wiki.mozilla.org/CA/Application_Verification#Detailed_Review
> [3] https://wiki.mozilla.org/CA/Application_Verification#Public_Discussion
> [4] https://wiki.mozilla.org/CA/Incident_Dashboard
> [5] 
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/
> [6] https://www.ccadb.org/policy
> [7] https://github.com/mozilla/pkipolicy/issues
> [8] https://wiki.mozilla.org/CA/Policy_Participants

Thanks for sharing Kathleen and I am thrilled to hear this news.  Ben, we 
certainly look forward to continuing working with you in this new role.  

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Auditing of CA facilities in lockdown because of an environmental disaster/pandemic

[Jeff Ward via dev-security-policy]

On Friday, March 20, 2020 at 3:55:08 PM UTC-5, Ryan Sleevi wrote:
> On Fri, Mar 20, 2020 at 4:07 PM Kathleen Wilson via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
> 
> > My question: What should "location" mean in the above requirement?
> >
> 
> The WebTrust Practitioner Guidance offers a reasonable definition:
> https://www.cpacanada.ca/en/business-and-accounting-resources/audit-and-assurance/overview-of-webtrust-services/practitioner-qualification-and-guidance
> 
> CA Processing Locations
> All reports issued should list the city, state/province (if applicable),
> and country of all physical locations
> used in CA operations. This includes data center locations (primary and
> alternate sites), registration
> authority locations (for registration authority operations performed by the
> CA), and all other locations
> where general IT and business process controls that are relevant to CA
> operations are performed.
> 
> 
> > For example, if a CA happens to have two facilities in the same city
> > that should be audited, how can the audit statement clearly indicate if
> > all of that CA's facilities were audited without providing the exact
> > physical addresses?
> 
> 
> We're primarily interested in making sure that the auditor examined /both/
> facilities for the appropriateness of controls. ETSI's lack of rigorous
> methodology leaves a lot to be desired here, but it's not difficult to
> disambiguate by indicating something like
> "Facility 1 in City, State, Country" vs "Facility 2 in City, State, Country"
> or
> "Primary Facility in City, State, Country" vs "Disaster Recovery Facility
> in City, State, Country"
> 
> (adjusted as appropriate)

Shortly before the COVID-19 pandemic, members of the WebTrust Task Force 
reviewed this guidance and had discussion focused on whether our reports were 
providing too much information in a publicly available report as to the 
operations of a CA.  Practitioners have been getting questioned in the past by 
CAs as to why such specific information should be disclosed to the level of 
city and state for the location of its operations.  It is a good point as 
certainly not all CAs provide this information freely to all of their 
employees, let alone outsiders.  This is especially true with the larger and 
more complex CAs.  For the more complex CAs, I can envision another Attachment 
in the audit report, similar to the thumbprint attachment, that lists the 
locations in a manner that Jeremy suggests that protects the physical location 
to some degree, yet provides the users of the report enough information to know 
what was able to be covered. That could be part of our guidance, which of 
course is jus
 t that - guidance.  Having our guidance adjusted in this manner would 
certainly help drive consistency that would be helpful to the CABF. I am sure 
there will be variations in reports, however, as guidance is non-authoritative 
for AICAP and CPA Canada. 

As far as the term "CA facility", I'd like to get thoughts from this group as 
to what that includes.  For instance, while a facility hosting an active HSM 
with CA private keys is a certainly a "CA facility", would you also include in 
this definition things like a bank safe deposit box that stores a deactivated 
and encrypted copy of a private key a CA facility?  Would you expect this level 
of information disclosed in an audit report?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #207: Require audit statements to provide information about which CA Locations were audited

[Jeff Ward via dev-security-policy]

On Sunday, January 3, 2021 at 8:38:05 AM UTC-6, Jeff Ward wrote:
> On Tuesday, December 15, 2020 at 2:41:10 PM UTC-6, Ben Wilson wrote: 
> > All, 
> > 
> > This email is part of the discussion for the next version of the Mozilla 
> > Root Store Policy (MSRP), version 2.7.1, to be published during of Q1-2021. 
> > 
> > For audit delays, we currently require that audit statements disclose the 
> > locations that were and were not audited, but that requirement has not been 
> > incorporated yet into the MRSP. See 
> > https://wiki.mozilla.org/CA/Audit_Statements#Minimum_Expectations. That 
> > provision reads as follows: 
> > 
> > Disclose each location (at the state/province level) that was included in 
> > the scope of the audit or should have been included in the scope of the 
> > audit, whether the inspection was physically carried out in person at each 
> > location, and which audit criteria were checked (or not checked) at each 
> > location. 
> > 
> > - If the CA has more than one location in the same state/province, then 
> > use terminology to clarify the number of facilities in that state/province 
> > and whether or not all of them were audited. For example: "Facility 1 in 
> > Province", "Facility 2 in Province, Facility 3 in Province" *or* 
> > "Primary Facility in Province", "Secondary Facility in Province", "Tertiary 
> > Facility in Province". 
> > - The public audit statement does not need to identify the type of 
> > Facility. 
> > - "Facility" includes: data center locations, registration authority 
> > locations, where IT and business process controls of CA operations are 
> > performed, facility hosting an active HSM with CA private keys, 
> > facility or 
> > bank deposit box storing a deactivated and encrypted copy of a 
> > private key. 
> > 
> > It is proposed by Issue #207 
> >  that this language 
> > requiring the disclosure of site locations--audited and unaudited--be made 
> > clearly part of the MSRP by reference to the language above. 
> > 
> > A similar method of incorporating by reference has been taken in section 
> > 2.4 of the MSRP 
> > 
> >  
> > with respect to incident reporting and in section 7.1 
> > 
> >  
> > with requirements for the CA inclusion process. 
> > 
> > It is proposed that we add a new subsection 10 to MRSP section 3.1.4 
> > 
> >  
> > that would require that audit documentation disclose the facility site 
> > locations that were, or were not, examined. 
> > 
> > One concern that has been raised previously is that the Baseline 
> > Requirements do not define "facility site location". However, we believe 
> > that the language above at 
> > https://wiki.mozilla.org/CA/Audit_Statements#Minimum_Expectations 
> > accomplishes that. We're open to suggestions for re-wording parts of it to 
> > make it even better. 
> > 
> > Currently, the audit letter template for WebTrust for CAs references the 
> > site location audited (at the level of specificity that is proposed 
> > above). Over this past year, due to COVID, some ETSI attestation letters 
> > have also explained which sites were and were not checked. This approach 
> > seems to work, and the additional information will be beneficial in the 
> > future as we evaluate the security and trust of PKI service providers. 
> > 
> > So, for the page cited above, we intend to move "Minimum Expectations" out 
> > from under "Audit Delay" so that it stands separately as a requirement for 
> > disclosing the facility site location. Then we will also revise MRSP 
> > section 3.1.4 by inserting a new subsection 10 to require "facility site 
> > locations that were, or were not, examined" with a hyperlink to the Minimum 
> > Expectations language cited above. 
> > 
> > We look forward to your comments and suggestions. 
> > 
> > Sincerely yours, 
> > 
> > Ben
> Hi Ben. Happy New Year. I have asked the WebTrust Task Force members to 
> provide their comments and Don and I will then provide a more detailed 
> response. I wanted to be sure to get each of the major firms' feedback before 
> responding. 
> 
> Thank you. 
> 
> Jeff

Ben, Don and I offer the following response, which has been vetted through the 
WebTrust Task Force:

Proposed Requirement
Disclose each location (at the state/province level) that was included in the 
scope of the audit or should have been included in the scope of the audit, 
whether the inspection was physically carried out in person at each location, 
and which audit criteria were checked (or not checked) at each location.
•   If the CA has more than one location in the same state/province, then 
use terminology to clarify the 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Saturday, November 7, 2020 at 10:36:58 AM UTC-6, Ryan Sleevi wrote:
> On Sat, Nov 7, 2020 at 9:21 AM Jeff Ward via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > Sure Ryan, the answer is quite simple. When I used the word "public" in 
> > my post, I should have been more clear as to the nuance of this concept. 
> > Public reports by definition are generally distributed (available to 
> > anyone). When reports are restricted in distribution and only intended for 
> > a certain user or users as specified in the report, they are no longer 
> > public. In each of the EY examples you reference, they all state in the 
> > last paragraph before the EY signature, "This report is intended solely for 
> > the information and use of GPO-CA and the Federal PKI Policy Authority and 
> > is not intended to be, and should not be, used by anyone other than GPO-CA 
> > and the Federal PKI Policy Authority." When reports are not generally 
> > distributed and available to the general public, the specifics of 
> > individuals performing the audit will not be present. When my firm issues 
> > reports for FPKI, we also provide the listing of individuals in a letter 
> > that is not public information. 
> >
> Thanks Jeff, 
> 
> This is useful for confirming that there is a clearly viable path forward 
> here. As you know, I appreciate the nuance here regarding public reporting, 
> as well as reports that are restricted in distribution but still made 
> public (e.g. as part of a regulatory function, such as the OIG in this 
> case). I think we agree in the substance: that this is possible, and are 
> merely working out the details here with regards to reporting. 
> 
> For example, Mozilla could require that, in addition to the "traditional" 
> WebTrust reporting , Mozilla be named as part of a restricted distribution 
> report that contains these details. Alternatively, Mozilla could require 
> that, as part of participating within their root program, CAs ensure such 
> reports include as restricted distribution those Subscribers, Relying 
> Parties, and Application Vendors that would rely upon the CAs' services, 
> much like widely practiced in industry today with respect to cloud 
> providers. 
> 
> Would you agree that it's fair to say that it is not fundamentally that 
> auditors cannot report such information, but focused here on the details of 
> how that report is delivered to Mozilla?

Thanks Ryan, I do agree that this information is better placed in a separate 
communication as you suggest.  As we already worked through the restricted use 
issue with the detailed controls report, we could adopt that same limited 
distribution language in the added communication you suggest.  So now you have 
me thinking.  This separate communication could also be used to discuss other 
items, especially the locations in scope and visited during the audit.  Without 
having this separate communication, this information will likely be added as 
another exhibit to the report, especially for the larger, more complex CAs, 
making a long report even longer in volume.  (Side note: length of some reports 
is an issue when the WebTrust seal is applied).  I think if this information 
was put into the separate communication, we could solve another problem herein 
by taking this information that is candidly sensitive (specific locations) and 
move it to a more restricted report.  I'll be interested to hea
 r your thoughts on this approach as well.  

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Friday, November 6, 2020 at 1:13:43 PM UTC-6, Ryan Sleevi wrote:
> On Fri, Nov 6, 2020 at 12:31 PM Jeff Ward via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > Audit reports, whether for WebTrust, financial statements, or other forms 
> > of engagement reports providing assurance to users of the information, do 
> > not include specific audit team members’ names. Simply stated, this desire 
> > to include individual auditor’s qualifications in a public report is not 
> > consistent with any other compliance reporting methods or reporting 
> > requirements for CAs, or any other auditee for that matter.
> Hi Jeff, 
> 
> Could you help me square this statement with the practical examples? For 
> example, here's a report [1] from a WebTrust TF member, Ernst and Young, 
> demonstrating how this works in practice. You can see there hasn't been an 
> issue for years [2][3], even with the direct involvement of individuals on 
> the WebTrust TF in preparing such an audit. 
> 
> So I'm having difficulty squaring your statement that they "do not", when 
> we've got examples from long-standing members of the WebTrust TF that 
> demonstrate that, in practice, they do. Could you help highlight what's 
> inconsistent here? 
> 
> Alternatively, and as mentioned to ETSI, here's an example of an ISAE 3000 
> based audit scheme, similar to WebTrust, that also discloses the relevant 
> professional qualifications and skills to clients [4], as discussed in 
> 4.4.8 and 4.4.9. 
> 
> [1] https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf 
> [2] 
> https://www.oversight.gov/sites/default/files/oig-reports/Assessment%20Report%2019-12%20GPO%20Federal%20PKI%20Compliance.pdf
>  
> [3] https://www.oversight.gov/sites/default/files/oig-reports/17-27.pdf 
> [4] 
> https://www.bsi.bund.de/EN/Topics/CloudComputing/Compliance_Criteria_Catalogue/Compliance_Criteria_Catalogue_node.html

Sure Ryan, the answer is quite simple.  When I used the word "public" in my 
post, I should have been more clear as to the nuance of this concept.  Public 
reports by definition are generally distributed (available to anyone).  When 
reports are restricted in distribution and only intended for a certain user or 
users as specified in the report, they are no longer public.  In each of the EY 
examples you reference, they all state in the last paragraph before the EY 
signature, "This report is intended solely for the information and use of 
GPO-CA and the Federal PKI Policy Authority and is not intended to be, and 
should not be, used by anyone other than GPO-CA and the Federal PKI Policy 
Authority."  When reports are not generally distributed and available to the 
general public, the specifics of individuals performing the audit will not be 
present.   When my firm issues reports for FPKI, we also provide the listing of 
individuals in a letter that is not public information.  

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Tuesday, November 3, 2020 at 5:53:52 PM UTC-6, Ben Wilson wrote:
> Historically, Mozilla Policy required that CAs "provide attestation of 
> their conformance to the stated verification requirements and other 
> operational criteria by a competent independent party or parties with 
> access to details of the CA's internal operations." 
> https://wiki.mozilla.org/CA:CertificatePolicyV1.0 "Competency" was "for 
> whom there is sufficient public information available to determine that the 
> party is competent to judge the CA's conformance to the stated criteria. In 
> the latter case the 'public information' referred to should include 
> information regarding the party's: 
> 
> - knowledge of CA-related technical issues such as public key 
> cryptography and related standards; 
> - experience in performing security-related audits, evaluations, or risk 
> analyses; *and* 
> - honesty and objectivity." 
> 
> Today, section 3.2 of the MRSP 
> 
>  
> states, "In normal circumstances, Mozilla requires that audits MUST be 
> performed by a Qualified Auditor, as defined in the Baseline Requirements 
> section 8.2," but under section 2.3 
> ,
>  
> "Mozilla reserves the right to accept audits by auditors who do not meet 
> the qualifications given in section 8.2 of the Baseline Requirements, or 
> refuse audits from auditors who do." 
> 
> Section 8.2 of the Baseline Requirements states an auditor must have: 
> 1. Independence from the subject of the audit; 
> 2. The ability to conduct an audit that addresses the criteria specified in 
> an Eligible Audit Scheme (see Section 8.1); 
> 3. Employs individuals who have proficiency in examining Public Key 
> Infrastructure technology, information security tools and techniques, 
> information technology and security auditing, and the third-party 
> attestation function; 
> 4. (For audits conducted in accordance with any one of the ETSI standards) 
> accredited in accordance with ISO 17065 applying the requirements specified 
> in ETSI EN 319 403; 
> 5. (For audits conducted in accordance with the WebTrust standard) licensed 
> by WebTrust; 
> 6. Bound by law, government regulation, or professional code of ethics; and 
> 7. Except in the case of an Internal Government Auditing Agency, maintains 
> Professional Liability/Errors & Omissions insurance with policy limits of 
> at least one million US dollars in coverage 
> 
> It is proposed in Issue #192 
>  that information about 
> individual auditor's qualifications be provided--identity, competence, 
> experience and independence. (For those interested as to this independence 
> requirement, Mozilla Policy v.1.0 required either disclosure of the 
> auditor's compensation or the establishment that the auditor "is bound by 
> law, government regulation, and/or a professional code of ethics to render 
> an honest and objective judgement regarding the CA.") 
> 
> While subsection 3 of BR 8.2 requires "individuals who have proficiency in 
> examining Public Key Infrastructure technology, information security tools 
> and techniques, information technology and security auditing, and the 
> third-party attestation function," that fact needs evidence in order to be 
> established. The proposed resolution of this Issue #192 intends to 
> accomplish that. 
> 
> This proposal to require disclosure of individual auditor qualifications is 
> very similar to the approach adopted by the U.S. Federal PKI 
> 
>  
> (see Appendices B-1 and C). E.g., "Did each Audit Opinion Letter identify 
> the auditor and the individuals performing the audit?" In practice, the 
> information about auditor qualifications could be in the form of a separate 
> document, such as a curriculum vitae. 
> 
> Some initial, draft language to address this issue is located here: 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/d0da7cb2b6db38e66c3a72e5c1db0e78e91d8df6
>  
> 
> A new subsection 3. would be added to the list of audit requirements that 
> would require "[the] name(s) and qualifications of individuals performing 
> the audit, as required by section 3.2" and a new paragrpah would be added 
> to section 3.2 that would say, "A Qualified Auditor MUST have relevant IT 
> Security experience, or have audited a number of CAs, and be independent 
> and not conflicted. Individuals have competence, partnerships and 
> corporations do not. Audit documentation of individual auditor 
> qualifications MUST be provided to Mozilla that is sufficient for Mozilla 
> to determine the competence, experience, and independence of the Qualified 
> Auditor. Mozilla will review each individual 

Re: Policy 2.7.1: MRSP Issue #187: Require disclosure of incidents in Audit Reports

[Jeff Ward via dev-security-policy]

On Thursday, October 22, 2020 at 1:53:40 PM UTC-5, Ben Wilson wrote:
> The purpose of this email is to begin public discussion on the addition of 
> a subsection 11 to section 3.1.4 of the Mozilla Root Store Policy. Issue 
> #187  in GitHub proposes 
> to require audit reports to list all incidents occurring (or open) during 
> the audit period of which the auditor has been made aware or to state that 
> the auditor is unaware of any incidents. This is related to Issue #154 
>  (management assertion 
> disclosures). That proposal is to have section 2.4 read as follows: "If 
> being audited to the WebTrust criteria, the Management Assertion letter 
> MUST include all known incidents that occurred or were still 
> open/unresolved at any time during the audit period." 
> 
> Proposed language may be found in the following commits: 
> 
> - 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/f6639f503b743aae402dc0f4841dc3dd5ba88753
>  
> - 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/6c07c44e4db473dc4d34009f1bc955a0e18cb4c1
>  
> - 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/5dec00e53b4c6361d85af7644660fe185fcf463d
>  
> 
> Proposed language for section 3.1.4 is: 
> 
> "11. all incidents (as defined in section 2.4) that occurred or were still 
> open/unresolved at any time during the audit period, or a statement that 
> the auditor is unaware of any;" 
> 
> I look forward to your comments, suggestions and discussions. 
> 
> Ben

Thanks for bringing this up Ben.  It is important to consider this requirement 
in conjunction with #154 and address them together. It seems reasonable to 
require a CA to disclose all known incidents that are applicable during a given 
period. It would be important, however, to define “known incident” as a 
“verified bug” and exclude items such as bugs closed as a duplicate, invalid, 
etc.  It would also make sense to clarify that an incident should only be 
disclosed once and eliminate duplication when an incident spans two audit 
periods. 

Also keep in mind an auditor typically issues an opinion on management’s 
assertion of its controls. Audit opinions do not make negative assurance 
statements, such as not being aware of any incidents during the period. If the 
CA is required to make this assertion, the auditor’s opinion will consider that 
statement. 

Thanks, 

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #207: Require audit statements to provide information about which CA Locations were audited

[Jeff Ward via dev-security-policy]

On Tuesday, December 15, 2020 at 2:41:10 PM UTC-6, Ben Wilson wrote:
> All, 
> 
> This email is part of the discussion for the next version of the Mozilla 
> Root Store Policy (MSRP), version 2.7.1, to be published during of Q1-2021. 
> 
> For audit delays, we currently require that audit statements disclose the 
> locations that were and were not audited, but that requirement has not been 
> incorporated yet into the MRSP. See 
> https://wiki.mozilla.org/CA/Audit_Statements#Minimum_Expectations. That 
> provision reads as follows: 
> 
> Disclose each location (at the state/province level) that was included in 
> the scope of the audit or should have been included in the scope of the 
> audit, whether the inspection was physically carried out in person at each 
> location, and which audit criteria were checked (or not checked) at each 
> location. 
> 
> - If the CA has more than one location in the same state/province, then 
> use terminology to clarify the number of facilities in that state/province 
> and whether or not all of them were audited. For example: "Facility 1 in 
> Province", "Facility 2 in Province, Facility 3 in Province" *or* 
> "Primary Facility in Province", "Secondary Facility in Province", "Tertiary 
> Facility in Province". 
> - The public audit statement does not need to identify the type of 
> Facility. 
> - "Facility" includes: data center locations, registration authority 
> locations, where IT and business process controls of CA operations are 
> performed, facility hosting an active HSM with CA private keys, 
> facility or 
> bank deposit box storing a deactivated and encrypted copy of a 
> private key. 
> 
> It is proposed by Issue #207 
>  that this language 
> requiring the disclosure of site locations--audited and unaudited--be made 
> clearly part of the MSRP by reference to the language above. 
> 
> A similar method of incorporating by reference has been taken in section 
> 2.4 of the MSRP 
> 
>  
> with respect to incident reporting and in section 7.1 
> 
>  
> with requirements for the CA inclusion process. 
> 
> It is proposed that we add a new subsection 10 to MRSP section 3.1.4 
> 
>  
> that would require that audit documentation disclose the facility site 
> locations that were, or were not, examined. 
> 
> One concern that has been raised previously is that the Baseline 
> Requirements do not define "facility site location". However, we believe 
> that the language above at 
> https://wiki.mozilla.org/CA/Audit_Statements#Minimum_Expectations 
> accomplishes that. We're open to suggestions for re-wording parts of it to 
> make it even better. 
> 
> Currently, the audit letter template for WebTrust for CAs references the 
> site location audited (at the level of specificity that is proposed 
> above). Over this past year, due to COVID, some ETSI attestation letters 
> have also explained which sites were and were not checked. This approach 
> seems to work, and the additional information will be beneficial in the 
> future as we evaluate the security and trust of PKI service providers. 
> 
> So, for the page cited above, we intend to move "Minimum Expectations" out 
> from under "Audit Delay" so that it stands separately as a requirement for 
> disclosing the facility site location. Then we will also revise MRSP 
> section 3.1.4 by inserting a new subsection 10 to require "facility site 
> locations that were, or were not, examined" with a hyperlink to the Minimum 
> Expectations language cited above. 
> 
> We look forward to your comments and suggestions. 
> 
> Sincerely yours, 
> 
> Ben

Hi Ben.  Happy New Year.  I have asked the WebTrust Task Force members to 
provide their comments and Don and I will then provide a more detailed 
response.  I wanted to be sure to get each of the major firms' feedback before 
responding.

Thank you.

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Thursday, February 11, 2021 at 12:41:44 PM UTC-6, Ben Wilson wrote:
> All, 
> 
> I've modified the proposed change to MRSP section 3.2 so that it would now 
> insert a middle paragraph that would read: 
> 
> "A Qualified Auditor MUST have relevant IT Security experience, or have 
> audited a number of CAs, and be independent and not conflicted. Individuals 
> have competence, partnerships and corporations do not. Each Audit Report 
> MUST be accompanied by documentation provided to Mozilla of individual 
> auditor qualifications sufficient for Mozilla to determine the competence, 
> experience, and independence of the Qualified Auditor." 
> 
> See 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/57063dc07f5b753184c94dbf5d0d30d0b9b90789
>  
> 
> The basis for further interpretation of the above language would still be 
> section 8.2 of the Baseline Requirements. ("In normal circumstances, 
> Mozilla requires that audits MUST be performed by a Qualified Auditor, as 
> defined in the Baseline Requirements section 8.2"). 
> 
> Section 3.1.4 still remains with a proposed subsection 3 - "name(s) and 
> qualifications of individuals performing the audit, as required by section 
> 3.2." 
> 
> I anticipate that additional guidance for how CAs should submit this 
> information will be made available here on the wiki - 
> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications. 
> 
> 
>  
> Ben
> On Thu, Jan 28, 2021 at 2:10 PM Ryan Sleevi  wrote: 
> 
> > 
> > On Thu, Jan 28, 2021 at 3:05 PM Ben Wilson  wrote: 
> > 
> >> Thanks. My current thinking is that we can leave the MRSP "as is" and 
> >> that we write up what we want in 
> >> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications, 
> >> which is, as you note, information about members of the audit team and how 
> >> individual members meet #2, #3, and #6. 
> >> 
> > 
> > Is this intended as a temporary fix until the issue is meaningfully 
> > addressed? Or are you seeing this as a long-term resolution of the issue? 
> > 
> > I thought the goal was to make the policy clearer on the expectations, and 
> > my worry is that it would be creating more work for you and Kathleen, and 
> > the broader community, because it puts the onus on you to chase down CAs to 
> > provide the demonstration because they didn't pay attention to it in the 
> > policy. This was the complaint previously raised about "CA Problematic 
> > Practices" and things that are forbidden, so I'm not sure I understand the 
> > distinction/benefit here from moving it out? 
> > 
> > I think the relevance to MRSP is trying to clarify whether Mozilla thinks 
> > of auditors as individuals (as it originally did), or whether it thinks of 
> > auditors as organizations. I think that if MRSP was clarified regarding 
> > that, then the path you're proposing may work (at the risk of creating more 
> > work for y'all to request that CAs provide the information that they're 
> > required to provide, but didn't know that). 
> > 
> > If the issue you're trying to solve is one about whether it's in the audit 
> > letter vs communicated to Mozilla, then I think it should be possible to 
> > achieve that within the MRSP and explicitly say that (i.e. not require it 
> > in the audit letter, but still requiring it). 
> > 
> > Just trying to make sure I'm not overlooking or misunderstanding your 
> > concerns there :) 
> > 
> >>
I wanted to clarify a couple of points.  Firms must be independent to do 
audit/assurance work.  If independence is impaired, for example, by one person 
in the firm performing management functions, the entire firm is no longer 
independent.  Firms have the responsibility to monitor activities of its 
professionals, which also includes personal investments, to ensure they remain 
independent.

Also, WebTrust practitioners provide information on the firm and the 
professionals used on these engagements.  The information provided is closely 
aligned with the Auditor Qualifications you are describing.  As you know, CPA 
Canada provides a listing of qualified audit firms on its website.  Working 
closely with them could also help in instances where auditor qualifications are 
in question.

And one last item, thank you for hearing us on the listing of auditors 
performing the engagement.  The only place I am aware that lists the audit 
partner in a comparable world is the signing audit partner on public company 
audits in the US, which is available on the SEC website.  Other than that, I am 
not aware of any other team member being listed.  We have seen listings of team 
members and related experience summarized on a non-publicly issued letter to 
management in the US Federal space.

Hope this helps!

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Monday, February 15, 2021 at 1:57:11 PM UTC-6, Ryan Sleevi wrote:
> On Mon, Feb 15, 2021 at 2:03 PM Jeff Ward via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > I wanted to clarify a couple of points. Firms must be independent to do 
> > audit/assurance work. If independence is impaired, for example, by one 
> > person in the firm performing management functions, the entire firm is no 
> > longer independent. Firms have the responsibility to monitor activities of 
> > its professionals, which also includes personal investments, to ensure they 
> > remain independent. 
> > 
> > Also, WebTrust practitioners provide information on the firm and the 
> > professionals used on these engagements. The information provided is 
> > closely aligned with the Auditor Qualifications you are describing. As you 
> > know, CPA Canada provides a listing of qualified audit firms on its 
> > website. Working closely with them could also help in instances where 
> > auditor qualifications are in question. 
> > 
> > And one last item, thank you for hearing us on the listing of auditors 
> > performing the engagement. The only place I am aware that lists the audit 
> > partner in a comparable world is the signing audit partner on public 
> > company audits in the US, which is available on the SEC website. Other 
> > than that, I am not aware of any other team member being listed. We have 
> > seen listings of team members and related experience summarized on a 
> > non-publicly issued letter to management in the US Federal space.
> Jeff, 
> 
> https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf 
> 
> Is an example, which is an audit of the U.S. Government Printing Office, 
> provided by a WTTF member, against the US Federal PKI CP. This doesn’t meet 
> the criteria you mentioned (public company, SEC), and itself was provided 
> several years ago. 
> 
> It is directed to a set of named parties, and made publicly available by 
> those parties, using the WebTrust for CAs criteria. On page 4 (report)/6 
> (FPKI submission)/9 (PDF page), you can see an enumerated list of audit 
> participants and their applicable skills, summarized. 
> 
> Since you mentioned “a comparable world”, the BSI C5 controls, which 
> provide a valuable model for improvements in transparency and thoroughness 
> of reporting (aka the so called “detailed controls” report), notes this 
> within Section 3.5.1 of the Controls [1] 
> 
> “As part of the reporting, it must be specified which of the professional 
> examinations/certifications are held by the audit team (e. g. in the 
> section “Independence and quality assurance of the auditor”). Upon request, 
> appropriate documents (e. g. certificates etc.) must be submitted to the 
> client.” 
> 
> Could you clarify whether you and the WTTF considered these two cases? The 
> former is an example of using an assurance scheme the FPKIMA has said on 
> its own is insufficient, namely WTCA, but with additional reporting can be 
> made sufficient. The latter is an example of a scheme specifically adapted 
> for cloud/vendor security controls against an ISAE 3000 reporting scheme, 
> which is nearly identical to WTBRs in that regard. It was unclear if y’all 
> were simply not familiar with these cases, or if you believe there is 
> substantive differences in the proposal here that may require addressing. 
> 
> [1] 
> https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/CloudComputing/ComplianceControlsCatalogue-Cloud_Computing-C5.pdf?__blob=publicationFile=3
>  
> 
> >
Correct Ryan.  This is one of the examples you provided previously.  This 
report is of course restricted:
"This report is intended solely for the information and use of {CA} and the 
Federal PKI Policy Authority and is not intended to be, and should not be, used 
by anyone other than {CA} and Federal PKI Policy Authority."

As you know, this report then is not generally / publicly distributed as it is 
a restricted use report.  This restriction does not appear in the public 
company audit opinions, hence the reference.  I called out the federal space 
with this very example in mind.  So yes, I am aware of and quite familiar with 
this scenario.  It is more about the restricted use (or in this case the lack 
thereof) as it is the framework being used.  The very point you are referencing 
an opinion that you are using outside of the restriction sums up my argument.  
I can't speak for all audit firms as this is more of a risk management issue, 
but my firm would be fine issuing this report in a restricted manner, unless it 
became known it would not actually be used in the restricted manner.  That 
defeats the whole purpose.  I've 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote:
> Apologies for belaboring the point, but I think we might be talking past 
> eachother. 
> 
> You originally stated “The only place I am aware that lists the audit
> partner in a comparable world is the signing audit partner on public
> company audits in the US, which is available on the SEC website.” I gave 
> two separate examples of such, and you responded to one (FPKI) by saying 
> the report was not public (even when it is made available publicly), and 
> the other I didn’t see a response to. 
> 
> This might feel like nit-picking, but I think this is a rather serious 
> point to work through, because I don’t think you’re fully communicating 
> what you judge to be a “comparable world”, as it appears you are dismissing 
> these examples. 
> 
> I can think of several possible dimensions you might be thinking are 
> relevant, but rather than assume, I’m hoping you can expand with a few 
> simple questions. Some admittedly seem basic, but I don’t want to take 
> anything for granted here. 
> 
> 1) Are you/the WTTF familiar with these audit schemes? 
> 
> 2) Are you aware of schemes that require disclosing the relevant skills and 
> experience of the audit team to the client? (E.g. as done by BSI C5 audits 
> under ISAE 3000) 
> 
> 3) Are you aware of such reports naming multiple parties for the use of the 
> report (e.g. as done by FPKI audits) 
> 
> 4) Are you aware of schemes in which a supplier requires a vendor to be 
> audited, and ensures that the customer of supplier are able to access such 
> audits as part of their reliance upon supplier? (Note, this doesn’t have to 
> be limited to ISMS systems) 
> 
> I’m trying to understand what, given the prevalence of these practices, 
> makes these instances *not* a comparable world, since it seems that helps 
> move closer to solutions.


Ryan, I hope you are not suggesting I am dodging you points.  That would be 
absurd.  Let me use different words as comparable world seems to be tripping 
you up.  You are not providing a general/public distribution example to make 
your point so it is baseless.  You are using a restricted opinion from EY and 
neither Ryan Sleevi nor Google are listed as the two intended users.  The 
closest I have seen to support your desire to name individual auditors is in 
public company audit reports, which are housed on the SEC website.  To be 
clear, of your two examples, one is an opinion, which is restricted, and the 
other represents the guidelines.  Perhaps you have seen a public/general 
distribution report from your second opinion as I do not see it in this thread. 
 I am aware, as mentioned previously, of the Federal PKI program desiring to 
know more about team members, but that is not listed in a non-restricted 
report, in a public/general distribution format.  

EY did the FPKI audit.  I am not sure why you keep tagging the as a WTTF 
member.  They are a global firm so if you are implying only they know the 
standards/rules (which I hope you are not) would be misleading.  But to answer 
you question #1, yes.  We even spoke last about this in our TF meeting last 
week and every member had the same response, including the one you have 
referenced.  #2 answered previously.  We are not arguing who wants what.  The 
fact this information is desired is not being debated, rather how it is 
reported to the user.  #3 question is unclear.  I am not aware of any report 
that restricts an opinion to certain users specifically, in the case you 
mentioned the CA and FPKI that allows additional users to get this information. 
 SOC2 for example has a broader restriction which allows the reports to go to a 
class or classes of users.  Your example is not that case.#4 I am 
definitely aware of this requirement.  A public/general distribution report can 
be shared with anyone.  The restriction dictates who gets the opinion.  This is 
the main point you are not understanding Ryan.  For example, I if perform an 
audit of a company and restrict it to them and one other user, say their bank, 
the engagement letter / statement of work would clearly reflect this 
restriction.  In addition, the standard terms would require the company to get 
permission to issue the report beyond the specified users.  The example you 
raise in this question is certainly covered under the broad type of restriction 
that SOC2 provides, as they would be knowledgeable about the subject matter.  
The EY report example you provided does not include the broader use.  And not 
to belabor this point, but the restriction precludes its public/general 
distribution.  The words matter.  When the distribution of a report is tightly 
binding two parties, as in your example, you can't distribute it broader.  
Restricted reports by definition are different.

This is a long dialogue supporting Ben's approach.  Firms will most likely be 
willing to provide the qualifications of auditors in a non-public 

Re: Policy 2.7.1: MRSP Issue #187: Require disclosure of incidents in Audit Reports

[Jeff Ward via dev-security-policy]

On Friday, February 12, 2021 at 10:27:11 AM UTC-6, Ben Wilson wrote:
> I'm fine with that suggestion.
> On Fri, Feb 12, 2021 at 5:06 AM malcol...--- via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > On Thursday, 11 February 2021 at 21:14:13 UTC, Ben Wilson wrote:
> > > 11. all incidents (as defined in section 2.4), including those reported 
> > in 
> > > Bugzilla, that were: 
> > > * disclosed by the CA or discovered by the auditor, and 
> > > * unresolved at any time during the audit period; 
> > >
> > > The idea is that all "incidents" must be reported if they were 
> > "unresolved" 
> > > - which would include those that occurred or were open - at any time 
> > during 
> > > the audit period. 
> > > 
> >
> > Wouldn't it be clearer to non-native English speakers to avoid the nuance 
> > associated with "unresolved at any time" needing to imply both those that 
> > occurred or those that were still open? 
> > 
> > Why not amend the language to just say: 
> >
> > 11. all incidents (as defined in section 2.4), including those reported in
> > Bugzilla, that: 
> > * were disclosed by the CA or discovered by the auditor, and 
> > * occurred or were open at any time during the audit period;
> > ___ 
> > dev-security-policy mailing list 
> > dev-secur...@lists.mozilla.org 
> > https://lists.mozilla.org/listinfo/dev-security-policy 
> >
This wording works from a WebTrust perspective as well.  Thanks!
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy