Thanks for the response Jeff. On Fri, Sep 6, 2019 at 4:17 PM jeffwardpki--- via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote:
> On Wednesday, August 21, 2019 at 11:46:37 PM UTC-5, Jeremy Rowley wrote: > > Hey all, > > > > An interesting issue came up recently with audits. Because the Mozilla > policy includes some requirements that diverge from the BRs, the audit > criteria don't necessarily cover everything Mozilla cares about. Thus, it's > possible to have an incident that doesn't show up on an audit. It's also > possible that the auditor determines the incident is not sufficiently > important/risky(?) to include it in an audit. For example: > https://bugzilla.mozilla.org/show_bug.cgi?id=1458024. Auditors aren't > controlled by the CA and operate independently which means the CA can't > dictate what goes into the opinion. One solution is to require CAs to list > all of the incidents that occur during their audit in the management > assertion letter. I posted an addendum to the management assertion on that > thread. Going forward, we'll just include it as part of the main body. I > need to look into whether I can get our existing audit reissued to the > appendix is part of the seal as well. > > > > What do you think about just requiring that as part of the Mozilla > policy? Ie - the management assertion letter must include a list of the > incidents active/opened during the audit period. Something like that could > ensure transparency and make sure all incidents are disclosed to the > auditor, distinguishing the CA's disclosures from the auditors. > > > > Jeremy > > Reposting as I don't see my original response to this thread: > > Sorry for the delay in responding, but I first wanted to gather feedback > from the members of the WebTrust Task Force. Keep in mind any > audit/assurance engagement entails professional judgment, which of course > may vary from firm to firm. We are certainly seeing a trend in audit > reports whereby “Other Matters” are described that do not modify/qualify > the auditor’s opinion but do allow the auditor to make mention of the > issues. Some firms use this section to list items noted in Bugzilla, for > instance, to demonstrate that these types of issues were addressed in the > course of the audit. > > Depending on the firm, some firms have been listing all issues noted in > Bugzilla, while others are listing only those that are significant. As a > Task Force, it is not our position, or even a possibility, for us to > mandate how these should be handled as professional judgment will dictate > their treatment based on the respective firms’ risk management policies. > That being said, we have as part of our draft practitioner guidance > commentary that the browser community prefers seeing all these types of > issues listed as either modifications/qualifications, or described in > “Other Matters” and encourage practitioners to use this approach. > > I would go beyond "all issues noted in Bugzilla" to "all issues, including those noted in Bugzilla". In most cases, management’s Assertion accompanies the audit report, and > there is greater flexibility for what goes into them. Our professional > standards require certain items be present in the Assertion, but other > items that management wants to add are permissible to include. Except in > the rare reporting scenario when an assertion by management is not > provided, most of the Task Force members agree there would not be any issue > with your proposed requirement to require a CA's management to detail > Bugzilla or similar issues relevant to the current audit period in their > assertion to the auditors. > > By doing this, a CA can eliminate any question about disclosure of incidents to auditors, which is terrific and encouraged. As a Mozilla requirement it has the problem - as Ryan pointed out - that ETSI audits don't include management assertions, at least not today. > As far as the management representation letter, which is a required > written communication to the auditors at the conclusion to the audit, > regulatory non-compliance would normally form part of the management > representation that is needed for each audit, and the assertion can form > part of that representation. Keep in mind that the management > representation letter is not part of the deliverable that is viewed > publicly as is the case with the auditor’s opinion and management assertion. > > To demonstrate that the auditor has addressed the significance of the > relevant issues, we can propose that the standard WebTrust reports have an > Exhibit that sets out the issues identified by reference to Bugzilla, as an > example. The audit report can then reflect the relevant significance > either as a report qualification or through an "Other Matters" paragraph > that provides further commentary. The main issue is that, even by > identifying and addressing them in the audit report, there is still the > potential for disagreements to exist as to significance due to auditor’s > professional judgment. Of course, in our new SOC-like report, these issues > will be discussed in the system description, audit report and potentially > an unaudited management comment section. > > This also sounds like a great idea, and one that is more in line with the solution Clemens described for ETSI - disclosing major and minor conformities in all attestation reports. I'm much less interested in the categorization of an issue as a qualification versus an "other matter" than I am with having it disclosed. Under this approach, Mozilla could require that all audit reports include a section listing all incidents that occurred during the audit period. Given that Mozilla already places specific requirements on audit reports in section 3.1.4, our policy is a good location for this requirement. I filed https://github.com/mozilla/pkipolicy/issues/187 to track this idea. Thanks, > > Jeff Ward > > _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
