Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Watson Ladd via dev-security-policy]

On Monday, February 15, 2021 at 3:07:12 PM UTC-8, Jeff Ward wrote:
> On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote: 
> > Apologies for belaboring the point, but I think we might be talking past 
> > eachother. 
> > 
> > You originally stated “The only place I am aware that lists the audit 
> > partner in a comparable world is the signing audit partner on public 
> > company audits in the US, which is available on the SEC website.” I gave 
> > two separate examples of such, and you responded to one (FPKI) by saying 
> > the report was not public (even when it is made available publicly), and 
> > the other I didn’t see a response to. 
> > 
> > This might feel like nit-picking, but I think this is a rather serious 
> > point to work through, because I don’t think you’re fully communicating 
> > what you judge to be a “comparable world”, as it appears you are dismissing 
> > these examples. 
> > 
> > I can think of several possible dimensions you might be thinking are 
> > relevant, but rather than assume, I’m hoping you can expand with a few 
> > simple questions. Some admittedly seem basic, but I don’t want to take 
> > anything for granted here. 
> > 
> > 1) Are you/the WTTF familiar with these audit schemes? 
> > 
> > 2) Are you aware of schemes that require disclosing the relevant skills and 
> > experience of the audit team to the client? (E.g. as done by BSI C5 audits 
> > under ISAE 3000) 
> > 
> > 3) Are you aware of such reports naming multiple parties for the use of the 
> > report (e.g. as done by FPKI audits) 
> > 
> > 4) Are you aware of schemes in which a supplier requires a vendor to be 
> > audited, and ensures that the customer of supplier are able to access such 
> > audits as part of their reliance upon supplier? (Note, this doesn’t have to 
> > be limited to ISMS systems) 
> > 
> > I’m trying to understand what, given the prevalence of these practices, 
> > makes these instances *not* a comparable world, since it seems that helps 
> > move closer to solutions.
> Ryan, I hope you are not suggesting I am dodging you points. That would be 
> absurd. Let me use different words as comparable world seems to be tripping 
> you up. You are not providing a general/public distribution example to make 
> your point so it is baseless. You are using a restricted opinion from EY and 
> neither Ryan Sleevi nor Google are listed as the two intended users. The 
> closest I have seen to support your desire to name individual auditors is in 
> public company audit reports, which are housed on the SEC website. To be 
> clear, of your two examples, one is an opinion, which is restricted, and the 
> other represents the guidelines. Perhaps you have seen a public/general 
> distribution report from your second opinion as I do not see it in this 
> thread. I am aware, as mentioned previously, of the Federal PKI program 
> desiring to know more about team members, but that is not listed in a 
> non-restricted report, in a public/general distribution format. 

I can click on the URL and read it.  This seems to be the very definition of 
public, even if the audit report says it is not for reliance upon by the 
general public. I fully appreciate that this may be a technicality in the world 
of auditing, but it is very confusing to those of us who are less familiar.

> Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Mon, Feb 15, 2021 at 6:07 PM Jeff Ward via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Ryan, I hope you are not suggesting I am dodging you points.  That would
> be absurd.  Let me use different words as comparable world seems to be
> tripping you up.


I'm not trying to suggest you're dodging intentionally, but as I said, I
think we were, and to a degree still are, talking past each other. I think
your reply captured that you don't think I understood your phrasing,
which I didn't, but I also don't think you understood why I was trying to
unpack that phrase more.


> You are not providing a general/public distribution example to make your
> point so it is baseless.  You are using a restricted opinion from EY and
> neither Ryan Sleevi nor Google are listed as the two intended users.


I'm not sure why you're bringing in Google with respect to
https://wiki.mozilla.org/CA/Policy_Participants . I'm also not sure how
this relates to the specific questions I asked, which were an attempt to
reset the conversation here to try and make progress on finding solutions.


> The closest I have seen to support your desire to name individual auditors
> is in public company audit reports, which are housed on the SEC website.
> To be clear, of your two examples, one is an opinion, which is restricted,
> and the other represents the guidelines.  Perhaps you have seen a
> public/general distribution report from your second opinion as I do not see
> it in this thread.  I am aware, as mentioned previously, of the Federal PKI
> program desiring to know more about team members, but that is not listed in
> a non-restricted report, in a public/general distribution format.


Sure, and the purpose of the questions was an attempt to reset the
conversation to a point where we're not talking past each other, by trying
to build up from basics. I know we both know there are many different audit
criteria that are used, and the WTTF member organizations are global
organizations. I don't want to assume that those who focus on the WTTF are
as familiar with, say, the BSI requirements or other audit criteria, much
the same way it wouldn't be reasonable to expect I know as much about AI as
PKI :)

The example report I provided wasn't about calling out an individual
member, but rather to highlight an area that, objectively, seems
comparable, particularly to Ben's proposed language, to better understand
what you meant by that, and whether you had concerns with the new approach
still.

You've anchored here on the public disclosure part, which I can understand
and appreciate the risk perceived by having auditor names attached to the
reports, instead of just firms. I see most of your answers focus on a
specific form of disclosure, but that wasn't what my previous mail was
trying to get to yet. I was trying to work back to understand where
"comparable" criteria diverge, so we can figure out possible paths.

To be clear, I support Ben's proposal as a very useful and meaningful
interim step. Your original message was a bit ambiguous about where you
stand, in large part because your use of "comparable" left quite a bit
ambiguous, and whether you thought the changes were sufficient. I'm quite
glad to hear that it sounds workable for those performing WebTrust audits,
because that helps make forward progress.

However, I still think it'd be useful to view this as an interim step, and
it certainly feels like you might be saying this as far as things can go.
In the cloud services space, we regularly see end users expecting audits of
their cloud provider, and those transitively apply to dependencies of that
cloud provider (e.g. datacenter security, software vendors used by the
cloud provider, etc). This is similarly true with respect to software
supply chain security: working out not just first order dependencies, but
working through second and third-order dependencies as well, to build up a
holistic risk picture. Browsers rely on and delegate to CAs, so it's no
surprise that browsers expect audits for CAs. Yet browsers have their own
customers, and need to provide those customers the assurances they need,
and their customers need, etc.

My Question #5 was an attempt to unpack "comparable" further, because these
are hardly unique problems to software vendor/service provider
relationships. It's not even unique to "cloud" or "PKI", as supply chain
assurances are pretty universal, as shown by goods such as coconut milk,
coffee, chocolate, or diamonds. You answered in the context of "public
report", but I wasn't trying to go there yet. I was trying to figure out
where "comparable" split off, and what schemes were and weren't
comparable, so that we can continue to make improvements.  Now that it
seems we have a path forward for direct disclosure, I think it's reasonable
to spend time thinking about how to bring that assurance to browser's
users/customers, such as this community, in the same way we help bring
transparency to cloud 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote:
> Apologies for belaboring the point, but I think we might be talking past 
> eachother. 
> 
> You originally stated “The only place I am aware that lists the audit
> partner in a comparable world is the signing audit partner on public
> company audits in the US, which is available on the SEC website.” I gave 
> two separate examples of such, and you responded to one (FPKI) by saying 
> the report was not public (even when it is made available publicly), and 
> the other I didn’t see a response to. 
> 
> This might feel like nit-picking, but I think this is a rather serious 
> point to work through, because I don’t think you’re fully communicating 
> what you judge to be a “comparable world”, as it appears you are dismissing 
> these examples. 
> 
> I can think of several possible dimensions you might be thinking are 
> relevant, but rather than assume, I’m hoping you can expand with a few 
> simple questions. Some admittedly seem basic, but I don’t want to take 
> anything for granted here. 
> 
> 1) Are you/the WTTF familiar with these audit schemes? 
> 
> 2) Are you aware of schemes that require disclosing the relevant skills and 
> experience of the audit team to the client? (E.g. as done by BSI C5 audits 
> under ISAE 3000) 
> 
> 3) Are you aware of such reports naming multiple parties for the use of the 
> report (e.g. as done by FPKI audits) 
> 
> 4) Are you aware of schemes in which a supplier requires a vendor to be 
> audited, and ensures that the customer of supplier are able to access such 
> audits as part of their reliance upon supplier? (Note, this doesn’t have to 
> be limited to ISMS systems) 
> 
> I’m trying to understand what, given the prevalence of these practices, 
> makes these instances *not* a comparable world, since it seems that helps 
> move closer to solutions.


Ryan, I hope you are not suggesting I am dodging you points.  That would be 
absurd.  Let me use different words as comparable world seems to be tripping 
you up.  You are not providing a general/public distribution example to make 
your point so it is baseless.  You are using a restricted opinion from EY and 
neither Ryan Sleevi nor Google are listed as the two intended users.  The 
closest I have seen to support your desire to name individual auditors is in 
public company audit reports, which are housed on the SEC website.  To be 
clear, of your two examples, one is an opinion, which is restricted, and the 
other represents the guidelines.  Perhaps you have seen a public/general 
distribution report from your second opinion as I do not see it in this thread. 
 I am aware, as mentioned previously, of the Federal PKI program desiring to 
know more about team members, but that is not listed in a non-restricted 
report, in a public/general distribution format.  

EY did the FPKI audit.  I am not sure why you keep tagging the as a WTTF 
member.  They are a global firm so if you are implying only they know the 
standards/rules (which I hope you are not) would be misleading.  But to answer 
you question #1, yes.  We even spoke last about this in our TF meeting last 
week and every member had the same response, including the one you have 
referenced.  #2 answered previously.  We are not arguing who wants what.  The 
fact this information is desired is not being debated, rather how it is 
reported to the user.  #3 question is unclear.  I am not aware of any report 
that restricts an opinion to certain users specifically, in the case you 
mentioned the CA and FPKI that allows additional users to get this information. 
 SOC2 for example has a broader restriction which allows the reports to go to a 
class or classes of users.  Your example is not that case.#4 I am 
definitely aware of this requirement.  A public/general distribution report can 
be shared with anyone.  The restriction dictates who gets the opinion.  This is 
the main point you are not understanding Ryan.  For example, I if perform an 
audit of a company and restrict it to them and one other user, say their bank, 
the engagement letter / statement of work would clearly reflect this 
restriction.  In addition, the standard terms would require the company to get 
permission to issue the report beyond the specified users.  The example you 
raise in this question is certainly covered under the broad type of restriction 
that SOC2 provides, as they would be knowledgeable about the subject matter.  
The EY report example you provided does not include the broader use.  And not 
to belabor this point, but the restriction precludes its public/general 
distribution.  The words matter.  When the distribution of a report is tightly 
binding two parties, as in your example, you can't distribute it broader.  
Restricted reports by definition are different.

This is a long dialogue supporting Ben's approach.  Firms will most likely be 
willing to provide the qualifications of auditors in a non-public 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

Apologies for belaboring the point, but I think we might be talking past
eachother.

You originally stated “The only place I am aware that lists the audit
partner in a comparable world is the signing audit partner on public
company audits in the US, which is available on the SEC website.” I gave
two separate examples of such, and you responded to one (FPKI) by saying
the report was not public (even when it is made available publicly), and
the other I didn’t see a response to.

This might feel like nit-picking, but I think this is a rather serious
point to work through, because I don’t think you’re fully communicating
what you judge to be a “comparable world”, as it appears you are dismissing
these examples.

I can think of several possible dimensions you might be thinking are
relevant, but rather than assume, I’m hoping you can expand with a few
simple questions. Some admittedly seem basic, but I don’t want to take
anything for granted here.

1) Are you/the WTTF familiar with these audit schemes?

2) Are you aware of schemes that require disclosing the relevant skills and
experience of the audit team to the client? (E.g. as done by BSI C5 audits
under ISAE 3000)

3) Are you aware of such reports naming multiple parties for the use of the
report (e.g. as done by FPKI audits)

4) Are you aware of schemes in which a supplier requires a vendor to be
audited, and ensures that the customer of supplier are able to access such
audits as part of their reliance upon supplier? (Note, this doesn’t have to
be limited to ISMS systems)

I’m trying to understand what, given the prevalence of these practices,
makes these instances *not* a comparable world, since it seems that helps
move closer to solutions.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Monday, February 15, 2021 at 1:57:11 PM UTC-6, Ryan Sleevi wrote:
> On Mon, Feb 15, 2021 at 2:03 PM Jeff Ward via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > I wanted to clarify a couple of points. Firms must be independent to do 
> > audit/assurance work. If independence is impaired, for example, by one 
> > person in the firm performing management functions, the entire firm is no 
> > longer independent. Firms have the responsibility to monitor activities of 
> > its professionals, which also includes personal investments, to ensure they 
> > remain independent. 
> > 
> > Also, WebTrust practitioners provide information on the firm and the 
> > professionals used on these engagements. The information provided is 
> > closely aligned with the Auditor Qualifications you are describing. As you 
> > know, CPA Canada provides a listing of qualified audit firms on its 
> > website. Working closely with them could also help in instances where 
> > auditor qualifications are in question. 
> > 
> > And one last item, thank you for hearing us on the listing of auditors 
> > performing the engagement. The only place I am aware that lists the audit 
> > partner in a comparable world is the signing audit partner on public 
> > company audits in the US, which is available on the SEC website. Other 
> > than that, I am not aware of any other team member being listed. We have 
> > seen listings of team members and related experience summarized on a 
> > non-publicly issued letter to management in the US Federal space.
> Jeff, 
> 
> https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf 
> 
> Is an example, which is an audit of the U.S. Government Printing Office, 
> provided by a WTTF member, against the US Federal PKI CP. This doesn’t meet 
> the criteria you mentioned (public company, SEC), and itself was provided 
> several years ago. 
> 
> It is directed to a set of named parties, and made publicly available by 
> those parties, using the WebTrust for CAs criteria. On page 4 (report)/6 
> (FPKI submission)/9 (PDF page), you can see an enumerated list of audit 
> participants and their applicable skills, summarized. 
> 
> Since you mentioned “a comparable world”, the BSI C5 controls, which 
> provide a valuable model for improvements in transparency and thoroughness 
> of reporting (aka the so called “detailed controls” report), notes this 
> within Section 3.5.1 of the Controls [1] 
> 
> “As part of the reporting, it must be specified which of the professional 
> examinations/certifications are held by the audit team (e. g. in the 
> section “Independence and quality assurance of the auditor”). Upon request, 
> appropriate documents (e. g. certificates etc.) must be submitted to the 
> client.” 
> 
> Could you clarify whether you and the WTTF considered these two cases? The 
> former is an example of using an assurance scheme the FPKIMA has said on 
> its own is insufficient, namely WTCA, but with additional reporting can be 
> made sufficient. The latter is an example of a scheme specifically adapted 
> for cloud/vendor security controls against an ISAE 3000 reporting scheme, 
> which is nearly identical to WTBRs in that regard. It was unclear if y’all 
> were simply not familiar with these cases, or if you believe there is 
> substantive differences in the proposal here that may require addressing. 
> 
> [1] 
> https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/CloudComputing/ComplianceControlsCatalogue-Cloud_Computing-C5.pdf?__blob=publicationFile=3
>  
> 
> >
Correct Ryan.  This is one of the examples you provided previously.  This 
report is of course restricted:
"This report is intended solely for the information and use of {CA} and the 
Federal PKI Policy Authority and is not intended to be, and should not be, used 
by anyone other than {CA} and Federal PKI Policy Authority."

As you know, this report then is not generally / publicly distributed as it is 
a restricted use report.  This restriction does not appear in the public 
company audit opinions, hence the reference.  I called out the federal space 
with this very example in mind.  So yes, I am aware of and quite familiar with 
this scenario.  It is more about the restricted use (or in this case the lack 
thereof) as it is the framework being used.  The very point you are referencing 
an opinion that you are using outside of the restriction sums up my argument.  
I can't speak for all audit firms as this is more of a risk management issue, 
but my firm would be fine issuing this report in a restricted manner, unless it 
became known it would not actually be used in the restricted manner.  That 
defeats the whole purpose.  I've issued auditor qualifications in this manner 
in a management letter, which is also restricted.  To my knowledge, that letter 
has never been made available to those outside of the restricted use, as it 
appears to have been in this case.  So unfortunately your statement is 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Mon, Feb 15, 2021 at 2:03 PM Jeff Ward via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> I wanted to clarify a couple of points.  Firms must be independent to do
> audit/assurance work.  If independence is impaired, for example, by one
> person in the firm performing management functions, the entire firm is no
> longer independent.  Firms have the responsibility to monitor activities of
> its professionals, which also includes personal investments, to ensure they
> remain independent.
>
> Also, WebTrust practitioners provide information on the firm and the
> professionals used on these engagements.  The information provided is
> closely aligned with the Auditor Qualifications you are describing.  As you
> know, CPA Canada provides a listing of qualified audit firms on its
> website.  Working closely with them could also help in instances where
> auditor qualifications are in question.
>
> And one last item, thank you for hearing us on the listing of auditors
> performing the engagement.  The only place I am aware that lists the audit
> partner in a comparable world is the signing audit partner on public
> company audits in the US, which is available on the SEC website.  Other
> than that, I am not aware of any other team member being listed.  We have
> seen listings of team members and related experience summarized on a
> non-publicly issued letter to management in the US Federal space.


Jeff,

https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf

Is an example, which is an audit of the U.S. Government Printing Office,
provided by a WTTF member, against the US Federal PKI CP. This doesn’t meet
the criteria you mentioned (public company, SEC), and itself was provided
several years ago.

It is directed to a set of named parties, and made publicly available by
those parties, using the WebTrust for CAs criteria. On page 4 (report)/6
(FPKI submission)/9 (PDF page), you can see an enumerated list of audit
participants and their applicable skills, summarized.

Since you mentioned “a comparable world”, the BSI C5 controls, which
provide a valuable model for improvements in transparency and thoroughness
of reporting (aka the so called “detailed controls” report), notes this
within Section 3.5.1 of the Controls [1]

“As part of the reporting, it must be specified which of the professional
examinations/certifications are held by the audit team (e. g. in the
section “Independence and quality assurance of the auditor”). Upon request,
appropriate documents (e. g. certificates etc.) must be submitted to the
client.”

Could you clarify whether you and the WTTF considered these two cases? The
former is an example of using an assurance scheme the FPKIMA has said on
its own is insufficient, namely WTCA, but with additional reporting can be
made sufficient. The latter is an example of a scheme specifically adapted
for cloud/vendor security controls against an ISAE 3000 reporting scheme,
which is nearly identical to WTBRs in that regard. It was unclear if y’all
were simply not familiar with these cases, or if you believe there is
substantive differences in the proposal here that may require addressing.

[1]
https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/CloudComputing/ComplianceControlsCatalogue-Cloud_Computing-C5.pdf?__blob=publicationFile=3

>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Thursday, February 11, 2021 at 12:41:44 PM UTC-6, Ben Wilson wrote:
> All, 
> 
> I've modified the proposed change to MRSP section 3.2 so that it would now 
> insert a middle paragraph that would read: 
> 
> "A Qualified Auditor MUST have relevant IT Security experience, or have 
> audited a number of CAs, and be independent and not conflicted. Individuals 
> have competence, partnerships and corporations do not. Each Audit Report 
> MUST be accompanied by documentation provided to Mozilla of individual 
> auditor qualifications sufficient for Mozilla to determine the competence, 
> experience, and independence of the Qualified Auditor." 
> 
> See 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/57063dc07f5b753184c94dbf5d0d30d0b9b90789
>  
> 
> The basis for further interpretation of the above language would still be 
> section 8.2 of the Baseline Requirements. ("In normal circumstances, 
> Mozilla requires that audits MUST be performed by a Qualified Auditor, as 
> defined in the Baseline Requirements section 8.2"). 
> 
> Section 3.1.4 still remains with a proposed subsection 3 - "name(s) and 
> qualifications of individuals performing the audit, as required by section 
> 3.2." 
> 
> I anticipate that additional guidance for how CAs should submit this 
> information will be made available here on the wiki - 
> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications. 
> 
> 
>  
> Ben
> On Thu, Jan 28, 2021 at 2:10 PM Ryan Sleevi  wrote: 
> 
> > 
> > On Thu, Jan 28, 2021 at 3:05 PM Ben Wilson  wrote: 
> > 
> >> Thanks. My current thinking is that we can leave the MRSP "as is" and 
> >> that we write up what we want in 
> >> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications, 
> >> which is, as you note, information about members of the audit team and how 
> >> individual members meet #2, #3, and #6. 
> >> 
> > 
> > Is this intended as a temporary fix until the issue is meaningfully 
> > addressed? Or are you seeing this as a long-term resolution of the issue? 
> > 
> > I thought the goal was to make the policy clearer on the expectations, and 
> > my worry is that it would be creating more work for you and Kathleen, and 
> > the broader community, because it puts the onus on you to chase down CAs to 
> > provide the demonstration because they didn't pay attention to it in the 
> > policy. This was the complaint previously raised about "CA Problematic 
> > Practices" and things that are forbidden, so I'm not sure I understand the 
> > distinction/benefit here from moving it out? 
> > 
> > I think the relevance to MRSP is trying to clarify whether Mozilla thinks 
> > of auditors as individuals (as it originally did), or whether it thinks of 
> > auditors as organizations. I think that if MRSP was clarified regarding 
> > that, then the path you're proposing may work (at the risk of creating more 
> > work for y'all to request that CAs provide the information that they're 
> > required to provide, but didn't know that). 
> > 
> > If the issue you're trying to solve is one about whether it's in the audit 
> > letter vs communicated to Mozilla, then I think it should be possible to 
> > achieve that within the MRSP and explicitly say that (i.e. not require it 
> > in the audit letter, but still requiring it). 
> > 
> > Just trying to make sure I'm not overlooking or misunderstanding your 
> > concerns there :) 
> > 
> >>
I wanted to clarify a couple of points.  Firms must be independent to do 
audit/assurance work.  If independence is impaired, for example, by one person 
in the firm performing management functions, the entire firm is no longer 
independent.  Firms have the responsibility to monitor activities of its 
professionals, which also includes personal investments, to ensure they remain 
independent.

Also, WebTrust practitioners provide information on the firm and the 
professionals used on these engagements.  The information provided is closely 
aligned with the Auditor Qualifications you are describing.  As you know, CPA 
Canada provides a listing of qualified audit firms on its website.  Working 
closely with them could also help in instances where auditor qualifications are 
in question.

And one last item, thank you for hearing us on the listing of auditors 
performing the engagement.  The only place I am aware that lists the audit 
partner in a comparable world is the signing audit partner on public company 
audits in the US, which is available on the SEC website.  Other than that, I am 
not aware of any other team member being listed.  We have seen listings of team 
members and related experience summarized on a non-publicly issued letter to 
management in the US Federal space.

Hope this helps!

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org

Re: Policy 2.7.1: MRSP Issue #207: Require audit statements to provide information about which CA Locations were audited

[Ben Wilson via dev-security-policy]

The current proposed draft of changes is at
https://github.com/BenWilson-Mozilla/pkipolicy/commit/443b4c5d5155942a216322480f3a6a273ea2

Right now, I'm considering having subsection of MRSP section 3.1.4 say,
"the CA locations that were or were not audited" - with a hyperlink to
https://wiki.mozilla.org/CA/Audit_Statements#Audited_Locations, and then
elaborating there as needed.


On Wed, Jan 13, 2021 at 10:25 AM Ben Wilson  wrote:

> Thanks, Jeff.  These are useful comments, and I will take them into
> consideration in revising our proposal.
>
> On Tue, Jan 12, 2021 at 8:38 AM Jeff Ward via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
>
>> On Sunday, January 3, 2021 at 8:38:05 AM UTC-6, Jeff Ward wrote:
>> > On Tuesday, December 15, 2020 at 2:41:10 PM UTC-6, Ben Wilson wrote:
>> > > All,
>> > >
>> > > This email is part of the discussion for the next version of the
>> Mozilla
>> > > Root Store Policy (MSRP), version 2.7.1, to be published during of
>> Q1-2021.
>> > >
>> > > For audit delays, we currently require that audit statements disclose
>> the
>> > > locations that were and were not audited, but that requirement has
>> not been
>> > > incorporated yet into the MRSP. See
>> > > https://wiki.mozilla.org/CA/Audit_Statements#Minimum_Expectations.
>> That
>> > > provision reads as follows:
>> > >
>> > > Disclose each location (at the state/province level) that was
>> included in
>> > > the scope of the audit or should have been included in the scope of
>> the
>> > > audit, whether the inspection was physically carried out in person at
>> each
>> > > location, and which audit criteria were checked (or not checked) at
>> each
>> > > location.
>> > >
>> > > - If the CA has more than one location in the same state/province,
>> then
>> > > use terminology to clarify the number of facilities in that
>> state/province
>> > > and whether or not all of them were audited. For example: "Facility 1
>> in
>> > > Province", "Facility 2 in Province, Facility 3 in Province" *or*
>> > > "Primary Facility in Province", "Secondary Facility in Province",
>> "Tertiary
>> > > Facility in Province".
>> > > - The public audit statement does not need to identify the type of
>> > > Facility.
>> > > - "Facility" includes: data center locations, registration authority
>> > > locations, where IT and business process controls of CA operations
>> are
>> > > performed, facility hosting an active HSM with CA private keys,
>> > > facility or
>> > > bank deposit box storing a deactivated and encrypted copy of a
>> > > private key.
>> > >
>> > > It is proposed by Issue #207
>> > >  that this language
>> > > requiring the disclosure of site locations--audited and unaudited--be
>> made
>> > > clearly part of the MSRP by reference to the language above.
>> > >
>> > > A similar method of incorporating by reference has been taken in
>> section
>> > > 2.4 of the MSRP
>> > > <
>> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#24-incidents>
>>
>> > > with respect to incident reporting and in section 7.1
>> > > <
>> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#71-inclusions>
>>
>> > > with requirements for the CA inclusion process.
>> > >
>> > > It is proposed that we add a new subsection 10 to MRSP section 3.1.4
>> > > <
>> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#314-public-audit-information>
>>
>> > > that would require that audit documentation disclose the facility
>> site
>> > > locations that were, or were not, examined.
>> > >
>> > > One concern that has been raised previously is that the Baseline
>> > > Requirements do not define "facility site location". However, we
>> believe
>> > > that the language above at
>> > > https://wiki.mozilla.org/CA/Audit_Statements#Minimum_Expectations
>> > > accomplishes that. We're open to suggestions for re-wording parts of
>> it to
>> > > make it even better.
>> > >
>> > > Currently, the audit letter template for WebTrust for CAs references
>> the
>> > > site location audited (at the level of specificity that is proposed
>> > > above). Over this past year, due to COVID, some ETSI attestation
>> letters
>> > > have also explained which sites were and were not checked. This
>> approach
>> > > seems to work, and the additional information will be beneficial in
>> the
>> > > future as we evaluate the security and trust of PKI service
>> providers.
>> > >
>> > > So, for the page cited above, we intend to move "Minimum
>> Expectations" out
>> > > from under "Audit Delay" so that it stands separately as a
>> requirement for
>> > > disclosing the facility site location. Then we will also revise MRSP
>> > > section 3.1.4 by inserting a new subsection 10 to require "facility
>> site
>> > > locations that were, or were not, examined" with a hyperlink to the
>> Minimum
>> > > Expectations language cited above.
>> > >
>> > 

Re: Policy 2.7.1: MRSP Issue #187: Require disclosure of incidents in Audit Reports

[Jeff Ward via dev-security-policy]

On Friday, February 12, 2021 at 10:27:11 AM UTC-6, Ben Wilson wrote:
> I'm fine with that suggestion.
> On Fri, Feb 12, 2021 at 5:06 AM malcol...--- via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > On Thursday, 11 February 2021 at 21:14:13 UTC, Ben Wilson wrote:
> > > 11. all incidents (as defined in section 2.4), including those reported 
> > in 
> > > Bugzilla, that were: 
> > > * disclosed by the CA or discovered by the auditor, and 
> > > * unresolved at any time during the audit period; 
> > >
> > > The idea is that all "incidents" must be reported if they were 
> > "unresolved" 
> > > - which would include those that occurred or were open - at any time 
> > during 
> > > the audit period. 
> > > 
> >
> > Wouldn't it be clearer to non-native English speakers to avoid the nuance 
> > associated with "unresolved at any time" needing to imply both those that 
> > occurred or those that were still open? 
> > 
> > Why not amend the language to just say: 
> >
> > 11. all incidents (as defined in section 2.4), including those reported in
> > Bugzilla, that: 
> > * were disclosed by the CA or discovered by the auditor, and 
> > * occurred or were open at any time during the audit period;
> > ___ 
> > dev-security-policy mailing list 
> > dev-secur...@lists.mozilla.org 
> > https://lists.mozilla.org/listinfo/dev-security-policy 
> >
This wording works from a WebTrust perspective as well.  Thanks!
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy