Re: Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

All,
As discussed previously, here is a draft amendment to the Audit Statements
wiki page for your review and comment:
https://wiki.mozilla.org/CA/Audit_Statements#Providing_Auditor_Qualifications
Sincerely yours,
Ben
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

AW: Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Wanko Clemens via dev-security-policy]

Dear Ben, Dear Kahtleen,

 

we suppose, there are no other changes intendent then apart from what you say 
below? 

If the rest of section 3.2 remains as it is, the specific matter of the ETSI 
auditor qualification would be addressed through the referrer back to BRG 
section 8.2. It would then read like this, which is okay for us: 

 

"3.2 Auditors

In normal circumstances, Mozilla requires that audits MUST be performed by a 
Qualified Auditor, as defined in the Baseline Requirements section 8.2.

A Qualified Auditor MUST have relevant IT Security experience, or have audited 
a number of CAs, and be independent. Each Audit Report MUST be accompanied by 
documentation provided to Mozilla of the audit team qualifications sufficient 
for Mozilla to determine the competence, experience, and independence of the 
auditor."

 

Otherwise a link to the Wiki would be necessary for clear definition of the 
details for the auditor qualification. 

 

Apart from that: is it also intended to change section 3.1.4?

 

All the best 

Clemens

 

-Ursprüngliche Nachricht-
Von: mozilla.dev.security.pol...@googlegroups.com 
 Im Auftrag von Ben Wilson
Gesendet: Dienstag, 9. März 2021 00:31
An: mozilla-dev-security-policy 
Betreff: EXT: Re: Policy 2.7.1: MRSP Issue #192: Require information about 
auditor qualifications in the audit report

 

All,

Kathleen and I discussed the language of this proposal and have modified it for 
MRSP section 3.2 as follows:  "A Qualified Auditor MUST have relevant IT 
Security experience, or have audited a number of CAs, and be independent.

Each Audit Report MUST be accompanied by documentation provided to Mozilla of 
the audit team qualifications sufficient for Mozilla to determine the 
competence, experience, and independence of the auditor."

Ben

 

 

On Thu, Feb 18, 2021 at 11:27 AM Ben Wilson < <mailto:bwil...@mozilla.com> 
bwil...@mozilla.com> wrote:

 

> All,

> 

> I have edited the proposed resolution of Issue #192 

> <https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f674807

> 5fa760555c1eabb81bd7914ee>

> as follows:

> 

> Subsection 3 of MRSP Section 3.1.4. would read:

> 

> "The publicly-available documentation relating to each audit MUST 

> contain at least the following clearly-labelled information: ...

> 

> 3. name of the lead auditor and qualifications of the team performing 

> the audit, as required by section 3.2; ..."

> 

> Section 3.2 would read:

> 

> "A Qualified Auditor MUST have relevant IT Security experience, or 

> have audited a number of CAs, and be independent and not conflicted. 

> People have competence, partnerships and corporations do not. Each 

> Audit Report MUST be accompanied by documentation provided to Mozilla 

> of the audit team qualifications 

> < <https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications> 
> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications>

> sufficient for Mozilla to determine the competence, experience, and 

> independence of the Qualified Auditor."

> 

> The wiki page linked above will provide further details on how to 

> submit documentation of the audit team's qualifications (which may be 

> separate from the audit letter itself).

> 

> Ben

> 

> 

> <https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f674807

> 5fa760555c1eabb81bd7914ee>

> 

> 

> 

> On Mon, Feb 15, 2021 at 6:01 PM Watson Ladd via dev-security-policy < 

>  <mailto:dev-security-policy@lists.mozilla.org> 
> dev-security-policy@lists.mozilla.org> wrote:

> 

>> On Monday, February 15, 2021 at 3:07:12 PM UTC-8, Jeff Ward wrote:

>> > On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote:

>> > > Apologies for belaboring the point, but I think we might be 

>> > > talking

>> past

>> > > eachother.

>> > >

>> > > You originally stated “The only place I am aware that lists the 

>> > > audit partner in a comparable world is the signing audit partner 

>> > > on public company audits in the US, which is available on the SEC 

>> > > website.” I

>> gave

>> > > two separate examples of such, and you responded to one (FPKI) by

>> saying

>> > > the report was not public (even when it is made available 

>> > > publicly),

>> and

>> > > the other I didn’t see a response to.

>> > >

>> > > This might feel like nit-picking, but I think this is a rather

>> serious

>> > > point to work through, because I don’t think you’re fully

>> communicating

>> > > what you judge to be a “comparable world”, as it appea

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

All,
Kathleen and I discussed the language of this proposal and have modified it
for MRSP section 3.2 as follows:  "A Qualified Auditor MUST have relevant
IT Security experience, or have audited a number of CAs, and be independent.
Each Audit Report MUST be accompanied by documentation provided to Mozilla
of the audit team qualifications sufficient for Mozilla to determine the
competence, experience, and independence of the auditor."
Ben


On Thu, Feb 18, 2021 at 11:27 AM Ben Wilson  wrote:

> All,
>
> I have edited the proposed resolution of Issue #192
> 
> as follows:
>
> Subsection 3 of MRSP Section 3.1.4. would read:
>
> "The publicly-available documentation relating to each audit MUST contain
> at
> least the following clearly-labelled information: ...
>
> 3. name of the lead auditor and qualifications of the team performing the
> audit, as required by section 3.2;
> ..."
>
> Section 3.2 would read:
>
> "A Qualified Auditor MUST have relevant IT Security experience, or have
> audited a number of CAs, and be independent and not conflicted. People
> have competence, partnerships and corporations do not. Each Audit Report
> MUST be accompanied by documentation provided to Mozilla of the audit team
> qualifications
> 
> sufficient for Mozilla to determine the competence, experience, and
> independence of the Qualified Auditor."
>
> The wiki page linked above will provide further details on how to submit
> documentation of the audit team's qualifications (which may be separate
> from the audit letter itself).
>
> Ben
>
>
> 
>
>
>
> On Mon, Feb 15, 2021 at 6:01 PM Watson Ladd via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
>
>> On Monday, February 15, 2021 at 3:07:12 PM UTC-8, Jeff Ward wrote:
>> > On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote:
>> > > Apologies for belaboring the point, but I think we might be talking
>> past
>> > > eachother.
>> > >
>> > > You originally stated “The only place I am aware that lists the audit
>> > > partner in a comparable world is the signing audit partner on public
>> > > company audits in the US, which is available on the SEC website.” I
>> gave
>> > > two separate examples of such, and you responded to one (FPKI) by
>> saying
>> > > the report was not public (even when it is made available publicly),
>> and
>> > > the other I didn’t see a response to.
>> > >
>> > > This might feel like nit-picking, but I think this is a rather
>> serious
>> > > point to work through, because I don’t think you’re fully
>> communicating
>> > > what you judge to be a “comparable world”, as it appears you are
>> dismissing
>> > > these examples.
>> > >
>> > > I can think of several possible dimensions you might be thinking are
>> > > relevant, but rather than assume, I’m hoping you can expand with a
>> few
>> > > simple questions. Some admittedly seem basic, but I don’t want to
>> take
>> > > anything for granted here.
>> > >
>> > > 1) Are you/the WTTF familiar with these audit schemes?
>> > >
>> > > 2) Are you aware of schemes that require disclosing the relevant
>> skills and
>> > > experience of the audit team to the client? (E.g. as done by BSI C5
>> audits
>> > > under ISAE 3000)
>> > >
>> > > 3) Are you aware of such reports naming multiple parties for the use
>> of the
>> > > report (e.g. as done by FPKI audits)
>> > >
>> > > 4) Are you aware of schemes in which a supplier requires a vendor to
>> be
>> > > audited, and ensures that the customer of supplier are able to access
>> such
>> > > audits as part of their reliance upon supplier? (Note, this doesn’t
>> have to
>> > > be limited to ISMS systems)
>> > >
>> > > I’m trying to understand what, given the prevalence of these
>> practices,
>> > > makes these instances *not* a comparable world, since it seems that
>> helps
>> > > move closer to solutions.
>> > Ryan, I hope you are not suggesting I am dodging you points. That would
>> be absurd. Let me use different words as comparable world seems to be
>> tripping you up. You are not providing a general/public distribution
>> example to make your point so it is baseless. You are using a restricted
>> opinion from EY and neither Ryan Sleevi nor Google are listed as the two
>> intended users. The closest I have seen to support your desire to name
>> individual auditors is in public company audit reports, which are housed on
>> the SEC website. To be clear, of your two examples, one is an opinion,
>> which is restricted, and the other represents the guidelines. Perhaps you
>> have seen a public/general distribution report from your second opinion as
>> I do not see it in this thread. I am aware, as mentioned previously, of the
>> Federal PKI program desiring to know more about team 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

All,

I have edited the proposed resolution of Issue #192

as follows:

Subsection 3 of MRSP Section 3.1.4. would read:

"The publicly-available documentation relating to each audit MUST contain
at
least the following clearly-labelled information: ...

3. name of the lead auditor and qualifications of the team performing the
audit, as required by section 3.2;
..."

Section 3.2 would read:

"A Qualified Auditor MUST have relevant IT Security experience, or have
audited a number of CAs, and be independent and not conflicted. People have
competence, partnerships and corporations do not. Each Audit Report MUST be
accompanied by documentation provided to Mozilla of the audit team
qualifications

sufficient for Mozilla to determine the competence, experience, and
independence of the Qualified Auditor."

The wiki page linked above will provide further details on how to submit
documentation of the audit team's qualifications (which may be separate
from the audit letter itself).

Ben





On Mon, Feb 15, 2021 at 6:01 PM Watson Ladd via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Monday, February 15, 2021 at 3:07:12 PM UTC-8, Jeff Ward wrote:
> > On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote:
> > > Apologies for belaboring the point, but I think we might be talking
> past
> > > eachother.
> > >
> > > You originally stated “The only place I am aware that lists the audit
> > > partner in a comparable world is the signing audit partner on public
> > > company audits in the US, which is available on the SEC website.” I
> gave
> > > two separate examples of such, and you responded to one (FPKI) by
> saying
> > > the report was not public (even when it is made available publicly),
> and
> > > the other I didn’t see a response to.
> > >
> > > This might feel like nit-picking, but I think this is a rather serious
> > > point to work through, because I don’t think you’re fully
> communicating
> > > what you judge to be a “comparable world”, as it appears you are
> dismissing
> > > these examples.
> > >
> > > I can think of several possible dimensions you might be thinking are
> > > relevant, but rather than assume, I’m hoping you can expand with a few
> > > simple questions. Some admittedly seem basic, but I don’t want to take
> > > anything for granted here.
> > >
> > > 1) Are you/the WTTF familiar with these audit schemes?
> > >
> > > 2) Are you aware of schemes that require disclosing the relevant
> skills and
> > > experience of the audit team to the client? (E.g. as done by BSI C5
> audits
> > > under ISAE 3000)
> > >
> > > 3) Are you aware of such reports naming multiple parties for the use
> of the
> > > report (e.g. as done by FPKI audits)
> > >
> > > 4) Are you aware of schemes in which a supplier requires a vendor to
> be
> > > audited, and ensures that the customer of supplier are able to access
> such
> > > audits as part of their reliance upon supplier? (Note, this doesn’t
> have to
> > > be limited to ISMS systems)
> > >
> > > I’m trying to understand what, given the prevalence of these
> practices,
> > > makes these instances *not* a comparable world, since it seems that
> helps
> > > move closer to solutions.
> > Ryan, I hope you are not suggesting I am dodging you points. That would
> be absurd. Let me use different words as comparable world seems to be
> tripping you up. You are not providing a general/public distribution
> example to make your point so it is baseless. You are using a restricted
> opinion from EY and neither Ryan Sleevi nor Google are listed as the two
> intended users. The closest I have seen to support your desire to name
> individual auditors is in public company audit reports, which are housed on
> the SEC website. To be clear, of your two examples, one is an opinion,
> which is restricted, and the other represents the guidelines. Perhaps you
> have seen a public/general distribution report from your second opinion as
> I do not see it in this thread. I am aware, as mentioned previously, of the
> Federal PKI program desiring to know more about team members, but that is
> not listed in a non-restricted report, in a public/general distribution
> format.
>
> I can click on the URL and read it.  This seems to be the very definition
> of public, even if the audit report says it is not for reliance upon by the
> general public. I fully appreciate that this may be a technicality in the
> world of auditing, but it is very confusing to those of us who are less
> familiar.
>
> > Jeff
> ___
> dev-security-policy mailing list
> dev-security-policy@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Watson Ladd via dev-security-policy]

On Monday, February 15, 2021 at 3:07:12 PM UTC-8, Jeff Ward wrote:
> On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote: 
> > Apologies for belaboring the point, but I think we might be talking past 
> > eachother. 
> > 
> > You originally stated “The only place I am aware that lists the audit 
> > partner in a comparable world is the signing audit partner on public 
> > company audits in the US, which is available on the SEC website.” I gave 
> > two separate examples of such, and you responded to one (FPKI) by saying 
> > the report was not public (even when it is made available publicly), and 
> > the other I didn’t see a response to. 
> > 
> > This might feel like nit-picking, but I think this is a rather serious 
> > point to work through, because I don’t think you’re fully communicating 
> > what you judge to be a “comparable world”, as it appears you are dismissing 
> > these examples. 
> > 
> > I can think of several possible dimensions you might be thinking are 
> > relevant, but rather than assume, I’m hoping you can expand with a few 
> > simple questions. Some admittedly seem basic, but I don’t want to take 
> > anything for granted here. 
> > 
> > 1) Are you/the WTTF familiar with these audit schemes? 
> > 
> > 2) Are you aware of schemes that require disclosing the relevant skills and 
> > experience of the audit team to the client? (E.g. as done by BSI C5 audits 
> > under ISAE 3000) 
> > 
> > 3) Are you aware of such reports naming multiple parties for the use of the 
> > report (e.g. as done by FPKI audits) 
> > 
> > 4) Are you aware of schemes in which a supplier requires a vendor to be 
> > audited, and ensures that the customer of supplier are able to access such 
> > audits as part of their reliance upon supplier? (Note, this doesn’t have to 
> > be limited to ISMS systems) 
> > 
> > I’m trying to understand what, given the prevalence of these practices, 
> > makes these instances *not* a comparable world, since it seems that helps 
> > move closer to solutions.
> Ryan, I hope you are not suggesting I am dodging you points. That would be 
> absurd. Let me use different words as comparable world seems to be tripping 
> you up. You are not providing a general/public distribution example to make 
> your point so it is baseless. You are using a restricted opinion from EY and 
> neither Ryan Sleevi nor Google are listed as the two intended users. The 
> closest I have seen to support your desire to name individual auditors is in 
> public company audit reports, which are housed on the SEC website. To be 
> clear, of your two examples, one is an opinion, which is restricted, and the 
> other represents the guidelines. Perhaps you have seen a public/general 
> distribution report from your second opinion as I do not see it in this 
> thread. I am aware, as mentioned previously, of the Federal PKI program 
> desiring to know more about team members, but that is not listed in a 
> non-restricted report, in a public/general distribution format. 

I can click on the URL and read it.  This seems to be the very definition of 
public, even if the audit report says it is not for reliance upon by the 
general public. I fully appreciate that this may be a technicality in the world 
of auditing, but it is very confusing to those of us who are less familiar.

> Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Mon, Feb 15, 2021 at 6:07 PM Jeff Ward via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Ryan, I hope you are not suggesting I am dodging you points.  That would
> be absurd.  Let me use different words as comparable world seems to be
> tripping you up.


I'm not trying to suggest you're dodging intentionally, but as I said, I
think we were, and to a degree still are, talking past each other. I think
your reply captured that you don't think I understood your phrasing,
which I didn't, but I also don't think you understood why I was trying to
unpack that phrase more.


> You are not providing a general/public distribution example to make your
> point so it is baseless.  You are using a restricted opinion from EY and
> neither Ryan Sleevi nor Google are listed as the two intended users.


I'm not sure why you're bringing in Google with respect to
https://wiki.mozilla.org/CA/Policy_Participants . I'm also not sure how
this relates to the specific questions I asked, which were an attempt to
reset the conversation here to try and make progress on finding solutions.


> The closest I have seen to support your desire to name individual auditors
> is in public company audit reports, which are housed on the SEC website.
> To be clear, of your two examples, one is an opinion, which is restricted,
> and the other represents the guidelines.  Perhaps you have seen a
> public/general distribution report from your second opinion as I do not see
> it in this thread.  I am aware, as mentioned previously, of the Federal PKI
> program desiring to know more about team members, but that is not listed in
> a non-restricted report, in a public/general distribution format.


Sure, and the purpose of the questions was an attempt to reset the
conversation to a point where we're not talking past each other, by trying
to build up from basics. I know we both know there are many different audit
criteria that are used, and the WTTF member organizations are global
organizations. I don't want to assume that those who focus on the WTTF are
as familiar with, say, the BSI requirements or other audit criteria, much
the same way it wouldn't be reasonable to expect I know as much about AI as
PKI :)

The example report I provided wasn't about calling out an individual
member, but rather to highlight an area that, objectively, seems
comparable, particularly to Ben's proposed language, to better understand
what you meant by that, and whether you had concerns with the new approach
still.

You've anchored here on the public disclosure part, which I can understand
and appreciate the risk perceived by having auditor names attached to the
reports, instead of just firms. I see most of your answers focus on a
specific form of disclosure, but that wasn't what my previous mail was
trying to get to yet. I was trying to work back to understand where
"comparable" criteria diverge, so we can figure out possible paths.

To be clear, I support Ben's proposal as a very useful and meaningful
interim step. Your original message was a bit ambiguous about where you
stand, in large part because your use of "comparable" left quite a bit
ambiguous, and whether you thought the changes were sufficient. I'm quite
glad to hear that it sounds workable for those performing WebTrust audits,
because that helps make forward progress.

However, I still think it'd be useful to view this as an interim step, and
it certainly feels like you might be saying this as far as things can go.
In the cloud services space, we regularly see end users expecting audits of
their cloud provider, and those transitively apply to dependencies of that
cloud provider (e.g. datacenter security, software vendors used by the
cloud provider, etc). This is similarly true with respect to software
supply chain security: working out not just first order dependencies, but
working through second and third-order dependencies as well, to build up a
holistic risk picture. Browsers rely on and delegate to CAs, so it's no
surprise that browsers expect audits for CAs. Yet browsers have their own
customers, and need to provide those customers the assurances they need,
and their customers need, etc.

My Question #5 was an attempt to unpack "comparable" further, because these
are hardly unique problems to software vendor/service provider
relationships. It's not even unique to "cloud" or "PKI", as supply chain
assurances are pretty universal, as shown by goods such as coconut milk,
coffee, chocolate, or diamonds. You answered in the context of "public
report", but I wasn't trying to go there yet. I was trying to figure out
where "comparable" split off, and what schemes were and weren't
comparable, so that we can continue to make improvements.  Now that it
seems we have a path forward for direct disclosure, I think it's reasonable
to spend time thinking about how to bring that assurance to browser's
users/customers, such as this community, in the same way we help bring
transparency to cloud 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Monday, February 15, 2021 at 4:11:15 PM UTC-6, Ryan Sleevi wrote:
> Apologies for belaboring the point, but I think we might be talking past 
> eachother. 
> 
> You originally stated “The only place I am aware that lists the audit
> partner in a comparable world is the signing audit partner on public
> company audits in the US, which is available on the SEC website.” I gave 
> two separate examples of such, and you responded to one (FPKI) by saying 
> the report was not public (even when it is made available publicly), and 
> the other I didn’t see a response to. 
> 
> This might feel like nit-picking, but I think this is a rather serious 
> point to work through, because I don’t think you’re fully communicating 
> what you judge to be a “comparable world”, as it appears you are dismissing 
> these examples. 
> 
> I can think of several possible dimensions you might be thinking are 
> relevant, but rather than assume, I’m hoping you can expand with a few 
> simple questions. Some admittedly seem basic, but I don’t want to take 
> anything for granted here. 
> 
> 1) Are you/the WTTF familiar with these audit schemes? 
> 
> 2) Are you aware of schemes that require disclosing the relevant skills and 
> experience of the audit team to the client? (E.g. as done by BSI C5 audits 
> under ISAE 3000) 
> 
> 3) Are you aware of such reports naming multiple parties for the use of the 
> report (e.g. as done by FPKI audits) 
> 
> 4) Are you aware of schemes in which a supplier requires a vendor to be 
> audited, and ensures that the customer of supplier are able to access such 
> audits as part of their reliance upon supplier? (Note, this doesn’t have to 
> be limited to ISMS systems) 
> 
> I’m trying to understand what, given the prevalence of these practices, 
> makes these instances *not* a comparable world, since it seems that helps 
> move closer to solutions.


Ryan, I hope you are not suggesting I am dodging you points.  That would be 
absurd.  Let me use different words as comparable world seems to be tripping 
you up.  You are not providing a general/public distribution example to make 
your point so it is baseless.  You are using a restricted opinion from EY and 
neither Ryan Sleevi nor Google are listed as the two intended users.  The 
closest I have seen to support your desire to name individual auditors is in 
public company audit reports, which are housed on the SEC website.  To be 
clear, of your two examples, one is an opinion, which is restricted, and the 
other represents the guidelines.  Perhaps you have seen a public/general 
distribution report from your second opinion as I do not see it in this thread. 
 I am aware, as mentioned previously, of the Federal PKI program desiring to 
know more about team members, but that is not listed in a non-restricted 
report, in a public/general distribution format.  

EY did the FPKI audit.  I am not sure why you keep tagging the as a WTTF 
member.  They are a global firm so if you are implying only they know the 
standards/rules (which I hope you are not) would be misleading.  But to answer 
you question #1, yes.  We even spoke last about this in our TF meeting last 
week and every member had the same response, including the one you have 
referenced.  #2 answered previously.  We are not arguing who wants what.  The 
fact this information is desired is not being debated, rather how it is 
reported to the user.  #3 question is unclear.  I am not aware of any report 
that restricts an opinion to certain users specifically, in the case you 
mentioned the CA and FPKI that allows additional users to get this information. 
 SOC2 for example has a broader restriction which allows the reports to go to a 
class or classes of users.  Your example is not that case.#4 I am 
definitely aware of this requirement.  A public/general distribution report can 
be shared with anyone.  The restriction dictates who gets the opinion.  This is 
the main point you are not understanding Ryan.  For example, I if perform an 
audit of a company and restrict it to them and one other user, say their bank, 
the engagement letter / statement of work would clearly reflect this 
restriction.  In addition, the standard terms would require the company to get 
permission to issue the report beyond the specified users.  The example you 
raise in this question is certainly covered under the broad type of restriction 
that SOC2 provides, as they would be knowledgeable about the subject matter.  
The EY report example you provided does not include the broader use.  And not 
to belabor this point, but the restriction precludes its public/general 
distribution.  The words matter.  When the distribution of a report is tightly 
binding two parties, as in your example, you can't distribute it broader.  
Restricted reports by definition are different.

This is a long dialogue supporting Ben's approach.  Firms will most likely be 
willing to provide the qualifications of auditors in a non-public 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

Apologies for belaboring the point, but I think we might be talking past
eachother.

You originally stated “The only place I am aware that lists the audit
partner in a comparable world is the signing audit partner on public
company audits in the US, which is available on the SEC website.” I gave
two separate examples of such, and you responded to one (FPKI) by saying
the report was not public (even when it is made available publicly), and
the other I didn’t see a response to.

This might feel like nit-picking, but I think this is a rather serious
point to work through, because I don’t think you’re fully communicating
what you judge to be a “comparable world”, as it appears you are dismissing
these examples.

I can think of several possible dimensions you might be thinking are
relevant, but rather than assume, I’m hoping you can expand with a few
simple questions. Some admittedly seem basic, but I don’t want to take
anything for granted here.

1) Are you/the WTTF familiar with these audit schemes?

2) Are you aware of schemes that require disclosing the relevant skills and
experience of the audit team to the client? (E.g. as done by BSI C5 audits
under ISAE 3000)

3) Are you aware of such reports naming multiple parties for the use of the
report (e.g. as done by FPKI audits)

4) Are you aware of schemes in which a supplier requires a vendor to be
audited, and ensures that the customer of supplier are able to access such
audits as part of their reliance upon supplier? (Note, this doesn’t have to
be limited to ISMS systems)

I’m trying to understand what, given the prevalence of these practices,
makes these instances *not* a comparable world, since it seems that helps
move closer to solutions.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Monday, February 15, 2021 at 1:57:11 PM UTC-6, Ryan Sleevi wrote:
> On Mon, Feb 15, 2021 at 2:03 PM Jeff Ward via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > I wanted to clarify a couple of points. Firms must be independent to do 
> > audit/assurance work. If independence is impaired, for example, by one 
> > person in the firm performing management functions, the entire firm is no 
> > longer independent. Firms have the responsibility to monitor activities of 
> > its professionals, which also includes personal investments, to ensure they 
> > remain independent. 
> > 
> > Also, WebTrust practitioners provide information on the firm and the 
> > professionals used on these engagements. The information provided is 
> > closely aligned with the Auditor Qualifications you are describing. As you 
> > know, CPA Canada provides a listing of qualified audit firms on its 
> > website. Working closely with them could also help in instances where 
> > auditor qualifications are in question. 
> > 
> > And one last item, thank you for hearing us on the listing of auditors 
> > performing the engagement. The only place I am aware that lists the audit 
> > partner in a comparable world is the signing audit partner on public 
> > company audits in the US, which is available on the SEC website. Other 
> > than that, I am not aware of any other team member being listed. We have 
> > seen listings of team members and related experience summarized on a 
> > non-publicly issued letter to management in the US Federal space.
> Jeff, 
> 
> https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf 
> 
> Is an example, which is an audit of the U.S. Government Printing Office, 
> provided by a WTTF member, against the US Federal PKI CP. This doesn’t meet 
> the criteria you mentioned (public company, SEC), and itself was provided 
> several years ago. 
> 
> It is directed to a set of named parties, and made publicly available by 
> those parties, using the WebTrust for CAs criteria. On page 4 (report)/6 
> (FPKI submission)/9 (PDF page), you can see an enumerated list of audit 
> participants and their applicable skills, summarized. 
> 
> Since you mentioned “a comparable world”, the BSI C5 controls, which 
> provide a valuable model for improvements in transparency and thoroughness 
> of reporting (aka the so called “detailed controls” report), notes this 
> within Section 3.5.1 of the Controls [1] 
> 
> “As part of the reporting, it must be specified which of the professional 
> examinations/certifications are held by the audit team (e. g. in the 
> section “Independence and quality assurance of the auditor”). Upon request, 
> appropriate documents (e. g. certificates etc.) must be submitted to the 
> client.” 
> 
> Could you clarify whether you and the WTTF considered these two cases? The 
> former is an example of using an assurance scheme the FPKIMA has said on 
> its own is insufficient, namely WTCA, but with additional reporting can be 
> made sufficient. The latter is an example of a scheme specifically adapted 
> for cloud/vendor security controls against an ISAE 3000 reporting scheme, 
> which is nearly identical to WTBRs in that regard. It was unclear if y’all 
> were simply not familiar with these cases, or if you believe there is 
> substantive differences in the proposal here that may require addressing. 
> 
> [1] 
> https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/CloudComputing/ComplianceControlsCatalogue-Cloud_Computing-C5.pdf?__blob=publicationFile=3
>  
> 
> >
Correct Ryan.  This is one of the examples you provided previously.  This 
report is of course restricted:
"This report is intended solely for the information and use of {CA} and the 
Federal PKI Policy Authority and is not intended to be, and should not be, used 
by anyone other than {CA} and Federal PKI Policy Authority."

As you know, this report then is not generally / publicly distributed as it is 
a restricted use report.  This restriction does not appear in the public 
company audit opinions, hence the reference.  I called out the federal space 
with this very example in mind.  So yes, I am aware of and quite familiar with 
this scenario.  It is more about the restricted use (or in this case the lack 
thereof) as it is the framework being used.  The very point you are referencing 
an opinion that you are using outside of the restriction sums up my argument.  
I can't speak for all audit firms as this is more of a risk management issue, 
but my firm would be fine issuing this report in a restricted manner, unless it 
became known it would not actually be used in the restricted manner.  That 
defeats the whole purpose.  I've issued auditor qualifications in this manner 
in a management letter, which is also restricted.  To my knowledge, that letter 
has never been made available to those outside of the restricted use, as it 
appears to have been in this case.  So unfortunately your statement is 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Mon, Feb 15, 2021 at 2:03 PM Jeff Ward via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> I wanted to clarify a couple of points.  Firms must be independent to do
> audit/assurance work.  If independence is impaired, for example, by one
> person in the firm performing management functions, the entire firm is no
> longer independent.  Firms have the responsibility to monitor activities of
> its professionals, which also includes personal investments, to ensure they
> remain independent.
>
> Also, WebTrust practitioners provide information on the firm and the
> professionals used on these engagements.  The information provided is
> closely aligned with the Auditor Qualifications you are describing.  As you
> know, CPA Canada provides a listing of qualified audit firms on its
> website.  Working closely with them could also help in instances where
> auditor qualifications are in question.
>
> And one last item, thank you for hearing us on the listing of auditors
> performing the engagement.  The only place I am aware that lists the audit
> partner in a comparable world is the signing audit partner on public
> company audits in the US, which is available on the SEC website.  Other
> than that, I am not aware of any other team member being listed.  We have
> seen listings of team members and related experience summarized on a
> non-publicly issued letter to management in the US Federal space.


Jeff,

https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf

Is an example, which is an audit of the U.S. Government Printing Office,
provided by a WTTF member, against the US Federal PKI CP. This doesn’t meet
the criteria you mentioned (public company, SEC), and itself was provided
several years ago.

It is directed to a set of named parties, and made publicly available by
those parties, using the WebTrust for CAs criteria. On page 4 (report)/6
(FPKI submission)/9 (PDF page), you can see an enumerated list of audit
participants and their applicable skills, summarized.

Since you mentioned “a comparable world”, the BSI C5 controls, which
provide a valuable model for improvements in transparency and thoroughness
of reporting (aka the so called “detailed controls” report), notes this
within Section 3.5.1 of the Controls [1]

“As part of the reporting, it must be specified which of the professional
examinations/certifications are held by the audit team (e. g. in the
section “Independence and quality assurance of the auditor”). Upon request,
appropriate documents (e. g. certificates etc.) must be submitted to the
client.”

Could you clarify whether you and the WTTF considered these two cases? The
former is an example of using an assurance scheme the FPKIMA has said on
its own is insufficient, namely WTCA, but with additional reporting can be
made sufficient. The latter is an example of a scheme specifically adapted
for cloud/vendor security controls against an ISAE 3000 reporting scheme,
which is nearly identical to WTBRs in that regard. It was unclear if y’all
were simply not familiar with these cases, or if you believe there is
substantive differences in the proposal here that may require addressing.

[1]
https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/CloudComputing/ComplianceControlsCatalogue-Cloud_Computing-C5.pdf?__blob=publicationFile=3

>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Thursday, February 11, 2021 at 12:41:44 PM UTC-6, Ben Wilson wrote:
> All, 
> 
> I've modified the proposed change to MRSP section 3.2 so that it would now 
> insert a middle paragraph that would read: 
> 
> "A Qualified Auditor MUST have relevant IT Security experience, or have 
> audited a number of CAs, and be independent and not conflicted. Individuals 
> have competence, partnerships and corporations do not. Each Audit Report 
> MUST be accompanied by documentation provided to Mozilla of individual 
> auditor qualifications sufficient for Mozilla to determine the competence, 
> experience, and independence of the Qualified Auditor." 
> 
> See 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/57063dc07f5b753184c94dbf5d0d30d0b9b90789
>  
> 
> The basis for further interpretation of the above language would still be 
> section 8.2 of the Baseline Requirements. ("In normal circumstances, 
> Mozilla requires that audits MUST be performed by a Qualified Auditor, as 
> defined in the Baseline Requirements section 8.2"). 
> 
> Section 3.1.4 still remains with a proposed subsection 3 - "name(s) and 
> qualifications of individuals performing the audit, as required by section 
> 3.2." 
> 
> I anticipate that additional guidance for how CAs should submit this 
> information will be made available here on the wiki - 
> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications. 
> 
> 
>  
> Ben
> On Thu, Jan 28, 2021 at 2:10 PM Ryan Sleevi  wrote: 
> 
> > 
> > On Thu, Jan 28, 2021 at 3:05 PM Ben Wilson  wrote: 
> > 
> >> Thanks. My current thinking is that we can leave the MRSP "as is" and 
> >> that we write up what we want in 
> >> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications, 
> >> which is, as you note, information about members of the audit team and how 
> >> individual members meet #2, #3, and #6. 
> >> 
> > 
> > Is this intended as a temporary fix until the issue is meaningfully 
> > addressed? Or are you seeing this as a long-term resolution of the issue? 
> > 
> > I thought the goal was to make the policy clearer on the expectations, and 
> > my worry is that it would be creating more work for you and Kathleen, and 
> > the broader community, because it puts the onus on you to chase down CAs to 
> > provide the demonstration because they didn't pay attention to it in the 
> > policy. This was the complaint previously raised about "CA Problematic 
> > Practices" and things that are forbidden, so I'm not sure I understand the 
> > distinction/benefit here from moving it out? 
> > 
> > I think the relevance to MRSP is trying to clarify whether Mozilla thinks 
> > of auditors as individuals (as it originally did), or whether it thinks of 
> > auditors as organizations. I think that if MRSP was clarified regarding 
> > that, then the path you're proposing may work (at the risk of creating more 
> > work for y'all to request that CAs provide the information that they're 
> > required to provide, but didn't know that). 
> > 
> > If the issue you're trying to solve is one about whether it's in the audit 
> > letter vs communicated to Mozilla, then I think it should be possible to 
> > achieve that within the MRSP and explicitly say that (i.e. not require it 
> > in the audit letter, but still requiring it). 
> > 
> > Just trying to make sure I'm not overlooking or misunderstanding your 
> > concerns there :) 
> > 
> >>
I wanted to clarify a couple of points.  Firms must be independent to do 
audit/assurance work.  If independence is impaired, for example, by one person 
in the firm performing management functions, the entire firm is no longer 
independent.  Firms have the responsibility to monitor activities of its 
professionals, which also includes personal investments, to ensure they remain 
independent.

Also, WebTrust practitioners provide information on the firm and the 
professionals used on these engagements.  The information provided is closely 
aligned with the Auditor Qualifications you are describing.  As you know, CPA 
Canada provides a listing of qualified audit firms on its website.  Working 
closely with them could also help in instances where auditor qualifications are 
in question.

And one last item, thank you for hearing us on the listing of auditors 
performing the engagement.  The only place I am aware that lists the audit 
partner in a comparable world is the signing audit partner on public company 
audits in the US, which is available on the SEC website.  Other than that, I am 
not aware of any other team member being listed.  We have seen listings of team 
members and related experience summarized on a non-publicly issued letter to 
management in the US Federal space.

Hope this helps!

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

All,

I've modified the proposed change to MRSP section 3.2 so that it would now
insert a middle paragraph that would read:

"A Qualified Auditor MUST have relevant IT Security experience, or have
audited a number of CAs, and be independent and not conflicted. Individuals
have competence, partnerships and corporations do not. Each Audit Report
MUST be accompanied by documentation provided to Mozilla of individual
auditor qualifications sufficient for Mozilla to determine the competence,
experience, and independence of the Qualified Auditor."

See
https://github.com/BenWilson-Mozilla/pkipolicy/commit/57063dc07f5b753184c94dbf5d0d30d0b9b90789

The basis for further interpretation of the above language would still be
section 8.2 of the Baseline Requirements. ("In normal circumstances,
Mozilla requires that audits MUST be performed by a Qualified Auditor, as
defined in the Baseline Requirements section 8.2").

Section 3.1.4 still remains with a proposed subsection 3 - "name(s) and
qualifications of individuals performing the audit, as required by section
3.2."

I anticipate that additional guidance for how CAs should submit this
information will be made available here on the wiki -
https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications.


Ben

On Thu, Jan 28, 2021 at 2:10 PM Ryan Sleevi  wrote:

>
> On Thu, Jan 28, 2021 at 3:05 PM Ben Wilson  wrote:
>
>> Thanks.  My current thinking is that we can leave the MRSP "as is" and
>> that we write up what we want in
>> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications,
>> which is, as you note, information about members of the audit team and how
>> individual members meet #2, #3, and #6.
>>
>
> Is this intended as a temporary fix until the issue is meaningfully
> addressed? Or are you seeing this as a long-term resolution of the issue?
>
> I thought the goal was to make the policy clearer on the expectations, and
> my worry is that it would be creating more work for you and Kathleen, and
> the broader community, because it puts the onus on you to chase down CAs to
> provide the demonstration because they didn't pay attention to it in the
> policy. This was the complaint previously raised about "CA Problematic
> Practices" and things that are forbidden, so I'm not sure I understand the
> distinction/benefit here from moving it out?
>
> I think the relevance to MRSP is trying to clarify whether Mozilla thinks
> of auditors as individuals (as it originally did), or whether it thinks of
> auditors as organizations. I think that if MRSP was clarified regarding
> that, then the path you're proposing may work (at the risk of creating more
> work for y'all to request that CAs provide the information that they're
> required to provide, but didn't know that).
>
> If the issue you're trying to solve is one about whether it's in the audit
> letter vs communicated to Mozilla, then I think it should be possible to
> achieve that within the MRSP and explicitly say that (i.e. not require it
> in the audit letter, but still requiring it).
>
> Just trying to make sure I'm not overlooking or misunderstanding your
> concerns there :)
>
>>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Thu, Jan 28, 2021 at 3:05 PM Ben Wilson  wrote:

> Thanks.  My current thinking is that we can leave the MRSP "as is" and
> that we write up what we want in
> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications,
> which is, as you note, information about members of the audit team and how
> individual members meet #2, #3, and #6.
>

Is this intended as a temporary fix until the issue is meaningfully
addressed? Or are you seeing this as a long-term resolution of the issue?

I thought the goal was to make the policy clearer on the expectations, and
my worry is that it would be creating more work for you and Kathleen, and
the broader community, because it puts the onus on you to chase down CAs to
provide the demonstration because they didn't pay attention to it in the
policy. This was the complaint previously raised about "CA Problematic
Practices" and things that are forbidden, so I'm not sure I understand the
distinction/benefit here from moving it out?

I think the relevance to MRSP is trying to clarify whether Mozilla thinks
of auditors as individuals (as it originally did), or whether it thinks of
auditors as organizations. I think that if MRSP was clarified regarding
that, then the path you're proposing may work (at the risk of creating more
work for y'all to request that CAs provide the information that they're
required to provide, but didn't know that).

If the issue you're trying to solve is one about whether it's in the audit
letter vs communicated to Mozilla, then I think it should be possible to
achieve that within the MRSP and explicitly say that (i.e. not require it
in the audit letter, but still requiring it).

Just trying to make sure I'm not overlooking or misunderstanding your
concerns there :)

>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

Thanks.  My current thinking is that we can leave the MRSP "as is" and that
we write up what we want in
https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications, which
is, as you note, information about members of the audit team and how
individual members meet #2, #3, and #6.




On Thu, Jan 28, 2021 at 12:44 PM Ryan Sleevi  wrote:

>
>
> On Thu, Jan 28, 2021 at 1:43 PM Ben Wilson via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
>
>> On second thought, I think that Mozilla can accomplish what we want
>> without
>> modifying the MRSP
>> <
>> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#32-auditors
>> >
>> (which says audits MUST be performed by a Qualified Auditor, as defined in
>> the Baseline Requirements section 8.2), and instead adding language to
>> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications that
>> explains what additional information we need submitted to determine that
>> an
>> auditor is "qualified" under Section 8.2 of the Baseline Requirements.
>>
>> In other words (paraphrasing from BR 8.2), we would need evidence that the
>> persons or entities:
>> 1. Are independent from the subject of the audit;
>> 2. Have the ability to conduct an audit that addresses the criteria;
>> 3. Have proficiency in examining Public Key Infrastructure technology,
>> information security tools and techniques, information technology and
>> security auditing, and the third-party attestation function;
>> 4. Are accredited in accordance with ISO 17065 applying the requirements
>> specified in ETSI EN 319 403  *OR*   5. Are licensed by WebTrust;
>> 6. Are bound by law, government regulation, or professional code of ethics
>> (to render an honest and objective opinion); and
>> 7. Maintain Professional Liability/Errors & Omissions insurance with
>> policy
>> limits of at least one million US dollars in coverage.
>>
>> We do some of this already when we check on an auditor's status to bring
>> an
>> auditor's record current in the CCADB.  The edits that we'll make will
>> just
>> make it easier for us to go through the list above.
>>
>> Thoughts?
>>
>
> I'm not sure this approach is very clear about the edits you're making,
> and whether pull requests or commits might be clearer, as Wayne did in the
> past. If there is a commit, happy to look at it and apologies if I missed
> it.
>
> I'm not sure this addresses the issue as raised, or at least, "or
> entities" seems to create the same issues that are trying to be addressed,
> by thinking in terms of "legal entities" rather than qualified persons.
>
> Your discussion about "auditor's" and "auditor's status" might be misread
> as "Audit firm", when I think the issue raised was thinking about "person
> performing the audit". The individual persons aren't necessarily licensed
> or accredited (e.g. #4/ #5), and may not be the ones that retain PL/E
> insurance (#7). Further, the individuals might be independent, but the firm
> not (#1)
>
> So I think you're really just left with wanting to have a demonstration as
> to the members of the audit team and how individual members meet (#2, #3,
> #6). Is that right? I think Kathleen's proposal from November got close to
> that, and then the remainder is clarifying the language that you've
> proposed for 2.7.1, namely "Individuals have competence, partnerships and
> corporations do not".
>
> I think the expectation goal is that "Individually, and as an audit team,
> they are independent (#1)" (e.g. you can't have a non-independent party
> running the audit with a bunch of independent parties reporting to them,
> since they're no longer independent), while that collectively the audit
> team meets #2/#3, with the burden being to demonstrate how the individuals
> on the team meet that.
>
> Is that what you were thinking? Or is my explanation a jumbled mess :)
>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Thu, Jan 28, 2021 at 1:43 PM Ben Wilson via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On second thought, I think that Mozilla can accomplish what we want without
> modifying the MRSP
> <
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#32-auditors
> >
> (which says audits MUST be performed by a Qualified Auditor, as defined in
> the Baseline Requirements section 8.2), and instead adding language to
> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications that
> explains what additional information we need submitted to determine that an
> auditor is "qualified" under Section 8.2 of the Baseline Requirements.
>
> In other words (paraphrasing from BR 8.2), we would need evidence that the
> persons or entities:
> 1. Are independent from the subject of the audit;
> 2. Have the ability to conduct an audit that addresses the criteria;
> 3. Have proficiency in examining Public Key Infrastructure technology,
> information security tools and techniques, information technology and
> security auditing, and the third-party attestation function;
> 4. Are accredited in accordance with ISO 17065 applying the requirements
> specified in ETSI EN 319 403  *OR*   5. Are licensed by WebTrust;
> 6. Are bound by law, government regulation, or professional code of ethics
> (to render an honest and objective opinion); and
> 7. Maintain Professional Liability/Errors & Omissions insurance with policy
> limits of at least one million US dollars in coverage.
>
> We do some of this already when we check on an auditor's status to bring an
> auditor's record current in the CCADB.  The edits that we'll make will just
> make it easier for us to go through the list above.
>
> Thoughts?
>

I'm not sure this approach is very clear about the edits you're making, and
whether pull requests or commits might be clearer, as Wayne did in the
past. If there is a commit, happy to look at it and apologies if I missed
it.

I'm not sure this addresses the issue as raised, or at least, "or entities"
seems to create the same issues that are trying to be addressed, by
thinking in terms of "legal entities" rather than qualified persons.

Your discussion about "auditor's" and "auditor's status" might be misread
as "Audit firm", when I think the issue raised was thinking about "person
performing the audit". The individual persons aren't necessarily licensed
or accredited (e.g. #4/ #5), and may not be the ones that retain PL/E
insurance (#7). Further, the individuals might be independent, but the firm
not (#1)

So I think you're really just left with wanting to have a demonstration as
to the members of the audit team and how individual members meet (#2, #3,
#6). Is that right? I think Kathleen's proposal from November got close to
that, and then the remainder is clarifying the language that you've
proposed for 2.7.1, namely "Individuals have competence, partnerships and
corporations do not".

I think the expectation goal is that "Individually, and as an audit team,
they are independent (#1)" (e.g. you can't have a non-independent party
running the audit with a bunch of independent parties reporting to them,
since they're no longer independent), while that collectively the audit
team meets #2/#3, with the burden being to demonstrate how the individuals
on the team meet that.

Is that what you were thinking? Or is my explanation a jumbled mess :)
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

On second thought, I think that Mozilla can accomplish what we want without
modifying the MRSP

(which says audits MUST be performed by a Qualified Auditor, as defined in
the Baseline Requirements section 8.2), and instead adding language to
https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications that
explains what additional information we need submitted to determine that an
auditor is "qualified" under Section 8.2 of the Baseline Requirements.

In other words (paraphrasing from BR 8.2), we would need evidence that the
persons or entities:
1. Are independent from the subject of the audit;
2. Have the ability to conduct an audit that addresses the criteria;
3. Have proficiency in examining Public Key Infrastructure technology,
information security tools and techniques, information technology and
security auditing, and the third-party attestation function;
4. Are accredited in accordance with ISO 17065 applying the requirements
specified in ETSI EN 319 403  *OR*   5. Are licensed by WebTrust;
6. Are bound by law, government regulation, or professional code of ethics
(to render an honest and objective opinion); and
7. Maintain Professional Liability/Errors & Omissions insurance with policy
limits of at least one million US dollars in coverage.

We do some of this already when we check on an auditor's status to bring an
auditor's record current in the CCADB.  The edits that we'll make will just
make it easier for us to go through the list above.

Thoughts?

Ben

On Tue, Jan 26, 2021 at 1:36 PM Ben Wilson  wrote:

> Thanks, Clemens. I'll take a look.
>
> Also, apparently my redlining was lost when my message was saved to the
> newsgroup.
>
> I'll see if I can re-post without the text formatting of strikeouts and
> underlines.
>
> On Tue, Jan 26, 2021 at 10:24 AM Clemens Wanko via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
>
>> Hi Ben,
>> looking at what was suggested so far for section 3.2, it seems that the
>> BR combine and summarize under "qualified" in the BR section 8.2 what you
>> and Kathleen describe with the definitions for "competent" and
>> "independent" parties.
>>
>> Based upon that, MRSP section 3.2 could be structured in the following
>> way:
>>
>> * 1st: definition of "competent party" **
>> By "competent party" we mean...
>>
>> * 2nd: definition of "independency" **
>> By "independent party" we mean...
>>
>> * 3rd: now refer to the BR summarizing 1 and 2 up in the term
>> "qualified assessor/auditor" *
>> By "qualified party" we mean a person or other entity or group of persons
>> who meet *is meeting * the combination of the requirements defined above
>> for a "competent party" and an "independent party" and as such meets
>> *meeting * the requirements of section 8.2 of the Baseline Requirements.
>>
>>
>> Further following that idea and syncing it with the wording also used by
>> the BR, the current suggestion for MRSP section 3.2 could be
>> revised/amended as follows:
>>
>> *
>> 3.2 Auditors
>> Mozilla requires that audits MUST be performed by a competent,
>> independent and herewith qualified party.
>> [...]
>> By "competent party" we mean a person or other entity *group of persons*
>> who has the proficiency and is authorized to perform audits according to
>> the stated criteria (e.g., by the organization responsible for the criteria
>> or by a relevant agency) and for whom is sufficient public information
>> available to determine and evidence that the party is competent *has
>> sufficient education, experience, and ability* to judge the CA’s
>> conformance to the stated criteria.
>> In the latter case, "Public information" referred to SHOULD *** -> SHALL
>> - Why not being more strict here?*** include information regarding the
>> party’s:
>> - evidence of being bound by law, government regulation, or professional
>> code of ethics;
>> - knowledge of CA-related technical issues such as public key
>> cryptography and related standards;
>> - experience in performing security-related audits, evaluations, and risk
>> analyses; and
>> - honesty and objectivity *ability to deliver an opinion as to the CA’s
>> compliance with applicable requirements*.
>> [...]
>> *
>>
>> Best regards
>> Clemens
>>
>> ___
>> dev-security-policy mailing list
>> dev-security-policy@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-security-policy
>>
>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

Thanks, Clemens. I'll take a look.

Also, apparently my redlining was lost when my message was saved to the
newsgroup.

I'll see if I can re-post without the text formatting of strikeouts and
underlines.

On Tue, Jan 26, 2021 at 10:24 AM Clemens Wanko via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Hi Ben,
> looking at what was suggested so far for section 3.2, it seems that the BR
> combine and summarize under "qualified" in the BR section 8.2 what you and
> Kathleen describe with the definitions for "competent" and "independent"
> parties.
>
> Based upon that, MRSP section 3.2 could be structured in the following way:
>
> * 1st: definition of "competent party" **
> By "competent party" we mean...
>
> * 2nd: definition of "independency" **
> By "independent party" we mean...
>
> * 3rd: now refer to the BR summarizing 1 and 2 up in the term
> "qualified assessor/auditor" *
> By "qualified party" we mean a person or other entity or group of persons
> who meet *is meeting * the combination of the requirements defined above
> for a "competent party" and an "independent party" and as such meets
> *meeting * the requirements of section 8.2 of the Baseline Requirements.
>
>
> Further following that idea and syncing it with the wording also used by
> the BR, the current suggestion for MRSP section 3.2 could be
> revised/amended as follows:
>
> *
> 3.2 Auditors
> Mozilla requires that audits MUST be performed by a competent, independent
> and herewith qualified party.
> [...]
> By "competent party" we mean a person or other entity *group of persons*
> who has the proficiency and is authorized to perform audits according to
> the stated criteria (e.g., by the organization responsible for the criteria
> or by a relevant agency) and for whom is sufficient public information
> available to determine and evidence that the party is competent *has
> sufficient education, experience, and ability* to judge the CA’s
> conformance to the stated criteria.
> In the latter case, "Public information" referred to SHOULD *** -> SHALL -
> Why not being more strict here?*** include information regarding the
> party’s:
> - evidence of being bound by law, government regulation, or professional
> code of ethics;
> - knowledge of CA-related technical issues such as public key cryptography
> and related standards;
> - experience in performing security-related audits, evaluations, and risk
> analyses; and
> - honesty and objectivity *ability to deliver an opinion as to the CA’s
> compliance with applicable requirements*.
> [...]
> *
>
> Best regards
> Clemens
>
> ___
> dev-security-policy mailing list
> dev-security-policy@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Clemens Wanko via dev-security-policy]

Hi Ben,
looking at what was suggested so far for section 3.2, it seems that the BR 
combine and summarize under "qualified" in the BR section 8.2 what you and 
Kathleen describe with the definitions for "competent" and "independent" 
parties.

Based upon that, MRSP section 3.2 could be structured in the following way:

* 1st: definition of "competent party" **
By "competent party" we mean...

* 2nd: definition of "independency" **
By "independent party" we mean... 

* 3rd: now refer to the BR summarizing 1 and 2 up in the term "qualified 
assessor/auditor" *
By "qualified party" we mean a person or other entity or group of persons who 
meet *is meeting * the combination of the requirements defined above for a 
"competent party" and an "independent party" and as such meets  *meeting * the 
requirements of section 8.2 of the Baseline Requirements.


Further following that idea and syncing it with the wording also used by the 
BR, the current suggestion for MRSP section 3.2 could be revised/amended as 
follows:

*
3.2 Auditors
Mozilla requires that audits MUST be performed by a competent, independent and 
herewith qualified party.
[...]
By "competent party" we mean a person or other entity *group of persons* who 
has the proficiency and is authorized to perform audits according to the stated 
criteria (e.g., by the organization responsible for the criteria or by a 
relevant agency) and for whom is sufficient public information available to 
determine and evidence that the party is competent *has sufficient education, 
experience, and ability* to judge the CA’s conformance to the stated criteria.
In the latter case, "Public information" referred to SHOULD *** -> SHALL - Why 
not being more strict here?*** include information regarding the party’s:
- evidence of being bound by law, government regulation, or professional code 
of ethics;
- knowledge of CA-related technical issues such as public key cryptography and 
related standards;
- experience in performing security-related audits, evaluations, and risk 
analyses; and
- honesty and objectivity *ability to deliver an opinion as to the CA’s 
compliance with applicable requirements*.
[...]
*

Best regards
Clemens

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

Here is my attempt to reword section 3.2 based on combining MRSP version
2.4.1 with version 2.7.
My approach was to align the concepts of "competent", "independent" and
"qualified" with their more-accepted meanings.
Version 2.4.1 and earlier versions of the Mozilla Root Store Policy mixed
some of these concepts together.

3.2 Auditors

Mozilla requires that audits MUST be performed by a competent, independent,
qualified party.

The burden is on the CA to prove *establish* that it*s auditor* has me*e*t
*s* the below requirements *below*.

However*,* the CA MAY request a preliminary determination from us regarding
the acceptability of the criteria and/or the competent, independent,
qualified party or parties by which it proposes to meet the requirements of
this policy.

By "competent party" we mean a person or other entity *group of persons* who
is authorized to perform audits according to the stated criteria (e.g., by
the organization responsible for the criteria or by a relevant agency) or
for whom there is sufficient public information available to determine that
the party is competent *has sufficient education, experience, and ability*
to judge the CA’s conformance to the stated criteria.

In the latter case, "Public information" referred to SHOULD include
information regarding the party’s:
- knowledge of CA-related technical issues such as public key cryptography
and related standards;
- experience in performing security-related audits, evaluations, or
risk analyses;
and
- honesty and objectivity *ability to deliver an opinion as to the CA’s
compliance with applicable requirements*.

By "independent party" we mean a person or other entity *group of persons* who
is not affiliated with the CA as an employee or director and for whom at
least one of the following statements is true:

the party is not financially compensated by the CA;

the nature and amount of the party's financial compensation by the CA is
publicly disclosed; or

the party is bound by law, government regulation, and/or a professional
code of ethics to render an honest and objective judgement regarding the CA.

By "qualified party" we mean a person or other entity or group of persons who
meets  *meeting *the requirements of section 8.2 of the Baseline
Requirements.

If a CA wishes to use auditors who do not fit the definition in section 8.2
of the Baseline Requirements, they MUST receive written permission from
Mozilla to do so in advance of the start of the audit engagement.

Mozilla will make its own determination as to the suitability of the
suggested party or parties, at its sole discretion.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Kathleen Wilson via dev-security-policy]

On 11/13/20 1:43 PM, Ryan Sleevi wrote:

In this regard, the principles from Mozilla's 1.0 Certificate Policy
provide a small minimum, along with some of the language from, say, the
FPKI, regarding technical competencies. The basis here is simply for the
auditor to *disclose* why they believe they meet the criteria or objectives
set. This avoids the need to address part of your questions (e.g. "How do
auditors apply to be considered qualified auditors"), because it leaves the
current policies and presumptions in place, but introduces the disclosure
requirement for why the auditor is relevant and reliable for the report.



I think it is reasonable to update section 3.2 of Mozilla's Root Store 
Policy in v2.7.1 to re-add information that appears to have been lost 
during the efforts to remove duplication with the BRs. And we could 
consider adding some incremental changes to improve transparency and 
clarify expectations regarding auditor experience.


For example, we could begin by updating section 3.2 to the following, 
which is a combination of the versions 2.7 and 2.4.1 
(https://github.com/mozilla/pkipolicy/blob/2.4.1/rootstore/policy.md) of 
Mozilla's Root Store Policy. And then see if there are incremental 
updates to this that will improve transparency while keeping the audit 
statements that we add to the CCADB as fully public-facing documents.


===

3.2 Auditors

Mozilla requires that audits MUST be performed by a competent, 
independent, qualified party.


The burden is on the CA to prove that it has met the below requirements. 
However the CA MAY request a preliminary determination from us regarding 
the acceptability of the criteria and/or the competent, independent, 
qualified party or parties by which it proposes to meet the requirements 
of this policy.


By "competent party" we mean a person or other entity who is authorized 
to perform audits according to the stated criteria (e.g., by the 
organization responsible for the criteria or by a relevant government 
agency) or for whom there is sufficient public information available to 
determine that the party is competent to judge the CA’s conformance to 
the stated criteria. In the latter case the "public information" 
referred to SHOULD include information regarding the party’s:
- knowledge of CA-related technical issues such as public key 
cryptography and related standards;
- experience in performing security-related audits, evaluations, or risk 
analyses; and

- honesty and objectivity.

By "independent party" we mean a person or other entity who is not 
affiliated with the CA as an employee or director and for whom at least 
one of the following statements is true:

- the party is not financially compensated by the CA;
- the nature and amount of the party’s financial compensation by the CA 
is publicly disclosed; or
- the party is bound by law, government regulation, and/or a 
professional code of ethics to render an honest and objective judgement 
regarding the CA.


By "qualified party" we mean a person or other entity who meets the 
requirements of section 8.2 of the Baseline Requirements. If a CA wishes 
to use auditors who do not fit the definition in section 8.2 of the 
Baseline Requirements, they MUST receive written permission from Mozilla 
to do so in advance of the start of the audit engagement. Mozilla will 
make its own determination as to the suitability of the suggested party 
or parties, at its sole discretion.


==

Thanks,
Kathleen
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Thu, Nov 12, 2020 at 7:27 PM Kathleen Wilson via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> I am very much in favor of increasing transparency about the
> qualifications of the auditors providing audit statements for CAs in our
> program. However, I think that we need to spend more time figuring out a
> few things before adding such a requirement to our policy. Therefore, I
> think we should add this to our list of things to spend some focused
> time to figure out in early 2021, and move this item to the next version
> of Mozilla’s root store policy.
>

I think that's a reasonable place, but I do worry that there's a risk that
either progress won't be made until then, and we'll be in the similar
situation of needing more time.

Given that this relates to audits and audit statements, does it make sense
to actually go forward with trying to find a narrowly banded approach, with
a forward dated requirement, such as 12 months after 2.7.1?

The benefit to this approach is that it allows the expectation to be
factored into contractual agreements with auditors, which may be done
several years in advance, as well as provides the necessary signals to
audit firms, that this is a thing of real and concrete value that Mozilla
is committed to. It also allows Mozilla to assess progress, throughout the
year, on the full set of requirements. If we think about other changes that
have required some degree of details to be worked out (e.g. the SHA-1
deprecation, RSA-1024 deprecation, certificate lifetimes), the benefit of
setting a clear expectation ahead of time helps move the conversation from
debating "if" to discussing "how", which can result in more productive
engagement.

Equally, I think there's a lot that can be borrowed from how approaches to
transparency have been done in other regards. For example, with the
introduction of CAA, there was "merely" a requirement to disclose, which
later turned into more concrete criteria and objectives. Similarly, with
respect to organizational validations, an approach being taken right now
for the EV Guidelines is to disclose, with the intent of using such
disclosures to better inform, analyze, and develop concrete requirements,
based on those collective disclosures and interpretations.

In this regard, the principles from Mozilla's 1.0 Certificate Policy
provide a small minimum, along with some of the language from, say, the
FPKI, regarding technical competencies. The basis here is simply for the
auditor to *disclose* why they believe they meet the criteria or objectives
set. This avoids the need to address part of your questions (e.g. "How do
auditors apply to be considered qualified auditors"), because it leaves the
current policies and presumptions in place, but introduces the disclosure
requirement for why the auditor is relevant and reliable for the report.

This is why I took exception to ETSI's approach, which I worry your
questions jump to as well, which is the first step to answering those
questions is understanding the existing set of qualifications and how those
relate to Mozilla's goals and principles, without seeking to establish a
full "qualification regime" out of the gate. By focusing on disclosure, it
allows us to evaluate both "is this what we expect/want", as well as better
address some of the issues (e.g. "We see auditors with skill X provide much
better reports than skill Y; we should consider making X a required skill")

As it relates to Ben's proposal, I think the language he's proposed largely
works, and we can avoid the set of questions you're worried about, by
removing the language "Mozilla will review each individual auditor's
credentials and ensure that any Qualified Auditor has the collective set of
skills required by Section 8.2 of the Baseline Requirements". Additionally,
introducing a forward-dated requirement (e.g. 12 months) helps work through
any of the delivery issues Jeff highlighted, in a way that's mutually
workable. This ensures transparency, without adding a hurdle here. Issues
related to auditors that Mozilla may lose trust in are already covered in
Section 2.3 of the policy, with
https://github.com/mozilla/pkipolicy/issues/146 existing to provide greater
clarity, but I think this can be seen as a contingency.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Kathleen Wilson via dev-security-policy]

PS: In the meantime, we will continue to verify auditor qualifications 
as described here:

https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications


On 11/12/20 4:27 PM, Kathleen Wilson wrote:

 > It is proposed in Issue #192
 >  that information about
 > individual auditor's qualifications be provided--identity, competence,
 > experience and independence. (For those interested as to this 
independence

 > requirement, Mozilla Policy v.1.0 required either disclosure of the
 > auditor's compensation or the establishment that the auditor "is 
bound by
 > law, government regulation, and/or a professional code of ethics to 
render

 > an honest and objective judgement regarding the CA.")


I am very much in favor of increasing transparency about the 
qualifications of the auditors providing audit statements for CAs in our 
program. However, I think that we need to spend more time figuring out a 
few things before adding such a requirement to our policy. Therefore, I 
think we should add this to our list of things to spend some focused 
time to figure out in early 2021, and move this item to the next version 
of Mozilla’s root store policy.


Below are some of the questions we need to be able to answer before 
adding this requirement to Mozilla's root store policy.


Please do NOT respond to these questions now. We will have future 
discussions about this when we are ready.


- What information is needed and in what format to demonstrate each 
individual auditor's qualifications?
- What are the criteria to be considered and what is sufficient to be 
considered a qualified auditor?

- How do auditors apply to be considered qualified auditors?
- How can new participants become involved in this space and become 
qualified auditors?

- What is the process to determine if an auditor is qualified?
- Does every auditor signing their name or listed in an audit statement 
need to be verified as a qualified auditor? Or just the lead auditor?
- How are the qualifications of the auditors communicated in conjunction 
with the audit statement(s)?

- Who is responsible for verifying auditor qualifications?
- Who is responsible for maintaining the list of known qualified auditors?
- How do CAs find out if their auditors are qualified?

I look forward to having these discussions in full later, but I think 
this effort is too large in scope for version 2.7.1 of Mozilla's Root 
Store Policy.


Thanks,
Kathleen



___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Kathleen Wilson via dev-security-policy]

> It is proposed in Issue #192
>  that information about
> individual auditor's qualifications be provided--identity, competence,
> experience and independence. (For those interested as to this 
independence

> requirement, Mozilla Policy v.1.0 required either disclosure of the
> auditor's compensation or the establishment that the auditor "is bound by
> law, government regulation, and/or a professional code of ethics to 
render

> an honest and objective judgement regarding the CA.")


I am very much in favor of increasing transparency about the 
qualifications of the auditors providing audit statements for CAs in our 
program. However, I think that we need to spend more time figuring out a 
few things before adding such a requirement to our policy. Therefore, I 
think we should add this to our list of things to spend some focused 
time to figure out in early 2021, and move this item to the next version 
of Mozilla’s root store policy.


Below are some of the questions we need to be able to answer before 
adding this requirement to Mozilla's root store policy.


Please do NOT respond to these questions now. We will have future 
discussions about this when we are ready.


- What information is needed and in what format to demonstrate each 
individual auditor's qualifications?
- What are the criteria to be considered and what is sufficient to be 
considered a qualified auditor?

- How do auditors apply to be considered qualified auditors?
- How can new participants become involved in this space and become 
qualified auditors?

- What is the process to determine if an auditor is qualified?
- Does every auditor signing their name or listed in an audit statement 
need to be verified as a qualified auditor? Or just the lead auditor?
- How are the qualifications of the auditors communicated in conjunction 
with the audit statement(s)?

- Who is responsible for verifying auditor qualifications?
- Who is responsible for maintaining the list of known qualified auditors?
- How do CAs find out if their auditors are qualified?

I look forward to having these discussions in full later, but I think 
this effort is too large in scope for version 2.7.1 of Mozilla's Root 
Store Policy.


Thanks,
Kathleen

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Mon, Nov 9, 2020 at 11:53 AM Clemens Wanko via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Hi Ryan, hi all,
> well, isn’t the point to make here just, that there are multiple ways to
> ensure proper auditor qualification? No matter which way you like to go
> however, you must define the details of your regime: what is the criteria
> you require the auditor to fulfill, how do you organize that these criteria
> are checked, how do you ensure the quality of this process, how do you
> publish any results with regard to the auditors qualification, etc.


Clemens,

I see you doubled down on this approach, but I already responded
previously. I think this mindset to compliance does a great disservice, and
also substantially misrepresents, the values and principles at play here.
That is, you've again anchored on the notion here tha compliance should be
a checklist - this exact approach and mindset that has led to countless
security issues and incidents. This is the fundamentally wrong approach
here that I think bears calling out, and it's worth again, emphasizing,
that no, it doesn't have to be like how you describe, and also
(unsurprisingly) overlooks the downsides.

If you examine the posts I referenced previously, you'll see this in
action, so I do hope you will. So your questions are a bit nonsense here,
because they start from a conceptually bad foundation.



> Certainly, there are always alternatives. But the alternative should be
> clearly defined and described in order to allow an evaluation of all the
> options before taking a decision. In the current case I like to understand
> the alternatives you suggest for Mozilla.
>

You've turned this thread into a broader discussion of ETSI standards, and
while many criticisms could be fairly stated, I think that detracts from
this thread, and ignores the very thing it was started to discuss. I'd like
to reorient you back to the original purpose, of bringing transparency to
the auditor skills and competencies. It would seem your position is that
there should be no independent evaluation, by affected software vendors, of
the skills and competencies of the auditor or how they perform the audit.
You would like us to rely solely on the NAB and Supervisory Body for
carrying that out, even for a totally unrelated audit scheme (the eIDAS
Regulation), which can incidentally make use of largely-unrelated standards
for audits (the EN 319 403 series). You seem to argue that's superior to
any form of transparency or accountability, and seem to reject the notion
that, were auditors skills and qualifications known and part of the report,
it can tangibly lead to improvements on the requirements.

Frankly, I think that idea is nonsense. We know, from the Supervisory
Bodies involved in the eIDAS Regulation, that there are vast differences in
quality and expectation from CABs. Browsers have shared those same concerns
to ENISA, and ACAB'c seems to recognize it as well, from its very
existence. Yet you seem to reject efforts to improve that, and suggest that
we simply "trust the process" that it'll get sorted out. We have ample
evidence, from Certificate Transparency, about how bringing transparency
reveals systemic issues and flaws. Yet, rather than embrace this for
audits, it's seemingly rejected with unsubstantiated roadblocks.

You've not responded, in substance, to my previous message, and so I'm not
sure how to interpret that, especially since this largely repeats the same
points.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Dimitris Zacharopoulos via dev-security-policy]

Thank you Ben, this is really helpful.

Dimitris.

On 2020-11-09 6:52 μ.μ., Ben Wilson via dev-security-policy wrote:

Hi Dimitris,

I intend to introduce the remaining discussion topics over the next three
weeks. I did not announce an end to the discussion period on purpose, so
that we can have as full of a discussion as possible. Also, in the next
three weeks, I intend to start summarizing the discussions and coming up
with new suggested language on those issues that have been discussed. I
expect that during December we will start to solidify the amendments to
MRSP (v.2.7.1), and that in January I'll announce a "last call" on the
amendments. Following that I will "summarize a consensus that has been
reached, and/or state the official position of Mozilla" - see
https://wiki.mozilla.org/CA/Updating_Root_Store_Policy.

Part of the discussion that will still need to take place deals with
implementation deadlines, timing, etc. Let's discuss that now for the
non-controversial items, and then in late December / early January for
those that are more contentious (assuming they remain in this batch of
changes).

Sincerely yours,
Ben Wilson
Mozilla Root Store


On Mon, Nov 9, 2020 at 2:45 AM Dimitris Zacharopoulos via
dev-security-policy  wrote:



On 7/11/2020 3:12 μ.μ., Ryan Sleevi wrote:


On Sat, Nov 7, 2020 at 4:52 AM Dimitris Zacharopoulos
mailto:ji...@it.auth.gr>> wrote:


 I will try to further explain my thoughts on this. As we all know,
 according to Mozilla Policy "CAs MUST follow and be aware of
 discussions in the mozilla.dev.security.policy
  forum,
 where Mozilla's root program is coordinated". I believe Mozilla
 Root store managers' minimum expectations from CAs are to _read
 the messages and understand the content of those messages_. Right
 now, we have [1], [2], [3], [4], [5], [6], [7], [8], [9]
 policy-related threads opened up for discussion since October 15th.

 If every post in these threads contained as much information and
 complexity as your recent reply to Clemens,


This seems like a strawman argument,  ht I don’t think it’s intentional.

You’re arguing that “if things were like this hypothetical situation,
that would be bad”. However, they aren’t like that situation, as the
evidence you provided shows. This also goes back to the “what is your
desired outcome from your previous mail”, and trying to work out what
a clear call to action to address your concerns. Your previous
message, especially in the context of your (hypothetical) concern,
reads like you’re suggesting “Mozilla shouldn’t discuss policy changes
with the community”. I think we’re all sensitive and aware of the
desire not to have too many parallels discussions, which is exactly
why Ben’s been only introducing a few points a week, to facilitate
that and make progress without overwhelming.

To the contrary, I want more people to be able to participate in these
discussions, which is precisely why I "complained" about the size of
your response to Clemens :-) Keeping our replies to reasonable levels,
with a mindset that this is an International Internet community and
people might be interested to participate (even auditors that are not
native-English speakers), I believe is a good thing.

I also see that Ben has introduced a lot of policy proposal topics for
discussion in a short period of time, but I don't know what the
expectations about their "discussion time" are. Should anyone just pick
any topic and start a discussion? That might introduce a lot of parallel
discussions and I'm not sure if this is desirable by Ben. Perhaps we
need some coordination on these topics, for example "please send
feedback for topics 1 and 2 before the end of week X. If no feedback is
received, we'll deem the proposal accepted", something like that, before
moving to other topics.


As it relates to this thread, or any other thread, it seems the first
order evaluation for any CA is “Will the policy change”, followed by
“What do I need to do to meet the policy?”, both of which are still
very early in this discussion. You’re aware of the policy discussion,
and you’re aware a decision has not been made yet: isn’t that all you
need at this point? Unlike some of the other proposals, which require
action by CAs, this is a proposal that largely requires action by
auditors, because it touches on the audit framework and scheme. It
seems like, in terms of expectations for CAs to participate,
discussing this thread with your auditor is the reasonable step, and
working with them to engage here.

Hopefully that helps. Your “but what if” is easily answered as “but
we’re not”, and the “this is a lot, what do I need to do” is simply
“talk with your auditor and make sure they’re aware of discussions
here”. That seems a very simple, digestible call to action?


It helps me understand your point of view but it seems that you don't
acknowledge the need to keep these 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Clemens Wanko via dev-security-policy]

Hi Ryan, hi all,
well, isn’t the point to make here just, that there are multiple ways to ensure 
proper auditor qualification? No matter which way you like to go however, you 
must define the details of your regime: what is the criteria you require the 
auditor to fulfill, how do you organize that these criteria are checked, how do 
you ensure the quality of this process, how do you publish any results with 
regard to the auditors qualification, etc. 

Now, what we have in place with the EU ETSI scheme (or with WebTrust) is a 
regime which provides 
-   normative options specifically adopted to address CA/B-Forum/Browser 
requirements as well as 
-   a system to ensure auditor qualification is there and kept upright over 
time.
The regime is there, proven and flexible to make proper use out of it. And 
constant effort is made from ETSI as well as from the auditors (ACAB’c) to the 
Forum in order to incorporate CA/B Forum requirements.

Certainly, there are always alternatives. But the alternative should be clearly 
defined and described in order to allow an evaluation of all the options before 
taking a decision. In the current case I like to understand the alternatives 
you suggest for Mozilla.
Best regards
Clemens


On Friday, 6 November 2020 at 20:35:40 UTC+1, Ryan Sleevi wrote:
> On Fri, Nov 6, 2020 at 12:00 PM Clemens Wanko via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > Hi Ryan, hi all, 
> > three things to comment on that: 
> > 
> > 1. How is the EU ETSI audit scheme thought and what is it intended to 
> > provide to Mozilla and the CA/Browser ecosystem? 
> 
> The European scheme of technical standards for CA/TSP developed by ETSI was 
> > made and is constantly adopted to integrate CA/Browser requirements. The 
> > main standards I am referring to are: ETSI EN 319 401, EN 319 411-1 and EN 
> > 319 411-2. With regard to auditor guidance specifically for the CA/Browser 
> > ecosystem, ETSI even developed and maintains the TS 119 403-2 “Part 2: 
> > Additional requirements for Conformity Assessment Bodies auditing Trust 
> > Service Providers that issue Publicly-Trusted Certificates”. 
> > For auditors using such technical standards Europe has established a well 
> > thought through scheme based upon organizations (accredited audit bodies) 
> > which job it is – amongst others – to ensure auditors (the natural person) 
> > qualification, independence, etc. This scheme turned out to be of high 
> > trustworthiness, reliability, robustness, etc. And it works very well over 
> > here since years. 
> >
> I'm sure something is lost in translation, but this does not address any of 
> the concerns raised. The ETSI standards here are not relevant; voluntary 
> standards, in and of themselves, are not binding. The fact that a standard 
> exists is not relevant, since what matters is how the standard is adhered 
> to and supervised. 
> 
> Your subsequent statement is simply "Well, they're auditors, people trust 
> them to do this", without any form of meaningful engagement on the why. 
> "It's their job to ensure independence" - yes, but that says nothing about 
> the performance of the job, that your measure of independence is consistent 
> with another measure, etc. Your entire appeal here simply is "We say we're 
> trustworthy, so of course we're trustworthy", which would be farcical if it 
> wasn't presented so seriously.
> > 2. Transparency 
> > The transparency lies in the European scheme with independent skilled 
> > bodies auditing each other in order to ensure conformant implementation of 
> > technical standards as well as of operational requirements for audit bodies 
> > as well as for the single auditor (natural person).
> This is, again, a restatement of "Trust us, because we declared we're 
> trustworthy". For something such as trust, there's no question of "but 
> verify". Your appeal to SDOs is not relevant here, because as you and I are 
> both aware, the SDO just makes (voluntary) standards, and doesn't enforce 
> them. And this gets to the heart of the matter: the question about how each 
> of these dimensions are interpreted is something you would prefer the CABs 
> to self-declare, and the suggestion here is "Sure, you can say that, but 
> you need to also help us understand why that's true".
> > Not only the requirements for the qualification/independence/etc. of the 
> > auditor (natural person) are clearly defined within the ISO/ETSI but also 
> > the management requirements of the body in order to ensure that they are 
> > kept upright – btw. BSI C5 controls, section 4.4.9 is meant similar to 
> > that. 
> > Every accredited body is audited at least once a year by its NAB which 
> > checks conformant audit processing (e.g. according to ISO/IEC17065 in 
> > combination with ETSI EN 319 403). 
> >
> This is the first time in this e-mail that you remotely come to 
> establishing a "why". As we both know full well, the authority of the CAB 
> (and its 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson via dev-security-policy]

Hi Dimitris,

I intend to introduce the remaining discussion topics over the next three
weeks. I did not announce an end to the discussion period on purpose, so
that we can have as full of a discussion as possible. Also, in the next
three weeks, I intend to start summarizing the discussions and coming up
with new suggested language on those issues that have been discussed. I
expect that during December we will start to solidify the amendments to
MRSP (v.2.7.1), and that in January I'll announce a "last call" on the
amendments. Following that I will "summarize a consensus that has been
reached, and/or state the official position of Mozilla" - see
https://wiki.mozilla.org/CA/Updating_Root_Store_Policy.

Part of the discussion that will still need to take place deals with
implementation deadlines, timing, etc. Let's discuss that now for the
non-controversial items, and then in late December / early January for
those that are more contentious (assuming they remain in this batch of
changes).

Sincerely yours,
Ben Wilson
Mozilla Root Store


On Mon, Nov 9, 2020 at 2:45 AM Dimitris Zacharopoulos via
dev-security-policy  wrote:

>
>
> On 7/11/2020 3:12 μ.μ., Ryan Sleevi wrote:
> >
> >
> > On Sat, Nov 7, 2020 at 4:52 AM Dimitris Zacharopoulos
> > mailto:ji...@it.auth.gr>> wrote:
> >
> >
> > I will try to further explain my thoughts on this. As we all know,
> > according to Mozilla Policy "CAs MUST follow and be aware of
> > discussions in the mozilla.dev.security.policy
> >  forum,
> > where Mozilla's root program is coordinated". I believe Mozilla
> > Root store managers' minimum expectations from CAs are to _read
> > the messages and understand the content of those messages_. Right
> > now, we have [1], [2], [3], [4], [5], [6], [7], [8], [9]
> > policy-related threads opened up for discussion since October 15th.
> >
> > If every post in these threads contained as much information and
> > complexity as your recent reply to Clemens,
> >
> >
> > This seems like a strawman argument,  ht I don’t think it’s intentional.
> >
> > You’re arguing that “if things were like this hypothetical situation,
> > that would be bad”. However, they aren’t like that situation, as the
> > evidence you provided shows. This also goes back to the “what is your
> > desired outcome from your previous mail”, and trying to work out what
> > a clear call to action to address your concerns. Your previous
> > message, especially in the context of your (hypothetical) concern,
> > reads like you’re suggesting “Mozilla shouldn’t discuss policy changes
> > with the community”. I think we’re all sensitive and aware of the
> > desire not to have too many parallels discussions, which is exactly
> > why Ben’s been only introducing a few points a week, to facilitate
> > that and make progress without overwhelming.
>
> To the contrary, I want more people to be able to participate in these
> discussions, which is precisely why I "complained" about the size of
> your response to Clemens :-) Keeping our replies to reasonable levels,
> with a mindset that this is an International Internet community and
> people might be interested to participate (even auditors that are not
> native-English speakers), I believe is a good thing.
>
> I also see that Ben has introduced a lot of policy proposal topics for
> discussion in a short period of time, but I don't know what the
> expectations about their "discussion time" are. Should anyone just pick
> any topic and start a discussion? That might introduce a lot of parallel
> discussions and I'm not sure if this is desirable by Ben. Perhaps we
> need some coordination on these topics, for example "please send
> feedback for topics 1 and 2 before the end of week X. If no feedback is
> received, we'll deem the proposal accepted", something like that, before
> moving to other topics.
>
> >
> > As it relates to this thread, or any other thread, it seems the first
> > order evaluation for any CA is “Will the policy change”, followed by
> > “What do I need to do to meet the policy?”, both of which are still
> > very early in this discussion. You’re aware of the policy discussion,
> > and you’re aware a decision has not been made yet: isn’t that all you
> > need at this point? Unlike some of the other proposals, which require
> > action by CAs, this is a proposal that largely requires action by
> > auditors, because it touches on the audit framework and scheme. It
> > seems like, in terms of expectations for CAs to participate,
> > discussing this thread with your auditor is the reasonable step, and
> > working with them to engage here.
> >
> > Hopefully that helps. Your “but what if” is easily answered as “but
> > we’re not”, and the “this is a lot, what do I need to do” is simply
> > “talk with your auditor and make sure they’re aware of discussions
> > here”. That seems a very simple, digestible call to action?
> >
>
> It 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Dimitris Zacharopoulos via dev-security-policy]

On 7/11/2020 3:12 μ.μ., Ryan Sleevi wrote:



On Sat, Nov 7, 2020 at 4:52 AM Dimitris Zacharopoulos 
mailto:ji...@it.auth.gr>> wrote:



I will try to further explain my thoughts on this. As we all know,
according to Mozilla Policy "CAs MUST follow and be aware of
discussions in the mozilla.dev.security.policy
 forum,
where Mozilla's root program is coordinated". I believe Mozilla
Root store managers' minimum expectations from CAs are to _read
the messages and understand the content of those messages_. Right
now, we have [1], [2], [3], [4], [5], [6], [7], [8], [9]
policy-related threads opened up for discussion since October 15th.

If every post in these threads contained as much information and
complexity as your recent reply to Clemens,


This seems like a strawman argument,  ht I don’t think it’s intentional.

You’re arguing that “if things were like this hypothetical situation, 
that would be bad”. However, they aren’t like that situation, as the 
evidence you provided shows. This also goes back to the “what is your 
desired outcome from your previous mail”, and trying to work out what 
a clear call to action to address your concerns. Your previous 
message, especially in the context of your (hypothetical) concern, 
reads like you’re suggesting “Mozilla shouldn’t discuss policy changes 
with the community”. I think we’re all sensitive and aware of the 
desire not to have too many parallels discussions, which is exactly 
why Ben’s been only introducing a few points a week, to facilitate 
that and make progress without overwhelming.


To the contrary, I want more people to be able to participate in these 
discussions, which is precisely why I "complained" about the size of 
your response to Clemens :-) Keeping our replies to reasonable levels, 
with a mindset that this is an International Internet community and 
people might be interested to participate (even auditors that are not 
native-English speakers), I believe is a good thing.


I also see that Ben has introduced a lot of policy proposal topics for 
discussion in a short period of time, but I don't know what the 
expectations about their "discussion time" are. Should anyone just pick 
any topic and start a discussion? That might introduce a lot of parallel 
discussions and I'm not sure if this is desirable by Ben. Perhaps we 
need some coordination on these topics, for example "please send 
feedback for topics 1 and 2 before the end of week X. If no feedback is 
received, we'll deem the proposal accepted", something like that, before 
moving to other topics.




As it relates to this thread, or any other thread, it seems the first 
order evaluation for any CA is “Will the policy change”, followed by 
“What do I need to do to meet the policy?”, both of which are still 
very early in this discussion. You’re aware of the policy discussion, 
and you’re aware a decision has not been made yet: isn’t that all you 
need at this point? Unlike some of the other proposals, which require 
action by CAs, this is a proposal that largely requires action by 
auditors, because it touches on the audit framework and scheme. It 
seems like, in terms of expectations for CAs to participate, 
discussing this thread with your auditor is the reasonable step, and 
working with them to engage here.


Hopefully that helps. Your “but what if” is easily answered as “but 
we’re not”, and the “this is a lot, what do I need to do” is simply 
“talk with your auditor and make sure they’re aware of discussions 
here”. That seems a very simple, digestible call to action?




It helps me understand your point of view but it seems that you don't 
acknowledge the need to keep these emails to a reasonable and digestible 
size, regardless if the intended recipients are auditors, CAs, Relying 
Parties. You seem to dismiss my point and the fact that some messages on 
this list have been, in fact, very long and very complicated which makes 
participation and contributions very difficult. I trust that we are both 
interested in truly meeting Mozilla's goal for an open Internet 
community (which includes contributions from International 
participants), so please help the community by trying to break down 
complicated responses into simpler ones, and let's all try to use 
shorter answers and to the point.


Indeed, this particular policy change proposal seems to mainly affect 
Auditors, but individual members of this community (either representing 
CAs or as Relying Parties) might also be interested to participate, just 
as Auditors and Relying Parties may participate in discussions around 
policy change proposals that affect CAs. FWIW, I think changing the 
rules for auditors also affects CAs because it creates an opportunity 
for CAs to have engagements with individual auditor persons, as long as 
they are accepted by Mozilla.


___

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Saturday, November 7, 2020 at 10:36:58 AM UTC-6, Ryan Sleevi wrote:
> On Sat, Nov 7, 2020 at 9:21 AM Jeff Ward via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > Sure Ryan, the answer is quite simple. When I used the word "public" in 
> > my post, I should have been more clear as to the nuance of this concept. 
> > Public reports by definition are generally distributed (available to 
> > anyone). When reports are restricted in distribution and only intended for 
> > a certain user or users as specified in the report, they are no longer 
> > public. In each of the EY examples you reference, they all state in the 
> > last paragraph before the EY signature, "This report is intended solely for 
> > the information and use of GPO-CA and the Federal PKI Policy Authority and 
> > is not intended to be, and should not be, used by anyone other than GPO-CA 
> > and the Federal PKI Policy Authority." When reports are not generally 
> > distributed and available to the general public, the specifics of 
> > individuals performing the audit will not be present. When my firm issues 
> > reports for FPKI, we also provide the listing of individuals in a letter 
> > that is not public information. 
> >
> Thanks Jeff, 
> 
> This is useful for confirming that there is a clearly viable path forward 
> here. As you know, I appreciate the nuance here regarding public reporting, 
> as well as reports that are restricted in distribution but still made 
> public (e.g. as part of a regulatory function, such as the OIG in this 
> case). I think we agree in the substance: that this is possible, and are 
> merely working out the details here with regards to reporting. 
> 
> For example, Mozilla could require that, in addition to the "traditional" 
> WebTrust reporting , Mozilla be named as part of a restricted distribution 
> report that contains these details. Alternatively, Mozilla could require 
> that, as part of participating within their root program, CAs ensure such 
> reports include as restricted distribution those Subscribers, Relying 
> Parties, and Application Vendors that would rely upon the CAs' services, 
> much like widely practiced in industry today with respect to cloud 
> providers. 
> 
> Would you agree that it's fair to say that it is not fundamentally that 
> auditors cannot report such information, but focused here on the details of 
> how that report is delivered to Mozilla?

Thanks Ryan, I do agree that this information is better placed in a separate 
communication as you suggest.  As we already worked through the restricted use 
issue with the detailed controls report, we could adopt that same limited 
distribution language in the added communication you suggest.  So now you have 
me thinking.  This separate communication could also be used to discuss other 
items, especially the locations in scope and visited during the audit.  Without 
having this separate communication, this information will likely be added as 
another exhibit to the report, especially for the larger, more complex CAs, 
making a long report even longer in volume.  (Side note: length of some reports 
is an issue when the WebTrust seal is applied).  I think if this information 
was put into the separate communication, we could solve another problem herein 
by taking this information that is candidly sensitive (specific locations) and 
move it to a more restricted report.  I'll be interested to hea
 r your thoughts on this approach as well.  

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Sat, Nov 7, 2020 at 9:21 AM Jeff Ward via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Sure Ryan, the answer is quite simple.  When I used the word "public" in
> my post, I should have been more clear as to the nuance of this concept.
> Public reports by definition are generally distributed (available to
> anyone).  When reports are restricted in distribution and only intended for
> a certain user or users as specified in the report, they are no longer
> public.  In each of the EY examples you reference, they all state in the
> last paragraph before the EY signature, "This report is intended solely for
> the information and use of GPO-CA and the Federal PKI Policy Authority and
> is not intended to be, and should not be, used by anyone other than GPO-CA
> and the Federal PKI Policy Authority."  When reports are not generally
> distributed and available to the general public, the specifics of
> individuals performing the audit will not be present.   When my firm issues
> reports for FPKI, we also provide the listing of individuals in a letter
> that is not public information.
>

Thanks Jeff,

This is useful for confirming that there is a clearly viable path forward
here. As you know, I appreciate the nuance here regarding public reporting,
as well as reports that are restricted in distribution but still made
public (e.g. as part of a regulatory function, such as the OIG in this
case). I think we agree in the substance: that this is possible, and are
merely working out the details here with regards to reporting.

For example, Mozilla could require that, in addition to the "traditional"
WebTrust reporting , Mozilla be named as part of a restricted distribution
report that contains these details. Alternatively, Mozilla could require
that, as part of participating within their root program, CAs ensure such
reports include as restricted distribution those Subscribers, Relying
Parties, and Application Vendors that would rely upon the CAs' services,
much like widely practiced in industry today with respect to cloud
providers.

Would you agree that it's fair to say that it is not fundamentally that
auditors cannot report such information, but focused here on the details of
how that report is delivered to Mozilla?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Friday, November 6, 2020 at 1:13:43 PM UTC-6, Ryan Sleevi wrote:
> On Fri, Nov 6, 2020 at 12:31 PM Jeff Ward via dev-security-policy < 
> dev-secur...@lists.mozilla.org> wrote: 
> 
> > Audit reports, whether for WebTrust, financial statements, or other forms 
> > of engagement reports providing assurance to users of the information, do 
> > not include specific audit team members’ names. Simply stated, this desire 
> > to include individual auditor’s qualifications in a public report is not 
> > consistent with any other compliance reporting methods or reporting 
> > requirements for CAs, or any other auditee for that matter.
> Hi Jeff, 
> 
> Could you help me square this statement with the practical examples? For 
> example, here's a report [1] from a WebTrust TF member, Ernst and Young, 
> demonstrating how this works in practice. You can see there hasn't been an 
> issue for years [2][3], even with the direct involvement of individuals on 
> the WebTrust TF in preparing such an audit. 
> 
> So I'm having difficulty squaring your statement that they "do not", when 
> we've got examples from long-standing members of the WebTrust TF that 
> demonstrate that, in practice, they do. Could you help highlight what's 
> inconsistent here? 
> 
> Alternatively, and as mentioned to ETSI, here's an example of an ISAE 3000 
> based audit scheme, similar to WebTrust, that also discloses the relevant 
> professional qualifications and skills to clients [4], as discussed in 
> 4.4.8 and 4.4.9. 
> 
> [1] https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf 
> [2] 
> https://www.oversight.gov/sites/default/files/oig-reports/Assessment%20Report%2019-12%20GPO%20Federal%20PKI%20Compliance.pdf
>  
> [3] https://www.oversight.gov/sites/default/files/oig-reports/17-27.pdf 
> [4] 
> https://www.bsi.bund.de/EN/Topics/CloudComputing/Compliance_Criteria_Catalogue/Compliance_Criteria_Catalogue_node.html

Sure Ryan, the answer is quite simple.  When I used the word "public" in my 
post, I should have been more clear as to the nuance of this concept.  Public 
reports by definition are generally distributed (available to anyone).  When 
reports are restricted in distribution and only intended for a certain user or 
users as specified in the report, they are no longer public.  In each of the EY 
examples you reference, they all state in the last paragraph before the EY 
signature, "This report is intended solely for the information and use of 
GPO-CA and the Federal PKI Policy Authority and is not intended to be, and 
should not be, used by anyone other than GPO-CA and the Federal PKI Policy 
Authority."  When reports are not generally distributed and available to the 
general public, the specifics of individuals performing the audit will not be 
present.   When my firm issues reports for FPKI, we also provide the listing of 
individuals in a letter that is not public information.  

Jeff
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Sat, Nov 7, 2020 at 4:52 AM Dimitris Zacharopoulos 
wrote:

>
> I will try to further explain my thoughts on this. As we all know,
> according to Mozilla Policy "CAs MUST follow and be aware of discussions in
> the mozilla.dev.security.policy
>  forum, where
> Mozilla's root program is coordinated". I believe Mozilla Root store
> managers' minimum expectations from CAs are to *read the messages and
> understand the content of those messages*. Right now, we have [1], [2],
> [3], [4], [5], [6], [7], [8], [9] policy-related threads opened up for
> discussion since October 15th.
>
> If every post in these threads contained as much information and
> complexity as your recent reply to Clemens,
>

This seems like a strawman argument,  ht I don’t think it’s intentional.

You’re arguing that “if things were like this hypothetical situation, that
would be bad”. However, they aren’t like that situation, as the evidence
you provided shows. This also goes back to the “what is your desired
outcome from your previous mail”, and trying to work out what a clear call
to action to address your concerns. Your previous message, especially in
the context of your (hypothetical) concern, reads like you’re suggesting
“Mozilla shouldn’t discuss policy changes with the community”. I think
we’re all sensitive and aware of the desire not to have too many parallels
discussions, which is exactly why Ben’s been only introducing a few points
a week, to facilitate that and make progress without overwhelming.

As it relates to this thread, or any other thread, it seems the first order
evaluation for any CA is “Will the policy change”, followed by “What do I
need to do to meet the policy?”, both of which are still very early in this
discussion. You’re aware of the policy discussion, and you’re aware a
decision has not been made yet: isn’t that all you need at this point?
Unlike some of the other proposals, which require action by CAs, this is a
proposal that largely requires action by auditors, because it touches on
the audit framework and scheme. It seems like, in terms of expectations for
CAs to participate, discussing this thread with your auditor is the
reasonable step, and working with them to engage here.

Hopefully that helps. Your “but what if” is easily answered as “but we’re
not”, and the “this is a lot, what do I need to do” is simply “talk with
your auditor and make sure they’re aware of discussions here”. That seems a
very simple, digestible call to action?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Dimitris Zacharopoulos via dev-security-policy]

I will try to further explain my thoughts on this. As we all know, 
according to Mozilla Policy "CAs MUST follow and be aware of discussions 
in the mozilla.dev.security.policy 
 forum, where 
Mozilla's root program is coordinated". I believe Mozilla Root store 
managers' minimum expectations from CAs are to _read the messages and 
understand the content of those messages_. Right now, we have [1], [2], 
[3], [4], [5], [6], [7], [8], [9] policy-related threads opened up for 
discussion since October 15th.


If every post in these threads contained as much information and 
complexity as your recent reply to Clemens, I think it eventually 
"abuses" the requirement that CAs must follow discussions in m.d.s.p. 
and leads to fatigue. Understanding the complicated English language 
used, especially for non-Native English speakers, is a very challenging 
and difficult task of its own. Therefore, I think it is unreasonable for 
Mozilla Root store managers to expect that CAs will follow and 
understand all of these discussions if these threads are bombarded with 
long and complicate emails that only very few will be able to read and 
understand.


I think sending specific questions is a good advice and I will try to do 
that next week, but please try to also consider and respect the fact 
that CAs have a finite set of resources to work on these issues, among 
other duties. An unexpected increase in the volume of information CAs 
must follow, creates a risk that something critical might be missed, 
despite the good efforts of CAs having allocated the necessary resources 
to monitor these lists and Bugzilla incidents.


I obviously can't suggest anyone to post more or less, each person has 
the right to post whatever he/she deems necessary. I just wanted you to 
know, as a peer to this Module, that some participants of this Root 
Program want to contribute and continue to do so, and it would help 
tremendously if some messages were shorter and simpler to read. Perhaps 
breaking down your long reply into more than one messages might make 
them easier to process, I don't know.


Thanks for listening :-)


Dimitris.



[1]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/4fhP4iV4ut4/m/WQknrWbhAAAJ
[2]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/ZFLsguJyFDo/m/Tmn5rcXhAAAJ
[3]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/oJiMmvAJXdI/m/ZhH6oLwpAAAJ
[4]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/3sW3_cRBrfo/m/ErldH8JWAQAJ
[5]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/Oqd2iKCFELI/m/f9Kfs0M0BAAJ
[6]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/DChXLJrMwag/m/uGpEqiEcBgAJ
[7]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/nMrORsPPcds/m/hVahATyTBwAJ
[8]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/rbSFMYKlfI4/m/3kvOhydWAQAJ
[9]: 
https://groups.google.com/g/mozilla.dev.security.policy/c/xk3BanrcljY/m/8dFyM-5pAQAJ




On 2020-11-07 1:40 π.μ., Ryan Sleevi via dev-security-policy wrote:

On Fri, Nov 6, 2020 at 6:08 PM Dimitris Zacharopoulos via
dev-security-policy  wrote:


Can other people, except Ryan, follow this thread? I certainly can't. Too
much information, too much text, too many assumptions, makes it impossible
to meaningfully participate in the discussion.


These are complex topics, for sure, but that’s unavoidable. Participation
requires a degree of understanding both about the goals to be achieved by
auditing, as well as the relevant legal and institutional frameworks for
these audits. So, admittedly, that’s not the easiest to jump into.

Could you indicate what you’re having trouble following? I don’t know that
we can do much about “too much information”, since that can be said about
literally anything unfamiliar, but perhaps if you would simply ask
questions, or highlight what you’d like to more about, it could be more
digestible?

What would you say your desired outcome from your email to be? Accepting,
for a second that this is a complex topic, and so discussion will
inherently be complex, and so a response such as “make it simpler for me”
is a bit unreasonable.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy



___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Fri, Nov 6, 2020 at 6:08 PM Dimitris Zacharopoulos via
dev-security-policy  wrote:

> Can other people, except Ryan, follow this thread? I certainly can't. Too
> much information, too much text, too many assumptions, makes it impossible
> to meaningfully participate in the discussion.


These are complex topics, for sure, but that’s unavoidable. Participation
requires a degree of understanding both about the goals to be achieved by
auditing, as well as the relevant legal and institutional frameworks for
these audits. So, admittedly, that’s not the easiest to jump into.

Could you indicate what you’re having trouble following? I don’t know that
we can do much about “too much information”, since that can be said about
literally anything unfamiliar, but perhaps if you would simply ask
questions, or highlight what you’d like to more about, it could be more
digestible?

What would you say your desired outcome from your email to be? Accepting,
for a second that this is a complex topic, and so discussion will
inherently be complex, and so a response such as “make it simpler for me”
is a bit unreasonable.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Dimitris Zacharopoulos via dev-security-policy]

Can other people, except Ryan, follow this thread? I certainly can't. Too much 
information, too much text, too many assumptions, makes it impossible to 
meaningfully participate in the discussion.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jakob Bohm via dev-security-policy]

On 2020-11-06 18:31, Jeff Ward wrote:
> ...


Audit reports, whether for WebTrust, financial statements, or other forms of 
engagement reports providing assurance to users of the information, do not 
include specific audit team members’ names.  Simply stated, this desire to 
include individual auditor’s qualifications in a public report is not 
consistent with any other compliance reporting methods or reporting 
requirements for CAs, or any other auditee for that matter.



Most paper-based auditing schemes for company financial records (the
historic work area of auditors) include, on each report, the personal
signature and corresponding printed name of the responsible auditor,
optionally with an abbreviation of their national qualification level
(such as an abbreviation of "Examplarian State Authorized Public
Accountant").  From there, it would be possible for interested parties
to check that a physical person by that name is/was indeed on the roster
of such authorized individuals, but not if/why the State of Exemplar
decided to so include that person.  Furthermore, the auditor person
and/or their company may have voluntarily published further details of
their qualifications (in brochures, on websites etc.) and may have
applicable original degree documents framed and hanging on their walls
for all concerned to readily inspect.

In terms of GDPR, the state would have published rules for how to get
added/removed from the public roster, and each auditor would have the
opportunity, at all times, to retract their self-descriptions and/or
remove some or all of their framed documents from their public office.

A modern equivalent procedure for CA audits could be:

1. Each Auditor has their name and a unique public nickname registered
in a non-public roster at either CPA Canada or the relevant European
counterpart.  This is done to fulfill the contractual obligation of
their professional oath of responsibility.  The roster organization
might optionally provide alias e-mails based on the nicks.

2. Each non-public roster operates a public online service which will
confirm or deny the presence of a name/nick pair, with appropriate
safeguards against attempts to extract the roster by systematic polling
of made up names.

Unless otherwise stated in public by Mozilla (such as the statements
made a few years ago about certain auditors from E), any auditor on
these rosters shall be presumed sufficiently qualified to sign audits
used by Mozilla.

3. Each auditor person signs his public audit letters with his name,
nick, a reference to the roster-keeping organization and any other
honorific titles he/she may legitimately choose to use.  He does this to
satisfy his contractual obligation to provide the CA with that letter.
Any official physical copies will have his physical signature above his
name and may also carry a physical stamp or seal of him or his
organization, as dictated by local legal traditions.

4. Each such public audit letter is submitted to a public repository
operated by the roster-keeping organization, using a procedure that
verifies that the letter was submitted exactly as given, by that named
auditor from their roster.  This is done to satisfy the contractual
obligation of the auditor towards the CA in accordance with a
contractual reference to terms of the roster-keeping organization.

5. The roster-keeping organization publishes the public audit letters in
both a traditional paper journal deposited at major public archives and
as an online readily accessible web site with a Merkel hash tree
providing public verification that each letter was in the public record
on or before the stated inclusion date.  As hash algorithms fail to
future cracks, the roster-keeping organization retains the ability to
regenerate the signatures using new algorithms, based on its offline
archive of originals, including a signed public statement of said
regeneration.  This publication of records that include the identity of
both the actual auditors as well as relevant principal CA Officers is
done to further satisfy the contractual obligations in #4.  As is common
in paper-based book-keeping, retractions can be filed as separate
letters of correction, and the retracted documents may be made invisible
to the public without invalidating the hash-tree.
  For public access, each public letter is given a unique permanent URL
to which the CA may publicly refer, including in the CADB and on its
website.

6. Each auditor shall submit for publication by the roster organization
a self-authored but roster verified statement of qualifications, usually
just a few paragraphs.  Each such statement similarly gets a permanent
URL, but remains visible only until superseded or retracted by either
the auditor or roster-keeper.  This publication is done as part of the
auditor's contractual obligations to the roster-keeping organization,
and the ability to retract provides the GDPR right of deletion of any
included details.  Links to the current 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Fri, Nov 6, 2020 at 12:00 PM Clemens Wanko via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Hi Ryan, hi all,
> three things to comment on that:
>
> 1.  How is the EU ETSI audit scheme thought and what is it intended to
> provide to Mozilla and the CA/Browser ecosystem?

The European scheme of technical standards for CA/TSP developed by ETSI was
> made and is constantly adopted to integrate CA/Browser requirements. The
> main standards I am referring to are: ETSI EN 319 401, EN 319 411-1 and EN
> 319 411-2. With regard to auditor guidance specifically for the CA/Browser
> ecosystem, ETSI even developed and maintains the TS 119 403-2 “Part 2:
> Additional requirements for Conformity Assessment Bodies auditing Trust
> Service Providers that issue Publicly-Trusted Certificates”.
> For auditors using such technical standards Europe has established a well
> thought through scheme based upon organizations (accredited audit bodies)
> which job it is – amongst others – to ensure auditors (the natural person)
> qualification, independence, etc. This scheme turned out to be of high
> trustworthiness, reliability, robustness, etc. And it works very well over
> here since years.
>

I'm sure something is lost in translation, but this does not address any of
the concerns raised. The ETSI standards here are not relevant; voluntary
standards, in and of themselves, are not binding. The fact that a standard
exists is not relevant, since what matters is how the standard is adhered
to and supervised.

Your subsequent statement is simply "Well, they're auditors, people trust
them to do this", without any form of meaningful engagement on the why.
"It's their job to ensure independence" - yes, but that says nothing about
the performance of the job, that your measure of independence is consistent
with another measure, etc. Your entire appeal here simply is "We say we're
trustworthy, so of course we're trustworthy", which would be farcical if it
wasn't presented so seriously.


> 2.  Transparency
> The transparency lies in the European scheme with independent skilled
> bodies auditing each other in order to ensure conformant implementation of
> technical standards as well as of operational requirements for audit bodies
> as well as for the single auditor (natural person).


This is, again, a restatement of "Trust us, because we declared we're
trustworthy". For something such as trust, there's no question of "but
verify". Your appeal to SDOs is not relevant here, because as you and I are
both aware, the SDO just makes (voluntary) standards, and doesn't enforce
them. And this gets to the heart of the matter: the question about how each
of these dimensions are interpreted is something you would prefer the CABs
to self-declare, and the suggestion here is "Sure, you can say that, but
you need to also help us understand why that's true".


> Not only the requirements for the qualification/independence/etc. of the
> auditor (natural person) are clearly defined within the ISO/ETSI but also
> the management requirements of the body in order to ensure that they are
> kept upright – btw. BSI C5 controls, section 4.4.9 is meant similar to
> that.
> Every accredited body is audited at least once a year by its NAB which
> checks conformant audit processing (e.g. according to ISO/IEC17065 in
> combination with ETSI EN 319 403).
>

This is the first time in this e-mail that you remotely come to
establishing a "why". As we both know full well, the authority of the CAB
(and its assessment along these relevant axis in 319 403) is imbued by the
NAB. The authority of the NAB is imbued by EA, established by Regulation
765/2008. The root of trust is, simply stated, "The state mechanisms of the
European Union trusts us, ergo, we are defacto trustworthy".

As a long-time participant in this group, I'm sure you can understand and
appreciate that the "Trust us, we're the government" argument rarely
improves trust. For CAs, we work on the basis of concrete evidence; after
all, this is why we have audits in the first place, even for government
CAs. The argument you're making here is that EA imbued sovereign member
states the ability to establish their NAB, and the NABs are responsible for
supervising the CABs. If we have a complaint with the CAB, the argument
goes, we should complain to the NAB, consistent with the ISO 17065
framework that EN 319 403 is based on.

Yet this misguided argument overlooks a host of salient details, no doubt
because of the convenience in doing so. Unlike, say, the ISAE 3000 approach
to audits, the approach taking by this EA/NAB/CAB chain is much more
limited with respect to a host of professional duties, and only directly
applicable within the contents of 319 403, and the relevant (supervised)
reports, such as 319 411-1. Thus, there's easy opportunity for there to be
a divergence in needs, by a CAB asserting that they are qualified within
the aegis of 319 403/319 411-1 with the necessary 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Fri, Nov 6, 2020 at 12:31 PM Jeff Ward via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Audit reports, whether for WebTrust, financial statements, or other forms
> of engagement reports providing assurance to users of the information, do
> not include specific audit team members’ names.  Simply stated, this desire
> to include individual auditor’s qualifications in a public report is not
> consistent with any other compliance reporting methods or reporting
> requirements for CAs, or any other auditee for that matter.


Hi Jeff,

Could you help me square this statement with the practical examples? For
example, here's a report [1] from a WebTrust TF member, Ernst and Young,
demonstrating how this works in practice. You can see there hasn't been an
issue for years [2][3], even with the direct involvement of individuals on
the WebTrust TF in preparing such an audit.

So I'm having difficulty squaring your statement that they "do not", when
we've got examples from long-standing members of the WebTrust TF that
demonstrate that, in practice, they do. Could you help highlight what's
inconsistent here?

Alternatively, and as mentioned to ETSI, here's an example of an ISAE 3000
based audit scheme, similar to WebTrust, that also discloses the relevant
professional qualifications and skills to clients [4], as discussed in
4.4.8 and 4.4.9.

[1] https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf
[2]
https://www.oversight.gov/sites/default/files/oig-reports/Assessment%20Report%2019-12%20GPO%20Federal%20PKI%20Compliance.pdf
[3] https://www.oversight.gov/sites/default/files/oig-reports/17-27.pdf
[4]
https://www.bsi.bund.de/EN/Topics/CloudComputing/Compliance_Criteria_Catalogue/Compliance_Criteria_Catalogue_node.html
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Jeff Ward via dev-security-policy]

On Tuesday, November 3, 2020 at 5:53:52 PM UTC-6, Ben Wilson wrote:
> Historically, Mozilla Policy required that CAs "provide attestation of 
> their conformance to the stated verification requirements and other 
> operational criteria by a competent independent party or parties with 
> access to details of the CA's internal operations." 
> https://wiki.mozilla.org/CA:CertificatePolicyV1.0 "Competency" was "for 
> whom there is sufficient public information available to determine that the 
> party is competent to judge the CA's conformance to the stated criteria. In 
> the latter case the 'public information' referred to should include 
> information regarding the party's: 
> 
> - knowledge of CA-related technical issues such as public key 
> cryptography and related standards; 
> - experience in performing security-related audits, evaluations, or risk 
> analyses; *and* 
> - honesty and objectivity." 
> 
> Today, section 3.2 of the MRSP 
> 
>  
> states, "In normal circumstances, Mozilla requires that audits MUST be 
> performed by a Qualified Auditor, as defined in the Baseline Requirements 
> section 8.2," but under section 2.3 
> ,
>  
> "Mozilla reserves the right to accept audits by auditors who do not meet 
> the qualifications given in section 8.2 of the Baseline Requirements, or 
> refuse audits from auditors who do." 
> 
> Section 8.2 of the Baseline Requirements states an auditor must have: 
> 1. Independence from the subject of the audit; 
> 2. The ability to conduct an audit that addresses the criteria specified in 
> an Eligible Audit Scheme (see Section 8.1); 
> 3. Employs individuals who have proficiency in examining Public Key 
> Infrastructure technology, information security tools and techniques, 
> information technology and security auditing, and the third-party 
> attestation function; 
> 4. (For audits conducted in accordance with any one of the ETSI standards) 
> accredited in accordance with ISO 17065 applying the requirements specified 
> in ETSI EN 319 403; 
> 5. (For audits conducted in accordance with the WebTrust standard) licensed 
> by WebTrust; 
> 6. Bound by law, government regulation, or professional code of ethics; and 
> 7. Except in the case of an Internal Government Auditing Agency, maintains 
> Professional Liability/Errors & Omissions insurance with policy limits of 
> at least one million US dollars in coverage 
> 
> It is proposed in Issue #192 
>  that information about 
> individual auditor's qualifications be provided--identity, competence, 
> experience and independence. (For those interested as to this independence 
> requirement, Mozilla Policy v.1.0 required either disclosure of the 
> auditor's compensation or the establishment that the auditor "is bound by 
> law, government regulation, and/or a professional code of ethics to render 
> an honest and objective judgement regarding the CA.") 
> 
> While subsection 3 of BR 8.2 requires "individuals who have proficiency in 
> examining Public Key Infrastructure technology, information security tools 
> and techniques, information technology and security auditing, and the 
> third-party attestation function," that fact needs evidence in order to be 
> established. The proposed resolution of this Issue #192 intends to 
> accomplish that. 
> 
> This proposal to require disclosure of individual auditor qualifications is 
> very similar to the approach adopted by the U.S. Federal PKI 
> 
>  
> (see Appendices B-1 and C). E.g., "Did each Audit Opinion Letter identify 
> the auditor and the individuals performing the audit?" In practice, the 
> information about auditor qualifications could be in the form of a separate 
> document, such as a curriculum vitae. 
> 
> Some initial, draft language to address this issue is located here: 
> https://github.com/BenWilson-Mozilla/pkipolicy/commit/d0da7cb2b6db38e66c3a72e5c1db0e78e91d8df6
>  
> 
> A new subsection 3. would be added to the list of audit requirements that 
> would require "[the] name(s) and qualifications of individuals performing 
> the audit, as required by section 3.2" and a new paragrpah would be added 
> to section 3.2 that would say, "A Qualified Auditor MUST have relevant IT 
> Security experience, or have audited a number of CAs, and be independent 
> and not conflicted. Individuals have competence, partnerships and 
> corporations do not. Audit documentation of individual auditor 
> qualifications MUST be provided to Mozilla that is sufficient for Mozilla 
> to determine the competence, experience, and independence of the Qualified 
> Auditor. Mozilla will review each individual 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Clemens Wanko via dev-security-policy]

Hi Ryan, hi all,
three things to comment on that:

1.  How is the EU ETSI audit scheme thought and what is it intended to 
provide to Mozilla and the CA/Browser ecosystem?
The European scheme of technical standards for CA/TSP developed by ETSI was 
made and is constantly adopted to integrate CA/Browser requirements. The main 
standards I am referring to are: ETSI EN 319 401, EN 319 411-1 and EN 319 
411-2. With regard to auditor guidance specifically for the CA/Browser 
ecosystem, ETSI even developed and maintains the TS 119 403-2 “Part 2: 
Additional requirements for Conformity Assessment Bodies auditing Trust Service 
Providers that issue Publicly-Trusted Certificates”. 
For auditors using such technical standards Europe has established a well 
thought through scheme based upon organizations (accredited audit bodies) which 
job it is – amongst others – to ensure auditors (the natural person) 
qualification, independence, etc. This scheme turned out to be of high 
trustworthiness, reliability, robustness, etc. And it works very well over here 
since years.

2.  Transparency
The transparency lies in the European scheme with independent skilled bodies 
auditing each other in order to ensure conformant implementation of technical 
standards as well as of operational requirements for audit bodies as well as 
for the single auditor (natural person). Not only the requirements for the 
qualification/independence/etc. of the auditor (natural person) are clearly 
defined within the ISO/ETSI but also the management requirements of the body in 
order to ensure that they are kept upright – btw. BSI C5 controls, section 
4.4.9 is meant similar to that.  
Every accredited body is audited at least once a year by its NAB which checks 
conformant audit processing (e.g. according to ISO/IEC17065 in combination with 
ETSI EN 319 403).
Now, requesting more of transparency with regard to auditor qualification for 
Mozilla insinuates this immediately translates into more confidence in the 
auditor. Please lets be careful here. If we like the community to follow that, 
shouldn’t be clearly stated:
- which documents/evidences are expected from the auditor to be handed in to 
Mozilla, 
- what criteria (details need to be published) shall Mozilla use to check the 
documents/evidences against, 
- a statement of how Mozilla comes to the conclusion: auditor is acceptable / 
not acceptable,
- how Mozilla shall organize themselves to perform this task (skilled staff 
members are required),
- how that can get organized in a way that it is compatible to CA projects (see 
Tim Hollebeeks mail),
and finally: 
what information is expected from Mozilla to be published for every single 
auditor (natural person) to show auditors qualification and make it transparent.
The tasks related to auditor qualification validation and management actually 
are performed by the participants in the European scheme already (and apart 
from that, not very different also in the WebTrust world). 

3.  Requiring all their externally-operated subordinate CAs review their 
auditors 
You said:
quote  <<< That sounds like a great idea, and sounds like a good compliment to 
an approach that several (Root) CAs have taken, of requiring all their 
externally-operated subordinate CAs review their auditors and audit scope with 
the root CAO, before finalizing the audit engagement.
An example of how the public review has been done in the past is available at 
https://groups.google.com/g/mozilla.dev.security.policy/c/sRJOPiePUvI/m/oRmRffaQmcYJ
 >>> 

If we like to suggest that, wouldn’t we then not also need to state at least 
1st  based upon what rule-set we like the CA to review auditors,
2nd what competences are required at CA level to validate auditors and how 
those shall be established and maintained,
3rd how the CA is required to do that with regard to process and timing?
In order to be clear and transparent, things like these would need to be 
defined at CA level then.

Best regards
Clemens
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

On Thu, Nov 5, 2020 at 7:00 PM Wojtek Porczyk 
wrote:

> On Thu, Nov 05, 2020 at 11:48:20AM -0500, Ryan Sleevi via
> dev-security-policy wrote:
> > competency is with individuals, not organizations.
>
> [snip]
>
> > I find the appeal to redundancy and the NAB, and further, the suggestion
> of
> > GDPR, to be a bit insulting to this community. This opposition to
> > transparency fundamentally undermines the trust in ETSI provided audits,
> or
> > this appeal to the eIDAS scheme, which has limited relevance given it's a
> > fundamentally different audit scheme, beggars belief. If you'd like to
> > raise Fear/Uncertainty/Doubt about GDPR, I believe you owe this
> community a
> > precise and detailed explanation about what you believe, relevant to the
> > auditor professional experience, would be problematic.
>
> Not the original poster, but
> 1) I understand that the very general language of OP, which you dismiss as
> FUD, is because this is "consult your own lawyer" area;
> 2) contrary to what you have written, the onus is on Mozilla to demonstrate
> the compliance with GDPR and not the other way around.
>

I think this is significantly misstating things.

Without opining on the legal statements you've offered, the reality is that
fundamentally, if we cannot achieve reasonable evidence to trust the
auditor - specifically, the individuals that make up the audit team (both
the audit members and any relevant technical experts that may have
contributed) - then there's no objective reason to accept the audit. The
concerns you raised are secondary to that, and so the objection to
something *cannot* be provided simply means that it would limit the
utility, reliability, and trustworthiness of those audits and potentially
make them unacceptable. The need for objective, transparent understanding
about the skills and competencies is as paramount as the need for an
objective, transparent understanding about the CA itself, and for which
audits exist. The attestation of the CAB is demonstrably insufficient for
purpose, in the same way as a CA saying "I solemnly swear I'm up to good"
would not somehow invite trust.

I understand that the appeal is "Well, the NAB oversees the CAB, you see,
and EA oversees the NABs, and through the power of Regulation 765/2008,
trust is imbued", but that of course fails the very basic test of having
concrete, objective data for the consumer of the audit (e.g. Mozilla, but
also this broader community). It entirely outsources the supervision,
without providing any evidence of meeting the requirement, for a core
product security requirement. Rather than accept such risk, it becomes just
as reasonable to reject such audits.

I highlight this because to suggest something *cannot* be done is to
unquestionably invite the assertion of FUD. If there are *challenges* to
doing so, we should try to find solutions. But outright dismissal, or
suggesting somehow that transparency is not necessary because *handwave*
this other scheme *handwave* doesn't inspire confidence, nor does it
achieve the necessary transparency. This is clearly not insurmountable, as
discussed previously, and in the worst case, might mean that rather than
arbitrarily accepting audits, Mozilla would contractually negotiate with
potential auditors to ensure the necessary skills and qualifications were
met (e.g. as widely practiced in other industries).
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Wojtek Porczyk via dev-security-policy]

On Thu, Nov 05, 2020 at 11:48:20AM -0500, Ryan Sleevi via dev-security-policy 
wrote:
> competency is with individuals, not organizations.

[snip]

> I find the appeal to redundancy and the NAB, and further, the suggestion of
> GDPR, to be a bit insulting to this community. This opposition to
> transparency fundamentally undermines the trust in ETSI provided audits, or
> this appeal to the eIDAS scheme, which has limited relevance given it's a
> fundamentally different audit scheme, beggars belief. If you'd like to
> raise Fear/Uncertainty/Doubt about GDPR, I believe you owe this community a
> precise and detailed explanation about what you believe, relevant to the
> auditor professional experience, would be problematic.

Not the original poster, but
1) I understand that the very general language of OP, which you dismiss as
FUD, is because this is "consult your own lawyer" area;
2) contrary to what you have written, the onus is on Mozilla to demonstrate
the compliance with GDPR and not the other way around.

If Mozilla (or you personally, in your capacity as peer, doesn't matter)
intend to keep track of competency of people (like "physical people" and not
corporations), those people (at least those, who perform audits in Europe)
have certain rights from Mozilla under GDPR. You can't have it both ways
-- either you keep trust in organisations and ignore GDPR, or you keep trust
in people, and then you have all those GDPR requirements. Those are not hard
to fulfill, but they would have to be thought through before the policy takes
effect. I have found nothing in either the proposed change, or your response,
that this problem has been thought through.

For example, art. 13 of GDPR specifies that the data subject (the auditor) is
to be provided with information that the data about her/him is processed. In
the spirit of transparency, could you post an example notice which would be
sent to the auditor in question?

What would be the legal basis? (art. 6) If (e) or (f), the auditor has a right
to object; what would happen after the objection?

Have Mozilla appointed a representative in the EU (art. 27)? (I just checked
and I have found only "Attn: Legal" address in USA). If not, why? If yes,
what's his/her name and contact details?


-- 
pozdrawiam / best regards
Wojtek Porczyk
 
 I do not fear computers,
 I fear lack of them.
-- Isaac Asimov


signature.asc
Description: PGP signature
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

That sounds like a great idea, and sounds like a good compliment to an
approach that several (Root) CAs have taken, of requiring all their
externally-operated subordinate CAs review their auditors and audit scope
with the root CAO, before finalizing the audit engagement.

An example of how the public review has been done in the past is available
at
https://groups.google.com/g/mozilla.dev.security.policy/c/sRJOPiePUvI/m/oRmRffaQmcYJ

On Thu, Nov 5, 2020 at 4:53 PM Tim Hollebeek via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Ben,
>
> We're concerned that these changes could unintentionally prevent many new
> auditors from being able to conduct audits, despite being Qualified
> Auditors.  The problem is that CAs will understandably be very hesitant to
> use a new auditor, as they cannot risk having an audit conducted, and then
> not accepted by Mozilla.  An unfortunate effect of that is that CAs will
> likely only consider auditors who have a previous history of being accepted
> as qualified.
>
> One potential solution is to allow CAs to ask Mozilla to "pre-vet" a
> potential auditor they would like to use.  Mozilla policy already has a
> process in the next paragraph to provide opinions on auditors who "do not
> fit the definition of Qualified Auditor" that could potentially be
> re-used.  Unfortunately, as the paragraph is currently worded, it can only
> be used for auditors who are *NOT* Qualified, which is really weird.
>
> It would be helpful to clarify that CAs MAY use the same procedure to work
> with Mozilla to determine whether a particular auditor is Qualified or not,
> prior to the beginning an engagement.
>
> -Tim
>
> > -Original Message-
> > From: dev-security-policy 
> On
> > Behalf Of Ben Wilson via dev-security-policy
> > Sent: Tuesday, November 3, 2020 6:53 PM
> > To: Mozilla 
> > Subject: Policy 2.7.1: MRSP Issue #192: Require information about auditor
> > qualifications in the audit report
> >
> > Historically, Mozilla Policy required that CAs "provide attestation of
> their
> > conformance to the stated verification requirements and other operational
> > criteria by a competent independent party or parties with access to
> details of
> > the CA's internal operations."
> > https://wiki.mozilla.org/CA:CertificatePolicyV1.0  "Competency" was "for
> > whom there is sufficient public information available to determine that
> the
> > party is competent to judge the CA's conformance to the stated criteria.
> In the
> > latter case the 'public information' referred to should include
> information
> > regarding the party's:
> >
> >- knowledge of CA-related technical issues such as public key
> >cryptography and related standards;
> >- experience in performing security-related audits, evaluations, or
> risk
> >analyses; *and*
> >- honesty and objectivity."
> >
> > Today, section 3.2 of the MRSP
> >  > group/certs/policy/#32-auditors>
> > states, "In normal circumstances, Mozilla requires that audits MUST be
> > performed by a Qualified Auditor, as defined in the Baseline Requirements
> > section 8.2," but under section 2.3  > US/about/governance/policies/security-group/certs/policy/#23-baseline-
> > requirements-conformance>,
> > "Mozilla reserves the right to accept audits by auditors who do not meet
> the
> > qualifications given in section 8.2 of the Baseline Requirements, or
> refuse
> > audits from auditors who do."
> >
> > Section 8.2 of the Baseline Requirements states an auditor must have:
> > 1. Independence from the subject of the audit; 2. The ability to conduct
> an
> > audit that addresses the criteria specified in an Eligible Audit Scheme
> (see
> > Section 8.1); 3. Employs individuals who have proficiency in examining
> Public
> > Key Infrastructure technology, information security tools and techniques,
> > information technology and security auditing, and the third-party
> attestation
> > function; 4. (For audits conducted in accordance with any one of the ETSI
> > standards) accredited in accordance with ISO 17065 applying the
> requirements
> > specified in ETSI EN 319 403; 5. (For audits conducted in accordance
> with the
> > WebTrust standard) licensed by WebTrust; 6. Bound by law, government
> > regulation, or professional code of ethics; and 7. Except in the case of
> an
> > Internal Government Auditing Agency, maintains Professional
> Liability/Errors
> > & Omissions insurance with policy limits of at least one million US
> dollars in
> > coverage
> >
> > It is proposed in Issue #192
> >  that information about
> > individual auditor's qualifications be provided--identity, competence,
> > experience and independence. (For those interested as to this
> independence
> > requirement, Mozilla Policy v.1.0 required either disclosure of the
> auditor's
> > compensation or the 

RE: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Tim Hollebeek via dev-security-policy]

Ben,

We're concerned that these changes could unintentionally prevent many new 
auditors from being able to conduct audits, despite being Qualified Auditors.  
The problem is that CAs will understandably be very hesitant to use a new 
auditor, as they cannot risk having an audit conducted, and then not accepted 
by Mozilla.  An unfortunate effect of that is that CAs will likely only 
consider auditors who have a previous history of being accepted as qualified.

One potential solution is to allow CAs to ask Mozilla to "pre-vet" a potential 
auditor they would like to use.  Mozilla policy already has a process in the 
next paragraph to provide opinions on auditors who "do not fit the definition 
of Qualified Auditor" that could potentially be re-used.  Unfortunately, as the 
paragraph is currently worded, it can only be used for auditors who are *NOT* 
Qualified, which is really weird.

It would be helpful to clarify that CAs MAY use the same procedure to work with 
Mozilla to determine whether a particular auditor is Qualified or not, prior to 
the beginning an engagement.

-Tim

> -Original Message-
> From: dev-security-policy  On
> Behalf Of Ben Wilson via dev-security-policy
> Sent: Tuesday, November 3, 2020 6:53 PM
> To: Mozilla 
> Subject: Policy 2.7.1: MRSP Issue #192: Require information about auditor
> qualifications in the audit report
> 
> Historically, Mozilla Policy required that CAs "provide attestation of their
> conformance to the stated verification requirements and other operational
> criteria by a competent independent party or parties with access to details of
> the CA's internal operations."
> https://wiki.mozilla.org/CA:CertificatePolicyV1.0  "Competency" was "for
> whom there is sufficient public information available to determine that the
> party is competent to judge the CA's conformance to the stated criteria. In 
> the
> latter case the 'public information' referred to should include information
> regarding the party's:
> 
>- knowledge of CA-related technical issues such as public key
>cryptography and related standards;
>- experience in performing security-related audits, evaluations, or risk
>analyses; *and*
>- honesty and objectivity."
> 
> Today, section 3.2 of the MRSP
>  group/certs/policy/#32-auditors>
> states, "In normal circumstances, Mozilla requires that audits MUST be
> performed by a Qualified Auditor, as defined in the Baseline Requirements
> section 8.2," but under section 2.3  US/about/governance/policies/security-group/certs/policy/#23-baseline-
> requirements-conformance>,
> "Mozilla reserves the right to accept audits by auditors who do not meet the
> qualifications given in section 8.2 of the Baseline Requirements, or refuse
> audits from auditors who do."
> 
> Section 8.2 of the Baseline Requirements states an auditor must have:
> 1. Independence from the subject of the audit; 2. The ability to conduct an
> audit that addresses the criteria specified in an Eligible Audit Scheme (see
> Section 8.1); 3. Employs individuals who have proficiency in examining Public
> Key Infrastructure technology, information security tools and techniques,
> information technology and security auditing, and the third-party attestation
> function; 4. (For audits conducted in accordance with any one of the ETSI
> standards) accredited in accordance with ISO 17065 applying the requirements
> specified in ETSI EN 319 403; 5. (For audits conducted in accordance with the
> WebTrust standard) licensed by WebTrust; 6. Bound by law, government
> regulation, or professional code of ethics; and 7. Except in the case of an
> Internal Government Auditing Agency, maintains Professional Liability/Errors
> & Omissions insurance with policy limits of at least one million US dollars in
> coverage
> 
> It is proposed in Issue #192
>  that information about
> individual auditor's qualifications be provided--identity, competence,
> experience and independence. (For those interested as to this independence
> requirement, Mozilla Policy v.1.0 required either disclosure of the auditor's
> compensation or the establishment that the auditor "is bound by law,
> government regulation, and/or a professional code of ethics to render an
> honest and objective judgement regarding the CA.")
> 
> While subsection 3 of BR 8.2 requires "individuals who have proficiency in
> examining Public Key Infrastructure technology, information security tools and
> techniques, information technology and security auditing, and the third-party
> attestation function," that fact needs evidence in order to be established. 
> The
> proposed resolution of this Issue #192 intends to accomplish that.
> 
> This proposal to require disclosure of individual auditor qualifications is 
> very
> similar to the approach adopted by the U.S. Federal PKI
> 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ryan Sleevi via dev-security-policy]

Hi Clemens,

I think this fundamentally misunderstands the proposal. As Ben mentioned,
and as countless other schemes have highlighted, competency is with
individuals, not organizations. While the eIDAS Scheme is relevant for
eIDAS qualification, I think it's important to highlight that browsers are
not, nor have they ever, relied upon the Qualified Trust List, nor on the
eIDAS Framework, as they are fundamentally separate trust frameworks. I
understand you see this redundant, but given that browsers are not relying
on the Supervisory Body function, since they are different trust
frameworks, it's absolutely vital for transparency to be able to provide
that information.

While something may be acceptable for the eIDAS Certification scheme,
audits exist to support those consuming them, and it's important to make
sure they are addressed. WebTrust equally has an approach like you
describe, but what is being suggested here is that is not sufficient for
the need and use case, and I fully support that. I can understand that
professional accountability is scary, because it might mean that audits
that might be acceptable for eIDAS are rejected as unacceptable for
Mozilla, but again, that's a reflection of the different nature of trust
frameworks.

I find the appeal to redundancy and the NAB, and further, the suggestion of
GDPR, to be a bit insulting to this community. This opposition to
transparency fundamentally undermines the trust in ETSI provided audits, or
this appeal to the eIDAS scheme, which has limited relevance given it's a
fundamentally different audit scheme, beggars belief. If you'd like to
raise Fear/Uncertainty/Doubt about GDPR, I believe you owe this community a
precise and detailed explanation about what you believe, relevant to the
auditor professional experience, would be problematic.

The suggestion here that this is somehow unique or untowards is deeply
concerning. I note, for example, that BSI's own C5 controls are designed
around transparency, and Section 4.4.9 on auditor qualification similarly
places provisions for transparency.

Without making an appeal to either the NAB or the Supervisory Body, both of
which are relevant for eIDAS but not acting in a function for browser trust
frameworks (nor can they), what alternative would you propose to help
community members here have verifiable evidence and confidence in the
auditor's qualifications?

On Thu, Nov 5, 2020 at 10:54 AM Clemens Wanko via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Hi Ben,
> in order to avoid for every single audit the compilation work for the
> auditor (in person) on his qualification, independence, etc. as well as the
> need to crosscheck the statements he made, that was covered for the EU
> ETSI/eIDAS scheme by the accreditation of the body (organization; example:
> I am member/employee of TUV Austria CERT as accredited body) employing and
> using that auditor (in person) for a specific audit. The body will
> receive/keep its accreditation only in case it proves in annual audits with
> its accreditation body, that the following auditor related aspects were
> covered in the audit projects performed throughout the past audit period:
>
> ISO/IEC17065 - I listed the relevant chapters only:
> 6 Resource requirements ... 31
> 6.1 Certification body personnel ... 31
> 6.1.1 General .. 31
> 6.1.2 Management of competence for personnel involved in the certification
> process . 31
> 6.1.3 Contract with the personnel  33
> 6.2 Resources for evaluation. 33
> 6.2.1 Internal resources  33
> 6.2.2 External resources (outsourcing) ... 33
>
> …amended by guidance documentation in the following areas:
> Annex A (informative) Principles for product certification bodies and
> their certification activities ... 63
> A.1 General .. 63
> A.2 Impartiality  63
> A.3 Competence .. 65
> A.4 Confidentiality and openness . 65
> A.4.1 General .. 65
> A.4.2 Confidentiality .. 65
> A.4.3 Openness .. 65
> A.4.4 Access to information .. 65
> A.5 Responsiveness to complaints and appeals
> .. 65
> A.6 Responsibility ... 67
>
> For ETSI/eIDAS auditors the ISO/IEC17065 is detailed by the following
> mandatory ETSI EN 319 403 (updated version: ETSI EN 319 403-1)
> requirements. I listed 

Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Clemens Wanko via dev-security-policy]

Hi Ben, 
in order to avoid for every single audit the compilation work for the auditor 
(in person) on his qualification, independence, etc. as well as the need to 
crosscheck the statements he made, that was covered for the EU ETSI/eIDAS 
scheme by the accreditation of the body (organization; example: I am 
member/employee of TUV Austria CERT as accredited body) employing and using 
that auditor (in person) for a specific audit. The body will receive/keep its 
accreditation only in case it proves in annual audits with its accreditation 
body, that the following auditor related aspects were covered in the audit 
projects performed throughout the past audit period:

ISO/IEC17065 - I listed the relevant chapters only:
6 Resource requirements ... 31
6.1 Certification body personnel ... 31
6.1.1 General .. 31
6.1.2 Management of competence for personnel involved in the certification 
process . 31
6.1.3 Contract with the personnel  33
6.2 Resources for evaluation. 33
6.2.1 Internal resources  33
6.2.2 External resources (outsourcing) ... 33

…amended by guidance documentation in the following areas: 
Annex A (informative) Principles for product certification bodies and their 
certification activities ... 63
A.1 General .. 63
A.2 Impartiality  63
A.3 Competence .. 65
A.4 Confidentiality and openness . 65
A.4.1 General .. 65
A.4.2 Confidentiality .. 65
A.4.3 Openness .. 65
A.4.4 Access to information .. 65
A.5 Responsiveness to complaints and appeals 
.. 65
A.6 Responsibility ... 67

For ETSI/eIDAS auditors the ISO/IEC17065 is detailed by the following mandatory 
ETSI EN 319 403 (updated version: ETSI EN 319 403-1) requirements. I listed the 
relevant chapters only:
“Electronic Signatures and Infrastructures (ESI); Trust Service Provider 
Conformity Assessment; Part 1: Requirements for conformity assessment bodies 
assessing Trust Service Providers”:
6 Resource requirements 
...
 11
6.1 Conformity Assessment Body personnel 
.
 11
6.1.1 General 

 11
6.1.2 Management of competence for personnel involved in the audit 
process... 11
6.1.2.0 General requirements 

 11
6.1.2.1 Management of 
competence..
 11
6.1.2.2 Training of audit teams 
.
 12
6.2 Resources for evaluation 
..
 12
6.2.0 General requirements 
..
 12
6.2.1 Internal resources 

 12
6.2.1.0 General requirement 
..
 12
6.2.1.1 Competence of Conformity Assessment Body personnel 
. 12
6.2.1.2 Competences for all functions 
...
 12
6.2.1.3 Competences for application review 
.
 13
6.2.1.4 Competences and prerequisites for 
auditing..
 13
6.2.1.5 Competences for review