On 31/10/2017 3:14 μμ, Ryan Sleevi via dev-security-policy wrote:
On Tue, Oct 31, 2017 at 8:34 AM, Dimitris Zacharopoulos via
dev-security-policy <dev-security-policy@lists.mozilla.org> wrote:
[...]
I think you are looking at this from the opposite side. Auditors have
their own scheme to follow which is governed under ISO 17065 and ETSI EN
319 403. These schemes provide guidance on how to conduct an effective
audit for ETSI EN 319 401, 411-1, 411-2, 421 and so on. In addition to
these, there are National Accreditation Body schemes for specific audits
which provide additional guidance. When the audit covers operations for one
year (mandated by the Baseline Requirements and which finds it's way in the
contract between the CA and the CAB), the sampling must include evidence
from the entire year.
I don't believe your statement is supported by the evidence - which is why
I'm pushing you to provide precise references. Consider from the
perspective as a consumer of such audits - there is zero awareness of the
contract as to whether or not the BRs were in scope - after all, 319 411-1
is meant to be inclusive of the normative requirements with respect to
audit supervision.
My statement that auditors are governed by 17065 and 403 is supported by
evidence (section1 of 411-1 where it says that ETSI EN 319 403 provides
guidance to auditors that wish to audit the 411-1 standard). Also, the
BRs are normative for 411-1 as stated in section 2.1 of the same
document. Normative references to the BRs are all over the 411-1
document, unless I misunderstood your statement.
But stepping back further from the contract, the claim that "the audit
covers operations for one year" is also not part of the 17065, 17021, or
319 403 oversight. That is, the certification is forward looking (as
evidenced by the expiration), and while it involves historic review, it is
not, in and of itself, a statement of assurance of the historic activities.
This is the core difference between the 17021/17065 evaluation of processes
and products versus, say, the ISAE3000 assurance evaluation.
I read the ISAE3000 and can't find specific language to support a core
difference in auditor guidance, especially related to the assurance of
the historic activities. Perhaps there is a more specific section you
can reference.
The eIDAS Regulation mandates for 2-year audits (not the ETSI EN 319
411-1). This has been reflected in the ETSI EN 319 403 audit scheme, under
7.4.6 (Audit Frequency), which states:
"There shall be a period of no greater than two years for a full
(re-)assessment audit unless otherwise required by the applicable
legislation or commercial scheme applying the present document.
NOTE: A surveillance audit can be required by an entitled party at any
time or by the conformity assessment
body as defined by the surveillance programme according to clause 7.9."
I'm patently aware of that, but I'm trying to highlight to you that this
statement itself lacks the specificity to give confidence of the
operations. We've seen CAs try to present surveillance audits as full
audits. We've seen CAs try to present point in time assessments as periods
of time.
This means that some CAs can't tell the difference between a
point-in-time and a period-in-time audit (nothing to do with the audit
scheme). This is why the definition was added by CA/B Forum ballot 196
(https://cabforum.org/2017/04/17/ballot-196-define-audit-period/). We
clarified the "surveillance audit" issue in Bilbao and re-stated it
here. A standard (the BRs) or a CA can mandate that a "full audit" is
conducted every year, and this should be documented in the audit report.
Also, the current BRs state in 8.1 "The period during which the CA
issues Certificates SHALL be divided into an unbroken sequence of audit
periods. An audit period MUST NOT exceed one year in duration".
"Full audit" is not specifically defined in the BRs but it is a Mozilla
Root program requirement.
I searched Webtrust for CAs 2.0 document to find specific language that
audits must be performed yearly but I couldn't find an explicit
requirement. Could you please point me to the right section?
Also, as we discussed at F2F 38 in Bilbao and is covered in the minutes <
https://cabforum.org/2016/05/25/2016-05/#ETSI-and-eIDAS-Update>, even
though the general guidance for ETSI EN 319 403 is full re-assessment audit
every two years, it is up to the CA/TSP to request that a full audit is
conducted yearly and that this information is declared in the audit report.
If you see ETSI audit reports that don't specifically state that a
full-audit took place, then this information probably wasn't clear to the
specific TSP. This is not related to the audit scheme.
Respectfully, I disagree. If the audit scheme routinely encourages audits
that are insufficient, then one MUST look at the root cause for that.
Nothing wrong with disagreeing but the basis of your disagreement seems
to be circumstantial.
The 17065 audit scheme provides more than sufficient guidance for
sufficient audits which is why it is used so broadly in critical areas
like food products, elevators, etc. Adding 319 403 on top, makes it even
more robust to adhere to the challenges of the CA industry.
I am very curious to learn if there is a core difference between the two
audit schemes and why one approach leads to the desired result and one
does not. I assume the desired result is that during the audit period,
auditors must look at historical data throughout the entire period and
not pick one fraction of the period to collect evidence and assess that
everything is good for the entire audit period. They should also report
major findings of the entire audit period in their final audit report.
Can we at least agree on that?
It would appear your argument - and apologies if I misrepresent, but I
rephrase it here in the hopes of highlighting our disagreement - is that
"319 401, 319 401, and 319 411-1 provide frameworks for audits, but it's up
the the CAB, NAB, and TSP to establish a procedure that will suitably
satisfy the user agents".
This does not seem to be an accurate statement because 319 401 and 319
411-1 are compliance standards, very - very similar to the Webtrust for
CAs criteria. On the other hand, ISO 17065 provides an audit framework
and ETSI EN 319 403 provides more specific guidance for audits against
ETSI EN 319 411-1, 411-2, 421 and perhaps more.
In addition to that, NABs apply additional rules and guidance to CABs,
making the framework even stricter. At the end of the day, CABs must
follow all the rulesets defined in standards (like ETSI EN 319 411-1),
audit criteria guidance (like ISO 17065, ETSI EN 319 403), other
regulations (like the ones mandated from NABs), other requirements (like
the ones specified in the Baseline Requirements and Root Programs) that
are in the audit contract. All the various criteria are documented in
the audit reports.
My argument is that, especially when combined
with the fact that Regulation No 910/2014 establishes a process that is
unquestionably insufficient, and TSPs and CABs are using that as the
framework for their procedures and agreements, this is not sufficient.
Or, put differently, your argument seems to be that 319 411-1 "could" be
used to satisfy the requirements of the Mozilla Root Program (if supplanted
with additional contractual and procedural requirements, documented
privately to the TSP as part of Phase 1), while my argument is that 319
411-1 "alone" does not satisfy the requirements (and that, given the
confidentiality aspects of the reporting, as required under 319 403 and
17065, cannot)
In what ways is Webtrust any different? Are there no confidentiality
aspects of the reporting? I am aware of cases where an ETSI audit failed
and no audit *certificate* was issued but there was a report listing all
the major non-conformances that was the result of the audit. In that
particular case, the audit report (with the major non-conformances) was
disclosed to an interested "audit consumer" but it was done privately. I
can only assume that the same applies to Webtrust and practically any
audit scheme in the world. Auditors are not required or forced to
reporting anything publicly. They do so only if allowed by their
contract with their client. As far as I can remember, either CAs or
auditors with permission from CAs were sending audit reports directly to
the browsers.
AFAIK, when major non-conformances exist, or a number of minor
non-conformances which cumulatively indicate a significant deviation
from the requirements, no certificate is issued. The audit report
documents this. I double-checked this with our auditors.
Is this requirement to opine on the past year of activities described in
WebTrust for CAs?
Yes. As noted in the relevant professional standards I cited (with respect
to AT101 or ISAE3000), and the underlying professional standards (for which
we discussed during "Period of Time" definition), the very foundation of
the reporting is based on this degree of analysis.
I searched through ISAE3000 document and couldn't find a specific
section that states that audits under this framework must be done yearly.
The degree of analysis is a separate issue and the level of information
that should be disclosed in audit reports has been discussed in the
past. IMHO, this is also adequately covered in ETSI. If during the audit
period, there are major non-conformances, CAs don't get a certificate (I
suppose it is similar to the case where you don't get a "seal" if you
have major deviations from the Webtrust standards). CAs can get an audit
letter (no Certificate) that describe the process and the major
non-conformances. This letter can be kept private by the CA or be
disclosed at their discretion.
It is also the CA's right to engage with another auditor if the previous
auditor does not demonstrate the level of competence necessary to
conduct the audit. In this case, for the same period-in-time there might
be two audit reports. I also suppose this may happen with both schemes
and couldn't find something to forbid it.
If the Mozilla Root Program wishes to request that audit reports include
specific details about the audit that took place, this is something that
should affect both Webtrust and ETSI reports.
If an auditor allowed for this scenario, then this audit is not
"effective" and the auditor should not be trusted. With a very quick search
in ETSI EN 319 403, Section 7.4.3.2 describes the Requirements of Sample
Based Approach. If a CAB does not effectively sample the audit period and
capture evidence only for the last 3 months, they jeopardize their
accreditation. Again, this is not related to the audit scheme.
I appreciate you highlighting the text, but this again highlights the
deficiency. You've highlighted the text with respect to sampling, but as
you can see, it does not note the period of sampling. More precisely,
you're referencing "audit period", but that itself is missing from the
requirements. Further, I don't think you can argue 7.4.3.2 is relevant with
respect to the historic process review - it's instead normative for
multi-site sampling.
You are right, I can't argue, but like I said it was a very quick search
:) It was a reference to a sampling guidance for something else.
This is precisely because it's designed around a
process/site overview, not a historic overview - as evidenced throughout
Section 7 (and further emphasized in 7.4.5.2)
However, this conclusion seems to be overloaded because site sampling
that meet the same security properties, is something that makes sense,
especially after reading 7.4.3.1. Site overview is definitely in audit
scope and I would be surprised if this was not the case with Webtrust.
IMO, historic overview of evidence is defined in ETSI EN 319 403 section
7.9 "In addition, a sample of records relating to the operation of TSP
over the historical period since the previous audit shall be examined by
the auditor" that should be enough to address this particular concern,
that the ETSI scheme does not require auditors to examine historical
data of the audit period.
It would appear, and again, apologies if I'm mistating, that your view is
that the 'audit period' is negotiated with the CAB as part of the
contractual negotiations, and that the documentation review of Phase 1
(which includes the historic analysis) would include a sampling analysis
over the period. I don't believe this is actually supported under 17021,
17065, or 319 403. Indeed, the only sampling of records covered within 319
403 is in Section 7.9.1 with respect to surveillance audits
The audit period is not negotiated. It is always starting after the last
day of the previous audit period.
According to the clarifications of our auditor, which I posted earlier
on this list, the only thing that is left as optional is whether the
audit after the first "full" audit is another "full" audit and not a
"surveillance".
I also looked at Webtrust for CAs SSL BASELINE WITH NETWORK SECURITY 2.1
and couldn't find a direct reference that audits under this scheme must
be done annually, as mandated by the BRs. It is very likely that I am
not looking at the right place so, again, please point me to the right
direction.
I really hope that some ETSI auditors monitoring this list will be able to
confirm or correct me if my statements are incorrect. In the past years, I
had to go through the documentation of both sides (auditors and CAs) to
better understand the ETSI scheme but I believe the same basic concepts
exist in all audit schemes, including the Webtrust scheme.
I really appreciate you engaging on the list - please don't take my
criticisms personally.
I don't :)
But I am trying to highlight the specific parts of
the texts that describe (insufficient) processes and levels of assurance.
Hopefully I've provided sufficient evidence of that, but I can always work
to clarify (with a specific section reference) why I'm asserting something
specific.
Audit standards are not the same as technical standards where you can
get a final assertion by examining one particular section. We need to
see this as a full process and specify the expected goals. Then, if we
detect flaws in the process (webtrust or ETSI), we could provide
recommendations for improvements in the appropriate governing bodies of
these schemes.
Dimitris.
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
