Issue T: RA Program Misissuances (January 2010 - January 2017)
Program Background:
Symantec has operated an RA program designed to deliver a superior customer
experience in global markets where language skills, understanding of local
business requirements, and physical local presence are necessary. RA partners
have supported various certificate types, including those for publicly trusted
SSL/TLS.
The RA program for publicly trusted SSL/TLS authorized appropriately trained
personnel at select RA partners to complete all steps for authentication,
review, and certificate issuance.
In 2011, prior to ratification of the CA/Browser Forum Baseline Requirements,
Symantec scaled back the scope of the RA program for publicly trusted SSL/TLS
to support only those partners whose scale of business and investment in the
future success of that business warranted the additional cost associated with
supporting the then-new BRs. Since 2013, there have been only 4 RA partners
still capable of processing and issuing publicly trusted SSL/TLS certificates:
CrossCert, Certisign, Certsuperior, and Certisur.
Symantec has had multiple controls in place to ensure these RAs' compliance
with the BRs:
Documentation:
1. Symantec operates an internal Knowledge Base ("KB") for its authentication
staff and RA partners that contains detailed step-by-step procedures for
performing each of the tasks required to validate the identity asserted in a
certificate request.
2. The KB reinforces acceptable and unacceptable sources of validation
information and processes using a subset of the information in the BRs.
3. The KB explains request flagging, flag reasons, and flag clearing procedures.
Training & Exams:
1. Topics include BR changes, CPS changes, process changes resulting from
industry incidents regardless of the CA involved, and a review of Symantec's
procedures that extend the Baseline Requirements.
2. Exams are modified and retaken annually as criteria to renew individual
access certificates or after significant internal or external process changes.
Technology During Authentication:
1. Each request is screened for trade compliance, high-risk names, potential
phishing (strings used in scam domains, high-profile brands), and other
potentially risky content such as "test". Potential failures are flagged,
preventing RA issuance, until and unless further review and analysis is
completed.
2. Risk flags require manual override by authentication personnel - internal or
RA personal as appropriate - for certificate issuance to proceed. Flag clearing
privileges are only granted to personnel who are have completed the requisite
training and passed appropriate exams.
Technology Pre- and Post- Issuance:
1. Each request is screened to ensure elements outside of the subject
information are BR compliant (e.g. SAN fields are complete, proper validity
limits are in place, 2048 bit or higher key lengths are used, etc.). This scan
is done after Authentication personnel approve the request and before it is
issued. These checks cannot be overridden.
2. Daily, we rescan all certificates issued on the prior day using these same
checks.
Audit:
1. We requested independent WebTrust audit reports from RAs and assessed them
for material findings pursuant to BR 8.4 regarding WebTrust audited Delegated
Third Parties. See issue V addressing audits.
Customer-Facing Controls:
1. Symantec supports Certification Authority Authorization, putting control of
authorized CAs in the hands of customers.
2. Symantec logs publicly trusted certificates to Certificate Transparency Logs
and offers a CT monitor to provide visibility for all customers to enable
detection of suspect certificates.
CrossCert Test Certificate Issue:
On January 19, 2017, Andrew Ayer, an independent researcher posted the results
of an analysis of public Certificate Transparency logs through which he
identified roughly 270 instances of suspicious certificates issued by multiple
CAs, including Symantec, containing the words "test" or "example" in the
subject information.
Symantec determined that 127 of these certificates were issued from Symantec
operated CAs and that all 127 had been issued by the RA CrossCert. All but 31
had already expired or been revoked.
Immediate Response
Andrew Ayer's report was a certificate problem report under BR 4.9.5 which
required us to begin an investigation within 24 hours, which we did. We
determined that 127 certificates were in scope of the problem report.
1. On January 19, 2017, after becoming aware of this issue, Symantec disabled
issuance privileges for all CrossCert staff.
2. On January 20, 2017, Symantec revoked the 31 still valid and active
certificates. These certificates had been issued between December 28, 2016 and
January 18, 2017.
3. Symantec promptly took over validation and issuance for all pending and new
orders submitted through CrossCert. Since then, Symantec's authentication team
has continued to fully process these orders independently, without relying on
any previous authentication work completed by CrossCert.
4. Symantec disabled customer issuance from enterprise accounts originally
authenticated by CrossCert. New issuance from these accounts was blocked until
the accounts were re-validated by Symantec personnel.
Root Cause
Following discussions with CrossCert and review of their audit logs, Symantec
confirmed that all 127 certificates were issued as part of either internal or
customer-specific testing. In mid-2016, CrossCert created a customer-focused
feature with good intent to help customers with testing, but they did so
without following the proper guidelines.
CrossCert personnel had completed the Symantec-required training mandating that
only fully validated certificates be issued. At no point was CrossCert
authorized to perform less vetting on certificates for their own internal use
or for customer testing. The personnel at CrossCert had also completed exams
designed to verify comprehension of this training. Nonetheless, CrossCert
management authorized overrides for certificates being used for internal and
customer-specific testing, and CrossCert authentication personnel complied with
that authorization.
Audit logs confirm that our controls to flag these certificates for compliance
failures worked appropriately but CrossCert management overrode those flags.
CrossCert did not consult with Symantec on the significance of the compliance
failure flags or their decisions to override the flags for any of the
certificates. Our flagging process is extensive and a normal part of every
business day. For example, our process catches popular domains/trademarks and
variants using fuzzy logic that might otherwise be issued by a CA that doesn't
test or block trademark abuse. Our flag/hold rate is conservative - as a
result, the false positive rate is high and, following manual analysis, most
flags are cleared. We appreciate the suggestion that additional flag monitoring
and trend analysis would provide greater insight and increase the opportunity
to catch issues like this more quickly in the future.
Review of the 127 certificates showed that all issuances occurred in two time
frames: Before June, 2012 or after June, 2016. Symantec received unqualified
WebTrust for CAs and WebTrust for CAs Baseline and Network Security audits from
Ernst & Young KR for the period July 1, 2015-June 30, 2016. Each of the 127
certificates issued for testing purposes were issued either years before these
audits or after these most recent audits and so not detectable as part of these
independent audits.
Subsequent Investigation
As part of investigating this incident, Symantec began a direct review of the
authentication practices at CrossCert and their engagement with Ernst & Young
KR. Through that investigation, Symantec identified two additional issues:
1. There were deficiencies in CrossCert's documentation of the validation step
in the authentication process, failing to meet both Symantec and Baseline
requirements.
2. The CrossCert 2015-2016 audit did not include in its scope all of the CAs
from which CrossCert was authorized to issue certificates. E&Y KR stated that
CrossCert provided a list of all their issuing CAs but reduced the list of
issuing CAs in scope for sampling for budgetary reasons. E&Y KR did not call
out this known limit in its report.
These additional findings caused Symantec to seriously question not only
CrossCert, but also the adequacy of E&Y KR's WebTrust for CAs Baseline and
Network Security audits. This episode also reinforced that we need to more
closely monitor and review the scope, methodology, and practices of external
auditors.
Remediation
1. Symantec fully terminated CrossCert as an RA partner. They no longer
participate in the authentication process or issuance of publicly trusted
SSL/TLS certificates.
2. Symantec has hired a team of Korean speaking authentication personnel who
have completed the requisite training and now independently process all new
orders from this market.
3. Symantec is in the process of fully and independently revalidating active
SSL/TLS certificates previously approved by CrossCert, and if necessary,
revoking and replacing certificates if we detect any errors. As we posted in
response to the assertion about reusing RA validation in m.d.s.policy, we
reiterate that we are not doing this. For CrossCert, we are validating each
existing certificate as if it was a new request.
4. Symantec announced the decision to wind down the RA program for publicly
trusted SSL/TLS.
5. Symantec is in the process of reviewing 100% of the authentication records
for the active SSL/TLS certificates approved and issued by Certisign,
Certsuperior, and Certisur. If we become aware of a certificate that is
misleading, we follow the Baseline Requirements regarding revocation.
It has been suggested that the issues identified above and those associated
with Issue V warrant immediate revocation of all certificates issued by each of
these RAs. Symantec disagrees with that assessment. Revocation by a CA of
certificates, outside of a customer request, can be materially disruptive to
the individuals and the organizations that rely on them, and so should be
reserved for cases where there are clear security risks.
In the case of CrossCert, Symantec immediately revoked the clearly problematic
certificates mis-issued for testing. With respect to individual certificates,
the key additional issue identified with CrossCert related to proper
record-keeping. Accordingly, using a risk-informed approach, Symantec is fully
re-validating these certificates.
In the cases of Certisign, Certsuperior, and Certisur, we relied on at least
one unqualified audit for their operations. With respect to their individual
certificates, there was no evidence of any problems with the certificates they
approved and issued. Accordingly, and again using a risk-informed approach,
Symantec is reviewing all of these certificates, which amounts to conducting an
audit with 100% sampling.
Symantec will provide updates to the community regarding our revalidation of
active CrossCert-issued SSL/TLS certificates and our review of the other RA
partners.
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
