Re: Symantec Issues List

[Ryan Sleevi via dev-security-policy]

On Mon, Apr 3, 2017 at 12:46 PM, Jakob Bohm via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:
>
> How about this simple explanation (purely a guess, not at all checked):
>

I think we should focus on objective facts and statements. While there are
a number of possible ways to interpret things, both positively and
negatively, the fact that such multiple interpretations exist highlight a
problem that needs public clarification and resolution - both on behalf of
Symantec and their auditors.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Gervase Markham via dev-security-policy]

On 03/04/17 02:37, Peter Bowen wrote:
> I believe Issue L is incorrectly dated.  

Thank you for this; I have updated Issue L to hopefully be more
accurate, but I intend to keep it as a separate issue.

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Gervase Markham via dev-security-policy]

On 01/04/17 05:57, Peter Bowen wrote:
> The GeoRoot program was very similar to that offered by many CAs a few
> years ago.  CyberTrust (then Verizon, now DigiCert) has the OmniRoot
> program, Entrust has a root signing program[1], and GlobalSign Trusted
> Root[2] are just a few examples.

While this is true, it's not just about the existence of the legacy
program with problems, but about how the situation is handled. Verizon
were not able to bring their program into BR compliance; DigiCert bought
it and worked closely with Mozilla to generate some breathing space for
them to clean the system up. They posted public plans, kept us informed
of the issues found and their plans for remediation, and completed the
work pretty much on time. The Web PKI is a better place for it.

This does not seem to be the approach taken by Symantec, if we accept
Ryan's account of events.

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Gervase Markham via dev-security-policy]

On 01/04/17 00:38, Ryan Sleevi wrote:
> On Fri, Mar 31, 2017 at 2:39 PM, Gervase Markham via dev-security-policy <
> Thanks for organizing this information, as much of it was related to and
> relevant to Google's recent announcement. I want to take this opportunity
> to share additional details regarding the interactions for UniCredit, which
> I believe may be useful and relevant both for understanding that issue and
> the GeoTrust audits.

Thank you for this. I will attempt to integrate it into the document and
I'm sure we will hear comments from Symantec on it.

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Peter Bowen via dev-security-policy]

On Sun, Apr 2, 2017 at 9:36 PM, Ryan Sleevi  wrote:
>
> On Sun, Apr 2, 2017 at 11:14 PM Peter Bowen via dev-security-policy
>  wrote:
>>
>> On Fri, Mar 31, 2017 at 11:39 AM, Gervase Markham via
>> dev-security-policy  wrote:
>> > As we continue to consider how best to react to the most recent incident
>> > involving Symantec, and given that there is a question of whether it is
>> > part of a pattern of behaviour, it seemed best to produce an issues list
>> > as we did with WoSign. This means Symantec has proper opportunity to
>> > respond to issues raised and those responses can be documented in one
>> > place and the clearest overayll picture can be seen by the community.
>> >
>> > So I have prepared:
>> > https://wiki.mozilla.org/CA:Symantec_Issues
>> >
>> > I will now be dropping Symantec an email asking them to begin the
>> > process of providing whatever comment, factual correction or input they
>> > feel appropriate.
>> >
>> > If anyone in this group feels they have an issue which it is appropriate
>> > to add to the list, please send me email with the details.
>>
>> Gerv,
>>
>> I'm afraid that Issue V: RA Program Audit Issues (2013 or earlier -
>> January 2017) has confused RAs with subordinate CAs.
>>
>> According to
>> https://bug1334377.bmoattachments.org/attachment.cgi?id=8843448,
>> Symantec has indicated that they have (had) four unconstrained third
>> party RAs: CrossCert, Certisign, Certisur, and Certsuperior.  These
>> appear to fall into what the BRs call "Delegated Third Parties".  No
>> audit report seems to mention any issue with these RAs.
>>
>> Separately Symantec owned CAs have issued CA-certificates to several
>> CAs that are not operated by Symantec.  These appear to include at
>> least Apple, Google, the US Government, Aetna, and Unicredit.  The
>> audit reports linked from Issue V appear to have qualifications
>> regarding these CA-certificates.
>>
>> There are notable differences between third party owned CAs and third
>> party operated RAs and the difference should be clearly noted.
>>
>> Thanks,
>> Peter
>> ___
>> dev-security-policy mailing list
>> dev-security-policy@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-security-policy
>
> Both
> https://www.symantec.com/content/en/us/about/media/repository/GeoTrust-WTBR-2015.pdf
> (Finding number 3) and
> https://www.symantec.com/content/en/us/about/media/repository/symantec-webtrust-audit-report.pdf
> (Finding number 1) call out Delegated Third Parties as lacking audits. This
> is called out separately from the matters related to sub-CAs, as
> "Furthermore".
>
> Given that at least some of the sub-CAs possessed and provided audits to
> Symantec, it does not seem to support your summary, but perhaps your point
> was misunderstood?

I think there are two parts:

1) There should be two different issues in the issues list -- one for
management of Subordinate CAs and one for management of unconstrained
RAs (i.e. Delegated Third Parties)

2) It is not clear that the audit reports for the GeoTrust brand roots
are calling out RAs as qualifications.  My read is that they were
considering the subordinate CAs as DTPs, not the RAs.  However I can
see the other interpretation as well.

Thanks,
Peter
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Ryan Sleevi via dev-security-policy]

On Sun, Apr 2, 2017 at 11:14 PM Peter Bowen via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Fri, Mar 31, 2017 at 11:39 AM, Gervase Markham via
> dev-security-policy  wrote:
> > As we continue to consider how best to react to the most recent incident
> > involving Symantec, and given that there is a question of whether it is
> > part of a pattern of behaviour, it seemed best to produce an issues list
> > as we did with WoSign. This means Symantec has proper opportunity to
> > respond to issues raised and those responses can be documented in one
> > place and the clearest overayll picture can be seen by the community.
> >
> > So I have prepared:
> > https://wiki.mozilla.org/CA:Symantec_Issues
> >
> > I will now be dropping Symantec an email asking them to begin the
> > process of providing whatever comment, factual correction or input they
> > feel appropriate.
> >
> > If anyone in this group feels they have an issue which it is appropriate
> > to add to the list, please send me email with the details.
>
> Gerv,
>
> I'm afraid that Issue V: RA Program Audit Issues (2013 or earlier -
> January 2017) has confused RAs with subordinate CAs.
>
> According to
> https://bug1334377.bmoattachments.org/attachment.cgi?id=8843448,
> Symantec has indicated that they have (had) four unconstrained third
> party RAs: CrossCert, Certisign, Certisur, and Certsuperior.  These
> appear to fall into what the BRs call "Delegated Third Parties".  No
> audit report seems to mention any issue with these RAs.
>
> Separately Symantec owned CAs have issued CA-certificates to several
> CAs that are not operated by Symantec.  These appear to include at
> least Apple, Google, the US Government, Aetna, and Unicredit.  The
> audit reports linked from Issue V appear to have qualifications
> regarding these CA-certificates.
>
> There are notable differences between third party owned CAs and third
> party operated RAs and the difference should be clearly noted.
>
> Thanks,
> Peter
> ___
> dev-security-policy mailing list
> dev-security-policy@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
Both
https://www.symantec.com/content/en/us/about/media/repository/GeoTrust-WTBR-2015.pdf
(Finding number 3) and
https://www.symantec.com/content/en/us/about/media/repository/symantec-webtrust-audit-report.pdf
(Finding number 1) call out Delegated Third Parties as lacking audits. This
is called out separately from the matters related to sub-CAs, as
"Furthermore".

Given that at least some of the sub-CAs possessed and provided audits to
Symantec, it does not seem to support your summary, but perhaps your point
was misunderstood?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Peter Bowen via dev-security-policy]

On Fri, Mar 31, 2017 at 11:39 AM, Gervase Markham via
dev-security-policy  wrote:
> As we continue to consider how best to react to the most recent incident
> involving Symantec, and given that there is a question of whether it is
> part of a pattern of behaviour, it seemed best to produce an issues list
> as we did with WoSign. This means Symantec has proper opportunity to
> respond to issues raised and those responses can be documented in one
> place and the clearest overayll picture can be seen by the community.
>
> So I have prepared:
> https://wiki.mozilla.org/CA:Symantec_Issues
>
> I will now be dropping Symantec an email asking them to begin the
> process of providing whatever comment, factual correction or input they
> feel appropriate.
>
> If anyone in this group feels they have an issue which it is appropriate
> to add to the list, please send me email with the details.

Gerv,

I believe Issue L is incorrectly dated.  As can be seen on crt.sh,
there are two CAs operated by the US federal Government which have
been repeatedly issued certificates by various CAs trusted by Mozilla:

https://crt.sh/?caid=1324 (Federal Bridge CA)
https://crt.sh/?caid=1410 (Federal Bridge CA 2013)

These two CAs have cross-certified each other and have been issued
several certificates by VeriSign/Symantec and Digital Signature
Trust/IdenTrust. The earliest date for VeriSign is 2011-02-03 and the
earliest date for DST is 2011-01-14.

I also think that Issue L should probably be combined with the GeoRoot
items.  Functionally they are the same issue: management and oversight
of external subordinate CAs.

Thanks,
Peter
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Ryan Sleevi via dev-security-policy]

On Sat, Apr 1, 2017 at 12:57 AM, Peter Bowen  wrote:

> (Wearing my personal hat)
>
> Ryan,
>
> I haven't reviewed the audit reports myself, but I'll assume all you
> wrote is true.  However, I think it is important to consider it in the
> appropriate context.


> The GeoRoot program was very similar to that offered by many CAs a few
> years ago.  CyberTrust (then Verizon, now DigiCert) has the OmniRoot
> program, Entrust has a root signing program[1], and GlobalSign Trusted
> Root[2] are just a few examples.
>
> In almost every case the transition to requiring complete unqualified
> audits of the subordinates by a licensed practitioner was a rocky one.
> See DigiCert's thread
> (https://groups.google.com/d/msg/mozilla.dev.security.
> policy/tHUcqnWPt3o/U2U__7-UBQAJ)
> about the OmniRoot program or look at the audits available for some of
> the Entrust subordinates.
>
> I'm not suggesting that the GeoRoot subordinate issues should not be
> considered, but it seems the GeoRoot program was not notably
> exceptional a few years ago.
>

(Wearing a personal hat)

Peter,

There are a few issues to unpack from your reply. I think we're in
agreement that GeoRoot was by no means unique as an offering. I think, when
considering severity, it's important to instead focus on what the CAs
obligations were, what they were aware of, and what they did in response.
Further, in considering the broader scope of attempted remediation, it's
important to consider what risks were or are present as a result of this,
because it significantly impacts the ability to trust the existing set of
issued certificates.

On 2014-05-13, Mozilla requested all participating CAs disclose their
externally operated subordinate CAs. [1]
On 2014-06-03, Symantec reported it disclosed its sub-CAs in [2]
On 2015-04-06, Kathleen pointed out Symantec's disclosure was incomplete,
in [3] and [4]
On 2016-03-29, Symantec informed Google that there were 5 participants in
their GeoRoot program - Aetna, Google, Unicredit, Apple, NTT Docomo (DKHS).
On 2016-05-11 (or later), Symantec received Aetna's audit.
On 2016-05-13, Symantec's most recent audit for the Geotrust roots was made
available [5], which states there were 5 external partner subordinate CAs.
The timing of Aetna's letter suggests that this may be the audit that
"Symantec subsequently received an audit report for the other" - but that
cannot be confirmed without further detail from KPMG and Symantec.
On 2016-06-28, Symantec informs Google that NTT Docomo is part of
Symantec's audit, not separately audited.

This timeline hopefully highlights a particular serious issue: If NTT
Docomo is operated as part of Symantec's operations, then there are several
ways to interpret Symantec's audit statements:
1) KPMG failed to include NTT Docomo as part of the 5 externally operated
sub-CAs noted, and instead treated it as part of Symantec's audit. If this
is true, then there is an as-yet-unidentified intermediate certificate
issued as part of the GeoRoot program
2) KPMG was treating NTT Docomo as part of the 5 externally operated
sub-CAs noted. If this is correct, then it is in one of three sets
  a) The 3/5 sub-CAs for which KPMG identified as having audit reports
  b) The 1/5 sub-CAs for which KPMG identified as having a deficient audit
report (not appropriate to the scheme)
  c) The 1/5 sub-CAs for which KPMG identified Symantec as having later
received an audit report for.

If 2 is correct, then it's unclear of which set Aetna belongs to - that is,
if NTT Docomo is 2a, then Aetna is either 2b/2c, and it suggests that KPMG
may have been incomplete in its examination of the 2a set. If NTT Docomo is
2b, then Aetna is either 2a/2c, but calls into question Symantec's
operations if they were themselves operating this root, as it was not part
of the scope of the audit. If NTT Docomo is 2c, then Aetna is either 2a/2b,
both of which would call into question KPMG. Any of these possibilities is
quite troubling, but nowhere near as troubling as the possibility of 1,
which would imply an undisclosed sub-CA.

Based on the information provided by Unicredit, Unicredit would appear to
be 2b, because it was not performed by a licensed WebTrust practitioner to
the appropriate standards. Based on the information provided, Aetna would
seem to be 2c, but that would require confirmation from Symantec or KPMG.
This means that NTT Docomo is either 2a or 2c - either of which should be
concerning.

Independent of any questions regarding how other CAs (such as the
critically mismanaged Omniroot program) responded to disclosure, the
questions about the scope of "which sub-CAs were examined by KPMG" is very
much relevant to the discussion at hand, and gets to the heart of whether
or not there can be sufficient confidence to trust the existing set of
certificates. This also sets aside the question about whether or not
Symantec can/should be trusted going forward. It also highlights the limits
of relying on a report such 

Re: Symantec Issues List

[Peter Bowen via dev-security-policy]

On Fri, Mar 31, 2017 at 4:38 PM, Ryan Sleevi via dev-security-policy
 wrote:
> On Fri, Mar 31, 2017 at 2:39 PM, Gervase Markham wrote:
>
>> As we continue to consider how best to react to the most recent incident
>> involving Symantec, and given that there is a question of whether it is
>> part of a pattern of behaviour, it seemed best to produce an issues list
>> as we did with WoSign. This means Symantec has proper opportunity to
>> respond to issues raised and those responses can be documented in one
>> place and the clearest overayll picture can be seen by the community.
>
> (Wearing a Google hat)

(Wearing my normal personal non-work hat)

> In March of last year, Symantec provided us a list of five sub-CAs which
> they termed GeoRoots: Apple, Google, Unicredit, Aetna, NTT Docomo - and
> requested they be excluded from this requirement. We asked Symantec to
> provide current audit statements for each of these CAs.
>
> Symantec indicated that the audit information for these sub-CAs would be
> added to the CCADB. This was on 3/29.
>
> We then followed-up with Symantec, again, because as of 6/28, there were
> several outstanding issues with Symantec's disclosures:
>
> - Apple IST CA 3 was not covered by the general set of Apple audits
> - No audit information for Aetna was provided, and its CPS was dated in 2011
> - No audit information for Unicredit was provided
> - NTT Docomo (DKHS and DKHS CA2) were disclosed as being part of Symantec's
> audit
>
> Upon follow-up, Symantec provided Aetna's WebTrust for BRs audit. On it,
> there were 15 qualifications, some of which would have spanned the totality
> of operation.
>
> Regarding Unicredit, Google requested that Symantec place us in direct
> contact with Unicredit. We had several calls with Unicredit's management
> team regarding the issues, attempting to find a path to see if they would
> be able to complete a Baseline Requirements audit.
>
> I want to share these details so that a fuller picture of the GeoRoot
> issues can be noted. Particularly concerning is the seriousness of the
> Aetna issues and the failure to remedy them, and the failure to identify
> the NTT Docomo (DKHS) roots as part of Symantec's infrastructure.
(some portions of the quoted text omitted)

Ryan,

I haven't reviewed the audit reports myself, but I'll assume all you
wrote is true.  However, I think it is important to consider it in the
appropriate context.

The GeoRoot program was very similar to that offered by many CAs a few
years ago.  CyberTrust (then Verizon, now DigiCert) has the OmniRoot
program, Entrust has a root signing program[1], and GlobalSign Trusted
Root[2] are just a few examples.

In almost every case the transition to requiring complete unqualified
audits of the subordinates by a licensed practitioner was a rocky one.
See DigiCert's thread
(https://groups.google.com/d/msg/mozilla.dev.security.policy/tHUcqnWPt3o/U2U__7-UBQAJ)
about the OmniRoot program or look at the audits available for some of
the Entrust subordinates.

I'm not suggesting that the GeoRoot subordinate issues should not be
considered, but it seems the GeoRoot program was not notably
exceptional a few years ago.

Thanks,
Peter

[1] 
https://web-beta.archive.org/web/20140818191044/http://www.entrust.net/about/third-party-sub-ca.htm
[2] https://www.globalsign.com/en/certificate-authority-root-signing/
and 
https://web-beta.archive.org/web/20101008151742/http://globalsign.com/certificate-authority-root-signing/
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Symantec Issues List

[Ryan Sleevi via dev-security-policy]

On Fri, Mar 31, 2017 at 2:39 PM, Gervase Markham via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> As we continue to consider how best to react to the most recent incident
> involving Symantec, and given that there is a question of whether it is
> part of a pattern of behaviour, it seemed best to produce an issues list
> as we did with WoSign. This means Symantec has proper opportunity to
> respond to issues raised and those responses can be documented in one
> place and the clearest overayll picture can be seen by the community.
>
> So I have prepared:
> https://wiki.mozilla.org/CA:Symantec_Issues
>
> I will now be dropping Symantec an email asking them to begin the
> process of providing whatever comment, factual correction or input they
> feel appropriate.
>
> If anyone in this group feels they have an issue which it is appropriate
> to add to the list, please send me email with the details.
>

(Wearing a Google hat)

Gerv,

Thanks for organizing this information, as much of it was related to and
relevant to Google's recent announcement. I want to take this opportunity
to share additional details regarding the interactions for UniCredit, which
I believe may be useful and relevant both for understanding that issue and
the GeoTrust audits.

As the Chrome team announced at
https://security.googleblog.com/2015/10/sustaining-digital-certificate-security.html
, we made steps to require that all Symantec-issued certificates be
disclosed via Certificate Transparency.

In March of last year, Symantec provided us a list of five sub-CAs which
they termed GeoRoots: Apple, Google, Unicredit, Aetna, NTT Docomo - and
requested they be excluded from this requirement. We asked Symantec to
provide current audit statements for each of these CAs.

Symantec indicated that the audit information for these sub-CAs would be
added to the CCADB. This was on 3/29.

We then followed-up with Symantec, again, because as of 6/28, there were
several outstanding issues with Symantec's disclosures:

- Apple IST CA 3 was not covered by the general set of Apple audits
- No audit information for Aetna was provided, and its CPS was dated in 2011
- No audit information for Unicredit was provided
- NTT Docomo (DKHS and DKHS CA2) were disclosed as being part of Symantec's
audit

Upon follow-up, Symantec provided Aetna's WebTrust for BRs audit. On it,
there were 15 qualifications, some of which would have spanned the totality
of operation. If Symantec is not willing or able to provide this audit, we
would be happy to, in the interest of public transparency.

This audit was dated May 11, 2016, but covered the period January 1, 2015
through December 31, 2015. I highlight this, because this means it was
provided 132 days after the close of the period, or 42 days after provided
for by the Baseline Requirements. This audit was performed by Symantec's
auditors, KPMG, and thus demonstrates the pattern of delayed audits that
KPMG has shown a tendency for, and extends beyond just Symantec.

I want to highlight some of the qualifications on this audit:
"It was noted that:
 physical access to the cage housing the CA system was not logged;
 event logs for the CA prior to July 2015 were not available; and
 logs are not reviewed periodically."
"It was noted that no training programs were in place for Trusted Role
personnel."
"It was noted that a security risk assessment of the Aetna CA operations
was not performed during the examination period."
"It was noted that a security risk assessment of the Aetna CA operations
was not performed during the examination period."
"It was noted that penetration testing was not performed on the PKI
environment during the examination period."

Symantec's proposed remediation for this was to allow their contract to
expire, and proposed revoking this certificate in October. Note that this
was still nearly 3 months later.

Regarding NTT Docomo, Symantec repeatedly asserted they controlled issuance
for these CAs. However, I highlight this, because the Geotrust audits note
5 sub-CAs, so the fifth sub-CA, if not NTT Docomo, remains unknown and
unidentified by Symantec during the same time as their audit. If NTT Docomo
was issued and managed by Symantec, then their auditor (KPMG) did not
examine this.

Regarding Apple's IST CA 3 (and subsequently issued by Symantec, IST CA 8 -
G1), Apple requested and Google accepted a discussion with their CA
operations team. After discussions with the team, we received suitable
assurances from Apple that these two CAs were operated in accordance with
their CP/CPS and would be part of the next audit. As has been discussed in
other threads, there is a natural period of time where the scope of the
audit does not cover any newly issued intermediates, but we ensured that
these were listed within Apple's next CP/CPS, and thus scoped for the next
audit. Because of this reason alone, we excluded these sub-CAs from our CT
requirement.

Regarding Unicredit, Google requested that 

Symantec Issues List

[Gervase Markham via dev-security-policy]

As we continue to consider how best to react to the most recent incident
involving Symantec, and given that there is a question of whether it is
part of a pattern of behaviour, it seemed best to produce an issues list
as we did with WoSign. This means Symantec has proper opportunity to
respond to issues raised and those responses can be documented in one
place and the clearest overayll picture can be seen by the community.

So I have prepared:
https://wiki.mozilla.org/CA:Symantec_Issues

I will now be dropping Symantec an email asking them to begin the
process of providing whatever comment, factual correction or input they
feel appropriate.

If anyone in this group feels they have an issue which it is appropriate
to add to the list, please send me email with the details.

Thanks,

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy