Ryan, Thanks for providing the update. One area that I do need to push back on is the disclosure of the 100K certificates mentioned.
As demonstrated through past CA distrust discussions and whose need is evidenced by past incident reports, one of the purposes of having CAs disclose the affected certificates is to provide assurance to the community that the CA has completed a rigorous investigation of the issue. While I realize that, as a systemic issue, it’s easy to highlight that it is 100% of issuance from a given CA, in the goal of ensuring consistent incident reporting, please provide the following necessary and required information: - The full list of certificates affected (past CAs, when faced with such large volumes due to systemic issues, have used CSVs with the necessary information) - The full list of certificates not revoked according to the BR timeline (again, CSVs suffice) Can you also confirm that, based on the information provided so far, that it is accurate to say that 100% of these certificates were issued to a single organization? As the discussions around revocation captured, because the needs and nature of BR revocation violations can vary across organizations, separate incident reports should be filed for each affected customer. It would appear that there is only one customer (including Affiliates) - Google - but I do not see that explicitly stated. The use of subscribers (plural) leaves it ambiguous as to whether there have been multiple Subscriber Agreements/Terms of Use, or whether that is simply accounting for different teams within Google and it’s Affiliates under the auspices of a single Agreement/Terms of Use. On Tue, Mar 5, 2019 at 8:08 PM Ryan Hurst via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Sleevi, > > Thanks you for the links to both the reporting requirements and the > underscore issue with DigiCert. > > Regarding the statement about the severity of the issue, it was not > intended to diminish the non-compliance. Instead it was an attempt to frame > the issue with sufficient context to help others who follow this thread > answer the question of impact on the community. > > As for the incident response, we have been making every effort to gather > complete and accurate information to enable us to provide a useful and > actionable incident report. This has unfortunately taken longer than we had > hoped. We now have that report ready and you can find it below. > > You are right that some affected certificates could not be revoked within > the 5 day requirement. In the last section of the report you'll find more > information about the reasons for this along with our plan for revoking the > remaining certificates. > > Ryan Hurst > Google Trust Services > Product Manager > > > Summary > --- > Some certificates issued by GTS utilize EJBCA and as a result had serial > numbers with an effective entropy of 63 bits. These serial numbers were > created from a 64 bit CSPRNG output and were believed to be in compliance > with Section 7.1 of the Baseline Requirements. Upon closer investigation we > learned that EJBCA’s logic for serial number generation only selected > output numbers having a leading 0 bit, which reduced their effective > entropy to 63 bits. > > Though GTS agree that the issuance of the certificates based on the above > behavior qualifies as missisuance, we also believe that this issue does not > represent a material security risk to the community. > > To ensure that all of our certificates comply with the community’s > interpretation of the Baseline Requirements (BR) we have updated the > associated EJBCA CAs to mitigate the problematic behaviour. > > At this time approximately 95% of the affected certificates have been > replaced and revoked. The remaining certificates expire over the next 3 > months. > > We are actively working with the subscribers of these remaining > certificates to facilitate a replacement with the goal of minimizing > disruption of services. Should this not be possible, we will revoke these > certificates no later than 2019-03-31.. > > Certificates issued from non-EJBCA CAs have been checked and are not > affected. > > Incident Report > --- > > 1. How your CA first became aware of the problem > We have been following the thread discussing Dark Matter’s root inclusion > request. When concerns regarding the EJBCA serial number generation logic > were raised, we analyzed the behaviour of our EJBCA installations and found > that they were affected as well. > > 2. A timeline of the actions your CA took in response. > > 2019-02-22 - A thread on m.d.s.p. mentions serial entropy issue of Dark > Matter certificates. > 2019-02-26 - GTS begins reviewing the serial number generation behaviour > of its CAs. > 2019-02-27 - A third-party reports that serial numbers in all certificates > issued from a specific GTS CA have a leading bit of 0 and suggests that we > may have the same issue as Dark Matter. > 2019-02-27 - GTS requests clarification from PrimeKey. It is confirmed > that EJBCA serial generation logic causes the issue and that in order to > create compliant serial numbers, the logic has to be replaced. > 2019-02-27 - The associated CAs used an earlier version of EJBCA where the > serial number logic was not configurable. As a result code from a newer > version of EJBCA that supports configurable serial number length is > backported and configured to use 16 byte serials. > 2019-02-28 - The backported code is deployed to production. > 2019-02-28 - Ongoing discussion on m.d.s.p. revolves around interpretation > of Section 7.1 BR. A consensus emerges that the affected certificates must > be considered to have been misissued. > 2019-02-28 - We inventory the number of certificates issued since Section > 7.1 BR went into effect in September 2016, the number that were currently > valid as well as their validity period. The results are provided in the > section on remediation actions below. > 2019-03-01 - GTS decides to replace and revoke all affected certificates. > Customers are contacted to work out revocation plans. > 2019-03-01 - Issuance of replacement certificates begins. > 2019-03-02 - A first notification is posted to m.d.s.p > 2019-03-04 - Certificate revocation begins. > 2019-03-05 - An update on progress is posted to m.d.s.p > 2019-03-05 - This post mortem is posted to m.d.s.p > > 3. Whether your CA has stopped, or has not yet stopped, issuing > certificates with the problem. > GTS has stopped using the incorrect serial number generation logic. As of > 2019-02-28, all GTS certificates issued from its EJBCA CAs have serial > numbers with at least 64 bits of entropy. > > 4. A summary of the problematic certificates. > All certificates issued from GIAG3 (https://crt.sh/?id=109354897, > https://crt.sh/?id=158511650) between 2016-09-30 and 2019-02-28 were > affected. > > 5. The complete certificate data for the problematic certificates. > Given the large number of certificates (> 100k) they are not enumerated > here. All certificates issued from the CAs mentioned above are affected. > The CA certificates themselves are not affected. All affected certs were > logged to multiple CT logs. > > 6. Explanation about how and why the mistakes were made or bugs > introduced, and how they avoided detection until now. > GTS relied on the affected EJBCA CA to generate 64 bit long certificate > serial numbers as per its configuration and relied on EJBCA to perform this > serial number generation in a BR compliant manner. > > Since the serial numbers had the expected length, the leading bit of 0 was > not discovered as limiting the serial number space. > > Regular internal audits examined the technical profile of samples of > issued certificates but did not detect the issue, because the audit was > performed on a certificate-by-certificate basis. A comparison of serial > numbers across a larger certificate population would have been required to > identify it. Such tests were not implemented for the affected EJBCA CA > because it is a legacy system scheduled for decommissioning later this year. > > 7. List of steps your CA is taking to resolve the situation and ensure > such issuance will not be repeated in the future. > > The issue only affects certificates from one of our legacy EJBCA CAs. Most > of the GTS issuance volume has previously been migrated to a bespoke CA > that is not affected. > > Since the affected CA was patched with updated logic, it is issuing with > 16 byte serials. At the end of August 2019, it will be decommissioned. > > An inventory of the affected certificates provided the following: > > Total number issued since September 2016 - 100,836 > Total valid as of 2019-02-28 - 7,171 > Expiring within 90 days - 7,137 > > Revocation Schedule > --- > > Revocation was performed with the objective of meeting the timeline > prescribed for cases that fall under the second paragraph of Section 4.9.1 > BR (5 days). > > by 2019-03-04 : 6,199 (86.5%) > by 2019-03-05 : 575 (8%) > > The remaining 397 certificates (5.5%) will expire or be revoked no later > than 2019-03-31. > > For 86.5% of the affected certificates the BR revocation timeline could be > met. > > 954 certificates could not be replaced within the five day target without > causing outages or other business issues. Revoking these certificates > without providing adequate time for the associated subscribers to replace > the certificates certificates without providing adequate time for the > associated subscribers to replace the certificates would have caused > significant business damage to the respective subscribers. Our goal being > to revoke all certificates as quickly as possible while minimizing > disruption on the subscribers and the users of their services. > > We are actively working with remaining subscribers who have affected certs > which have not been revoked to replace their certificates in the shortest > time window possible. The 397 remaining certificates will expire or be > revoked by 2019-03-31. > > _______________________________________________ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
