Thank you for the detailed incident report Scott. I have created https://bugzilla.mozilla.org/show_bug.cgi?id=1531800 to track this issue, and attributed it to QuoVadis as the responsible root CA program member.
On Thu, Feb 28, 2019 at 4:43 PM Scott Rea via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > This incident report relates to the 64-bit serial numbers in all > certificates that DarkMatter CAs have issued since their inception. The > dialog surrounding CABF Ballot 164 “Certificate Serial Number Entropy” was > unknown to DarkMatter until shared with us recently by Ryan Sleevi of > Google, and demonstrated to DM that the industry expected at least 64-bits > of entropy in resulting serialNumbers despite the actual wording of the BRs > Section 7.1. > > 1. How your CA first became aware of the problem (e.g. via a problem > report submitted to your Problem Reporting Mechanism, a discussion in > mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and > the time and date. > > A post by Corey Bonnell to the mozilla.dev.security.policy group on > February 23, 2019 at 1:28am UAE time. Corey referenced Section 7.1 of the > Baseline Requirements and the specific sentence he believed DM was > violating: “Effective September 30, 2016, CAs SHALL generate non-sequential > Certificate serial numbers greater than zero (0) containing at least 64 > bits of output from a CSPRNG”. Corey also listed 23 certificates (CAs & > EE’s) and their serialNumbers as evidence. Corey followed this up 2 days > later with a more comprehensive list that included DMs pre-certs and final > certs published to various CT Logs. Corey also pointed out a second issue > in respect to nameConstrained intermediates issued by Quo Vadis to DM as > also a potential violation, but this shall be dealt with in a separate > incident report, since DM was not the issuer of these ICAs. Corey’s second > post was received on the list at 12:50am UAE time February 25, 2019. > > 2. A timeline of the actions your CA took in response. A timeline is a > date-and-time-stamped sequence of all relevant events. This may include > events before the incident was reported, such as when a particular > requirement became applicable, or a document changed, or a bug was > introduced, or an audit was done. > > 29-Apr-16: DM having previously engaged with QuoVadis for the > establishment of a National PKI utilizing a platform and hardware that > would initially be located in QV DCs, held key ceremonies for Private PKIs > on EJBCA platform. Platform was configured for random 64-bit serialNumbers > which at the time was considered far in excess of the then current BR > requirement of 20-bit entropy. > 30-Apr-16: QV creates DM intermediates intended for issuance of EV and OV > public trust TLS. These intermediates are instantiated in separate > partitions to the DM Private Trust PKIs, but issuance from them is still > subject to the same platform setting of 64-bit random serialNumbers. > 08-Jul-16: CABF Ballot 164 closes and passes. The phrase Corey highlighted > above is added in place of the previous 20-bit entropy statement. > 28-Mar-17: Transition of DM private PKIs and platform to DM DC’s is > complete under supervision of EY as WebTrust Auditors. Public Trust CAs > continue to operate out of QV DCs. > 08-Jun-17: DM Point In Time Audit is complete. NOTE: DM is now responsible > for platform configuration in compliance with BRs. DM considers current > serialNumber setting of platform (64-bit SNs) as compliant with BRs Section > 7.1 > 27-Oct-17: DM successfully completes WebTrust Period In Time. > 04-Nov-17: QV & DM complete transfer of DM public trust intermediates from > QV to DM under EY & KMPG audit observation > 18-Dec-17: DM & UAE Roots submitted to Mozilla > 02-Oct-18: DM successfully completes 2nd WebTrust Audit > 06-Nov-18: DM submits new WebTrust Audits to Mozilla > 23-Feb-19: 1:28am: Corey posts to MDSP claiming DM has mis-issuance of > certs due to BRs Section 7.1 > 23-Feb-19: 3:36am: Post is observed by DM, internal investigation > requested immediately. > 23-Feb-19: 6:14pm: Internal investigation confirms 64-bit setting for all > certificates, certlint/cablint/zlint do not complain about certs submitted > with this setting. Confirm that settings appear to be compliant with BRs. > Further validation sought from platform provider, support ticket opened. > 25-Feb-19: 12:50am: Second email from Corey with expanded list of > pre-certs and issued certs is posted to MDSP > 25-Feb-19: 5:08am: Response posted to Corey on MDSP regarding DM’s > investigation and the conclusion that DM is in compliance with BRs Section > 7.1. NOTE: over next two days several folks weigh in on the 64-bit entropy > vs 64-bit CSPRNG output wording of BRs and what that means. > 26-Feb-19: 7:41pm: Wayne Thayer posts to MSDP that 64-bit serialNumber > with 63-bit entropy are mis-issued under the BRs. > 27-Feb-19:10:55am: DM responds to Wayne reiterating our position of > compliance since BRs only say 64-bit output from a CSPRNG and do not say > anything about entropy. However, DM commits to make a move to 128-bit > serialNumbers from a CSPRNG in next change control to seek to alleviate > concerns. More discussion continues on the list. > 27-Feb-19: 11:55am: Ryan Sleevi posts historical data/discussions links to > MDSP regarding the change resulting from Ballet 164 that detailed where > subsequent discussion was had regarding some CAs that included serials of > exactly 64 bits, and how those were considered that they could not be > compliant, and they made the necessary changes. > 27-Feb-19: 5:00pm: DM, following a review of data/links provided by Ryan, > stops issuance of public trust certificates. Plans are begun for an > immediate migration to 128-bit serial numbers with far exceeding the > required entropy. DM announces decision on MDSP at 5:09pm > 28-Feb-19: 500pm: DM action is past 24 hours: Testing of manual upgrade of > platform without waiting for patch from vendor, roadmap of patch > application so that other profiles i.e. private trust (not TLS public > trust), can later be downgraded back to 64-bit – issuance is stopped for > those profiles until patch can be applied. Change scripts created, tested > and QA on pre-production, validated. Production change control scheduled > and executed successfully. TLS issuance restored in production with > confirmed 128-bit serialNumber from CSPRNG. Full list of still active > certificates with 64-bit serialNumber is generated. Notices to respective > customers are sent, and where possible phone calls made. Revocation and > re-issuance of affected certificates begins for customers who are > responsive. NOTE: Thursday is last day of work week in UAE, Friday & > Saturday are week-end, most businesses open again on Sunday. > 28-Feb-19: 9:30pm: Progress to date: 157 total certificates identified as > still active that will be revoked during this exercise. Currently 45 have > been re-issued. 18 revoked. > > > 3. Whether your CA has stopped, or has not yet stopped, issuing > certificates with the problem. A statement that you have will be considered > a pledge to the community; a statement that you have not requires an > explanation. > Issuance was stopped at 5:00pm 27 Feb 2019. Platform is now upgraded to > 128-bit serialNumbers, and issuance now with larger bit sized serialNumbers > is active. > > > 4. A summary of the problematic certificates. For each problem: number of > certs, and the date the first and last certs with that problem were issued. > all TLS certificates up until 5:00pm yesterday (27 Feb 2019) were issued > with 64-bit serialNumber from 29 April 2016 until 27- Feb 2019. 157 TLS > certificates still active as of yesterday. 45 have now been reissued, 18 > revoked. > > > 5. The complete certificate data for the problematic certificates. The > recommended way to provide this is to ensure each certificate is logged to > CT and then list the fingerprints or crt.sh IDs, either in the report or as > an attached spreadsheet, with one list per distinct problem. > This post will be updated with this info shortly, but please reference > Corey Bonnel’s earlier post of the same > > > 6. Explanation about how and why the mistakes were made or bugs > introduced, and how they avoided detection until now. > CA platform was originally provisioned with 64-bit serialNumber during > period this was explicitly allowed under BRs. When DM migrated private > trust platform and began preparing for public trust issuance through > WebTrust Audit preparations, this was after Ballot 164 had updated the BRs. > DM was not privy to the discussions around Ballet 164 and the subsequent > community discussions. DM analyzed the BRs post Ballot 164 and concluded > based on the definition in Section 7.1, that the platform was complaint > with the BRs. This continued to be our position until Ryan Sleevi posted > historical data regarding the interpretations provided to other CAs after > the adoption of Ballet 164. At this point DM realized that despite what the > BRs said, the general community interpretation of Section 7.1 was > different, and it had been consistently applied to all other CAs in the > community. DM is not seeking an exception, and only wishes to be held to > the same standard as all other CAs in the community, therefore it became > necessary to make an immediate change. It was not bug originally, it was > complaint at the time it was introduced. It avoided detection until now > because the update to the BRs appears to have lost some specificity in it’s > re-wording which existing CAs knew about at the time, but any new CAs > coming after the change may not make the same conclusion without the > benefit of the background information. > > > 7. List of steps your CA is taking to resolve the situation and ensure > such issuance will not be repeated in the future, accompanied with a > timeline of when your CA expects to accomplish these things. > Change is already active – CA now using 128-bit serialNumbers for any new > certificates. There are 157 active TLS certificates at the time that were > impacted and will be revoked. Expected completion for revocations is next > 24 to 48 hours. > > > Regards, > > -- > Scott Rea > > Scott Rea > Senior Vice President - Trust Services > > [cid:image45b117.PNG@3c439482.4391aafb]<http://www.darkmatter.ae> > > Level 15, Aldar HQ > Abu Dhabi, United Arab Emirates > T +971 2 417 1417<tel:+971%202%20417%201417> > M +971 52 847 5093<tel:+971%2052%20847%205093> > E scott....@darkmatter.ae<mailto:scott....@darkmatter.ae> > > darkmatter.ae<http://darkmatter.ae> > > [Linkedin]<https://www.linkedin.com/company/dark-matter-llc> [Twitter] < > https://twitter.com/GuardedbyGenius> > [Year of Zayed] [expo] > > > > The information in this email is intended only for the person(s) or entity > to whom it is addressed and may contain confidential or privileged > information. If you receive this email by error, please notify us > immediately, delete the original message and do not disclose the contents > to any other person, use or store or copy the information in any medium and > for whatever purpose. Any unauthorized use is strictly prohibited. > > _______________________________________________ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
