Thanks Ryan!
Honestly I would prefer things to be clean, but obviously new Root ceremonies
also come at a significant cost...
I am happy to be guided by Kathleen and Co on this to ensure the community
standards are maintained without playing favorites.
But if it becomes necessary for the community to consider past incidents while
considering our case, then it would surely be a service to have such data
available.
I am very grateful for your offer of assistance, and for the previous data
points you also provided that gave us a better understanding of community
consensus that was not apparent from reading the BRs.
Regards,
-Scott
Sent from my iPhone
Scott Rea
Senior Vice President - Trust Services
[cid:image59a03a.PNG@a459c918.459fb124]<http://www.darkmatter.ae>
Level 15, Aldar HQ
Abu Dhabi, United Arab Emirates
T +971 2 417 1417<tel:+971%202%20417%201417>
M +971 52 847 5093<tel:+971%2052%20847%205093>
E scott....@darkmatter.ae<mailto:scott....@darkmatter.ae>
darkmatter.ae<http://darkmatter.ae>
[Linkedin]<https://www.linkedin.com/company/dark-matter-llc> [Twitter]
<https://twitter.com/GuardedbyGenius>
[Year of Zayed] [expo]
The information in this email is intended only for the person(s) or entity to
whom it is addressed and may contain confidential or privileged information. If
you receive this email by error, please notify us immediately, delete the
original message and do not disclose the contents to any other person, use or
store or copy the information in any medium and for whatever purpose. Any
unauthorized use is strictly prohibited.
On Mar 3, 2019, at 11:58 PM, Ryan Sleevi
<r...@sleevi.com<mailto:r...@sleevi.com>> wrote:
This is a question for Kathleen, as Module Owner.
In the past, CAs which have had BR violations in their root certificates - such
as negative serial numbers, improper DER encodings, or RFC5280 violations (such
as keyUsages) - have been required to create new roots before inclusion
progresses. There have also been a few exceptions, depending on the nature of
the non-compliance.
Please let me know if it would be helpful to dig up those past incidents for
such examples.
On Sun, Mar 3, 2019 at 2:47 PM Scott Rea via dev-security-policy
<dev-security-policy@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org>>
wrote:
G’day Folks,
we have updated https://bugzilla.mozilla.org/show_bug.cgi?id=1531800 with the
latest actions taken by DarkMatter
A question I am posing to this list relates to the trust anchors produced with
64-bit serial numbers...
A Root certificate is included by explicit trust, and therefore pre-image
attacks (which is what the increased randomness in serialNumber is meant to
mitigate) do not apply. As such, is it still necessary to create new Root
certificates to update serialNumber for submission to Mozilla?
Regards,
--
Scott Rea
On 3/1/19, 8:23 PM, "dev-security-policy on behalf of Wayne Thayer via
dev-security-policy"
<dev-security-policy-boun...@lists.mozilla.org<mailto:dev-security-policy-boun...@lists.mozilla.org>
on behalf of
dev-security-policy@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org>>
wrote:
Thank you for the detailed incident report Scott. I have created
https://bugzilla.mozilla.org/show_bug.cgi?id=1531800 to track this issue,
and attributed it to QuoVadis as the responsible root CA program member.
On Thu, Feb 28, 2019 at 4:43 PM Scott Rea via dev-security-policy <
dev-security-policy@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org>>
wrote:
> This incident report relates to the 64-bit serial numbers in all
> certificates that DarkMatter CAs have issued since their inception. The
> dialog surrounding CABF Ballot 164 “Certificate Serial Number Entropy” was
> unknown to DarkMatter until shared with us recently by Ryan Sleevi of
> Google, and demonstrated to DM that the industry expected at least 64-bits
> of entropy in resulting serialNumbers despite the actual wording of the
BRs
> Section 7.1.
>
> 1. How your CA first became aware of the problem (e.g. via a problem
> report submitted to your Problem Reporting Mechanism, a discussion in
> mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and
> the time and date.
>
> A post by Corey Bonnell to the mozilla.dev.security.policy group on
> February 23, 2019 at 1:28am UAE time. Corey referenced Section 7.1 of the
> Baseline Requirements and the specific sentence he believed DM was
> violating: “Effective September 30, 2016, CAs SHALL generate
non-sequential
> Certificate serial numbers greater than zero (0) containing at least 64
> bits of output from a CSPRNG”. Corey also listed 23 certificates (CAs &
> EE’s) and their serialNumbers as evidence. Corey followed this up 2 days
> later with a more comprehensive list that included DMs pre-certs and final
> certs published to various CT Logs. Corey also pointed out a second issue
> in respect to nameConstrained intermediates issued by Quo Vadis to DM as
> also a potential violation, but this shall be dealt with in a separate
> incident report, since DM was not the issuer of these ICAs. Corey’s second
> post was received on the list at 12:50am UAE time February 25, 2019.
>
> 2. A timeline of the actions your CA took in response. A timeline is a
> date-and-time-stamped sequence of all relevant events. This may include
> events before the incident was reported, such as when a particular
> requirement became applicable, or a document changed, or a bug was
> introduced, or an audit was done.
>
> 29-Apr-16: DM having previously engaged with QuoVadis for the
> establishment of a National PKI utilizing a platform and hardware that
> would initially be located in QV DCs, held key ceremonies for Private PKIs
> on EJBCA platform. Platform was configured for random 64-bit serialNumbers
> which at the time was considered far in excess of the then current BR
> requirement of 20-bit entropy.
> 30-Apr-16: QV creates DM intermediates intended for issuance of EV and OV
> public trust TLS. These intermediates are instantiated in separate
> partitions to the DM Private Trust PKIs, but issuance from them is still
> subject to the same platform setting of 64-bit random serialNumbers.
> 08-Jul-16: CABF Ballot 164 closes and passes. The phrase Corey highlighted
> above is added in place of the previous 20-bit entropy statement.
> 28-Mar-17: Transition of DM private PKIs and platform to DM DC’s is
> complete under supervision of EY as WebTrust Auditors. Public Trust CAs
> continue to operate out of QV DCs.
> 08-Jun-17: DM Point In Time Audit is complete. NOTE: DM is now responsible
> for platform configuration in compliance with BRs. DM considers current
> serialNumber setting of platform (64-bit SNs) as compliant with BRs
Section
> 7.1
> 27-Oct-17: DM successfully completes WebTrust Period In Time.
> 04-Nov-17: QV & DM complete transfer of DM public trust intermediates from
> QV to DM under EY & KMPG audit observation
> 18-Dec-17: DM & UAE Roots submitted to Mozilla
> 02-Oct-18: DM successfully completes 2nd WebTrust Audit
> 06-Nov-18: DM submits new WebTrust Audits to Mozilla
> 23-Feb-19: 1:28am: Corey posts to MDSP claiming DM has mis-issuance of
> certs due to BRs Section 7.1
> 23-Feb-19: 3:36am: Post is observed by DM, internal investigation
> requested immediately.
> 23-Feb-19: 6:14pm: Internal investigation confirms 64-bit setting for all
> certificates, certlint/cablint/zlint do not complain about certs submitted
> with this setting. Confirm that settings appear to be compliant with BRs.
> Further validation sought from platform provider, support ticket opened.
> 25-Feb-19: 12:50am: Second email from Corey with expanded list of
> pre-certs and issued certs is posted to MDSP
> 25-Feb-19: 5:08am: Response posted to Corey on MDSP regarding DM’s
> investigation and the conclusion that DM is in compliance with BRs Section
> 7.1. NOTE: over next two days several folks weigh in on the 64-bit entropy
> vs 64-bit CSPRNG output wording of BRs and what that means.
> 26-Feb-19: 7:41pm: Wayne Thayer posts to MSDP that 64-bit serialNumber
> with 63-bit entropy are mis-issued under the BRs.
> 27-Feb-19:10:55am: DM responds to Wayne reiterating our position of
> compliance since BRs only say 64-bit output from a CSPRNG and do not say
> anything about entropy. However, DM commits to make a move to 128-bit
> serialNumbers from a CSPRNG in next change control to seek to alleviate
> concerns. More discussion continues on the list.
> 27-Feb-19: 11:55am: Ryan Sleevi posts historical data/discussions links to
> MDSP regarding the change resulting from Ballet 164 that detailed where
> subsequent discussion was had regarding some CAs that included serials of
> exactly 64 bits, and how those were considered that they could not be
> compliant, and they made the necessary changes.
> 27-Feb-19: 5:00pm: DM, following a review of data/links provided by Ryan,
> stops issuance of public trust certificates. Plans are begun for an
> immediate migration to 128-bit serial numbers with far exceeding the
> required entropy. DM announces decision on MDSP at 5:09pm
> 28-Feb-19: 500pm: DM action is past 24 hours: Testing of manual upgrade of
> platform without waiting for patch from vendor, roadmap of patch
> application so that other profiles i.e. private trust (not TLS public
> trust), can later be downgraded back to 64-bit – issuance is stopped for
> those profiles until patch can be applied. Change scripts created, tested
> and QA on pre-production, validated. Production change control scheduled
> and executed successfully. TLS issuance restored in production with
> confirmed 128-bit serialNumber from CSPRNG. Full list of still active
> certificates with 64-bit serialNumber is generated. Notices to respective
> customers are sent, and where possible phone calls made. Revocation and
> re-issuance of affected certificates begins for customers who are
> responsive. NOTE: Thursday is last day of work week in UAE, Friday &
> Saturday are week-end, most businesses open again on Sunday.
> 28-Feb-19: 9:30pm: Progress to date: 157 total certificates identified as
> still active that will be revoked during this exercise. Currently 45 have
> been re-issued. 18 revoked.
>
>
> 3. Whether your CA has stopped, or has not yet stopped, issuing
> certificates with the problem. A statement that you have will be
considered
> a pledge to the community; a statement that you have not requires an
> explanation.
> Issuance was stopped at 5:00pm 27 Feb 2019. Platform is now upgraded to
> 128-bit serialNumbers, and issuance now with larger bit sized
serialNumbers
> is active.
>
>
> 4. A summary of the problematic certificates. For each problem: number of
> certs, and the date the first and last certs with that problem were
issued.
> all TLS certificates up until 5:00pm yesterday (27 Feb 2019) were issued
> with 64-bit serialNumber from 29 April 2016 until 27- Feb 2019. 157 TLS
> certificates still active as of yesterday. 45 have now been reissued, 18
> revoked.
>
>
> 5. The complete certificate data for the problematic certificates. The
> recommended way to provide this is to ensure each certificate is logged to
> CT and then list the fingerprints or crt.sh IDs, either in the report or
as
> an attached spreadsheet, with one list per distinct problem.
> This post will be updated with this info shortly, but please reference
> Corey Bonnel’s earlier post of the same
>
>
> 6. Explanation about how and why the mistakes were made or bugs
> introduced, and how they avoided detection until now.
> CA platform was originally provisioned with 64-bit serialNumber during
> period this was explicitly allowed under BRs. When DM migrated private
> trust platform and began preparing for public trust issuance through
> WebTrust Audit preparations, this was after Ballot 164 had updated the
BRs.
> DM was not privy to the discussions around Ballet 164 and the subsequent
> community discussions. DM analyzed the BRs post Ballot 164 and concluded
> based on the definition in Section 7.1, that the platform was complaint
> with the BRs. This continued to be our position until Ryan Sleevi posted
> historical data regarding the interpretations provided to other CAs after
> the adoption of Ballet 164. At this point DM realized that despite what
the
> BRs said, the general community interpretation of Section 7.1 was
> different, and it had been consistently applied to all other CAs in the
> community. DM is not seeking an exception, and only wishes to be held to
> the same standard as all other CAs in the community, therefore it became
> necessary to make an immediate change. It was not bug originally, it was
> complaint at the time it was introduced. It avoided detection until now
> because the update to the BRs appears to have lost some specificity in
it’s
> re-wording which existing CAs knew about at the time, but any new CAs
> coming after the change may not make the same conclusion without the
> benefit of the background information.
>
>
> 7. List of steps your CA is taking to resolve the situation and ensure
> such issuance will not be repeated in the future, accompanied with a
> timeline of when your CA expects to accomplish these things.
> Change is already active – CA now using 128-bit serialNumbers for any new
> certificates. There are 157 active TLS certificates at the time that were
> impacted and will be revoked. Expected completion for revocations is next
> 24 to 48 hours.
>
>
> Regards,
>
> --
> Scott Rea
>
> Scott Rea
> Senior Vice President - Trust Services
>
> [cid:image45b117.PNG@3c439482.4391aafb]<http://www.darkmatter.ae>
>
> Level 15, Aldar HQ
> Abu Dhabi, United Arab Emirates
> T +971 2 417 1417<tel:+971%202%20417%201417>
> M +971 52 847 5093<tel:+971%2052%20847%205093>
> E
scott....@darkmatter.ae<mailto:scott....@darkmatter.ae><mailto:scott....@darkmatter.ae<mailto:scott....@darkmatter.ae>>
>
> darkmatter.ae<http://darkmatter.ae><http://darkmatter.ae>
>
> [Linkedin]<https://www.linkedin.com/company/dark-matter-llc> [Twitter] <
> https://twitter.com/GuardedbyGenius>
> [Year of Zayed] [expo]
>
>
>
> The information in this email is intended only for the person(s) or entity
> to whom it is addressed and may contain confidential or privileged
> information. If you receive this email by error, please notify us
> immediately, delete the original message and do not disclose the contents
> to any other person, use or store or copy the information in any medium
and
> for whatever purpose. Any unauthorized use is strictly prohibited.
>
>
Scott Rea | Senior Vice President - Trust Services
Tel: +971 2 417 1417 | Mob: +971 52 847 5093
scott....@darkmatter.ae<mailto:scott....@darkmatter.ae>
The information transmitted, including attachments, is intended only for the
person(s) or entity to which it is addressed and may contain confidential
and/or privileged material. Any review, retransmission, dissemination or other
use of, or taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you received this
in error, please contact the sender and destroy any copies of this information.
_______________________________________________
> dev-security-policy mailing list
>
dev-security-policy@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-security-policy
>
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
