Hi all,
On Friday, Sep 15, we discovered that 1090 certificates were rekeyed using expired domain validation documents. In each case, the original certificate's domain was properly verified at time of issuance using an approved method. Organization verification properly completed for each rekey prior to issuance. This means, to get a rekeyed certificate with expired validation data the following had to be true: A. Unexpired organization validation information, B. A previously issued, unexpired, and unrevoked certificate for the same domain as the rekeyed certificate, C. Valid domain documentation for the original certificate at time of issuance, and D. All necessary access and permissions on the account issuing the original certificate. In parallel to the on-going investigation, we are reverifying the affected domains using one of the approved domain methods. So far, we have reverified 1021 of the 1090 affected certificates, leaving 69 domains outstanding. We are working with the domain owners to re-establish domain approval, but plan to revoke any certs that cannot be revalidated by end of this week. 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date. [DC] We first became aware that there was a potential issue on Sep 14th during part of a routine internal review. We noticed some of the domain control validation documents were older than permitted under both the Baseline Requirements and EV Guidelines. On Sep 14th, we confirmed that rekeys in a validation system did not follow the proper path for verification of the timestamp associated with domain documents, instead using the timestamp of the domain documents on the original certificate. We patched our system early Sep 15th, preventing the system from further re-using expired domain documentation to rekey a certificate. Over the weekend, we worked on this report while revalidating domains associated with the misissued rekeys. 2. A timeline of the actions your CA took in response. [DC] The same day the staff reported the potential issue, we began investigating the root cause. On Sep 15th, we confirmed that older validation code improperly reused domain validation when rekeying a certificate. We thought we patched this after the CAB Forum's previous discussion on what constitutes a new certificate. If you recall those discussions, DigiCert mistakenly believed rekeyed certificates were considered the same certificate, not requiring domain reverification. The original system believed duplicate certificates and rekeyed certificates were identical to the original cert (despite a new serial number) and relied on a previously completed validation outside of the permitted reuse period. Ultimately, we decided to amend our issuance process because of the CAB Forum discussions but failed to properly communicate this requirement internally. Although most workflows were patched to follow our "new issuance" process, one workflow path was not. Validation documents properly expired in the system, preventing issuance of "new" certificate orders using expired documentation. All organization documents expired in the system according to the proper guidelines, blocking both new issuance and issuance of rekeyed certificates. 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem. [DC] Confirmed. The system was patched on Sep 15th to require valid domain validation prior to rekey. This was within 24 hours of receiving an internal message about the issue and within two hours of confirming the issue's existence. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. [DC] 1090 certificate rekeys are known. Impacted certificates will be provided in the bug list and will also be available through CT (we're still working on this part). 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. [DC] Provided as an excel file on the bug list (once compiled). 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. [DC] The issue was intentionally introduced during DigiCert's earlier days because of a mis-understanding by the original DigiCert team about whether rekeys constituted new issuance rather than duplication of a previous certificate. We thought we fixed the issue back when the CAB Forum discussed this topic, but apparently missed a work flow path. One mitigating factor is that all rekeys were initiated at the request of a previously verified and currently active account holder associated with the domains and a currently active certificate holder. Although the certificates had improper documentation, there was an active cert with the same name in each case. Also note that all rekeys have the same expiration date as the original certificate, meaning the rekeyed certificate would expire the same time as the currently active, non-revoked certificate. Over the weekend, we revalidated 1021 domains, limited the non-verified pool to 69 domains. The error avoided detection because there was a misunderstanding by former DigiCert staff that all certificates, regardless of whether the certificate is a rekey, duplicate, modification, etc, requires domain verification completed within the established BR timeframe. The new team discovered this exception for rekeys during their routine code review. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. [DC] Everyone is now fully aware that all certificates are "new" certificates, meaning they follow the same process regardless of whether the certificate is a duplicate, rekey, or modification. We've patched the system to remove this alternate work flow and prevent additional issuance without proper validation. We are currently revalidating the 1090 certificates. We are also deprecating this validation service in favor of a new, robust system. Although the system is still heavily used, a complete switch to DigiCert's v3 validation services will occur early next year. Jeremy
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
