Hello, I've been watching the Entrust saga of issues over the past month and keeping an eye on their incident response like many in the community. Given that, I was curious how the public could reasonably audit a non-compliant CA and ascertain how honest their claims were, as well as what bottlenecks exist for public analysis.
As part of an incident response CAs generally provide a list of impacted certs in the form of SHA256 sums and a link to crt.sh for each certificate. Additionally each certificate will need revoked and we have a Certificate Revocation List to check timestamps and reasons. In order to audit we must be able to take a list of crt.sh SHA256 links, and turn them into something we can check against a CRL: serial numbers. This is the biggest barrier currently. Note that there is a limitation in obtaining only the latest copy of a CRL as they generally do not include expired domains and are trimmed automatically. To reflect this limitation this dataset will be working on a CRL obtained 2024-04-20 and 2024-04-23 with merged de-duped records. The vast majority of certificates issued by Entrust lie in the 3-month period going by samples. *Proof of Concept 1: Entrust: CPS typographical (text placement) error* Original Issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1890896 Impacted Certs: 6008* The naive approach for a proof of concept is to check crt.sh's postgres db, or website for each url and grab them one-by-one to extract the serial number. In practice this took over 20 hours - noting that there isn't a good mechanism for checking in bulk and this is not a shortcoming of crt.sh as a public service. Thank you for your work Sectigo. Next I grabbed the most relevant CRL (L1K) and parsed it to provide a lookup table - but only from 2024-03-12. This date was chosen as it is a day before their major issues recently started snowballing. A decision was made to give a cutoff at 2024-04-23 08:00:00 UTC applying to all future notes. Combining both of these aspects onto a single cursed google sheet gets us some brief analysis and should suffice as a proof of concept. Full google sheet: https://docs.google.com/spreadsheets/d/1QuwPJh0sCHQCQaQEtUsr9qw_QRlf8fpwf8B7W6MPOqw/edit?usp=sharing Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vT8ulLxKWSdCo80Xqf5UyAaqznIOcGDl_TEd2T2ayCZbaItnWfvLs4AC_F8BYmI16R-F1UgHJiWgu5J/pubhtml NOTE: Avoid CPS_6K_List_56K_WARNING (5,995 rows), and CRL_L1K_56K_WARNING (12,013 rows) This issue in particular is one where Entrust insist they will not revoke so it's a good testbed for a worst-case scenario. There is a breakdown on just the 6,008* cert batch, and also on the CRL attached to 99% of their certificates. *Caveat: This was only checking Entrust's L1K CRL, and it turns out there are 35 certificates from L1F unaccounted for in this preliminary test. Additionally there are also 14 certificates not logged on crt.sh despite being on that list for inclusion.[1] With that in mind we can see a pattern to their revocation at the CRL level: 4am is an automated system for processing, they generally only take requests/process them Monday-Friday. The first time we breach over 500 revocations per-day occurs 2024-04-19 (Friday). *Proof of Concept 2: Entrust's Publicly Listed CRLs* Given the sample above and handling one of their largest pools went fine I expanded the test to all reasonably available CRLs and every entry. Given Entrust's scale this should test the limits of Google Sheets and provide a wider picture. NOTE: Avoid CRL_DATA_56K_WARNING (86k rows) Full google sheet: https://docs.google.com/spreadsheets/d/1V2vLoOdD5CxDeCfVbTFHP_8HWYCViS9wo5Q23VWDJPE/edit?usp=sharing Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vQ9RhMAJVmaBWZ_chwUIK_KYOMgERe-PZH2MbZhkS16EDJiOkRJJVO8QnCfdbbGcak6EEI_wvIT2ak_/pubhtml The advantage of this approach is that we can take a step back from checking an individual certificate against each CRL. If a CA is responding to an incident where they must revoke 10k certicates within 5 days, then we should see approximately 10k revocations in a short time period, no? While this has the full data I've also trimmed CRL_GRAPH to show from 2024-03-12 onward. There have been 24,154 revocations across every CRL, but there's an unusual spike this weekend (see linked image below or the google sheets link). 12,012 on L1K - the main OV intermediary, and 11,370 on L1M - the main EV intermediary. CRL Graph: https://i.imgur.com/xN3U8O3.png Between 04-20 and 04-22 on L1K (OV) accounts for 710 revocations, while L1M (EV) accounts for 2,598 revocations alone. The weekend is normally quiet however EV certificates started being handled differently, but that isn't the topic of this post. Now we can check these revocations against a single larger issue and get an idea of what's happening. Notably we won't be checking each certificate given the SHA256<->serial bottleneck mentioned above for a smaller separate pool. https://bugzilla.mozilla.org/show_bug.cgi?id=1883843#c32 Entrust: EV TLS Certificate cPSuri missing 26653 pre-certs provided, confirmed as mis-issued by Entrust 2024-03-05 but only action started 2024-03-18. Given these are EV certificates we know they will be primarily from intermediary L1M (and sampling confirms this). As noted early L1M has had 11,370 revocations since 2024-03-12, or if we count from Entrust acknowledging revocation as necessary (2024-03-18) a total of 11,268. This is imperfect as Entrust have decided that expiry is the same as revocation, and includes certificates issued outside of the affected batch. Additionally CRLs are pruned for expired certificates, however given the issue was from 2024-03-22 and a CRL from 2024-04-20 is sampled these should be accounted for. Looking at a statement made 2024-04-15 (https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c10) we are told 8,572 revoked or expired. The CRL for L1M at that time notes from 2024-03-12 to 2024-04-15 6,989 revocations (inclusive). From acknowledgement we get 2024-03-18 to 2024-04-15 with 6,887. Regardless both would be approximately 80% revoked:expired read generously. The latest statement (https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c11) by Entrust on revocation and expiring (2024-04-19) notes 10,013 revoked/expired. Against the total pool of 26,653 requiring revocation within 5 days we get 37.5%. This is noting a ramp-up in revocations is starting, but given the information provided this could be a separate issue unrelated to this batch. *Closing Remarks* There is room for a more thorough analysis by someone far more talented, but I hope this brief picture gives enough of an idea of what can be done in the future. *Takeaways* - We need a better method for the public to transform SHA256 certs en-mass to serial numbers to check CRLs - We need CAs to be more forthcoming with breakdowns of issuing intermediaries for each batch of certificates requiring revocation - There isn't any big technical barrier or processing issue to ascertaining a CA's compliance in an ongoing issue from someone with better tooling - There is further research required in checking for undisclosed issues via CRLs and encouraging transparency - A CA shouldn't collapse in their public trust to evoke this level of public research *Baseline Requirements Notes:* 4.10.1 Operational characteristics Revocation entries on a CRL or OCSP Response MUST NOT be removed until after the Expiry Date of the revoked Certificate. 7.2 CRL profile nextUpdate Indicates the date by which the next CRL will be issued. For CRLs covering Subscriber Certificates, at most 10 days after the thisUpdate. For other CRLs, at most 12 months after the thisUpdate. revokedCertificates MUST be present if the CA has issued a Certificate that has been revoked and the corresponding entry has yet to appear on at least one regularly scheduled CRL beyond the revoked Certificate’s validity period. The CA SHOULD remove an entry for a corresponding Certificate after it has appeared on at least one regularly scheduled CRL beyond the revoked Certificate’s validity period. See the “revokedCertificates Component” table for additional requirements. *Possible CRL Issue:* While compiling this I noticed that 'Entrust Enterprise Intermediate CA-ICA1' isn't refreshing the CRL daily (same CRL 2024-04-20 and 2024-04-23): 2048.crl: Last Update: Apr 5 15:55:58 2024 GMT Next Update: Apr 5 15:55:57 2025 GMT I presume that'd still cover: https://crt.sh/?caid=32 which has 13 unexpired certs, but this may be a misinterpretation on my behalf. *[1]*: Certificates not on crt.sh but listed in 20240409-OV-TLS-Certificates-typo-finalcerts: ----- https://crt.sh/?sha256=1086E495E6D4CDC926CDFE3C3F083272D70BE65F3BF75336393C6975A11E6D8A https://crt.sh/?sha256=125E7D5030D1B4C57DFF13DAD4E9525024DEC3649F77745C906C229765E24F5F https://crt.sh/?sha256=4323087C85F6FB5F7F2D8E66D9C5EC8BC4007D37CBF93ACC159D6173D0793AFA https://crt.sh/?sha256=637D768E1D386ACDEBED435C84B0B08C2C6B09B840045A34503DB518645657B5 https://crt.sh/?sha256=692991CF2926851395CE4C15A37AC4B6AD13D46979E5A7912F39FDD9FFFAC4A9 https://crt.sh/?sha256=749109CEAEE9CF2D88E6999D4768697696E7C67E4C4DB2648A19137A5D1669ED https://crt.sh/?sha256=76ADEF1237C4B07652D8A597337580947E6B6E1A83F3D25332BE3863BA180CAC https://crt.sh/?sha256=7DD2FF3533BD7C64DEA51E89F469EDD29B007C5DC4708597A9F0594B76954EE1 https://crt.sh/?sha256=AF4274B727520335A80F82C0E615EFB87D37C5F03971AFDF9114ACBCFA5882EA https://crt.sh/?sha256=C5E3974E1DBE2A36450B662A6867988B31361F9DFC67AC4ACA38B386B43432F5 https://crt.sh/?sha256=EF5F20F6B5913C3A5F9477E2BE2D6801D85DD80B9BB42A5363AA3B1E82566FD9 https://crt.sh/?sha256=F5455CBE0B50EC0002A5C1F83AC49CC38842B30D84B830D8A23257AEFA406117 https://crt.sh/?sha256=F7F70E1FD16BB7B09A21486AAC9295CFD8259BB1C6EF363D5A61C1230B1E9426 https://crt.sh/?sha256=FE5BEA6F8D4333CFEFA3277AC2F4D544987FFF7C3ED83514EAF92A34919F19E6 ----- - Wayne -- You received this message because you are subscribed to the Google Groups "dev-security-policy@mozilla.org" group. To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-policy+unsubscr...@mozilla.org. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/c5c77c36-b48d-46b1-b14b-c13b07ef1a31n%40mozilla.org.
