Continuing on with this research I've analysed the open issues on bugzilla for recent mis-issuances requiring revocation with more than 100 certificates.
The full dataset is held at: https://docs.google.com/spreadsheets/d/1DmqupuUfx5bQvtm99I6OZZnYj3BL9Eawo1gu0uPST0E/edit?usp=sharing Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vRM7a7yCe3jYJAvrb-dR90iIh5xu3WAbj7EU42r14pgjLS1G7z7W4Vp2jOAesmStkMM6hw4emLKw9-K/pubhtml To reiterate this is checking mis-issued certificate batches against Certificate Revocation Lists. For every mis-issued batch (bar some exceptionally large issues) I've obtained the serial number and checked the CRL to see revocation timeframes and progress at 5 days, 15 days, and 30 days. This is to also to see if there's any data in the murmured proposals of an extension. This information is provided to give an overall picture of compliance across all prominent incidents recently. If there are any questions or corrections feel free to email me. There are a number of caveats per-CA that I'll detail below alongside some figures. Consider this a rough summation given the various fuzzy factors involved. Information was all updated as of 2024-04-27 14:00 UTC *General Caveats:* - There is imprecise wording and different interpretations on when the revocation start timer begins, and how CAs note this in their incident reports. To that end I've separated the time that a CA seems to have internally confirmed the issue, and when they actively started some process of revocation: contacting subscribers or actively revoking. - A number of these issues are still pending revocation and will be updated as more information arrives, consider the spreadsheet a living document This covers 20 issues across 17 CAs - 17 of which get a detailed breakdown of individual certs to revocations: *1: ACCV* Issue: ACCV: Certificates issued with cRLIssuer in CDP extension Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884532 Impacted: 837 Missed at 5d: 0 (0.00%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - An outlier for all surveyed CAs in this dataset: the CRL involved is 24MB and has 537k entries, useful for testing - CA provided serial numbers speeding up this report *2: Actalis* Issue: Actalis: Certificates issued with invalid RDN order Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884532 Impacted: 263 Missed at 5d: 0 (0.00%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - CA provided serial numbers speeding up this report *3: Buypass (1)* Issue: Buypass: TLS certificates with incorrect Subject attribute order Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1864204 Impacted: 591 Missed at 5d: 544 (92.05%) Missed at 15d: 15 (2.54%) Missed at 30d: 2 (0.34%) - There is an extreme outlier (so far) as full revocation time is over 65 days - There isn't a clear start hour for when revocation procedures occurred *4: Buypass (2)* Issue: Buypass: Using an external DNS Resolver for DNS lookups Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1872371 Impacted: 177060 Missed at 5d: 177060 (100%) Missed at 15d: 90576 (51.16%) Missed at 30d: 0 (0.00%) - This is included based off of comments from the CA - No detailed analysis is provided, the scale of this is out of scope *5: Certigna* Issue: Certigna: TLS certificates with Basic constraint non-critical Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883416 Impacted: 3794 Missed at 5d: 1243 (32.76%) Missed at 15d: 1 (0.03%) Missed at 30d: 0 (0.00%) - Note: This is an ongoing incident - There isn't a clear start hour for when revocation procedures occurred - Counting it generously from 2024-03-21 (revocation start) to 2024-03-26 23:59 gives us 1243, when CA reports 1241? *6: certSIGN* Issue: certSIGN: Certificates with incorrect Subject attribute order Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886624 Impacted: 625 Missed at 5d: 306 (48.96%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - Casualty of the fuzziness of the precise revocation start time, the 306 certs are revoked within 68 minutes after the deadline *7: CFCA* Issue: CFCA: certificate basicConstraints extension not marked as critical Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886135 Impacted: 2095* Missed at 5d: 2089 (99.71%) Missed at 15d: 1268 (60.53%) Missed at 30d: 526 (25.11%) - Note: This is an ongoing incident - *Incident provides 2098 SHA256 sums, however this is 2095 de-duped *8: Chunghwa Telecom* Issue: Chunghwa Telecom: Wrong Extended Key Usage setting by GTLSCA Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1887096 Impacted: 6453 Missed at 5d: 6450 (99.95%) Missed at 15d: 6450 (99.95%) Missed at 30d: 6448 (99.92%) - Note: This is an ongoing incident - The CRL is 6315 entries long, and this incident is 6298 of them currently *9: D-Trust* Issue: D-Trust: LDAP-URL in Subscriber Certificate Authority Information Access field Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884714 Impacted: 2601* Missed at 5d: 0 (0.00%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - *2 certificates are not CT-logged and not included in this figure *10: Entrust (1)* Issue: Entrust: EV TLS Certificate cPSuri missing Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 Impacted: 26668 Missed at 5d: 22838 (85.64%) Missed at 15d: 20554 (77.07%) Missed at 30d: 16655 (62.45%) - Note: This is an ongoing incident - This is included based off of comments from the CA - 15d figure is taken from a comment at 14 days - 30d figure is taken from a comment at 31 days - No detailed analysis is provided, the scale of this is out of scope *11: Entrust (2)* Issue: Entrust: clientAuth TLS Certificates without serverAuth EKU Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886467 Impacted: 1176 Missed at 5d: 1076 (91.50%) Missed at 15d: 1062 (90.31%) Missed at 30d: 998 (84.87%) - No caveats *12: Entrust (3)* Issue: Entrust: CPS typographical (text placement) error Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1890896 Impacted: 6008 Missed at 5d: 6008* (100%) Missed at 15d: 6008* (100%) Missed at 30d: 6008* (100%) - CA has refused to revoke as per the Baseline Requirements and their own CPS - No detailed analysis is provided due to this *13: Firmaprofesional* Issue: Firmaprofesional: Policy Qualifiers other than id-qt-cps present for certificate Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1889420 Impacted: 499 Missed at 5d: 1 (0.20%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - No caveats *14: FNMT* Issue: FNMT: Certificates issued included Policy qualifiers other than id-qt-cps Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1875942 Impacted: 712 Missed at 5d: 0 (0.00%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - CA provided serial numbers speeding up this report *15: Hongkong Post* Issue: Hongkong Post: TLS certificates with Certificate Policies extension that does not assert http scheme Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886406 Impacted: 1176 Missed at 5d: 1160 (98.64%) Missed at 15d: 1160 (98.64%) Missed at 30d: 1160 (98.64%) - Note: This is an ongoing incident *16: NETLOCK* Issue: NETLOCK: Policy Qualifiers other than id-qt-cps is included in TLS certificates Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1889570 Impacted: 522 Missed at 5d: 521 (99.81%) Missed at 15d: 255 (48.85%) Missed at 30d: N/A - Note: This is an ongoing incident - No 30d figure as incident ongoing, at time of writing (22d) *17: Sectigo* Issue: Sectigo: EV Certificate issuance with incorrect subject:serialNumber attribute value Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1891245 Impacted: 166 Missed at 5d: 0 (0.00%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - CA provided serial numbers speeding up this report *18: Telekom Security* Issue: Telekom Security: TLS certificates with basicConstraints not marked as critical Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1875820 Impacted: 816 Missed at 5d: 336 (41.18%) Missed at 15d: 0 (0.00%) Missed at 30d: 0 (0.00%) - No caveats *19: TWCA* Issue: TWCA: TLS certificates with non-critical basicConstraints Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1885132 Impacted: 16481 Missed at 5d: 4330* (22.23%) Missed at 15d: 1933* (11.73%) Missed at 30d: 1846* (11.20%) - CA provided serial numbers speeding up this report - However incident data included expired certificates inflating the missed certificate figures *20: VikingCloud* Issue: VikingCloud: OV Precertificates with incorrect Subject RDN encoding order Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883779 Impacted: 3208 Missed at 5d: 3123 (97.35%) Missed at 15d: 2433 (75.84%) Missed at 30d: 1596 (49.75%) - Strictly speaking this is two incidents in one: additional small batch of EV certs were discovered 21 days into start of revocation I hope this information is useful and gives an insight into how CAs are performing their incident response currently. As mentioned any questions or comments feel free to comment publicly or privately. - Wayne On Wednesday, April 24, 2024 at 8:12:47 AM UTC+1 Wayne wrote: > Hello, > > I've been watching the Entrust saga of issues over the past month and > keeping an eye on their incident response like many in the community. Given > that, I was curious how the public could reasonably audit a non-compliant > CA and ascertain how honest their claims were, as well as what bottlenecks > exist for public analysis. > > As part of an incident response CAs generally provide a list of impacted > certs in the form of SHA256 sums and a link to crt.sh for each certificate. > Additionally each certificate will need revoked and we have a Certificate > Revocation List to check timestamps and reasons. > > In order to audit we must be able to take a list of crt.sh SHA256 links, > and turn them into something we can check against a CRL: serial numbers. > This is the biggest barrier currently. > > Note that there is a limitation in obtaining only the latest copy of a CRL > as they generally do not include expired domains and are trimmed > automatically. To reflect this limitation this dataset will be working on a > CRL obtained 2024-04-20 and 2024-04-23 with merged de-duped records. The > vast majority of certificates issued by Entrust lie in the 3-month period > going by samples. > > *Proof of Concept 1: Entrust: CPS typographical (text placement) error* > Original Issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1890896 > Impacted Certs: 6008* > > The naive approach for a proof of concept is to check crt.sh's postgres > db, or website for each url and grab them one-by-one to extract the serial > number. In practice this took over 20 hours - noting that there isn't a > good mechanism for checking in bulk and this is not a shortcoming of crt.sh > as a public service. Thank you for your work Sectigo. > > Next I grabbed the most relevant CRL (L1K) and parsed it to provide a > lookup table - but only from 2024-03-12. This date was chosen as it is a > day before their major issues recently started snowballing. A decision was > made to give a cutoff at 2024-04-23 08:00:00 UTC applying to all future > notes. > > Combining both of these aspects onto a single cursed google sheet gets us > some brief analysis and should suffice as a proof of concept. > > Full google sheet: > https://docs.google.com/spreadsheets/d/1QuwPJh0sCHQCQaQEtUsr9qw_QRlf8fpwf8B7W6MPOqw/edit?usp=sharing > Published HTML version: > https://docs.google.com/spreadsheets/d/e/2PACX-1vT8ulLxKWSdCo80Xqf5UyAaqznIOcGDl_TEd2T2ayCZbaItnWfvLs4AC_F8BYmI16R-F1UgHJiWgu5J/pubhtml > > NOTE: Avoid CPS_6K_List_56K_WARNING (5,995 rows), and CRL_L1K_56K_WARNING > (12,013 rows) > > This issue in particular is one where Entrust insist they will not revoke > so it's a good testbed for a worst-case scenario. There is a breakdown on > just the 6,008* cert batch, and also on the CRL attached to 99% of their > certificates. > > *Caveat: This was only checking Entrust's L1K CRL, and it turns out there > are 35 certificates from L1F unaccounted for in this preliminary test. > Additionally there are also 14 certificates not logged on crt.sh despite > being on that list for inclusion.[1] > > With that in mind we can see a pattern to their revocation at the CRL > level: 4am is an automated system for processing, they generally only take > requests/process them Monday-Friday. The first time we breach over 500 > revocations per-day occurs 2024-04-19 (Friday). > > *Proof of Concept 2: Entrust's Publicly Listed CRLs* > > Given the sample above and handling one of their largest pools went fine I > expanded the test to all reasonably available CRLs and every entry. Given > Entrust's scale this should test the limits of Google Sheets and provide a > wider picture. > > NOTE: Avoid CRL_DATA_56K_WARNING (86k rows) > > Full google sheet: > https://docs.google.com/spreadsheets/d/1V2vLoOdD5CxDeCfVbTFHP_8HWYCViS9wo5Q23VWDJPE/edit?usp=sharing > Published HTML version: > https://docs.google.com/spreadsheets/d/e/2PACX-1vQ9RhMAJVmaBWZ_chwUIK_KYOMgERe-PZH2MbZhkS16EDJiOkRJJVO8QnCfdbbGcak6EEI_wvIT2ak_/pubhtml > > The advantage of this approach is that we can take a step back from > checking an individual certificate against each CRL. If a CA is responding > to an incident where they must revoke 10k certicates within 5 days, then we > should see approximately 10k revocations in a short time period, no? > > While this has the full data I've also trimmed CRL_GRAPH to show from > 2024-03-12 onward. There have been 24,154 revocations across every CRL, but > there's an unusual spike this weekend (see linked image below or the google > sheets link). 12,012 on L1K - the main OV intermediary, and 11,370 on L1M - > the main EV intermediary. > > CRL Graph: https://i.imgur.com/xN3U8O3.png > > Between 04-20 and 04-22 on L1K (OV) accounts for 710 revocations, while > L1M (EV) accounts for 2,598 revocations alone. The weekend is normally > quiet however EV certificates started being handled differently, but that > isn't the topic of this post. > > Now we can check these revocations against a single larger issue and get > an idea of what's happening. Notably we won't be checking each certificate > given the SHA256<->serial bottleneck mentioned above for a smaller separate > pool. > > https://bugzilla.mozilla.org/show_bug.cgi?id=1883843#c32 > Entrust: EV TLS Certificate cPSuri missing > > 26653 pre-certs provided, confirmed as mis-issued by Entrust 2024-03-05 > but only action started 2024-03-18. Given these are EV certificates we know > they will be primarily from intermediary L1M (and sampling confirms this). > > As noted early L1M has had 11,370 revocations since 2024-03-12, or if we > count from Entrust acknowledging revocation as necessary (2024-03-18) a > total of 11,268. This is imperfect as Entrust have decided that expiry is > the same as revocation, and includes certificates issued outside of the > affected batch. Additionally CRLs are pruned for expired certificates, > however given the issue was from 2024-03-22 and a CRL from 2024-04-20 is > sampled these should be accounted for. > > Looking at a statement made 2024-04-15 ( > https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c10) we are told > 8,572 revoked or expired. The CRL for L1M at that time notes from > 2024-03-12 to 2024-04-15 6,989 revocations (inclusive). From > acknowledgement we get 2024-03-18 to 2024-04-15 with 6,887. Regardless both > would be approximately 80% revoked:expired read generously. > > The latest statement ( > https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c11) by Entrust on > revocation and expiring (2024-04-19) notes 10,013 revoked/expired. Against > the total pool of 26,653 requiring revocation within 5 days we get 37.5%. > This is noting a ramp-up in revocations is starting, but given the > information provided this could be a separate issue unrelated to this batch. > > *Closing Remarks* > > There is room for a more thorough analysis by someone far more talented, > but I hope this brief picture gives enough of an idea of what can be done > in the future. > > *Takeaways* > - We need a better method for the public to transform SHA256 certs en-mass > to serial numbers to check CRLs > - We need CAs to be more forthcoming with breakdowns of issuing > intermediaries for each batch of certificates requiring revocation > - There isn't any big technical barrier or processing issue to > ascertaining a CA's compliance in an ongoing issue from someone with better > tooling > - There is further research required in checking for undisclosed issues > via CRLs and encouraging transparency > - A CA shouldn't collapse in their public trust to evoke this level of > public research > > *Baseline Requirements Notes:* > 4.10.1 Operational characteristics > Revocation entries on a CRL or OCSP Response MUST NOT be removed until > after the Expiry Date of the revoked Certificate. > > 7.2 CRL profile > > nextUpdate > Indicates the date by which the next CRL will be issued. For CRLs covering > Subscriber Certificates, at most 10 days after the thisUpdate. For other > CRLs, at most 12 months after the thisUpdate. > > revokedCertificates > MUST be present if the CA has issued a Certificate that has been revoked > and the corresponding entry has yet to appear on at least one regularly > scheduled CRL beyond the revoked Certificate’s validity period. The CA > SHOULD remove an entry for a corresponding Certificate after it has > appeared on at least one regularly scheduled CRL beyond the revoked > Certificate’s validity period. See the “revokedCertificates Component” > table for additional requirements. > > *Possible CRL Issue:* > While compiling this I noticed that 'Entrust Enterprise Intermediate > CA-ICA1' isn't refreshing the CRL daily (same CRL 2024-04-20 and > 2024-04-23): > 2048.crl: > Last Update: Apr 5 15:55:58 2024 GMT > Next Update: Apr 5 15:55:57 2025 GMT > > I presume that'd still cover: https://crt.sh/?caid=32 which has 13 > unexpired certs, but this may be a misinterpretation on my behalf. > > *[1]*: Certificates not on crt.sh but listed in > 20240409-OV-TLS-Certificates-typo-finalcerts: > ----- > > https://crt.sh/?sha256=1086E495E6D4CDC926CDFE3C3F083272D70BE65F3BF75336393C6975A11E6D8A > > https://crt.sh/?sha256=125E7D5030D1B4C57DFF13DAD4E9525024DEC3649F77745C906C229765E24F5F > > https://crt.sh/?sha256=4323087C85F6FB5F7F2D8E66D9C5EC8BC4007D37CBF93ACC159D6173D0793AFA > > https://crt.sh/?sha256=637D768E1D386ACDEBED435C84B0B08C2C6B09B840045A34503DB518645657B5 > > https://crt.sh/?sha256=692991CF2926851395CE4C15A37AC4B6AD13D46979E5A7912F39FDD9FFFAC4A9 > > https://crt.sh/?sha256=749109CEAEE9CF2D88E6999D4768697696E7C67E4C4DB2648A19137A5D1669ED > > https://crt.sh/?sha256=76ADEF1237C4B07652D8A597337580947E6B6E1A83F3D25332BE3863BA180CAC > > https://crt.sh/?sha256=7DD2FF3533BD7C64DEA51E89F469EDD29B007C5DC4708597A9F0594B76954EE1 > > https://crt.sh/?sha256=AF4274B727520335A80F82C0E615EFB87D37C5F03971AFDF9114ACBCFA5882EA > > https://crt.sh/?sha256=C5E3974E1DBE2A36450B662A6867988B31361F9DFC67AC4ACA38B386B43432F5 > > https://crt.sh/?sha256=EF5F20F6B5913C3A5F9477E2BE2D6801D85DD80B9BB42A5363AA3B1E82566FD9 > > https://crt.sh/?sha256=F5455CBE0B50EC0002A5C1F83AC49CC38842B30D84B830D8A23257AEFA406117 > > https://crt.sh/?sha256=F7F70E1FD16BB7B09A21486AAC9295CFD8259BB1C6EF363D5A61C1230B1E9426 > > https://crt.sh/?sha256=FE5BEA6F8D4333CFEFA3277AC2F4D544987FFF7C3ED83514EAF92A34919F19E6 > ----- > > - Wayne -- You received this message because you are subscribed to the Google Groups "dev-security-policy@mozilla.org" group. To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-policy+unsubscr...@mozilla.org. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/da25a846-1bb5-4abf-84fe-38e9086a6ac1n%40mozilla.org.
