Hi Jeremy,
Thanks for pulling the Bugzilla data and BR-overlap figures. I would be very cautious about relying on this LLM analysis. Studies show over 80% hallucination rates on legal queries (Dahl et al., 2024), and even specialized models like Westlaw produce 17-33% unsupported assertions with one in six "very confident" answers being wrong (Magesh et al., 2024). A CPS is essentially a contract densely woven with BR, RFC 5280, ETSI, root program, and national law references. If a model can invent a Supreme Court case, it can just as easily invent a BR exception or declare similarity that does not exist. When reviewing these documents, we must consider the context of what they reference. CPSs often say "not stipulated," "as per XYZ," or "when not stated the [referenced standard] govern." This requires topological analysis within the broader regulatory framework that domain experts inherently apply. For example, a CPS might claim RFC 5280 compliance, but you need to verify that against the actual RFC. It's like legal document analysis where federal law can preempt local law. Beyond that, two CPSs can look 80% identical yet behave very differently if one CA blocks non-compliant certs pre-issuance and auto-updates its CPS, while another relies on quarterly manual checks. This is why I doubt these similarity figures are accurate, but even if they are, the larger question is whether CPSs should clone the BRs or describe actual practices. Some BR sections make sense to duplicate when you implement the corresponding mechanisms, but edits are often needed to keep documents cohesive and understandable. The operational differences that really matter for compliance are often invisible to text-based analysis, which is why we need CPSs that actually represent practices with enough detail that outsiders can understand what promises are being made. That is not to say that LLMs are not useful in the context of CPS analysis but I can say with confidence to do it reasonably well requires more than just a LLM. Best, Ryan On Monday, June 16, 2025 at 5:35:57 PM UTC-7 Jeremy Rowley wrote: > I was trying to use AI to analyze CPS docs to see where interesting > information might be. I've only looked at three different CAs so far, but > figured I'd share the results: > > Sectigo – 70% of the CPS is nearly identical to the BRs. The primary > variation is in section 9 and the use of a reseller network. > Sectigo does not use: > - Method 3.2.2.4.12 (Validating Applicant as Domain Contact) > - Method 3.2.2.4.21 (DNS Labeled with Account ID - ACME) > > DigiCert – about 80% overlap in language with the BRs. The primary > differences are that the DigiCert CPS covers public trust (not just TLS) > and the legal section. > Digicert does not use: > 1) Method 3.2.2.4.20 - TLS Using ALPN > > 2) Method 3.2.2.4.21 - DNS Labeled with Account ID - ACME > > > > Comparing the two CPS docs together, the AI found they were about 85% > similar on TLS. Excluding the business sections (section 1 and section 9), > the CPS docs are 95% similar. > > Let's Encrypt has about a 77% overlap with both DigiCert and Sectigo. The > major differences in the LE CPS are: > 1) Business terms and the lack of OV certificates > 2) Automation requirements for issuing certificates > 3) No language around the use of RAs (because LE doesn't use RAs) > > 82% of all documentation is about how the CA matches the BRs. > > This is, of course, subject to some interpretation by the AI used and I > haven't reviewed it in full. All CPS docs provide value in that they list > the CPR, the CAA records used, and the BR methods permitted for validation. > Is there a CPS I can look at that provides substantial additional > information beyond the BRs? > > On Mon, Jun 16, 2025 at 5:58 PM Jeremy Rowley <[email protected]> wrote: > >> Good question. I went through the last year of bugs and found the ones >> listed below. Determining what is a CPS violation vs. a BR violation is >> difficult because so many BR violations are also a CPS violation (as a lot >> of CPS documents mirror the BRs). I split it up between profile errors (at >> the bottom) and CPS related issues (at the top), both of which would be >> solved by automated CPS generation and a shift to treat the CPS document as >> a technical disclosure instead of a contract. >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=1970567 - Failed to list >> the full revocation reasons in its CPS >> https://bugzilla.mozilla.org/show_bug.cgi?id=1969842 - This is about >> T&Cs but since the T&Cs generally incorporate the CPS I thought I'd count >> it? >> https://bugzilla.mozilla.org/show_bug.cgi?id=1969036 - violates the CPS >> and the BRs >> https://bugzilla.mozilla.org/show_bug.cgi?id=1965808 - Conflicting info >> in the CPS >> https://bugzilla.mozilla.org/show_bug.cgi?id=1965806 - Missing OID on >> T&Cs (which would incorporate the CPS) >> https://bugzilla.mozilla.org/show_bug.cgi?id=1965804 - CPS clarity issues >> https://bugzilla.mozilla.org/show_bug.cgi?id=1963778 - CPS unavailability >> https://bugzilla.mozilla.org/show_bug.cgi?id=1963629 - CPR in CPS not >> working >> https://bugzilla.mozilla.org/show_bug.cgi?id=1962829 - policy document >> mis-paste >> https://bugzilla.mozilla.org/show_bug.cgi?id=1962830 - Cert change not >> compliant with CPS >> https://bugzilla.mozilla.org/show_bug.cgi?id=1955365 - Reused keys in >> violation of CPS >> https://bugzilla.mozilla.org/show_bug.cgi?id=1954580 - OCSP not >> published in time. This violated the BRs but would also violate the CPS if >> such items were actually dictated by the CPS instead of just the BRs. >> https://bugzilla.mozilla.org/show_bug.cgi?id=1948600 - outdated CPS >> https://bugzilla.mozilla.org/show_bug.cgi?id=1942241 - CPR in CPS not >> accepting attachments >> https://bugzilla.mozilla.org/show_bug.cgi?id=1938236 - CAA issue >> https://bugzilla.mozilla.org/show_bug.cgi?id=1939809 - This violated the >> ETSI requirement but not the BRs I think? Which would make it a CPS >> violation. >> https://bugzilla.mozilla.org/show_bug.cgi?id=1935393 - Failed to update >> CPS docs (note that the proposal would help remediate this by requiring >> automatic updates to CPS docs as things change). >> https://bugzilla.mozilla.org/show_bug.cgi?id=1933353 - violation of CPS >> on OCSP responses >> https://bugzilla.mozilla.org/show_bug.cgi?id=1932973 - violation of CAA >> checking >> https://bugzilla.mozilla.org/show_bug.cgi?id=1931413 - violation of >> onboarding SOP >> https://bugzilla.mozilla.org/show_bug.cgi?id=1925106 - incorrect CP >> provided >> https://bugzilla.mozilla.org/show_bug.cgi?id=1921573 - CPS issue on DN >> https://bugzilla.mozilla.org/show_bug.cgi?id=1918380 - Business entity >> not permitted in CPS >> https://bugzilla.mozilla.org/show_bug.cgi?id=1914911 - CAA disclosure >> issue >> https://bugzilla.mozilla.org/show_bug.cgi?id=1904749 - CAA record issue >> https://bugzilla.mozilla.org/show_bug.cgi?id=1904257 - Incorrect CPR >> address >> >> >> I'm listing the profiles issues as well as the proposal would address >> this issue, or at least make these issues more readily identifiable. If CAs >> are required to provide the profile directly from the CA, the profile can >> easily be compared to the BRs and issues identified. Right now the profile >> may not match the CPS so the CPS will be compliant but the profile will not >> match the requirements. >> Profiles mismatch: >> https://bugzilla.mozilla.org/show_bug.cgi?id=1965459 - AIA not correct >> https://bugzilla.mozilla.org/show_bug.cgi?id=1963663 - Multiple cert >> policies >> https://bugzilla.mozilla.org/show_bug.cgi?id=1963456 - HTTPS in AIA >> https://bugzilla.mozilla.org/show_bug.cgi?id=1952591 - SCT issue in certs >> https://bugzilla.mozilla.org/show_bug.cgi?id=1946921 - DV cert format >> issue >> https://bugzilla.mozilla.org/show_bug.cgi?id=1936908 - Incorrect encoding >> https://bugzilla.mozilla.org/show_bug.cgi?id=1922906 - :LDAP URI issue >> https://bugzilla.mozilla.org/show_bug.cgi?id=1921598 - Cert Policies >> extension issue >> https://bugzilla.mozilla.org/show_bug.cgi?id=1921254 - Duplicate >> attribute >> https://bugzilla.mozilla.org/show_bug.cgi?id=1919162 - incorrect profile >> https://bugzilla.mozilla.org/show_bug.cgi?id=1916489 - LDAP in CRLDP >> https://bugzilla.mozilla.org/show_bug.cgi?id=1916392 - 2 Localities >> listed >> >> >> On Sun, Jun 15, 2025 at 7:36 AM Mike Shaver <[email protected]> wrote: >> >>> On Sun, Jun 15, 2025 at 12:13 AM Jeremy Rowley <[email protected]> >>> wrote: >>> >>>> Given the number of bugs related to CPS errors, >>>> >>> >>> Perhaps you’re in a position to answer this question: how many bugs >>> *have* there been in the last few years related to CPS errors, and how many >>> certs have been subject to revocation for that reason, pre-Microsoft? >>> >>> Mike >>> >>> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/9f791033-5d53-44c8-aa6b-91cabf296512n%40mozilla.org.
