> > Thanks for filing this, Wojciech. This is definitely one of the better incident reports in terms of providing details and structure, while also speaking to the steps the CA has taken in response. There was sufficient detail here that I don't have a lot of questions - if anything, it sounds like a number of best practices that all CAs should abide by result. The few questions I do have are inline below:
> On Mon, Dec 3, 2018 at 6:06 AM Wojciech Trapczyński via dev-security-policy <dev-security-policy@lists.mozilla.org> wrote: > (All times in UTC±00:00) > >> > 10.11.2018 10:10 – We received a notification from our internal > >> monitoring system for issuing certificates and CRLs concerning issues > with publishing CRLs. We started verification. > Understanding what system you had in place before hand is valuable in understanding what changes you propose to make. In particular, in remediation, you note "We have deployed additional verification of certificate and CRL signatures in the external component" It's unclear here what the monitoring system monitored, or what the challenges in publishing were. It sounds like there was already monitoring in place in the internal system that detected the issue with corrupted signatures. Is that a misunderstanding? I could also see an interpretation being that "It was having trouble publishing large files", which would seem a different issue. Thus, it's helpful if you could discuss a bit more about what this monitoring system already monitors, and how you're improving it to catch this sort of issue. This may reveal other possible gaps, or it may be so comprehensive as to also serve as a best practice that all CAs should follow. In either event, the community wins :) > 6. Explanation about how and why the mistakes were made or bugs > introduced, and how they avoided detection until now. > <snip> > All issued certificates were unusable due to corrupted signature. > Could you speak to more about how you assessed this? An incorrect signature on the CRL would not necessarily prevent the certificate from being used; it may merely prevent it from being revoked. That is, all 30,000 (revoked) certificates may have been usable due to the corrupted signature. > 7. List of steps your CA is taking to resolve the situation and ensure > >> such issuance will not be repeated in the future, accompanied with a > >> timeline of when your CA expects to accomplish these things. > >> > We have deployed a new version of the signing module that correctly > >> signs large CRLs. From now, we are able to sign a CRL that is up to 128 > >> MB. In addition, we have improved the part of the signing module > >> responsible for verification of signatures (at the time of failure it > >> did not work properly). > >> > We have deployed additional verification of certificate and CRL > >> signatures in the external component, in addition to the signing module. > >> This module blocks the issuance of certificates and CRLs that have an > >> corrupted signature. > >> > We have extended the monitoring system tests that will allow us to > faster detection of incorrectly signed certificates or CRLs. > As others have highlighted, there is still an operational gap, in that 1MB CRLs are rather large and unwieldy. To help manage this, CRLs support "sharding", by virtue of the CRL distribution point URL and the (critical) CRL extension of Issuing Distribution Point ( https://tools.ietf.org/html/rfc5280#section-5.2.5 ). For example, the same (Subject DN + key) intermediate CA can divide the certificates it issues into an arbitrary number of CRLs. It does this by ensuring distinct URLs in the certificates' CRLDP extension, and then, for each of the URLs referenced, hosting a CRL for all certificates bearing that URL, and with a critical IDP extension in the CRL (ensuring the IDP is present and critical is a critical security function). By doing this, you can roll a new CRL for every X number of subscriber certificates you've issued, allowing you to bound the worst-case revocation. For example, if the average size of your CRL entry was 32 bytes (easier for the math), then every 2,000 certificates, you could create a new CRL URL, and the maximum size your CRL would be (in the worst case) for those 2,000 certificates is 64K. Have you considered such an option? Several other CAs already apply this practice, at varying degrees of scale and size, but it seems like it would be a further mitigation to a root cause, which is that the revocation of 30,000 certificates would not balloon things so much. _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
