Hey Matt, Ryan's post was the part I thought was relevant, but I understood it differently. The cert was issued, but we should have now revoked it (24 hours after receiving notice). I do see your interpretation though, and the language does support 24 hours after issuing the new cert. What I need is a tool that scans after revocation to ensure there are no additional certs with the same key. The frustration is that this was where the cert was issued after our scan of all keys but just before revocation. As a side note, our system blacklists the keys when a cert is revoked for key compromise, which means I don't have a way to blacklist a key before a cert is ever issued.
>> I don't think that supports your point, though, so I wonder if I've got the >> wrong part. That last part of Ryan's: "shenanigans, such as CAs arguing >> the[y?] require per-cert evidence rather than systemic demonstrations", >> seems to me like it's describing your statement, above, that you (only?) >> "need to revoke a cert that is key compromised once we're the key is >> compromised *for that cert*" (emphasis added). I don't read Ryan's use of >> "shenanigans" as approving of that sort of thing. I don't think its shenanigans, but I do think it's a pretty common interpretation. Such information would help determine the common interpretation of this section. I agree that CAs should scan all certs for the same key and revoke where they find them. Is that actually happening? Do other CAs object to there being a lack of specificity if you give the keys without a cert attached? >> Bim bam bom, all done and dusted, and we can get back to washing our hands. That you're *not* doing that is perplexing, and a little disconcerting. That's an oversimplification of the incident report process. I'm not resisting the incident report itself since incident reports are a good and healthy way for the public to have transparency into a CAs operation. In fact, I wouldn't mind filing one just to give public documentation on what DigiCert is doing for revocation. I was objecting to actually calling this an incident/breach of the guidelines since I disagreed with that, and there's a trend where the interpretation of these sections seems to evolve over time. My main emphasis was that the guidelines are ambiguous and need refinement. I'd support refining them to be more clear, but calling something shenanigans when the language is unclear is unfair. For the SPKI, you're right. The slow part is downloading it from your website. I guess I should have said "a link to the key compromise proof" rather than the SPKI. The additional time required to do that was 5 hours. Although the automation isn't quite complete yet, the end-state goal is to do the following when we get a key compromise notification: 1) Run the CSR through a tool and confirm key compromise 2) Have the tool email the impacted subscriber 3) Have the tool process the revocation at the 24 hour mark. The two parts that are done is having the tool process the revocation and parse out all the key compromises and processing the revocation within 24 hours. Unfortunately, we haven't automated the cert problem report yet so we can only accept an email. Once I get the revocation tool working the way I want it, we'll add a tool where you can submit a CSR directly and avoid the round trip. >> Even if Digicert would just prefer that I did something differently in my >> problem reports in the future -- something that would make it legitimately >> easier for Digicert to process my reports, without making my life a misery >> (after all, y'all have a lot more resources than I do) -- all you have to do >> is ask. My e-mail address is right there in every problem report I send. Nah - although I'd prefer just to get the CSRs in a zip file, I can work with what you send us. In fact, I wouldn't change your process at all since I want to accommodate all sorts of ways for people to send us private keys. I'd rather not force you to do extra work to send us the keys in a different way as anything that creates barriers to key compromise reports is a bad thing. The reason I brought up the format is that I'm not sure when the 24 hour clock actually kicks off for revocation. Is it 24 hours after we get your email or 24 hours after we can confirm key compromise? I've always regarded these as a certificate problem report under 4.9.5 (requiring us to kick off the investigation) but then revocation should happen 24 hours after we have reason to suspect the key compromise is legit. Usually those are the same time, but with the email link (and number of keys in the last batch) it may take longer to actually do the investigation to show the key is compromised. I think there's ambiguity in when that timer starts, and clearing up that ambiguity in the CAB forum would be good. As you pointed out "obtains" can mean different things. >> The time to say "hey, while we *can* process these SPKIs and CSRs, they are >> kinda teh suck, what else you got?" would have been, ideally, when I sent >> that first trickle of compromise reports that were acted upon. At worst, >> when the Big Bomb of Keys dropped, you stick your hand up and say "uhm, >> these kinda suck to process, any chance it's easy for you to send us X >> instead?", while simultaneously working on the pile as best you can. Worst >> case I've gone to bed, and you do what you did anyway, but depending on what >> X was, I might have been able to oblige. To be clear, I'm not complaining about the format. I'm wondering when we obtained the private key for the 24 hour purposes. With automation, the time between when we get the email and when we confirm key compromise should be nearly zero. However, with a more manual process, that time is not insignificant. What I don't like about the interpretation that the revocation event is 24 hours from when we get an email is some emails are very vague about key compromise. With that reading, if we get an email without proof that is later followed up by proof, the 24 hour period could start when we get the initial email even if the proof is provided 25 hours later. That does happen, which is why I think the time period should be 24 hours from after the CA receives proof of key compromise. But even that is ambiguous. When did we receive proof of key compromise? I'd say it's when all the CSRs finished downloading. If that's not the case, then you are encouraging CAs to be myopic in the way they accept key compromise information. Jeremy -----Original Message----- From: dev-security-policy <dev-security-policy-boun...@lists.mozilla.org> On Behalf Of Matt Palmer via dev-security-policy Sent: Monday, March 23, 2020 4:45 AM To: Mozilla <mozilla-dev-security-pol...@lists.mozilla.org> Subject: Re: Digicert: failure to revoke certificate with previously compromised key On Mon, Mar 23, 2020 at 06:14:29AM +0000, Jeremy Rowley wrote: > That's not the visible consensus IMO. The visible consensus is we > need to revoke a cert that is key compromised once we're informed the > key is compromised for that cert > (https://groups.google.com/forum/m/#!topic/mozilla.dev.security.policy/1ftkqbsnEU4). I think that link might not be doing what you expect, as it (at least for me) is collapsing all the replies in that topic before Doug Beattie's post. The only response that seems relevant in that topic to was Ryan's reply to me up-thread from Doug's post, which was, in (I believe) relevant part, when I asked the question: > 3. Can a CA be deemed to have "obtained evidence" of key compromise prior > to the issuance of a certificate, via a previously-submitted key > compromise problem report for the same private key? If so, it would > seem that, even if the issuance of the certificate is OK, it is a > failure-to-revoke incident if the cert doesn't get revoked within 24 > hours... To which Ryan replied: > Correct, that was indeed the previous conclusion around this. The CA > can issue, but then are obligated to revoke within 24 hours. There’s > not a statute of limitation on “obtains evidence” here, precisely > because it could allow a host of shenanigans, such as CAs arguing the > require per-cert evidence rather than systemic demonstrations. I don't think that supports your point, though, so I wonder if I've got the wrong part. That last part of Ryan's: "shenanigans, such as CAs arguing the[y?] require per-cert evidence rather than systemic demonstrations", seems to me like it's describing your statement, above, that you (only?) "need to revoke a cert that is key compromised once we're the key is compromised *for that cert*" (emphasis added). I don't read Ryan's use of "shenanigans" as approving of that sort of thing. > The certificate you mentioned was issued before the keys were > blacklisted and not part of a certificate problem report. When > revoking a cert we scan to see if additional certs are issued with the > same key t, but this particular cert one was issued after the scan but > before the revocation, largely because the way you are submitting > certificate problem reports breaks automation. We currently don't > have a way to blacklist private keys until a certificate is revoked, > although that would be a nice enhancement for us to add in the future. > Anyway, I don't think anything reported violated the BR since 1) this > cert was not part of a certificate problem report and 2) we will be > revoking within 24 hours of your Mozilla posting. The way I see it, once a CA is provided with accepted evidence of key compromise, every certificate issued by that CA using the same key -- past, present, and future -- is implicitly part of that certificate problem report. *I* think that's supported by Ryan's confirmation of my question, quoted above, but presumably you disagree? At the end of the day the only call that matters is that of the CA module owner and peers, so I guess we'll have to leave it up to them to make the call as to whether Digicert's behaviour that I described -- and the facts of which you don't seem to dispute -- constitutes a BR violation. Honestly, I'm a bit surprised you're trying so hard to argue against taking responsibility for this occurrence (probably should use "incident" yet, I guess). Based on purely on what you wrote in the above paragraph, it seems like it was a simple oversight in your systems for what is, I won't deny, a bit of a corner case. Even I, as gung-ho as I am about throwing out misbehaving CAs, wouldn't argue that this in and of itself is anything worth a rap over the knuckles for. As far as I can see, the meat of an incident report for this occurrence could be something like "our key blacklist is only populated at the time revocation occurs; the certificate in question was issued between problem report and revocation; we've added a story to the backlog to modify our CA systems to allow keys to be blacklisted before revocation, and until then we've modified our procedures to do a sweep of our certificate archive after the initial revocation has taken place, to catch a similarly situated certificate in the future." Bim bam bom, all done and dusted, and we can get back to washing our hands. That you're *not* doing that is perplexing, and a little disconcerting. > I support the idea of swift revocation of compromised private keys and > do appreciate you reporting them. I think this is helpful in ensuring > the safety of users online. However, using the SPKI to submit > information breaks our automation, making finding and revoking certs > difficult. The more standards way (IMO) is the SHA2 thumbprint or > serial number or a good old CSR. This confuses me a bit; an SPKI *is* a "SHA2[56] thumbprint" of the compromised key, and keys don't have serial numbers. As for a CSR, I provide one, signed by the compromised key, which contains (as is the nature of the beast) the entire public key, which you can extract and wield in whichever manner you choose. If you're referring to the SHA2 thumbprint or serial number of a *certificate*, then we're back to "CAs arguing the[y?] require per-cert evidence", which comes under the (disputed?) territory of "shenanigan". Meanwhile, I don't understand how *any* CSR -- even the original CSR which was submitted to the CA (which I wouldn't necessarily have, nor would I be expected to) -- could even uniquely identify a certificate, since it doesn't have a serial number or SHA2 thumbprint or *any* unique identifier that ties it to the certificate. > Because submitting the SPKI breaks automation, getting evidence of key > compromise took an additional 5 hours after you submitted the report. What automation *does* Digicert have in place to process key compromise reports? If I'd, say, just e-mailed a PEM format private key to rev...@digicert.com (or even 800 of them), how much quicker would that have been processed, as compared to the SPKIs I provided? I collect so many private keys of your customers' certs (I've got, IIRC, nine queued up that have come in over the last few hours that I'm yet to process), that it's worth my while spending an hour or two coding up something if it means I can just squirt key compromise reports into your systems somehow and avoid the whole e-mail round-trip dance. > We still revoked all of the current certs with submitted keys within > 24 hours of the report (since compromised private keys are bad and > there is nothing that says we can't revoke earlier than 24 hours), but > I did want to clarify that I don't think the time starts until we can > actually get the information necessary to do an investigation (because > there is not sufficient evidence of a key compromise until then). I certainly agree that the wording in the BRs, that says that the 24 hour period begins when the CA "obtains" evidence of key compromise, could stand to be improved. "Obtains" is a word open to much interpretation, and one could, potentially, argue that a CA only "obtains" evidence when it takes an active step -- such as opening an e-mail, retrieving a provided link, or you could even say that the evidence isn't really "obtained" until the CA actually validates the signature on a CSR and compares public keys. However, by that argument, a CA could delay revocation indefinitely by claiming that they hadn't yet "obtained" evidence, before finally saying "oh, yes, OK, now we've got it" and doing the revocation. Hopefully we agree that a CA which deliberately did that would be worthy of a trip to the naughty step, which means that the BRs should not be interpreted that way, based on my recollection of precedent. So, let's talk quality of evidence provided. If Digicert had a legitimate belief that the evidence I provided was insufficient, or was otherwise problematic, the correct action, IMO, would have been to mention that in the preliminary report, which has to come within the first 24 hours. I don't know how closely you watch the queues in the Digicert salt mines, so you might already be aware of this, but the "your evidence sucks" situation already happened, with the first problem reports I submitted. I was providing a JWS attesting to compromise, your people said "we can't handle this", I checked with m.d.s.p as to whether that was reasonable, Ryan called me a goose for choosing to use JWS, and we moved on. CSRs for everyone! So I started submitting problem reports with links to crt.sh and a CSR, and everything seemed to be going swimmingly. I did dump a whole pile of compromised keys on y'all at once, I admit, but I had an 800+ key backlog, and at the rate new keys were coming in, the backlog would have *never* been cleared if I'd kept reporting just a few a day. (See above, re: finding nine in the past few hours). The time to say "hey, while we *can* process these SPKIs and CSRs, they are kinda teh suck, what else you got?" would have been, ideally, when I sent that first trickle of compromise reports that were acted upon. At worst, when the Big Bomb of Keys dropped, you stick your hand up and say "uhm, these kinda suck to process, any chance it's easy for you to send us X instead?", while simultaneously working on the pile as best you can. Worst case I've gone to bed, and you do what you did anyway, but depending on what X was, I might have been able to oblige. Only raising this *now*, after something bad (may have) happened, strikes me as a bit of a cop-out. (Is that an Australianism?) Even if Digicert would just prefer that I did something differently in my problem reports in the future -- something that would make it legitimately easier for Digicert to process my reports, without making my life a misery (after all, y'all have a lot more resources than I do) -- all you have to do is ask. My e-mail address is right there in every problem report I send. > Going to the previous discussion, I'd definitely support seeing a > standardized way to report key compromise. Trying to account for the > various formats they come in and through the various channels creates > a lot of manual work on a process that can easily be automated. Weeeeeeeell... I'm in two minds here. Given the volume of reports I'm doing, a standard format that I can easily automate into, that I *know* all CAs would accept, would be awesome -- *for me*. There's no need to develop any new protocols, even, because a quite serviceable one already exists -- the revokeCert endpoint in ACME. (And don't think I'm not a bit salty that Digicert implements ACME but hides it behind customer registration, either <grin>). However, if someone wants to report a compromised key as a one-off, making them jump through unnecessary hoops to beat an ACME client into doing their bidding just so they can report a compromised key is *massively* counter-productive. Whilst you might say that a CA should also continue to accept key compromise reports via e-mail -- and I would agree -- I have little-to-no confidence that there would be at least one CA who would try to force everyone to use their not-particularly-casual-user-friendly ACME endpoint to submit key compromise notifications. So, if someone does manage to wedge "all CAs must provide an ACME revokeCert endpoint, *and* publish the directory URL for that ACME server in CCADB" into the BRs (and my oh my, how I pray for that day), I hope that the same amendment will also include language saying that CAs must also accept, without kicking, key compromise reports via e-mail (at an e-mail address also recorded in its own field in CCADB). - Matt _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
