Re: Digicert: failure to revoke certificate with previously compromised key

[Matt Palmer via dev-security-policy]

On Mon, Mar 23, 2020 at 06:15:00PM +, Jeremy Rowley wrote:
> There are two things worth discussing in general:
> 
> 1. I’m very interested in seeing the Let’s Encrypt response to this issue
> since the biggest obstacle in trying to find all of the keys with the same
> private key is the sheer volume of the certs.  Trying to do a
> comprehensive search when a private key is provided leaves some window
> between when we start the analysis and when we revoke.

Well, Let's Encrypt has committed to automatically blacklisting keys
reported for keyCompromise in its CA software
(https://community.letsencrypt.org/t/116762), so it won't be nearly as
problematic for them as it appears to be for Digicert.  At any rate, though,
I can get a list of all certs with a certain SPKI out of crt.sh in a matter
of a couple of seconds, and crt.sh has a *lot* more certs in it than
Digicert has issued.  The schema for the certwatch database is publicly
available, so, if nothing else, you could stand up a copy of that database,
skip deploying the CT scrapers, and just stuff a copy of every cert you
issue into the certificate table, and you're off to the races.

> 1. Another issue in trying to report keys that aren’t affiliated with
> any cert is that the process becomes subject to abuse.  Without knowing a
> cert affiliated with a key, someone can continuously generate keys and
> submit them as compromised.  You end up just blacklisting random keys,
> DDOSing the revocation system as it kicks off another request to  search
> for those keys.  I don’t think it’s feasible.  This is why the disclosures
> need to be affiliated with actual certs.

I don't think that anyone has, as yet, claimed that it is a BR violation for
a CA to issue a certificate with a key for which they have not yet received
a valid certificate problem report.  Nor do I believe that anyone has
claimed that a certificate problem report without any indication of a
problematic certificate is valid.  So, it appears to me that you're arguing
against something that nobody has proposed.

- Matt

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Digicert: failure to revoke certificate with previously compromised key

[Matt Palmer via dev-security-policy]

On Mon, Mar 23, 2020 at 12:53:43PM -0400, Ryan Sleevi wrote:
> To make sure I understand the timeline correctly:
> 2020-03-20 02:05:49 UTC - Matt reports SPKI 4310b6bc0841efd7fcec6ba0ed1f36
> e7a28bf9a707ae7f7771e2cd4b6f31b5af, associated with
> https://crt.sh/?id=1760024320 , as compromised
> 2020-03-21 01:56:31 UTC - DigiCert issues https://crt.sh/?id=2606438724 with
> that same SPKI
> 2020-03-21 02:09:12 UTC - DigiCert revokes https://crt.sh/?id=1760024320
> 2020-03-23 03:16:18 UTC - DigiCert revokes https://crt.sh/?id=2606438724
> 
> Is that roughly correct?

Yes, that appears to be a correct summary of the timeline as I see it.

- Matt

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Digicert: failure to revoke certificate with previously compromised key

[Matt Palmer via dev-security-policy]

On Mon, Mar 23, 2020 at 03:01:34PM +, Jeremy Rowley wrote:
> Ryan's post was the part I thought was relevant, but I understood it
> differently.  The cert was issued, but we should have now revoked it (24
> hours after receiving notice).  I do see your interpretation though, and
> the language does support 24 hours after issuing the new cert.

Aha, righto.  Glad we've gotten on the same page there.

> What I need is a tool that scans after revocation to ensure there are no
> additional certs with the same key.

I can give you a certwatch SQL query that'll do that, if you like -- "show
me all certs with this SPKI (or set of SPKIs) which aren't expired or
OCSP-revoked".  I use pretty much that query to get periodic reports of new
certificates that have appeared with keys already in the dungeon.  It's not
ideal for your purposes, though -- you might get some false positives
because your OCSP responders aren't up-to-date, and false negatives are
possible if certwatch is backlogged (or a cert wasn't logged).

Beyond that, though, if your internal certificate archive isn't indexed on
SPKI fingerprint, and updated in near-real-time, those are problems that I
think you'll have to fix, because the Internet's propensity to post their
private keys on the Internet, and then reuse them to get new certs after the
old one got revoked for key compromise, does not seem to be one that is
going away any time soon.

> The frustration is that this was
> where the cert was issued after our scan of all keys but just before
> revocation.  As a side note, our system blacklists the keys when a cert is
> revoked for key compromise, which means I don't have a way to blacklist a
> key before a cert is ever issued.

To the software developers!  *blows trumpets*

> >> I don't think that supports your point, though, so I wonder if I've got
> >> the wrong part.  That last part of Ryan's: "shenanigans, such as CAs
> >> arguing the[y?] require per-cert evidence rather than systemic
> >> demonstrations", seems to me like it's describing your statement,
> >> above, that you (only?) "need to revoke a cert that is key compromised
> >> once we're the key is compromised *for that cert*" (emphasis added).  I
> >> don't read Ryan's use of "shenanigans" as approving of that sort of
> >> thing.
> 
> I don't think its shenanigans, but I do think it's a pretty common
> interpretation.  Such information would help determine the common
> interpretation of this section.  I agree that CAs should scan all certs
> for the same key and revoke where they find them.  Is that actually
> happening?

A lot to unpack here, let me make up some specific questions and answer them
as best I can.

"Have any other CAs failed to revoke certificates issued for keys for
which they had previously revoked a certificate for key compromise?"

Yes, one CA has failed to do so, and I've reported that to this list.

"Have any other CAs successfully revoked a certificate within the
BR-mandated (am I OK using that phrase now?) time period, that they issued
for a private key for which they had previously revoked a certificate due to
key compromise?"

I don't know, I haven't checked (yet).  It's on my (lengthy) list of Interesting
Questions For Which I Need To Write Insanely Complicated SQL Queries.

"Have any other CAs blacklisted a private key that was reported as
compromised and prevented issuance before they had issued a certificate for
that key?"

Naturally I can't answer that one directly, because I don't have internal
access to CA systems.  *But*, I have one test case, in which a private key
known to have "hopped" between CAs after revocation was reported to a number
of CAs proactively, but there hasn't been a resolution to that test case one
way or the other.  No new certs have been issued for the same name, with the
compromised key or any other, so it's impossible to infer what the outcome
may be.

Other very specific questions welcomed.  The answers will probably be "I
haven't looked yet", but there are a lot of questions I've got at least a
vague idea of how to answer, once I have time2SQL.

> Do other CAs object to there being a lack of specificity if you give the
> keys without a cert attached?

Since I have been sending (links to) CSR format compromise attestations, no
CA has communicated an objection to the format of my reports, nor have they
failed to act (in some fashion) to any of my reports, as far as I am aware.

> >> Bim bam bom, all done and dusted, and we can get back to washing our 
> >> hands. 
>
> That you're *not* doing that is perplexing, and a little disconcerting.

My wife is immuno-suppressed -- about the only time I'm not washing my
hands is when I'm typing at my keyboard (the suds get in the switches and
cause all sorts of problems).  If I could get a reliable supply of
sanitizer, I'd be swimming in the stuff.

> That's an oversimplification of the incident report process.  I'm not
> resisting the incident report itself since incident reports are a 

Re: Auditing of CA facilities in lockdown because of an environmental disaster/pandemic

[Kathleen Wilson via dev-security-policy]

It's still very much a work-in-progress, but I updated the first bullet 
point in the "Minimum Expectations" section again.


https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay

""
Both ETSI and WebTrust Audits should:
- Disclose each location (at the state/province level) that was included 
in the scope of the audit or should have been included in the scope of 
the audit, whether the inspection was physically carried out in person 
at each location, and which audit criteria were checked (or not checked) 
at each location.
-- If the CA has more than one location in the same state/province, then 
use terminology to clarify the number of facilities in that 
state/province and whether or not all of them were audited. For example: 
"Facility 1 in Province", "Facility 2 in Province, Facility 3 in 
Province" or "Primary Facility in Province", "Secondary Facility in 
Province", "Tertiary Facility in Province".

""

TO DO: Clarify the types of CA locations that should be disclosed in the 
audit statement. e.g. data center locations, registration authority 
locations, where IT and business process controls of CA operations are 
performed, facility hosting an active HSM with CA private keys, facility 
or bank deposit box storing a deactivated and encrypted copy of a 
private key, other?


I will continue to appreciate your feedback on this, and the entire 
"Audit Delay" section.



I also filed an issue in GitHub regarding adding this to Mozilla's root 
store policy.

https://github.com/mozilla/pkipolicy/issues/207

Thanks,
Kathleen
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: COVID-19 and CA Operational Status

[Ryan Sleevi via dev-security-policy]

On Mon, Mar 23, 2020 at 6:18 PM Burton  wrote:

> Hi Ryan,
>
> I’m in the believe that CAs are a public service and as such they should
> provide public information regarding their operational status. The
> questions outlined below were open ended to provide CAs flexibility in the
> way they approach answering the questions.
>

While I appreciate that explanation, I disagree that there's much more
value to be had versus asking CAs what their favorite color was or how they
were feeling today. These questions are vague, and your further comments
(quoted) only emphasize the "damned if they do, damned if they don't"
approach you're proposing, which I think is quite unhealthy.

I believe that the questions are of value to the community only if CAs
> cooperate by providing answers that are useful, brief and to the point
> instead of non useful throwaway answers.
>
> To end, I do hope that most if not all CAs answer these simple questions
>

As a party deeply invested and interested in the trustworthiness of CAs, I
cannot help but again stress that these are anything but simple.


> as it doesn’t look good if they don’t because CAs are about trust and
> trust is fulfilled by being open as they can be regarding their operations.
> Without said openness there is no trust.
>

I think it's worth asking whether that same justification applies to asking
CAs what their favorite color was, or how much each person makes, or what
they had for breakfast. You might think that's not fair, because
operational status is, yanno, operational, but the lack of definition here
is so profound that it makes the questions meaningless.

I also worry that rather than improve participation here by CAs, it serves
as a barrier and a justification for why they shouldn't, if "just anyone"
can submit questions that they have to answer, are judged if they don't,
and judged if they do. The policy doesn't require that degree of
participation, as it says (emphasis added):

CAs MUST follow and be aware of discussions in the
mozilla.dev.security.policy forum, where Mozilla's root program is
coordinated. They are encouraged, *but not required,* to contribute to
those discussions.

I think the process of formal CA Communications is designed to try to ask
questions that are crisp, clear, and actionable - which is why these
communications undergo public review for feedback before they're sent, to
try to make sure they're asking the right questions. It also helps clarify
what's required, and what isn't.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: COVID-19 and CA Operational Status

[Kathleen Wilson via dev-security-policy]

All,

If Mozilla decides to ask each CA in our program these types of 
questions, we will do so via a CA Communication 
(https://wiki.mozilla.org/CA/Communications).


I appreciate Burton's curiosity, but your participation in this 
particular discussion thread is optional, and will not be considered to 
be your CA's response to an official Mozilla CA Communication and survey.


Thanks,
Kathleen
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: COVID-19 and CA Operational Status

[Burton via dev-security-policy]

Hi Ryan,

I’m in the believe that CAs are a public service and as such they should
provide public information regarding their operational status. The
questions outlined below were open ended to provide CAs flexibility in the
way they approach answering the questions.

I believe that the questions are of value to the community only if CAs
cooperate by providing answers that are useful, brief and to the point
instead of non useful throwaway answers.

To end, I do hope that most if not all CAs answer these simple questions as
it doesn’t look good if they don’t because CAs are about trust and trust is
fulfilled by being open as they can be regarding their operations. Without
said openness there is no trust.

Thank you

Burton





I didn’t want CAs to disclose confidential informatio publicly I wanted to
know

On Mon, 23 Mar 2020 at 21:42, Ryan Sleevi  wrote:

>
>
> On Mon, Mar 23, 2020 at 3:13 PM Burton via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
>
>> CAs,
>>
>> Please can you give a brief statement regarding these questions below:
>>
>> a) What’s your operational status at this time?
>>
>> b) Do you expect in the next six months to maintain an adequate
>> operational
>> status?
>>
>> c) If the worst case scenario does happen, what have you planned to
>> maintain operationally?
>>
>
> I think it's unlikely to get responses from many CAs. There is purely risk
> here, with an unclear goal. I appreciate the clarity, but I also don't
> think m.d.s.p. is necessarily a good venue for CA communications. For
> example, if you'd like to submit this to Kathleen as a suggestion for a CA
> communication, that might be a more productive endeavor.
>
> It also means that open ended questions like this may not get consistent
> answers across CAs. For example, CA Foo might say their operational status
> is "Case Nightmare Green"  while CA Bar might say that their operational
> status is "Major Tom" and CA Baz says "The eagle has left the nest". Those
> are answers, but are they helpful? Similarly, "adequate" operational status
> and "worst case" are equally ill-defined.
>
> In short, while I appreciate the curiosity, I don't think anything of
> value can be gained from this thread, at least holistically. And it just
> seems inherently risky for CAs to respond without that shared context.
>
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: COVID-19 and CA Operational Status

[Ryan Sleevi via dev-security-policy]

On Mon, Mar 23, 2020 at 3:13 PM Burton via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> CAs,
>
> Please can you give a brief statement regarding these questions below:
>
> a) What’s your operational status at this time?
>
> b) Do you expect in the next six months to maintain an adequate operational
> status?
>
> c) If the worst case scenario does happen, what have you planned to
> maintain operationally?
>

I think it's unlikely to get responses from many CAs. There is purely risk
here, with an unclear goal. I appreciate the clarity, but I also don't
think m.d.s.p. is necessarily a good venue for CA communications. For
example, if you'd like to submit this to Kathleen as a suggestion for a CA
communication, that might be a more productive endeavor.

It also means that open ended questions like this may not get consistent
answers across CAs. For example, CA Foo might say their operational status
is "Case Nightmare Green"  while CA Bar might say that their operational
status is "Major Tom" and CA Baz says "The eagle has left the nest". Those
are answers, but are they helpful? Similarly, "adequate" operational status
and "worst case" are equally ill-defined.

In short, while I appreciate the curiosity, I don't think anything of value
can be gained from this thread, at least holistically. And it just seems
inherently risky for CAs to respond without that shared context.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Is issuing a certificate for a previously-reported compromised private key misissuance?

[Ryan Sleevi via dev-security-policy]

On Mon, Mar 23, 2020 at 2:43 PM Bruce via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Thursday, March 19, 2020 at 2:02:39 AM UTC-4, Matt Palmer wrote:
>
> > 1. *Are* there explicit prohibitions on issuing a certificate for a
> private
> >key which has been previously submitted *to that CA* as compromised
> >(assuming, of course, that the prior submission was valid), and I'm
> just
> >not good at finding said prohibitions?
> >
> BR 6.1.1.3 has a weak key clause, "The CA SHALL reject a certificate
> request if the requested Public Key does not meet the requirements set
> forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key
> (such as a Debian weak key, see http://wiki.debian.org/SSLkeys)."
>
> I would think that "issuing a certificate for a private key which has been
> previously submitted *to that CA* as compromised" is not in the spirit of
> the weak key clause. It would be best if the CA would blacklist the public
> key to prevent future issuance for the compromised private key.
>

Yeah, https://github.com/cabforum/documents/issues/171 is filed to track
this.

I've held off preparing a CA/Browser Forum ballot so that we can make sure
to address the issue(s) holistically here, as we see the incidents coming
in.

(Sorry for the double post, from the right address this time)
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: Auditing of CA facilities in lockdown because of an environmental disaster/pandemic

[Jeremy Rowley via dev-security-policy]

Although I’m sure every CA has business continuity plans, I think that extended 
blocked access to every data center they have may not be part of that plan.  
I’m not sure, but I think if the required shelter’s are in place for long 
periods you may start to see problems. Early disclosure sounds like the best 
policy, but I thought the early disclosure requirement may be worth calling out 
in the Mozilla policy. Then again, that really should be standard procedure at 
that point.

From: Ryan Sleevi 
Sent: Friday, March 20, 2020 2:57 PM
To: Jeremy Rowley 
Cc: Kathleen Wilson ; Mozilla 

Subject: Re: Auditing of CA facilities in lockdown because of an environmental 
disaster/pandemic

On Fri, Mar 20, 2020 at 4:15 PM Jeremy Rowley via dev-security-policy 
mailto:dev-security-policy@lists.mozilla.org>>
 wrote:
What about issues other than audits? For example, with certain locations 
closing, key ceremonies may become impossible, leading to downed CRLs/OCSP for 
intermediates. There's also a potential issue with trusted roles even being 
able to access the data center if something goes down and Sub CAs can't be 
revoked. Should that be mentioned, requiring CAs to file an incident report as 
soon as the event becomes likely?

Yes. I think those are, quite honestly, much more concerning, because that's 
not about a CA's relationship with an external party, but about a CA's own 
preparedness for disaster. In any event, as with /any/ incident, the sooner 
it's filed, and the more information and context is provided, the more 
effective a response can be.


For the location issue, I think including the locations audited and the 
locations not audited (to the full criteria) as an emphasis of matter would be 
helpful. So maybe an emphasis like we audited the offices in x, y, and z. 
Office z was inaccessible to evaluate criteria 1-n. It give you the list of 
locations and where there were issues in getting access due t o he emergency.

Yup. That is the model WebTrust is using, and that reasonably meets the 
objective here of informing relying parties when the auditor faced limitations 
that should be considered when evaluating their report.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

COVID-19 and CA Operational Status

[Burton via dev-security-policy]

CAs,

Please can you give a brief statement regarding these questions below:

a) What’s your operational status at this time?

b) Do you expect in the next six months to maintain an adequate operational
status?

c) If the worst case scenario does happen, what have you planned to
maintain operationally?

Thank you,

Burton
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Is issuing a certificate for a previously-reported compromised private key misissuance?

[Bruce via dev-security-policy]

On Thursday, March 19, 2020 at 2:02:39 AM UTC-4, Matt Palmer wrote:

> 1. *Are* there explicit prohibitions on issuing a certificate for a private
>key which has been previously submitted *to that CA* as compromised 
>(assuming, of course, that the prior submission was valid), and I'm just
>not good at finding said prohibitions?
> 
BR 6.1.1.3 has a weak key clause, "The CA SHALL reject a certificate request if 
the requested Public Key does not meet the requirements set forth in Sections 
6.1.5 and 6.1.6 or if it has a known weak Private Key (such as a Debian weak 
key, see http://wiki.debian.org/SSLkeys)."

I would think that "issuing a certificate for a private key which has been 
previously submitted *to that CA* as compromised" is not in the spirit of the 
weak key clause. It would be best if the CA would blacklist the public key to 
prevent future issuance for the compromised private key.

Bruce.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: Digicert: failure to revoke certificate with previously compromised key

[Jeremy Rowley via dev-security-policy]

Yeah  - that’s about the sum of it. I’ll file an incident report.

There are two things worth discussing in general:

  1.  I’m very interested in seeing the Let’s Encrypt response to this issue 
since the biggest obstacle in trying to find all of the keys with the same 
private key is the sheer volume of the certs. Trying to do a comprehensive 
search when a private key is provided leaves some window between when we start 
the analysis and when we revoke.


  1.  Another issue in trying to report keys that aren’t affiliated with any 
cert is that the process becomes subject to abuse. Without knowing a cert 
affiliated with a key, someone can continuously generate keys and submit them 
as compromised. You end up just blacklisting random keys, DDOSing the 
revocation system as it kicks off another request to  search for those keys.  I 
don’t think it’s feasible. This is why the disclosures need to be affiliated 
with actual certs.




From: Ryan Sleevi 
Sent: Monday, March 23, 2020 10:54 AM
To: Jeremy Rowley 
Cc: Matt Palmer ; Mozilla 

Subject: Re: Digicert: failure to revoke certificate with previously 
compromised key



On Mon, Mar 23, 2020 at 11:01 AM Jeremy Rowley via dev-security-policy 
mailto:dev-security-policy@lists.mozilla.org>>
 wrote:
Hey Matt,

Ryan's post was the part I thought was relevant, but I understood it 
differently. The cert was issued, but we should have now revoked it (24 hours 
after receiving notice). I do see your interpretation though, and the language 
does support 24 hours after issuing the new cert.  What I need is a tool that 
scans after revocation to ensure there are no additional certs with the same 
key.  The frustration is that this was where the cert was issued after our scan 
of all keys but just before revocation.  As a side note, our system blacklists 
the keys when a cert is revoked for key compromise, which means I don't have a 
way to blacklist a key before a cert is ever issued.

Matt, Jeremy,

To make sure I understand the timeline correctly:
2020-03-20 02:05:49 UTC - Matt reports SPKI 
4310b6bc0841efd7fcec6ba0ed1f36e7a28bf9a707ae7f7771e2cd4b6f31b5af, associated 
with https://crt.sh/?id=1760024320 , as compromised
2020-03-21 01:56:31 UTC - DigiCert issues https://crt.sh/?id=2606438724 with 
that same SPKI
2020-03-21 02:09:12 UTC - DigiCert revokes https://crt.sh/?id=1760024320
2020-03-23 03:16:18 UTC - DigiCert revokes https://crt.sh/?id=2606438724

Is that roughly correct?

If so, it does seem like an Incident Report is warranted here, so we can 
understand why:
a) https://crt.sh/?id=2606438724 wasn't revoked when 
https://crt.sh/?id=1760024320 was revoked (assuming those timestamps in the CRL 
are accurate)
b) The key wasn't blocklisted as known compromised (if the timestamps are 
incorrect)

That is, it doesn't seem unreasonable that, for situations of key compromise, 
the CA has the necessary data to scan their systems for potential reuse of that 
key. Given DigiCert's data 
lake, it should be 
possible to scan for issues.

If I've misunderstood the timing here, please feel free to correct. This is 
where the incident report process is useful, and Resolved/Invalid is a 
perfectly fine state to end in.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: QuoVadis: Failure to revoke key-compromised certificates within 24 hours

[Stephen Davidson via dev-security-policy]

Hi Ryan:
As you wish.  I will start an incident report.  I do not believe there is a 
compliance failure here.
Regards, Stephen


From: Ryan Sleevi 
Sent: Monday, March 23, 2020 1:57 PM
To: Stephen Davidson 
Cc: Mozilla ; Matt Palmer 

Subject: Re: QuoVadis: Failure to revoke key-compromised certificates within 24 
hours



On Sun, Mar 22, 2020 at 10:03 PM Stephen Davidson via dev-security-policy 
mailto:dev-security-policy@lists.mozilla.org>>
 wrote:
Hello:
(Apologies if multiple copies of this are received.  The initial send was 
bounced by mdsp.)

Summary:  The certificates noted in Matt Palmer's email below were not in his 
original problem report to QuoVadis.  The certificates he reported were revoked 
in a time manner, and we acknowledged that additional certificates existed 
using the compromised private keys, and that they would be revoked as we 
identified them.  The client was notified of these additional certificates this 
morning which are scheduled to be revoked tonight.

Stephen:

This seems like a valid incident report, and worth following up on in Bugzilla. 
Would you like to open one with your preliminary findings, or would you like me 
to create one to be filled in by QuoVadis?

When it comes to reports of private key compromises, it seems the CA should be 
able to effectively determine the affected certificates (based on SPKI) and 
ensure these are all revoked in a timely fashion. Revoking some of them, but 
not all of them, seems like a BR violation.

It may be there are facts or understanding that's missing, and an incident 
report can help identify those, as well as any root causes or systemic 
mitigations to be deployed.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: QuoVadis: Failure to revoke key-compromised certificates within 24 hours

[Ryan Sleevi via dev-security-policy]

On Sun, Mar 22, 2020 at 10:03 PM Stephen Davidson via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Hello:
> (Apologies if multiple copies of this are received.  The initial send was
> bounced by mdsp.)
>
> Summary:  The certificates noted in Matt Palmer's email below were not in
> his original problem report to QuoVadis.  The certificates he reported were
> revoked in a time manner, and we acknowledged that additional certificates
> existed using the compromised private keys, and that they would be revoked
> as we identified them.  The client was notified of these additional
> certificates this morning which are scheduled to be revoked tonight.
>

Stephen:

This seems like a valid incident report, and worth following up on in
Bugzilla. Would you like to open one with your preliminary findings, or
would you like me to create one to be filled in by QuoVadis?

When it comes to reports of private key compromises, it seems the CA should
be able to effectively determine the affected certificates (based on SPKI)
and ensure these are all revoked in a timely fashion. Revoking some of
them, but not all of them, seems like a BR violation.

It may be there are facts or understanding that's missing, and an incident
report can help identify those, as well as any root causes or systemic
mitigations to be deployed.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Digicert: failure to revoke certificate with previously compromised key

[Ryan Sleevi via dev-security-policy]

On Mon, Mar 23, 2020 at 11:01 AM Jeremy Rowley via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> Hey Matt,
>
> Ryan's post was the part I thought was relevant, but I understood it
> differently. The cert was issued, but we should have now revoked it (24
> hours after receiving notice). I do see your interpretation though, and the
> language does support 24 hours after issuing the new cert.  What I need is
> a tool that scans after revocation to ensure there are no additional certs
> with the same key.  The frustration is that this was where the cert was
> issued after our scan of all keys but just before revocation.  As a side
> note, our system blacklists the keys when a cert is revoked for key
> compromise, which means I don't have a way to blacklist a key before a cert
> is ever issued.
>

Matt, Jeremy,

To make sure I understand the timeline correctly:
2020-03-20 02:05:49 UTC - Matt reports SPKI 4310b6bc0841efd7fcec6ba0ed1f36
e7a28bf9a707ae7f7771e2cd4b6f31b5af, associated with
https://crt.sh/?id=1760024320 , as compromised
2020-03-21 01:56:31 UTC - DigiCert issues https://crt.sh/?id=2606438724 with
that same SPKI
2020-03-21 02:09:12 UTC - DigiCert revokes https://crt.sh/?id=1760024320
2020-03-23 03:16:18 UTC - DigiCert revokes https://crt.sh/?id=2606438724

Is that roughly correct?

If so, it does seem like an Incident Report is warranted here, so we can
understand why:
a) https://crt.sh/?id=2606438724 wasn't revoked when
https://crt.sh/?id=1760024320 was revoked (assuming those timestamps in the
CRL are accurate)
b) The key wasn't blocklisted as known compromised (if the timestamps are
incorrect)

That is, it doesn't seem unreasonable that, for situations of key
compromise, the CA has the necessary data to scan their systems for
potential reuse of that key. Given DigiCert's data lake
, it should be
possible to scan for issues.

If I've misunderstood the timing here, please feel free to correct. This is
where the incident report process is useful, and Resolved/Invalid is a
perfectly fine state to end in.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: Digicert: failure to revoke certificate with previously compromised key

[Jeremy Rowley via dev-security-policy]

Hey Matt,

Ryan's post was the part I thought was relevant, but I understood it 
differently. The cert was issued, but we should have now revoked it (24 hours 
after receiving notice). I do see your interpretation though, and the language 
does support 24 hours after issuing the new cert.  What I need is a tool that 
scans after revocation to ensure there are no additional certs with the same 
key.  The frustration is that this was where the cert was issued after our scan 
of all keys but just before revocation.  As a side note, our system blacklists 
the keys when a cert is revoked for key compromise, which means I don't have a 
way to blacklist a key before a cert is ever issued. 

>> I don't think that supports your point, though, so I wonder if I've got the 
>> wrong part.  That last part of Ryan's: "shenanigans, such as CAs arguing 
>> the[y?] require per-cert evidence rather than systemic demonstrations", 
>> seems to me like it's describing your statement, above, that you (only?) 
>> "need to revoke a cert that is key compromised once we're the key is 
>> compromised *for that cert*" (emphasis added).  I don't read Ryan's use of 
>> "shenanigans" as approving of that sort of thing.

I don't think its shenanigans, but I do think it's a pretty common 
interpretation. Such information would help determine the common interpretation 
of this section.  I agree that CAs should scan all certs for the same key and 
revoke where they find them.  Is that actually happening? Do other CAs object 
to there being a lack of specificity if you give the keys without a cert 
attached? 

>> Bim bam bom, all done and dusted, and we can get back to washing our hands. 
That you're *not* doing that is perplexing, and a little disconcerting.

That's an oversimplification of the incident report process. I'm not resisting 
the incident report itself since incident reports are a good and healthy way 
for the public to have transparency into a CAs operation. In fact, I wouldn't 
mind filing one just to give public documentation on what DigiCert is doing for 
revocation.  I was objecting to actually calling this an incident/breach of the 
guidelines since I disagreed with that, and there's a trend where the 
interpretation of these sections seems to evolve over time. My main emphasis 
was that the guidelines are ambiguous and need refinement.  I'd support 
refining them to be more clear, but calling something shenanigans when the 
language is unclear is unfair.

For the SPKI, you're right. The slow part is downloading it from your website. 
I guess I should have said "a link to the key compromise proof" rather than the 
SPKI. The additional time required to do that was 5 hours. Although the 
automation isn't quite complete yet, the end-state goal is to do the following 
when we get a key compromise notification:
1) Run the CSR through a tool and confirm key compromise
2) Have the tool email the impacted subscriber
3) Have the tool process the revocation at the 24 hour mark. 

The two parts that are done is having the tool process the revocation and parse 
out all the key compromises and processing the revocation within 24 hours.  
Unfortunately, we haven't automated the cert problem report yet so we can only 
accept an email.  Once I get the revocation tool working the way I want it, 
we'll add a tool where you can submit a CSR directly and avoid the round trip. 

>>  Even if Digicert would just prefer that I did something differently in my 
>> problem reports in the future -- something that would make it legitimately 
>> easier for Digicert to process my reports, without making my life a misery 
>> (after all, y'all have a lot more resources than I do) -- all you have to do 
>> is ask.  My e-mail address is right there in every problem report I send. 

Nah - although I'd prefer just to get the CSRs in a zip file,  I can work with 
what you send us. In fact, I wouldn't change your process at all since I want 
to accommodate all sorts of ways for people to send us private keys. I'd rather 
not force you to do extra work to send us the keys in a different way as 
anything that creates barriers to key compromise reports is a bad thing. The 
reason I brought up the format is that I'm not sure when the 24 hour clock 
actually kicks off for revocation. Is it 24 hours after we get your email or 24 
hours after we can confirm key compromise?  I've always regarded these as a 
certificate problem report under 4.9.5 (requiring us to kick off the 
investigation) but then revocation should happen 24 hours after we have reason 
to suspect the key compromise is legit. Usually those are the same time, but 
with the email link (and number of keys in the last batch) it may take longer 
to actually do the investigation to show the key is compromised. I think 
there's ambiguity in when that timer starts, and clearing up that ambiguity in 
the CAB forum would be good.  As you pointed out "obtains" can mean different 
things. 

>> The time to say 

Re: Digicert: failure to revoke certificate with previously compromised key

[Matt Palmer via dev-security-policy]

On Mon, Mar 23, 2020 at 06:14:29AM +, Jeremy Rowley wrote:
> That's not the visible consensus IMO.  The visible consensus is we need to
> revoke a cert that is key compromised once we're informed the key is
> compromised for that cert
> (https://groups.google.com/forum/m/#!topic/mozilla.dev.security.policy/1ftkqbsnEU4).
>  

I think that link might not be doing what you expect, as it (at least for
me) is collapsing all the replies in that topic before Doug Beattie's post. 
The only response that seems relevant in that topic to was Ryan's reply to
me up-thread from Doug's post, which was, in (I believe) relevant part, when
I asked the question:

> 3. Can a CA be deemed to have "obtained evidence" of key compromise prior 
>to the issuance of a certificate, via a previously-submitted key
>compromise problem report for the same private key?  If so, it would
>seem that, even if the issuance of the certificate is OK, it is a
>failure-to-revoke incident if the cert doesn't get revoked within 24
>hours...

To which Ryan replied:

> Correct, that was indeed the previous conclusion around this. The CA can 
> issue, but then are obligated to revoke within 24 hours. There’s not a 
> statute of limitation on “obtains evidence” here, precisely because it 
> could allow a host of shenanigans, such as CAs arguing the require per-cert 
> evidence rather than systemic demonstrations. 

I don't think that supports your point, though, so I wonder if I've got the
wrong part.  That last part of Ryan's: "shenanigans, such as CAs arguing
the[y?] require per-cert evidence rather than systemic demonstrations",
seems to me like it's describing your statement, above, that you (only?)
"need to revoke a cert that is key compromised once we're the key is
compromised *for that cert*" (emphasis added).  I don't read Ryan's use of
"shenanigans" as approving of that sort of thing.

> The certificate you mentioned was issued before the keys were blacklisted
> and not part of a certificate problem report.  When revoking a cert we
> scan to see if additional certs are issued with the same key t, but this
> particular cert one was issued after the scan but before the revocation,
> largely because the way you are submitting certificate problem reports
> breaks automation.  We currently don't have a way to blacklist private
> keys until a certificate is revoked, although that would be a nice
> enhancement for us to add in the future.  Anyway, I don't think anything
> reported violated the  BR since 1) this cert was not part of a certificate
> problem report and 2) we will be revoking within 24 hours of your Mozilla
> posting.

The way I see it, once a CA is provided with accepted evidence of key
compromise, every certificate issued by that CA using the same key -- past,
present, and future -- is implicitly part of that certificate problem
report.  *I* think that's supported by Ryan's confirmation of my question,
quoted above, but presumably you disagree?

At the end of the day the only call that matters is that of the CA module
owner and peers, so I guess we'll have to leave it up to them to make the
call as to whether Digicert's behaviour that I described -- and the facts of
which you don't seem to dispute -- constitutes a BR violation.

Honestly, I'm a bit surprised you're trying so hard to argue against taking
responsibility for this occurrence (probably should use "incident" yet, I
guess).  Based on purely on what you wrote in the above paragraph, it seems
like it was a simple oversight in your systems for what is, I won't deny, a
bit of a corner case.  Even I, as gung-ho as I am about throwing out
misbehaving CAs, wouldn't argue that this in and of itself is anything worth
a rap over the knuckles for.

As far as I can see, the meat of an incident report for this occurrence
could be something like "our key blacklist is only populated at the time
revocation occurs; the certificate in question was issued between problem
report and revocation; we've added a story to the backlog to modify our CA
systems to allow keys to be blacklisted before revocation, and until then
we've modified our procedures to do a sweep of our certificate archive after
the initial revocation has taken place, to catch a similarly situated
certificate in the future."

Bim bam bom, all done and dusted, and we can get back to washing our hands. 
That you're *not* doing that is perplexing, and a little disconcerting.

> I support the idea of swift revocation of compromised private keys and do
> appreciate you reporting them.  I think this is helpful in ensuring the
> safety of users online.  However, using the SPKI to submit information
> breaks our automation, making finding and revoking certs difficult.  The
> more standards way (IMO) is the SHA2 thumbprint or serial number or a good
> old CSR.

This confuses me a bit; an SPKI *is* a "SHA2[56] thumbprint" of the
compromised key, and keys don't have serial numbers.  As for a CSR, I
provide one, signed 

RE: Digicert: failure to revoke certificate with previously compromised key

[Jeremy Rowley via dev-security-policy]

That's not the visible consensus IMO. The visible consensus is we need to 
revoke a cert that is key compromised once we're informed the key is 
compromised for that cert 
(https://groups.google.com/forum/m/#!topic/mozilla.dev.security.policy/1ftkqbsnEU4).
 The certificate you mentioned was issued before the keys were blacklisted and 
not part of a certificate problem report.  When revoking a cert we scan to see 
if additional certs are issued with the same key t, but this particular cert 
one was issued after the scan but before the revocation, largely because the 
way you are submitting certificate problem reports breaks automation. We 
currently don't have a way to blacklist private keys until a certificate is 
revoked, although that would be a nice enhancement for us to add in the future. 
 Anyway, I don't think anything reported violated the  BR since 1) this cert 
was not part of a certificate problem report and 2) we will be revoking within 
24 hours of your Mozilla posting. 

I support the idea of swift revocation of compromised private keys and do 
appreciate you reporting them. I think this is helpful in ensuring the safety 
of users online. However, using the SPKI to submit information breaks our 
automation, making finding and revoking certs difficult. The more standards way 
(IMO) is the SHA2 thumbprint or serial number or a good old CSR.  Because 
submitting the SPKI breaks automation, getting evidence of key compromise took 
an additional 5 hours after you submitted the report. We still revoked all of 
the current certs with submitted keys within 24 hours of the report (since 
compromised private keys are bad and there is nothing that says we can't revoke 
earlier than 24 hours), but I did want to clarify that I don't think the time 
starts until we can actually get the information necessary to do an 
investigation (because there is not sufficient evidence of a key compromise 
until then). 

Going to the previous discussion, I'd definitely support seeing a standardized 
way to report key compromise. Trying to account for the various formats they 
come in and through the various channels creates a lot of manual work on a 
process that can easily be automated. 

Jeremy


-Original Message-
From: dev-security-policy  On 
Behalf Of Matt Palmer via dev-security-policy
Sent: Saturday, March 21, 2020 11:01 PM
To: Mozilla 
Subject: Digicert: failure to revoke certificate with previously compromised key

Certificate https://crt.sh/?id=2606438724, issued either at 2020-03-21
00:00:00 UTC (going by notBefore) or 2020-03-21 01:56:31 UTC (going by SCTs), 
is using a private key with SPKI 
4310b6bc0841efd7fcec6ba0ed1f36e7a28bf9a707ae7f7771e2cd4b6f31b5af, which was 
reported to Digicert as compromised on 2020-03-20 02:05:49 UTC (and for which 
https://crt.sh/?id=1760024320 was revoked for keyCompromise soon after 
certificate 2606438724 was issued).

As previously discussed on this list, the visible consensus is that, according 
to the BRs, certificates for which the CA already had evidence of key 
compromise must be revoked within 24 hours of issuance.  That 24 hour period 
has passed for the above certificate, and thus it would appear that Digicert 
has failed to abide by the BRs.

- Matt

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy