On Wednesday, March 11, 2020 at 5:41:00 PM UTC-5, Matt Palmer wrote:
> On Wed, Mar 11, 2020 at 10:46:05AM -0700, Chris Kemmerer via
> dev-security-policy wrote:
> > On Tuesday, March 10, 2020 at 8:44:49 PM UTC-5, Matt Palmer wrote:
> > > On Tue, Mar 10, 2020 at 01:48:49PM -0700, Chris Kemmerer via
> > > dev-security-policy wrote:
> > > > For the purpose of identifying whether a Private Key is weak, SSL.com
> > > > uses
> > > > a set of Debian weak keys that was provided by our CA software vendor as
> > > > the basis for our blacklist.
> > >
> > > I think it's worth getting additional, *very* detailed, information from
> > > your CA software vendor as to where *they* got their Debian weak key list
> > > from. That appears to be the fundamental breakdown here -- you relied on
> > > a
> > > third-party to give you good service, and they didn't. So I think that
> > > digging into your vendor's practices is an important line of enquiry to go
> > > down.
> >
> > As mentioned on our report, we used that list as a basis, and paid
> > attention to augment it with other weak keys from available blacklists,
>
> So presumably if there are other Mozilla-trusted CAs using the same CA
> vendor, who *are* doing the bare minimum and just using the CA vendor's key
> list, they're even more vulnerable to a potential misissuance. As you
> mentioned that your CA software vendor does read this list, I *really* hope
> they speak up soon so we can figure out how they got their key list.
>
> > weak keys from available blacklists, even for the ROCA vulnerability.
>
> Sidenote: my understanding of ROCA is that it is of a different form to the
> Debian weak key problem, in that you can't a priori enumerate ROCA-impacted
> keys, but can only identify them as you find them. As such, my
> understanding is that there isn't, and in fact *cannot*, be a comprehensive
> "blacklist", as such, of keys affected by ROCA.
>
> Is your understanding of the ROCA vulnerability different to my description
> above, and if not, can you explain how a "blacklist"-based approach is a
> suitable mitigation for avoiding issuance of certificates using
> ROCA-impacted private keys?
>
> (Conversely, if it *is* possible get a comprehensive list of ROCA-impacted
> keys, I know what I'm doing this weekend...)
ROCA vulnerability detection is part of our “weak keys detection mechanism”,
not part of a blacklist. Our original language “we do have a weak keys
detection mechanism in place, it does detect Debian weak keys (although it's
not perfect) and it also detects ROCA vulnerable keys” makes that clear.
>
> > For what it's worth, we believe that the current language in the BRs could
> > be less ambiguous as far as the Debian weak keys are concerned. For
> > example, it seems that the community's expectations are for CAs to detect
> > and block weak Debian keys generated by vulnerable RNG using OpenSSL in
> > popular architectures.
>
> The problem with using the argument that "the BRs are ambiguous" to try and
> defend a breach of them is that there are always potential ambiguities in
> all language -- in many ways, "ambiguity is in the eye of the beholder"
> ("ambiguer"?). My understanding of the consensus from past discussions on
> this list is that if a CA believes there is an ambiguity in the BRs, the
> correct action is to raise that in the CA/B Forum *before* they fall foul of
> it.
>
> CAs should be reading the BRs, as I understand it, in a "defensive" mode,
> looking for requirements that could be read multiple ways, and when they are
> found, the CA needs to ensure either that they are complying with the
> strictest possible reading, or else bringing the ambiguity to the attention
> of the CA/B Forum and suggesting less ambiguous wording.
>
> At any rate, it would be helpful to know what, precisely, SSL.com's
> understanding of this requirement of the BRs prior to the commencement of
> this incident. Can you share this with us? Presumably SSL.com did a
> careful analysis of all aspects of the BRs, and came to a conclusion as to
> precisely what was considered "compliant". With regards to this
> requirement, what was SSL.com's position as to what was necessary to be
> compliant with this aspect of the BRs?
We have already described our understanding of the expectations expressed in BR
6.1.1.3 and the steps we took to comply with it. We provide details in bug
1620772, which remains the primary channel for this issue. As always, we
attempt to be as transparent as possible, because we strongly feel that this is
exactly the approach which better serves the ecosystem.
Our implementation did not meet these expectations, as it was missing direct
checks of keys matching the "openssl-blacklist" package. Immediately upon our
coming to this understanding, we initiated development of a fix to meet this
expectation. This fix was tested and pushed to production last Friday. In
parallel, we are conducting an analysis of which control(s) failed. Details on
our findings thus far and steps we are taking in remediation are also included
in our incident report.
>
> > That could be added in the BRs:
> >
> > Change:
> > "The CA SHALL reject a certificate request if the requested Public Key
> > does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if
> > it has a known weak Private Key (such as a Debian weak key, see
> > http://wiki.debian.org/SSLkeys)"
> >
> > to something like:
> > "The CA SHALL reject a certificate request if the requested Public Key
> > does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if
> > it has a known weak Private Key using, as a minimum set the Debian weak
> > keys produced by OpenSSL in i386 and x64 architectures (see
> > http://wiki.debian.org/SSLkeys)"
>
> It would appear that SSL.com is a member in good standing of the CA/B Forum.
> Is there any intention on the part of SSL.com to propose this change as a
> ballot? While you're at it, if you could include a fix for the issue
> described in https://github.com/cabforum/documents/issues/164, that would be
> appreciated, since it is the same sentences that need modification, and for
> much the same reasons.
Yes, this is reasonable, and we treated such key as compromised, revoking it
within 24 hours.
We would support a ballot that makes this clear. We also monitor the discussion
in https://github.com/cabforum/documents/issues/164.
>
> > Then, it would be clear that all CAs would need to block "at least" the
> > vulnerable keys produced by OpenSSL and could add other keys produced by
> > OpenSSH or other applications if they wanted a "more complete" list.
>
> Well, you're still missing the rnd/nornd/noreadrnd variations, and there's
> no specification as to what key sizes are considered the bare minimum.
We mention these variations in bug 1620772, but thank you for repeating it here
for completeness.
For the record, this fact (“there's no specification as to what key sizes are
considered the bare minimum”) is exactly our point too.
The BRs allow specific RSA key sizes. If we want to be unambiguous about this
requirement, we need to include our minimum (baseline) expectations.
>
> > > > This information was disclosed on 2020-03-07 to the person that
> > > > submitted
> > > > the Certificate Problem Report.
> > >
> > > I don't see anything from SSL.com that looks relevant in my inbox, but
> > > it's
> > > not an important point, just thought I'd mention it in passing.
> >
> > For the record, we replied to mpal...@hezmatt.org at 2020-03-07 8:41 PM
> > (UTC)
> > Email subject: "Problem Report for certificate(s) with compromised private
> > key"
> > Can you please confirm?
> > We did disclose the source of our weak keys in that email.
>
> Unfortunately that's the default subject line I use on my problem reports,
> so it's not as helpful as it might otherwise be. Looking through recent
> correspondence, though, I'm certainly not finding anything. If you've got a
> message ID, or even better the queue ID that my MTA would have responded
> with when it accepted the message (which could be obtained from your MTA's
> logs), that'd help me track down what happened to it. At any rate, there's
> no delivery attempts at *all* in my mail logs between 20:37:19 and 21:03:30
> on 2020-03-07, so it looks like it got in the post to at least some degree.
The reply was submitted to the MTA at 2020-03-07 04:04 UTC. Apologies that the
timezone conversion was incorrect. The ticket number was QGI-680-16751.
>
> > > > The above practices comply with our CP/CPS, version 1.8, section
> > > > 6.1.1.2,
> > > > which states:
> > > >
> > > > "SSL.com shall reject a certificate request if the request has a known
> > > > weak Private Key".
> > >
> > > Hmm, "known" is doing a lot of work in that sentence. Known to *whom* is
> > > the important, but unanswered, question.
>
> [snip]
>
> > This language was mainly taken from the Baseline Requirements.
>
> Sure, but SSL.com presumably came to some understanding as to what that
> sentence meant at the time it was included in SSL.com's CPS, given that the
> organisation was committing to do it. I think it would be valuable to know
> what that understanding was, if for no other reason than so that Mozilla can
> say, in the next CA communication, "if you think that "known weak Private
> Key" means "known to <X>, be advised that that that is not an appropriate
> interpretation, and you'll want to get onto that."
We have responded as to how we interpreted and implemented this requirement.
Our blacklist did not include the entire set of Debian weak keys. We have been
completely transparent about this issue and we have improved our weak key
detection mechanism to include the openssl-blacklist package.
We examined similar failures/incidents, such as:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1472052
- https://bugzilla.mozilla.org/show_bug.cgi?id=1435770
-
https://community.letsencrypt.org/t/2017-09-09-late-weak-key-revocation/42519
The last one shows that at least one more CA had a similar interpretation of
the BRs.
For us, this is an indication that this issue is not one particular
interpretation of a single CA. Updating this section in the BRs to reduce
misinterpretation would be beneficial for the greater PKI ecosystem.
>
> > Unfortunately the format of the blacklists in openssl-blacklist is
> > incompatible with our CA software which expects the SHA256 hash of the
> > modulus, that's why we mention about acquiring the actual public keys.
>
> That is an... interesting approach. How does your CA software check for
> known-bad EC keys? Yes, OpenSSL 0.9.8 didn't support EC keys, but Debian
> weak keys aren't the only known-bad keys in existence.
For EC keys, the CA software uses a different method to calculate the
fingerprints.
>
> > In our understanding, many individuals are in possession of such lists and
> > it would certainly expedite things, along with reducing waste from
> > duplicate effort, if you or others with similar lists, decided to share
> > the actual keys with the broader community.
>
> Given the number of active, unrevoked certificates using private keys that
> I've already found laying around (see, for example,
> https://misissued.com/batch/64/), dumping all the known-compromised private
> keys in existence in one place *might* not be considered a winning idea by
> all and sundry. I do appreciate your desire to watch the world burn,
> though. It's very Cyberpunk.
We only expected a list of weak or compromised public keys, not private keys,
to be disclosed.
>
> Further, a point-in-time publication of any list of keys would not be
> particularly useful, as new private keys are *regularly* being disclosed,
> and thus any list would very quickly be out-of-date. Even if the published
> list were regularly augmented with new entries, history has shown that
> people are far too keen on downloading a list once, stuffing into their (in
> this case) CA software, and saying "job done!", and thus they lull
> themselves into a false sense of security.
>
> I've seen this happen in many contexts over the years, such as in network
> operations -- bogon lists get loaded into routers and firewalls, they don't
> get updated, and it causes problems everywhere and forever. I'm not going
> to be party to creating yet another instance of the same problem with
> private keys.
Once a private key has been compromised, it's compromised. It's an "add-only"
database. Therefore, we believe it would be a large contribution to this
community if the *public keys* of compromised private keys (and reason of
compromise for verification) were available for download. CAs could
periodically synchronize their local blacklists with new entries of this
blacklist. This is one way we see to deal with the interoperability issue of
different implementations for weak/compromised key detection.
We would welcome discussion with security researchers regarding collecting,
documenting and disclosing such weak/compromised public keys as a means of
improving weak/compromised key detection by CAs and other participants in the
PKI community.
>
> > The same applies for private keys that have been publicly disclosed.
> > Maintaining such a list and justifying each entry is a great contribution
> > to the community. Even getting the modulus of those keys would solve many
> > compatibility issues with various "key detection" solutions.
>
> I am not averse to the provision of alternate formats of the pwnedkeys.com
> dataset, however I don't have enough spare time to generate every possible
> permutation of key representation, especially when they're as odd as "SHA256
> of the modulus" (even assuming there were a single unambiguous
> representation of an RSA modulus, which there most certainly isn't).
This was not our suggestion. Providing the public key associated with each
compromised private key would allow any implementation to do the necessary
conversions independently.
>
> > Regarding the use of external third-party systems, such as the
> > pwnedkeys.com database, we are of the opinion that no CA's certificate
> > issuance process and/or their compliance should rely on an external system
> > which has not been evaluated and without any SLA guarantees.
>
> There are commercial solutions to the problem of a lack of SLA guarantees,
> if that is the only roadblock.
SSL.com has been and continues to be transparent, by disclosing practices and
sources of information along with its understanding of the situation. Overall,
although we regret the circumstances, we view this issue as an opportunity for
improvement of both our own and the community’s understanding and operations.
We attempt to capture this, and to provide concrete steps for this improvement,
in our full incident report as posted in the corresponding bug.
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
