On Tuesday, March 10, 2020 at 8:44:49 PM UTC-5, Matt Palmer wrote: > On Tue, Mar 10, 2020 at 01:48:49PM -0700, Chris Kemmerer via > dev-security-policy wrote: > > We have updated https://bugzilla.mozilla.org/show_bug.cgi?id=1620772 with > > the findings of our current investigation. > > Thanks for this update. I have... comments. > > Before I get into the nitty-gritty, though, I'd just like to say that I > found your response to be unnecessarily defensive. So much of it reads -- > to me, at least -- as "please don't blame us, it wasn't our fault!". Fault > isn't really the issue at hand. Discovering what went wrong, and making > sure that the issues are fixed, both for SSL.com and, if appropriate, other > CAs, is what I, at least, am trying to achieve. > > Therefore, while a lot of my responses below address specific points that > you've made in your defence, please understand that I'm not trying to rebut > them in an attempt to say "nee nah nee nah, it *is* your fault!", but rather > to try and provide further information that SSL.com and other CAs could > benefit from considering for improvement in the future, given my experience > dealing with Debian weak keys.
We share the same view. Our analysis so far confirms that this is a tricky issue, and it gets trickier if one considers compatibility issues between CA software and lists of weak keys which are publicly available. This is exactly why it would be a useful contribution to the ecosystem, if anyone in possession of such weak key databases publishes them along with the keys. We have included more details for this below. > > For the purpose of identifying whether a Private Key is weak, SSL.com uses > > a set of Debian weak keys that was provided by our CA software vendor as > > the basis for our blacklist. > > I think it's worth getting additional, *very* detailed, information from > your CA software vendor as to where *they* got their Debian weak key list > from. That appears to be the fundamental breakdown here -- you relied on a > third-party to give you good service, and they didn't. So I think that > digging into your vendor's practices is an important line of enquiry to go > down. As mentioned on our report, we used that list as a basis, and paid attention to augment it with other weak keys from available blacklists, even for the ROCA vulnerability. So, instead of just relying on the CA software vendor, we definitely did - and do, on a regular basis - our own homework. > Also, given that there is a non-zero chance that other CAs trusted by > Mozilla may be using the same software, with the same incomplete list of > weak keys, I think it's important that the fact that the vendor is using a > demonstrably incomplete list be circulated to the other customers of this > vendor. How would you suggest this is best accomplished? Our CA software vendor is monitoring this list so they should be able to address this issue independently. For what it's worth, we believe that the current language in the BRs could be less ambiguous as far as the Debian weak keys are concerned. For example, it seems that the community's expectations are for CAs to detect and block weak Debian keys generated by vulnerable RNG using OpenSSL in popular architectures. That could be added in the BRs: Change: "The CA SHALL reject a certificate request if the requested Public Key does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys)" to something like: "The CA SHALL reject a certificate request if the requested Public Key does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key using, as a minimum set the Debian weak keys produced by OpenSSL in i386 and x64 architectures (see http://wiki.debian.org/SSLkeys)" Then, it would be clear that all CAs would need to block "at least" the vulnerable keys produced by OpenSSL and could add other keys produced by OpenSSH or other applications if they wanted a "more complete" list. > > This information was disclosed on 2020-03-07 to the person that submitted > > the Certificate Problem Report. > > I don't see anything from SSL.com that looks relevant in my inbox, but it's > not an important point, just thought I'd mention it in passing. For the record, we replied to mpal...@hezmatt.org at 2020-03-07 8:41 PM (UTC) Email subject: "Problem Report for certificate(s) with compromised private key" Can you please confirm? We did disclose the source of our weak keys in that email. > > The above practices comply with our CP/CPS, version 1.8, section 6.1.1.2, > > which states: > > > > "SSL.com shall reject a certificate request if the request has a known > > weak Private Key". > > Hmm, "known" is doing a lot of work in that sentence. Known to *whom* is > the important, but unanswered, question. It may be a question which should > be answered explicitly in SSL.com's CPS, as well as the CPS of all other CAs > (and even, potentially, in the BRs -- although my understanding is that that > little bit of the BRs may at some point in the near future get a tidy-up, > per https://github.com/cabforum/documents/issues/164). > > If the appropriate answer is "known to SSL.com", then you could run your CA > software with an empty blacklist, issue certs for all manner of known-weak > keys, and still be compliant with your CPS. As such, that's probably not an > acceptable answer to Mozilla. "Known to SSL.com's CA vendor" is similarly > problematic. > > If, on the other hand, the appropriate interpretation is "known to anyone", > then because the key was known to me, and I think I count as "anyone", your > CPS was not followed. > > "Known to the vendor of the software which generated the known-bad key" is > also an answer that results in your violating your CPS and the BRs, because > the key you issued a certificate for was, as previously mentioned, included > in the `openssl-blacklist` Debian package. > > Do you have another answer to the question "known to whom?" which SSL.com is > using in that sentence? This language was mainly taken from the Baseline Requirements. We will update this section in our next CP/CPS version to make it clearer, but we agree that Baseline Requirements should be updated to express the exact requirement more clearly. We elaborate further on that below. > > Our understanding is that there is no single or complete list with all > > known Debian weak keys, either one that is normative for use by the CAs > > included in the Mozilla Root Program, nor one specified in the Baseline > > Requirements. > > That is, I believe, correct, however (at the risk of tooting my own horn) > there is quite a comprehensive collection of Debian weak keys in the > pwnedkeys.com database. You are welcome to encourage your CA software > vendor to perform lookups against the public API if you wish. I don't claim > that any possible key you could generate from a buggy Debian system is > already in pwnedkeys, but I've accumulated a fair collection of likely > candidates, at great cost of (mostly emulated) CPU cycles. Unfortunately the format of the blacklists in openssl-blacklist is incompatible with our CA software which expects the SHA256 hash of the modulus, that's why we mention about acquiring the actual public keys. While we could use the "openssl-blacklist" package to conduct a short-term scan, it would be inefficient compared to the final solution. We already scanned our entire certificate database to detect weak Debian keys included in "openssl-blacklist" and found only 1 entry (the certificate that was reported and is now revoked). In our understanding, many individuals are in possession of such lists and it would certainly expedite things, along with reducing waste from duplicate effort, if you or others with similar lists, decided to share the actual keys with the broader community. The same applies for private keys that have been publicly disclosed. Maintaining such a list and justifying each entry is a great contribution to the community. Even getting the modulus of those keys would solve many compatibility issues with various "key detection" solutions. Regarding the use of external third-party systems, such as the pwnedkeys.com database, we are of the opinion that no CA's certificate issuance process and/or their compliance should rely on an external system which has not been evaluated and without any SLA guarantees. > > This can be demonstrated by submitting the following CSR > > That CSR uses a 2048 bit key generated on an i386 system, using OpenSSH with > an exponent of 3, with PID 23747. It might not be in certwatch, but it's in > pwnedkeys. :grin: > > > We have strong indications that there are several different lists of > > precalculated vulnerable keys whose precise populations depend on > > combinations of: > > > > architecture > > Yes, different CPU word sizes and endianness produce different keys. > That is covered in https://wiki.debian.org/SSLkeys#How_weak.3F. > > > key size > > That different key sizes would produce different keys should not be > surprising. > > > process ID > > Yes, that is the fundamental nature of the bug, that the random seed is > taken entirely from the ID of the process that generates the key. > > > (under certain conditions) application (openssh, openvpn, openssl) > > Yes, insofar as OpenSSL and OpenSSH use slightly different key generation > parameters (for example, older versions of OpenSSH used e=3, which produce a > different set of public keys than when e=65537 or otherwise). OpenVPN > itself only generates static (pre-shared) keys, not RSA keys, so is not > relevant for TLS. > > > openssl environment (for example, the use of .rnd file in the user's home > > directory and whether openssl can read this file). > > Yes, whether `~/.rnd` exists and/or is readable is one of the parameters > that is varied during generation of a complete set of weak private keys > using OpenSSL, for a given architecture/keysize pair. This is probably the > hardest aspect of the weak key generation process to know about, because it > does require examination of the method by which `openssl-blacklist` was > generated, but if your CA software vendor only included keys generated by > OpenSSH, it's somewhat of a moot point. This is useful. It confirms some of our investigation findings. We first wanted to understand the underlying technology before developing an improved Debian weak key detection mechanism. > > We were not aware that the application and variables produce different keys. > > If you're relying on your CA software vendor for your key blacklist, then I > wouldn't say that you would *need* to be particularly aware of the > permutations for generating a list of known-weak keys. You merely have to > have a CA software vendor that *does* have that awareness. > > > The lists provided by our CA software vendor and > > https://github.com/g0tmi1k/debian-ssh were probably produced using > > openssh. > > Quite possibly; once again, you really should ask your CA software vendor > for full and complete details of how the key blacklist they gave you came to > be. If they only used a key list that was generated by OpenSSH, I'd be > concerned on your behalf, because by far the more common source of keys used > in SSL certificates would be generated by OpenSSL, since that is, I would > expect, what most people use to generate CSRs. This was not required in the Baseline Requirements. We agree that this SHOULD be the common source of keys for TLS Certificates and we will work to include this list in our weak key detection engine. > > While openssh uses libcrypto/ssl for the key generation, it seems to > > generate different keys than openssl itself. > > Yes, it does, by using different parameters (such as e=3 rather than > e=65537) in older versions of OpenSSH. > > > In our understanding, the events detailed in this bug do not constitute a > > compliance violation that rises to the level of an incident. > > My understanding is that *all* compliance violations are an incident, in > Mozilla's eyes. However if my understanding is incorrect, I'm sure a module > peer or owner will be willing to correct me. You are correct, each compliance violation is considered an incident. However in our opinion we have not violated our CP/CPS or the current Baseline Requirements. Although this is a complex issue with no definite consensus on which authoritative list to use (only suggestions), we do have a weak keys detection mechanism in place, it does detect Debian weak keys (although it's not perfect) and it also detects ROCA vulnerable keys. Of course, we continue to use this discussion as an opportunity for improvement, for our own operations and for the community at large. csk > - Matt _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
