Sorry for the extra message but my description below is not quite correct. The problem is not that the key size is > 4096. It seems to happen whenever the key size is not a multiple of 8.
Graeme From: owner-openssl-...@openssl.org [mailto:owner-openssl-...@openssl.org] On Behalf Of Perrow, Graeme Sent: Friday, August 01, 2014 11:50 AM To: openssl-dev@openssl.org Subject: FIPS version of RSA_generate_key_ex puts FIPS library in a bad state If I attempt to create an RSA key pair with a size of >4096 bits using the FIPS library (FIPS 2.0.5, OpenSSL 1.0.1h), I get an error ("data too large for modulus"), but doing so seems to put the FIPS library into a bad state. Subsequent calls return failure and the error stack indicates that the FIPS self-test has failed. Source for a short C program is attached. If I attempt to use a small key size (say 500 bits), generating the key fails but subsequent actions are OK. A key size between 1024 and 4096 works as expected. If I try to use a key size of 4097, generating the key fails but then subsequent FIPS calls also fail. I am seeing this on both Windows 7 and Linux. Graeme Perrow
