Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and pkey_rsa_decrypt().
They cause a crash when called with out==NULL. Normally it should not happen – but when an engine is called, and it cannot process the padding – it reverts to the original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate). OpenSSL pkeyutl makes two calls when the key is not directly available (aka not presented in a disk file), and the first call with out==NULL crashes when RSA_private_decrypt() or RSA_public_encrypt() tries to copy the result to out. The fix should be adding something like if (out == NULL) { int klen = RSA_size(ctx->pkey->pkey.rsa); *outlen = klen; return 1; } right before the call to RSA_public_encrypt(). P.S. It’s more critical in pkey_rsa_encrypt(), because it’s more likely that the engine would handle the decryption operation completely by itself. -- Regards, Uri Blumenthal
smime.p7s
Description: S/MIME cryptographic signature
-- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev