Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and pkey_rsa_decrypt().
They cause a crash when called with out==NULL. Normally it should not happen –
but when an engine is called, and it cannot process the padding – it reverts to
the original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as
appropriate). OpenSSL pkeyutl makes two calls when the key is not directly
available (aka not presented in a disk file), and the first call with out==NULL
crashes when RSA_private_decrypt() or RSA_public_encrypt() tries to copy the
result to out.
The fix should be adding something like
if (out == NULL) {
int klen = RSA_size(ctx->pkey->pkey.rsa);
*outlen = klen;
return 1;
}
right before the call to RSA_public_encrypt().
P.S. It’s more critical in pkey_rsa_encrypt(), because it’s more likely that
the engine would handle the decryption operation completely by itself.
--
Regards,
Uri Blumenthal
smime.p7s
Description: S/MIME cryptographic signature
-- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
