> Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in
pkey_rsa_encrypt() and pkey_rsa_decrypt().
>
> They cause a crash when called with out==NULL. Normally it should not
happen
> but when an engine is called, and it cannot process the padding it
reverts to the
> original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as
appropriate).
The original RSA pkey method has the flag EVP_PKEY_FLAG_AUTOARGLEN set which
handles the NULL output automatically so it is not handled in pkey_rsa_*().
The ENGINE should either set this flag itself too or deal with NULL
arguments
manually if that is not appropriate.
Since hardware tokens I’m dealing with do not perform any public key operations
(the engine in this case is used to merely pull and provide the public key to
the requestor) I’m somewhat ambivalent about writing engine Encrypt function
specifically for handling the NULL argument case. On the one hand, it’s the
simplest solution, and it avoids going through OpenSSL modification process.;)
On the other hand, it’s not as clean as I’d like.
Where would I set this flag ? And would it work when the public key is on the
token, and needs to be retrieved via engine?
smime.p7s
Description: S/MIME cryptographic signature
-- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
