> Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in 
pkey_rsa_encrypt() and pkey_rsa_decrypt().
    > 
    > They cause a crash when called with out==NULL. Normally it should not 
happen 
    > but when an engine is called, and it cannot process the padding it 
reverts to the
    > original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as 
appropriate).

    The original RSA pkey method has the flag EVP_PKEY_FLAG_AUTOARGLEN set which
    handles the NULL output automatically so it is not handled in pkey_rsa_*().
    
    The ENGINE should either set this flag itself too or deal with NULL 
arguments
    manually if that is not appropriate.
    
Since hardware tokens I’m dealing with do not perform any public key operations 
(the engine in this case is used to merely pull and provide the public key  to 
the requestor) I’m somewhat ambivalent about writing engine Encrypt function 
specifically for handling the NULL argument case. On the one hand, it’s the 
simplest solution, and it avoids going through OpenSSL modification process.;) 
On the other hand, it’s not as clean as I’d like.

Where would I set this flag ? And would it work when the public key is on the 
token, and needs to be retrieved via engine?

Attachment: smime.p7s
Description: S/MIME cryptographic signature

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Reply via email to