On Wed, Sep 27, 2017, Blumenthal, Uri - 0553 - MITLL wrote: > > Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c > in pkey_rsa_encrypt() and pkey_rsa_decrypt(). > > > > They cause a crash when called with out==NULL. Normally it should not > happen > > but when an engine is called, and it cannot process the padding it > reverts to the > > original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as > appropriate). > > The original RSA pkey method has the flag EVP_PKEY_FLAG_AUTOARGLEN set > which > handles the NULL output automatically so it is not handled in > pkey_rsa_*(). > > The ENGINE should either set this flag itself too or deal with NULL > arguments > manually if that is not appropriate. > > Since hardware tokens I???m dealing with do not perform any public key > operations (the engine in this case is used to merely pull and provide the > public key to the requestor) I???m somewhat ambivalent about writing engine > Encrypt function specifically for handling the NULL argument case. On the one > hand, it???s the simplest solution, and it avoids going through OpenSSL > modification process.;) On the other hand, it???s not as clean as I???d like. > > Where would I set this flag ? And would it work when the public key is on the > token, and needs to be retrieved via engine?
It's set when the method is created via EVP_PKEY_meth_new(). If you set it it assumes the public key components are set in the EVP_PKEY and calls EVP_PKEY_size() appropriately to handle the NULL argument and if the supplied buffer is too small. Checkout the M_check_autoarg macro in crypto/evp/pmeth_fn.c to see exactly what it does. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
