I am using the openssl 3.0.0 library in an embedded environment and added a unit test for RSA based on test/rsa_mp_test.c:test_rsa_mp(). When the test is run, I see: 1. crypto/rsa/rsa_pk1.c:RSA_padding_check_PKCS1_type_2() calls RSAerr() just before returning at the bottom of the function in what appears to be the success path. 2. crypto/rsa/rsa_ossl.c:rsa_ossl_private_decrypt() calls RSAerr() near the bottom of the function, regardless of the value of ‘r’ – so also in what appears to be the success path.
Both errors indicate padding problems for what are presumably good test data. If I comment out the rsa_pk1.c case and add if (r < 0) to the rsa_ossl.c case, the final comparison in the test indicates that the encrypt/decrypt resulted in the original data. What am I missing here? --tnx --tom
