OpenSSL implementation of OAEP wrongly refuses to set the hash algorithm, preventing one from using SHA-2 family:
You'll probably need to pick up master and its -rsa_mgf1_md argument to pkeyutl. Thank you – better with “-pkeyopt rsa_mgf1_md:sha256”. But still broken – as it affects only the MGF1 setting, but not the hash setting. I’d say it still needs to allow “-pkeyutl digest:xxx” parameter. $ ~/openssl-1.1/bin/openssl version OpenSSL 1.1.1-dev xx XXX xxxx $ ~/openssl-1.1/bin/openssl pkeyutl -encrypt -in t1264.dat -out t1264.dat.enc2.oaep -keyform DER -pubin -inkey rsa3072pub.der -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_mgf1_md:sha256 $ yhsm2-tool --decrypt -m RSA-PKCS-OAEP --id 0301 -i t1264.dat.enc2.oaep -o t1264.dat.dec2 --hash-algorithm SHA256 Using slot 0 with a present token (0x0) Logging in to "YubiHSM". Please enter User PIN: Using decrypt algorithm RSA-PKCS-OAEP OAEP parameters: hashAlg=SHA256, mgf=MGF1-SHA256, source_type=0, source_ptr=0x0, source_len=0 error: PKCS11 function C_Decrypt failed: rv = CKR_FUNCTION_FAILED (0x6) Aborting. $ yhsm2-tool --decrypt -m RSA-PKCS-OAEP --id 0301 -i t1264.dat.enc2.oaep -o t1264.dat.dec2 --hash-algorithm SHA-1 --mgf MGF1-SHA256 Using slot 0 with a present token (0x0) Logging in to "YubiHSM". Please enter User PIN: Using decrypt algorithm RSA-PKCS-OAEP OAEP parameters: hashAlg=SHA-1, mgf=MGF1-SHA256, source_type=0, source_ptr=0x0, source_len=0 $ cmp t1264.dat t1264.dat.dec2 $
smime.p7s
Description: S/MIME cryptographic signature
-- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev