Thanks for the detailed explanation. So rsaEncryption cert can do both RSASSA-PKCS-v1_5 and RSASSA-PSS type signature. And also the digital signature present on the cert can be of type RSASSA-PKCS-v1_5 or RSASSA-PSS.
Currently in 1.1.1c's has_usable_cert() function, digital signature (Issuer's signature) present on rsaEncryption cert type is not checked. So if TLS1.3 client sends rsa_pss_rsae_xxx in "signature_algorithm" extension and if the server's rsaEncrypted cert has digital signature of type RSASSA-PKCS-v1_5, then it should not use that certificate but it is using currently. And also the hash algorithm used on rsaEncryption cert's digital signature is not checked with "signature_algorithm" extension received from client. If TLS1.3 client sends rsa_pss_rsae_sha384 and rsa_pss_rsae_sha512 in "signature_algorithm" extension, and if server has a rsaEncrypted cert with signature of type rsa_pss_rsae_sha256 then it should not use that certificate, but it is using currently. On Thu, Jun 6, 2019 at 9:11 PM Matt Caswell <m...@openssl.org> wrote: > > > On 06/06/2019 16:15, Raja Ashok wrote: > > Hi, > > > > Currently has_usable_cert() function is called on tls_choose_sigalg() to > find > > out the suitable certificate available. But currently rsa_pkcs1_xxx and > > rsa_pss_rsae_xxx certs are stored on same index SSL_PKEY_RSA. Because of > this it > > may ends in choosing rsa_pkcs1_xxx cert for rsa_pss_rsae_xxx extension. > Is this > > behaviour correct ? > > There are two things to consider: > > 1) The OID in the RSA cert, which can be one of rsaEncryption or > RSASSA-PSS. The > former is for "traditional" RSA certs, whilst the latter *only* allows use > of > the key for signing (it cannot be used for encryption). > > 2) The type of signing in use, e.g. RSASSA-PKCS-v1_5 or RSASSA-PSS. > > rsaEncryption certs are capable of doing *either* form of signing, whilst > RSASSA-PSS certs can only do PSS signing. > > We store rsaEncryption certs under the SSL_PKEY_RSA index, and RSASSA-PSS > certs > under the SSL_PKEY_RSA_PSS_SIGN index. > > TLSv1.2 and below signs handshake messages using PKCS v1.5. which > corresponds to > these signature algorithms: > > rsa_pkcs1_sha256(0x0401) > rsa_pkcs1_sha384(0x0501) > rsa_pkcs1_sha512(0x0601) > > These sig algs cannot be used in TLSv1.3 for signing handshake messages, > although they may appear in a ClientHello for backwards compatibility with > TLSv1.2. You can only use these sig algs with "traditional" RSA certs (not > PSS > RSA certs). > > TLSv1.3 signs handshake messages using PSS which corresponds to these > signature > algorithms for "traditional" (rsaEncryption) certs: > > rsa_pss_rsae_sha256(0x0804) > rsa_pss_rsae_sha384(0x0805) > rsa_pss_rsae_sha512(0x0806) > > Or these signature algorithms for PSS certs: > > rsa_pss_pss_sha256(0x0809) > rsa_pss_pss_sha384(0x080a) > rsa_pss_pss_sha512(0x080b) > > Therefore it is perfectly correct behaviour that a cert stored under the > SSL_PKEY_RSA index could be used for signing handshake message using either > rsa_pkcs1_xxx or for rsa_pss_rsae_xxx. The former is used in TLSv1.2 and > the > latter is used in TLSv1.3. > > Matt >