On 9/4/26 7:35 AM, Dmitry Dolgov wrote:
On Mon, Aug 31, 2026 at 09:37:39AM -0700, Si, Evan wrote:
Regardless, if RSA key exchange is used, SSL_get_negotiated_group is
supposed to return NID_undef. Things will error (Openssl 3.5 example):
Interesting, good to know, thanks. After a quick look I couldn't find
any documentation as to why it's happening this way, I only see OpenSSL
returning NID_undef if using tls1.2 and the ssl state has no session. Is
there any explanation?
The documentation doesn't look terribly clear about this to me either,
but I think its sensible. In the RSA case, there is no negotiation for
the key, so getting undef out of "SSL_get_negotiated_group" sounds
reasonable.
Poking around a bit more, there is some further nuance for non-EC DHE
key exchange (e.g. ssl_ciphers=DHE-RSA-AES128-GCM-SHA256). The server
always passes the FILE_DH2048 (or content of ssl_dh_params_file) into
SSL_CTX_set_tmp_dh. This case would also mean there is no negotiation
happening.
In TLSv1.3 there's no more support for custom dh though, so from my
testing this SSL_CTX_set_tmp_dh is completely ignored there and things
work fine (it has to all go through
ssl_groups/SSL_CTX_set1_groups_list). In other words only for TLSv1.2
and lower, non-EC DHE key exchange has no negotiation and thus NID_undef
comes out of the API.
Evan
