Hi Viktor,
I have assigned the task find out the root cause where the API is failing with
this composite number. I see that with this composite number, the API
BN_mod_inverse(Ri, R, &tmod, ctx) is returning NULL. (This is being called in
bn_mont.c).
This function is defined in bn_gcd.c
Because of this API failed to return non-null value, the final API
DH_generate_key() is failed to generate the DH public and private keys.
Can you explain what does the BN_mod_inverse() actually does.
Is this API related to the prime check on the DH Algorithm input prime number?
Regards,
Vishal
General
-----Original Message-----
From: openssl-users <openssl-users-boun...@openssl.org> On Behalf Of Viktor
Dukhovni
Sent: Friday, May 31, 2024 06:14 PM
To: openssl-users@openssl.org
Subject: Re: Issue in DH Algorithm Keys Generation in OpenSSL 3.3.0
[External email: Use caution with links and attachments]
________________________________
On Fri, May 31, 2024 at 12:39:12PM +0000, Vishal Kevat via openssl-users wrote:
> Is there any way to make this prime number work by doing some
> modifications in the openssl source code.
It ISN'T a *prime* number.
> Like bypassing the OpenSSL DH prime check?
Why do you want to use a broken DH group? Even if that 128-bit composite
number were instead prime, it would still be way too small to offer any
security.
It is hard to imagine how what you're asking for makes any sense.
--
Viktor.
