Problems with decrypting using openssl-0.9.8b and windows 64 (x64). Same test works on windows 32 bit os.
We are having difficulties trying to decrypt a encrypted text string
using a simple test application on a windows 64 bit os. The same test
application works as expected on windows 32 bit os.
The code is currently failing in the call to RSA_private_decrypt which
in turn eventually calls BN_num_bits.
int len1 = RSA_private_decrypt(modulusSize, (const unsigned char
*)encryptedtext, decryptedtext, rsaKeyFromClient, RSA_PKCS1_PADDING);
In windows 64 bit os, the function BN_num_bits returns
128 instead of 127. The windows 32 bit os correctly returns 127 from
the call to BN_num_bits. See code snippet below for the function that
is returning the incorrect value. Defined variables are BN_BITS2 = 64
and i = 15 based on the compile flags.
int BN_num_bits(const BIGNUM *a)
{
int i = a->top - 1;
bn_check_top(a);
if (BN_is_zero(a))
return 0;
return ((i*BN_BITS2) +
BN_num_bits_word(a->d[i]));
}
Please see attached program which can be compiled and linked on windows
32 and 64 bit Visual Studio IDEs.
Add the openssl 64 bit libs and includes to your VS solution to get it
to compile and link. I have tested this using the pre-canned openssl
libs and also built the openssl libs from the source code.
Win64server3.cpp
Description: Win64server3.cpp
