Problems with decrypting using openssl-0.9.8b and windows 64 (x64). Same test works on windows 32 bit os.
We are having difficulties trying to decrypt a encrypted text string using a simple test application on a windows 64 bit os. The same test application works as expected on windows 32 bit os. The code is currently failing in the call to RSA_private_decrypt which in turn eventually calls BN_num_bits. int len1 = RSA_private_decrypt(modulusSize, (const unsigned char *)encryptedtext, decryptedtext, rsaKeyFromClient, RSA_PKCS1_PADDING); In windows 64 bit os, the function BN_num_bits returns 128 instead of 127. The windows 32 bit os correctly returns 127 from the call to BN_num_bits. See code snippet below for the function that is returning the incorrect value. Defined variables are BN_BITS2 = 64 and i = 15 based on the compile flags. int BN_num_bits(const BIGNUM *a) { int i = a->top - 1; bn_check_top(a); if (BN_is_zero(a)) return 0; return ((i*BN_BITS2) + BN_num_bits_word(a->d[i])); } Please see attached program which can be compiled and linked on windows 32 and 64 bit Visual Studio IDEs. Add the openssl 64 bit libs and includes to your VS solution to get it to compile and link. I have tested this using the pre-canned openssl libs and also built the openssl libs from the source code.
Win64server3.cpp
Description: Win64server3.cpp