I'm hoping someone has some insight into what I'm doing wrong or if I've just bumped up against a bug. I'm using Visual Studio 2008 on Windows 8 and am in the process of running existing code with FIPS enabled. As expected a few things don't work, but in this case, I'm a bit stumped. I've narrowed it down to the reproducer test shown below at the bottom of this email. If in not FIPS mode, it runs fine. If in FIPS mode, X509_sign() crashes. After the crash the call stack looks like it is corrupted so I can't tell where in openssl or the fips module things are falling apart.
The error in the crash indicates it is a memory read violation: Unhandled exception at 0x77ace3be (ntdll.dll) in Test.OpenSSL.exe: 0xC0000005: Access violation reading location 0x2ed31dc7. The call stack indicates it is trying to free up memory: > ntdll.dll!@RtlpLowFragHeapFree@8() + 0x2c bytes ntdll.dll!_RtlFreeHeap@12() + 0x7e bytes kernel32.dll!_HeapFree@12() + 0x14 bytes msvcr90.dll!_free() + 0xcd bytes libeay32.dll!0fb024dd() [Frames below may be incorrect and/or missing, no symbols loaded for libeay32.dll] libeay32.dll!0fbb45ec() kernel32.dll!_HeapFree@12() + 0x14 bytes Visual Studio also has a runtime memory checker which is available when run as "Start with Application Verifier" in the "Debug" drop-down menu. When I do this, the application verifier reports the following: Heap violation detected Memory access operation in the context of an allocated block: heap overrun or heap underrun. The call stack indicates that the failure is in libeay32.dll: libeay32.dll!0fbc0316() [Frames below may be incorrect and/or missing, no symbols loaded for libeay32.dll] libeay32.dll!0fbb357e() libeay32.dll!0fbb1619() libeay32.dll!0fb5eb3b() libeay32.dll!0fb5f87e() libeay32.dll!0fb63989() libeay32.dll!0fb63a96() libeay32.dll!0fb81bd1() > Test.OpenSSL.exe!TestCreateCertificate() Line 339 + 0x11 bytes C++ 00000041() So by all indication, in FIPS mode, I'm bumping up against memory corruption problems that is crashing my app. Any ideas? Thanks in advance for taking the time to read and respond to my request, Lester ================================================================ Test case. Crashes on X509_sign() only if in FIPS mode: int TestCreateCertificate() { //Enable FIPS if (FIPS_mode_set(1) != 1) return E_FAIL; //Initialize some randomness DWORD dw = GetCurrentProcessId(); RAND_add(&dw, sizeof(dw), 0); dw = GetCurrentThreadId(); RAND_add(&dw, sizeof(dw), 0); dw = GetTickCount(); RAND_add(&dw, sizeof(dw), 0); //Create an X509 Cert and sign it. RSA *rsa = RSA_generate_key(2048, RSA_F4, 0, 0); if (!rsa) { printf("Failed to generate the RSA key, certificate generation failed"); return E_FAIL; } EVP_PKEY *pKey = EVP_PKEY_new(); if (!pKey) { printf("Failed to generated a new pkey, certificate generation failed"); return E_FAIL; } if (!EVP_PKEY_set1_RSA(pKey, rsa)) { printf("An error occured accessing the private key, certificate generation failed"); return E_OUTOFMEMORY; } X509 *x509cert = X509_new(); if (!x509cert) { printf("Failed to allocate memory for the X509 certificate"); return E_FAIL; } if (!X509_set_pubkey(x509cert, pKey)) { printf("Error setting the public key for the X509 certificate\n"); goto egress; } //Crash occurs here on the call to X509_sign() if in FIPS mode. Disable FIPS mode and it works. const EVP_MD *digest = EVP_sha1(); if (!X509_sign(x509cert, pKey, digest)) { printf("Error signing the X509 certificate\n"); goto egress; } egress: if (NULL != x509cert) { X509_free(x509cert); } if (NULL != pKey) { EVP_PKEY_free(pKey); } //Turn off FIPS mode FIPS_mode_set(0); return 0; } ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager majord...@openssl.org