RE: Only in FIPS mode: Crash in X509_sign() with memory corruption
For anyone who happens to bump up against this, we found that it was because we
unintentionally were linking in the 64-bit fipscanister.lib into a 32-bit
application. The default on a 64-bit version of Windows is a 64-bit library.
We rebuilt as a 32-bit library and it resolved the problem.
Thanks,
Lester
-Original Message-
From: Memmott, Lester
Sent: Friday, February 08, 2013 12:07 PM
To: openssl-users@openssl.org
Subject: Only in FIPS mode: Crash in X509_sign() with memory corruption
I'm hoping someone has some insight into what I'm doing wrong or if I've just
bumped up against a bug. I'm using Visual Studio 2008 on Windows 8 and am in
the process of running existing code with FIPS enabled. As expected a few
things don't work, but in this case, I'm a bit stumped. I've narrowed it down
to the reproducer test shown below at the bottom of this email. If in not FIPS
mode, it runs fine. If in FIPS mode, X509_sign() crashes. After the crash the
call stack looks like it is corrupted so I can't tell where in openssl or the
fips module things are falling apart.
The error in the crash indicates it is a memory read violation:
Unhandled exception at 0x77ace3be (ntdll.dll) in Test.OpenSSL.exe:
0xC005:
Access violation reading location 0x2ed31dc7.
The call stack indicates it is trying to free up memory:
ntdll.dll!@RtlpLowFragHeapFree@8() + 0x2c bytes
ntdll.dll!_RtlFreeHeap@12() + 0x7e bytes
kernel32.dll!_HeapFree@12() + 0x14 bytes
msvcr90.dll!_free() + 0xcd bytes
libeay32.dll!0fb024dd()
[Frames below may be incorrect and/or missing, no symbols loaded for
libeay32.dll]
libeay32.dll!0fbb45ec()
kernel32.dll!_HeapFree@12() + 0x14 bytes
Visual Studio also has a runtime memory checker which is available when run as
Start with Application Verifier in the Debug drop-down menu. When I do
this, the application verifier reports the following:
Heap violation detected
Memory access operation in the context of an allocated block: heap
overrun or heap underrun.
The call stack indicates that the failure is in libeay32.dll:
libeay32.dll!0fbc0316()
[Frames below may be incorrect and/or missing, no symbols loaded for
libeay32.dll]
libeay32.dll!0fbb357e()
libeay32.dll!0fbb1619()
libeay32.dll!0fb5eb3b()
libeay32.dll!0fb5f87e()
libeay32.dll!0fb63989()
libeay32.dll!0fb63a96()
libeay32.dll!0fb81bd1()
Test.OpenSSL.exe!TestCreateCertificate() Line 339 + 0x11 bytes C++
0041()
So by all indication, in FIPS mode, I'm bumping up against memory corruption
problems that is crashing my app. Any ideas?
Thanks in advance for taking the time to read and respond to my request, Lester
Test case. Crashes on X509_sign() only if in FIPS mode:
int TestCreateCertificate()
{
//Enable FIPS
if (FIPS_mode_set(1) != 1)
return E_FAIL;
//Initialize some randomness
DWORD dw = GetCurrentProcessId();
RAND_add(dw, sizeof(dw), 0);
dw = GetCurrentThreadId();
RAND_add(dw, sizeof(dw), 0);
dw = GetTickCount();
RAND_add(dw, sizeof(dw), 0);
//Create an X509 Cert and sign it.
RSA *rsa = RSA_generate_key(2048, RSA_F4, 0, 0);
if (!rsa)
{
printf(Failed to generate the RSA key, certificate generation
failed);
return E_FAIL;
}
EVP_PKEY *pKey = EVP_PKEY_new();
if (!pKey)
{
printf(Failed to generated a new pkey, certificate generation
failed);
return E_FAIL;
}
if (!EVP_PKEY_set1_RSA(pKey, rsa))
{
printf(An error occured accessing the private key, certificate
generation failed);
return E_OUTOFMEMORY;
}
X509 *x509cert = X509_new();
if (!x509cert)
{
printf(Failed to allocate memory for the X509 certificate);
return E_FAIL;
}
if (!X509_set_pubkey(x509cert, pKey))
{
printf(Error setting the public key for the X509
certificate\n);
goto egress;
}
//Crash occurs here on the call to X509_sign() if in FIPS mode.
Disable FIPS mode and it works.
const EVP_MD *digest = EVP_sha1();
if (!X509_sign(x509cert, pKey, digest))
{
printf(Error signing the X509 certificate\n);
goto egress;
}
egress:
if (NULL != x509cert)
{
X509_free(x509cert);
}
if (NULL != pKey)
{
EVP_PKEY_free(pKey);
}
Only in FIPS mode: Crash in X509_sign() with memory corruption
I'm hoping someone has some insight into what I'm doing wrong or if I've just
bumped up against a bug. I'm using Visual Studio 2008 on Windows 8 and am in
the process of running existing code with FIPS enabled. As expected a few
things don't work, but in this case, I'm a bit stumped. I've narrowed it down
to the reproducer test shown below at the bottom of this email. If in not FIPS
mode, it runs fine. If in FIPS mode, X509_sign() crashes. After the crash the
call stack looks like it is corrupted so I can't tell where in openssl or the
fips module things are falling apart.
The error in the crash indicates it is a memory read violation:
Unhandled exception at 0x77ace3be (ntdll.dll) in Test.OpenSSL.exe:
0xC005:
Access violation reading location 0x2ed31dc7.
The call stack indicates it is trying to free up memory:
ntdll.dll!@RtlpLowFragHeapFree@8() + 0x2c bytes
ntdll.dll!_RtlFreeHeap@12() + 0x7e bytes
kernel32.dll!_HeapFree@12() + 0x14 bytes
msvcr90.dll!_free() + 0xcd bytes
libeay32.dll!0fb024dd()
[Frames below may be incorrect and/or missing, no symbols loaded for
libeay32.dll]
libeay32.dll!0fbb45ec()
kernel32.dll!_HeapFree@12() + 0x14 bytes
Visual Studio also has a runtime memory checker which is available when run as
Start with Application Verifier in the Debug drop-down menu. When I do
this, the application verifier reports the following:
Heap violation detected
Memory access operation in the context of an allocated block: heap
overrun or heap underrun.
The call stack indicates that the failure is in libeay32.dll:
libeay32.dll!0fbc0316()
[Frames below may be incorrect and/or missing, no symbols loaded for
libeay32.dll]
libeay32.dll!0fbb357e()
libeay32.dll!0fbb1619()
libeay32.dll!0fb5eb3b()
libeay32.dll!0fb5f87e()
libeay32.dll!0fb63989()
libeay32.dll!0fb63a96()
libeay32.dll!0fb81bd1()
Test.OpenSSL.exe!TestCreateCertificate() Line 339 + 0x11 bytes C++
0041()
So by all indication, in FIPS mode, I'm bumping up against memory corruption
problems that is crashing my app. Any ideas?
Thanks in advance for taking the time to read and respond to my request,
Lester
Test case. Crashes on X509_sign() only if in FIPS mode:
int TestCreateCertificate()
{
//Enable FIPS
if (FIPS_mode_set(1) != 1)
return E_FAIL;
//Initialize some randomness
DWORD dw = GetCurrentProcessId();
RAND_add(dw, sizeof(dw), 0);
dw = GetCurrentThreadId();
RAND_add(dw, sizeof(dw), 0);
dw = GetTickCount();
RAND_add(dw, sizeof(dw), 0);
//Create an X509 Cert and sign it.
RSA *rsa = RSA_generate_key(2048, RSA_F4, 0, 0);
if (!rsa)
{
printf(Failed to generate the RSA key, certificate generation
failed);
return E_FAIL;
}
EVP_PKEY *pKey = EVP_PKEY_new();
if (!pKey)
{
printf(Failed to generated a new pkey, certificate generation
failed);
return E_FAIL;
}
if (!EVP_PKEY_set1_RSA(pKey, rsa))
{
printf(An error occured accessing the private key, certificate
generation failed);
return E_OUTOFMEMORY;
}
X509 *x509cert = X509_new();
if (!x509cert)
{
printf(Failed to allocate memory for the X509 certificate);
return E_FAIL;
}
if (!X509_set_pubkey(x509cert, pKey))
{
printf(Error setting the public key for the X509
certificate\n);
goto egress;
}
//Crash occurs here on the call to X509_sign() if in FIPS mode.
Disable FIPS mode and it works.
const EVP_MD *digest = EVP_sha1();
if (!X509_sign(x509cert, pKey, digest))
{
printf(Error signing the X509 certificate\n);
goto egress;
}
egress:
if (NULL != x509cert)
{
X509_free(x509cert);
}
if (NULL != pKey)
{
EVP_PKEY_free(pKey);
}
//Turn off FIPS mode
FIPS_mode_set(0);
return 0;
}
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager majord...@openssl.org
