Hello,
After digging some more I saw that the cause of the problem was that the
BIO callback function "b->method->bgets" that should have pointed to
file_gets actually pointed to BIO_new_fp which obviously crashed my
program. I don't know what caused this, but I can guess that this is
related to the special build required for OpenSSL 0.9.7 FIPS support
(using MSYS & GCC for the FIPS canister). This does not happen in 0.9.8.
I was able to find a workaround by using different API to load the
certificate, this code works:
bioCert = BIO_new(BIO_s_file());
if(bioCert == NULL) {
printf ("problem\n");
return;
}
if (BIO_read_filename(bioCert," CA.cer") <= 0)
{
printf ("problems\n");
}
hCert=PEM_read_bio_X509_AUX(bioCert,NULL,NULL, NULL);
Cheers,
Hagai.
________________________________
From: Hagai Yaffe
Sent: Thursday, December 04, 2008 11:08 PM
To: openssl-users@openssl.org
Subject: Crash when using FIPS OpenSSL
Hello,
I am working on using OpenSSL in FIPS mode in my application, I am using
OpenSSL 0.9.7m on windows. I have successfully built OpenSSL according
to the FIPS user guide, but on specific operation my application crashes
inside OpenSSL.
I have created a small program that demonstrates the problem, when
linked with regular OpenSSL build in works successfully, when working
with FIPS build OpenSSL (in FIPS mode or not) it crashes (in a call to
BIO_gets).
The certificate I am trying to read is a base 64 encoded CA certificate.
Any ideas on what is causing this? I can't seem to debug into the
BIO_gets command (I guess that this happens since it is in the FIPS
canister), any help would be great.
Thanks,
Hagai.
Program:
----------------
#include <stdio.h>
#include <windows.h>
#include <openssl/fips.h>
#include <openssl/x509.h>
#include <openssl/pem.h>
#define OPENSSL_FIPS
void main()
{
#ifdef _FIPS
if (!FIPS_mode_set(1))
{
fprintf(stderr,"*** Problem entering fips mode ***\n");
return;
}
else
{
fprintf(stderr,"*** IN FIPS MODE ***\n");
}
#endif
int iRc = 1;
BIO *bioCert = NULL;
FILE *hStoreFile;
X509 *hCert = NULL;
hStoreFile = fopen("CA.cer", "r");
// Create new BIO and set it to point our store file
bioCert = BIO_new(BIO_s_file());
if(bioCert == NULL) {
printf ("problem\n");
return;
}
iRc = BIO_set_fp(bioCert,hStoreFile, BIO_NOCLOSE);
if (iRc <= 0) {
printf ("problem\n");
}
hCert = (X509 *)PEM_read_bio_X509(bioCert, NULL, NULL, NULL);
}
Crash occurs at line: i=BIO_gets(bp,buf,254); (pem_lib.c:643)
> libeay32.dll!PEM_read_bio(bio_st * bp=0x0172acb8, char * *
name=0x0012d14c, char * * header=0x0012d148, unsigned char * *
data=0x0012d150, long * len=0x0012d158) Line 643 C
libeay32.dll!PEM_bytes_read_bio(unsigned char * *
pdata=0x0012d180, long * plen=0x0012d188, char * * pnm=0x00000000, const
char * name=0x0fbee77c, bio_st * bp=0x0172acb8, int (char *, int, int,
void *)* cb=0x00000000, void * u=0x00000000) Line 231 + 0x19 C
libeay32.dll!PEM_ASN1_read_bio(char * (void)* d2i=0x0fb3af70,
const char * name=0x0fbee77c, bio_st * bp=0x0172acb8, char * *
x=0x00000000, int (char *, int, int, void *)* cb=0x00000000, void *
u=0x00000000) Line 77 + 0x1f C
libeay32.dll!PEM_read_bio_X509(bio_st * bp=0x0172acb8, x509_st *
* x=0x00000000, int (char *, int, int, void *)* cb=0x00000000, void *
u=0x00000000) Line 68 + 0x22 C
