I've been investigating a memory leak in using a FIPS-capable OpenSSL in
non-FIPS mode.
For example, the following code does not seem to be correct in evp_enc.c:
int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *cipher, ENGINE
*impl,
...
#ifndef OPENSSL_NO_ENGINE
if(impl)
{
if (!ENGINE_init(impl))
{
EVPerr(EVP_F_EVP_CIPHERINIT_EX,
EVP_R_INITIALIZATION_ERROR);
return 0;
}
}
else
/* Ask if an ENGINE is reserved for this job */
impl = ENGINE_get_cipher_engine(cipher->nid);
if(impl)
{
/* There's an ENGINE for this job ... (apparently) */
const EVP_CIPHER *c = ENGINE_get_cipher(impl, cipher->nid);
if(!c)
{
/* One positive side-effect of US's export
* control history, is that we should at least
* be able to avoid using US mispellings of
* "initialisation"? */
EVPerr(EVP_F_EVP_CIPHERINIT_EX,
EVP_R_INITIALIZATION_ERROR);
return 0;
}
/* We'll use the ENGINE's private cipher definition */
cipher = c;
/* Store the ENGINE functional reference so we know
* 'cipher' came from an ENGINE and we need to release
* it when done. */
ctx->engine = impl;
}
else
ctx->engine = NULL;
#endif
#ifdef OPENSSL_FIPS
return FIPS_cipherinit(ctx, cipher, key, iv, enc);
#else
So the code goes through all the motions of honoring the engine that exists and
incrementing reference counts, etc, but then blindly calls FIPS_cipherinit(),
which ends up removing the engine pointer from the context. I see in some
cases this behaviour, yet in others the call is wrapped with a test for FIPS
mode. E.g.:
int EVP_DigestInit_ex(EVP_MD_CTX *ctx, const EVP_MD *type, ENGINE *impl)
...
#ifdef OPENSSL_FIPS
if (FIPS_mode())
{
if (FIPS_digestinit(ctx, type))
return 1;
OPENSSL_free(ctx->md_data);
ctx->md_data = NULL;
return 0;
}
#endif
return ctx->digest->init(ctx);
}
Note that the Update call however has:
int EVP_DigestUpdate(EVP_MD_CTX *ctx, const void *data, size_t count)
{
#ifdef OPENSSL_FIPS
return FIPS_digestupdate(ctx, data, count);
#else
return ctx->update(ctx,data,count);
#endif
}
Should *all* calls be protected with a test for FIPS_mode()? Or is it
documented that a FIPS-capable OpenSSL is not compatible with the usage of
engines, even in non-FIPS mode?
....................................
Erik Tkal
Juniper OAC/UAC/Pulse Development
