RE: Possible memory leak with RSA_private_encrypt?

[Kevin Regan]

I found the issue.  There is a custom patch used locally that seems to
be leaking memory.  I'm going to find the responsible coworker and make
them pay for wasting out time. ;-)

--Kevin 

-----Original Message-----
From: Kevin Regan 
Sent: Monday, January 14, 2008 10:59 AM
To: 'openssl-users@openssl.org'
Subject: RE: Possible memory leak with RSA_private_encrypt?


After recompiling with debugging information, I'm getting a bit more
information:

==21592== 24 bytes in 1 blocks are definitely lost in loss record 1 of 1
==21592==    at 0x4421846: malloc (vg_replace_malloc.c:149)
==21592==    by 0x4464D40: (within /usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x44653F6: CRYPTO_malloc (in
/usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x4495FB9: (within /usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x449628A: bn_expand2 (in /usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x4496ACA: BN_bin2bn (in /usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x44B9F63: (within /usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x44B8631: (within /usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x44BB0EA: RSA_private_encrypt (in
/usr/lib/libcrypto.so.0.9.8)
==21592==    by 0x80490E7: main (in /shared/rsa_test1)

--Kevin

-----Original Message-----
From: Kevin Regan
Sent: Sunday, January 13, 2008 10:26 AM
To: openssl-users@openssl.org
Subject: RE: Possible memory leak with RSA_private_encrypt?


I've also tested a version with the following change:

    ...

    //
    // Encrypt test data.
    //

    for (int i = 0; i < 99999999; i++) {
       ...
    }

    //
    // Free RSA key.
    //

    ...

This causes the application to continually grow in memory size.  So, the
memory definitely seems to be leaked in the RSA_private_encrypt call.

--Kevin

-----Original Message-----
From: [EMAIL PROTECTED] on behalf of Kevin Regan
Sent: Sun 1/13/2008 12:30 AM
To: openssl-users@openssl.org
Subject: Possible memory leak with RSA_private_encrypt?
 

There is a good chance that I am doing something wrong.  However, the
program below produces the following output from valgrind with OpenSSL
v0.9.8g:

==12647== 1,032 bytes in 43 blocks are definitely lost in loss record 2
of 2
==12647==    at 0x4421846: malloc (vg_replace_malloc.c:149)
==12647==    by 0x4464FFD: default_malloc_ex (in
/usr/lib/libcrypto.so.0.9.8)

Here is the program (replace keyFile with valid path to key):

#include <iostream>
#include <vector>

#include <cstdio>
#include <cstdlib>

#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/engine.h>
#include <openssl/err.h>
#include <openssl/conf.h>


namespace {
    std::string keyFile = "/my/pem/key";
    const size_t PKCS1_PADDING_LENGTH = 11; }


int
main(int argc, char *argv[])
{
    //
    // Open key file.
    //

    std::FILE* file = std::fopen(keyFile.c_str(), "r");

    if (NULL == file) {
        std::cerr << "Could not open key file" << std::endl;
        std::exit(1);
    }

    //
    // Read in private key.
    //

    RSA* privKey = PEM_read_RSAPrivateKey(file, NULL, NULL, (void*)"");

    std::fclose(file);

    if (NULL == privKey) {
        std::cerr << "Could not read private key" << std::endl;
        std::exit(1);
    }

    //
    // Create test data.
    //

    std::vector<unsigned char> data;

    for (int i = 0; i < 5000; i++) {
        data.push_back(i % 256);
    }

    //
    // Encrypt test data.
    //

    int outputBlockSize = RSA_size(privKey);
    int inputBlockSize = outputBlockSize - PKCS1_PADDING_LENGTH;

    std::vector<unsigned char> encryptedData;
    std::vector<unsigned char> tempEncryptedData(outputBlockSize);

    for (size_t i = 0; i < data.size(); i += inputBlockSize) {
        int bytesLeft = data.size() - i;
        int bytesToEncrypt =
            bytesLeft > inputBlockSize ? inputBlockSize : bytesLeft;

        int result = RSA_private_encrypt(bytesToEncrypt,
                                         &(data[i]),
                                         &(tempEncryptedData[0]),
                                         privKey,
                                         RSA_PKCS1_PADDING);

        if (result != outputBlockSize) {
            std::cerr << "Could not RSA encrypt authentication token."
                      << std::endl;
            std::exit(1);
        }

        encryptedData.insert(encryptedData.end(),
                             tempEncryptedData.begin(),
                             tempEncryptedData.end());
    }

    //
    // Free RSA key.
    //

    RSA_free(privKey);

    CONF_modules_unload(1);
    EVP_cleanup();
    ENGINE_cleanup();
    CRYPTO_cleanup_all_ex_data();
    ERR_remove_state(0);
    ERR_free_strings(); 

    std::cout << "Data encrypted" << std::endl;

    return 0;

} // main

<<winmail.dat>>