Using RC4 vs EVP_CIPHER

[Sean Covel]

Now that I've updated to the latest OpenSSL (7g) I've got a question:

I implemented some RC4 functionality using the low-level rc4 functions.(
RC4_Set_Key, RC4), and its working fine.

The OpenSSL documentation recommends using the EVP_* functions instead
of the lower level functions.  It seems like it might give easy access
to additional functionality, so I'm converting.

Here's the problem.  When I switched the code from RC4 to EVP, the
program broke, in a big way.  The project freezes and I have to either
logout or shutdown in order to get Visual Studio 6 to die.  If I'm
REALLY patient (and I'm not) I can eventually get Task Manager to come
up, and then click on msdev.exe (and then wait.....) and then click on
end-process (and wait) and then click on OK (and wait....).  You get the
idea.

A brief explanation of what the code is doing.  The code is a plugin for
another program.  The program initializes the plugin, passes it a
password, and then starts handing it chunks of text to encrypt.  The
program has a socket open to another identical program, doing the same
thing "on the other side."  Kinda like a chat program.  The encrypt key
is for sending, the decrypt key is for receiving.  All the plugin knows
is the key, the input/output buffer, and the length of the chunk.

I have the RC4 code and the EVP code both in there.  They are controlled
by a #define.

Here are the relevant chunks for code.  I'm trying to give enough code
without over-burdening you.  If you need more, let me know.

unsigned char keystr[MD5_DIGEST_LENGTH];
#ifdef EVP
EVP_CIPHER_CTX Ectx;
EVP_CIPHER_CTX Dctx;
#else
RC4_KEY Ekey;
RC4_KEY Dkey;
#endif

unsigned char iv[] = "12345678";

PLUGIN_API int Startup(void)
{
        //hash the password into a 128bit key
        EVP_Digest((unsigned char *)szExternalKey,(unsigned
long)strlen((const char *)szExternalKey),keystr,NULL,EVP_md5(),NULL);

#ifdef EVP
        EVP_CIPHER_CTX_init(&Ectx);
        EVP_CipherInit_ex(&Ectx, EVP_rc4(), NULL, NULL, NULL, 1);
        EVP_CIPHER_CTX_set_key_length(&Ectx, 128);
        EVP_CipherInit_ex(&Ectx, NULL, NULL, keystr, iv, 1);

        EVP_Digest((unsigned char *)szExternalKey,(unsigned
long)strlen((const char *)szExternalKey),keystr,NULL,EVP_md5(),NULL);

        EVP_CIPHER_CTX_init(&Dctx);
        EVP_CipherInit_ex(&Dctx, EVP_rc4(), NULL, NULL, NULL, 0);
        EVP_CIPHER_CTX_set_key_length(&Dctx, 128);
        EVP_CipherInit_ex(&Dctx, NULL, NULL, keystr, iv, 0);
#else
        RC4_set_key(&Ekey,MD5_DIGEST_LENGTH,keystr);
        RC4_set_key(&Dkey,MD5_DIGEST_LENGTH,keystr);
#endif
}

PLUGIN_API BYTE* TransformBuffer(BYTE* pDataBuffer, int nDataLen, int*
pnTransformedDataLen)
{
    int       dwByteCount = 0;

    BYTE* pTransBuffer =
CheckLocalTransBufferSize(GiveTransDataLen(nDataLen));
    if (pTransBuffer == NULL)
    {
        *pnTransformedDataLen = -1;
        return NULL;
    }


#ifdef EVP
        if(!EVP_CipherUpdate(&Ectx, pTransBuffer, &dwByteCount,
pDataBuffer, nDataLen))
        {
                // Error
                PrintLog((DEST,"TransformBuffer failed"));
                return NULL;
        }
#else
        RC4(&Ekey,nDataLen,(unsigned char *)pDataBuffer,(unsigned char
*)pTransBuffer);
#endif

    // return the transformed data length
    *pnTransformedDataLen = GiveTransDataLen(nDataLen);

    return pTransBuffer;
}


PLUGIN_API BYTE* RestoreBuffer(BYTE* pRestoredDataBuffer, int nDataLen,
int* pnRestoredDataLen)
{
    int         dwByteCount = 0;

    if (pRestoredDataBuffer == NULL)
    {
                // Give the size of the transformed data buffer, based
on the original data length
        *pnRestoredDataLen = GiveRestDataLen(nDataLen);

        // Ensure the pLocalRestBuffer that receive transformed data is
big enough
        BYTE* pBuffer = CheckLocalRestBufferSize(*pnRestoredDataLen);
        return pBuffer;
    }

#ifdef EVP
        if(!EVP_CipherUpdate(&Dctx, pRestoredDataBuffer, &dwByteCount,
pLocalRestBuffer, nDataLen))
        {
                // Error
                PrintLog((DEST,"RestoreBuffer failed"));
                return NULL;
        }
#else
    RC4(&Dkey,nDataLen,(unsigned char *)pLocalRestBuffer,(unsigned char
*)pRestoredDataBuffer);
#endif

    // return the resulting data length
    *pnRestoredDataLen = GiveRestDataLen(nDataLen);

    return pLocalRestBuffer;
}

PLUGIN_API int Shutdown(void)
{
    // Cleanup everything
#ifdef EVP
        EVP_CIPHER_CTX_cleanup(&Ectx);
        EVP_CIPHER_CTX_cleanup(&Dctx);
#endif
        return 1;
}

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@openssl.org
Automated List Manager                           [EMAIL PROTECTED]