When calling the MD5 function on very large data sets (around 2GB) in memory or from a memory map, the computed MD5 sum is false and even worth, can cause the program to crash with a seg-fault. By tracking down the calculation of the sum, I found out that this behaviour occurs when the data pointer crosses the address 0X80000000L in the MD5_Update() function, thus wrapping around from a positive to negative integer. The reason for this error lies in the comparison of two signed numbers rather than two unsigned numbers in the assembly code found in crypto/md5/asm/md5-586.pl at line 296. Instead of a "jge" (greater equal) instruction, there should be a "jae" (above equal) instruction for evaluating an unsigned compare.
An even better fix, IMHO, would be to omit the 64 bytes subtraction from the target address register right at the begining of the function call and then just compare the pointers for equality (jne). In order to compile, the "jae" instruction must also be added somewhere in the crypto/perlasm/x86unix.pl and x86ms.pl perl scripts. OpenSSL self-test report: OpenSSL version: 0.9.7c-dev Last change: In ssl3_get_client_hello() (ssl/s3_srvr.c), tolerate ad... Options: no-krb5 OS (uname): Linux mx040 2.4.4-4GB #1 Wed May 16 00:37:55 GMT 2001 i686 unknown OS (config): i686-whatever-linux2 Target (default): linux-pentium Target: linux-pentium Compiler: Configured with: Thread model: posix gcc version 3.2.2 Eric -- ================================================================== Eric Allamanche mailto:[EMAIL PROTECTED] Fraunhofer Institut Integrierte Schaltungen Phone :+49 9131 776-322 Dept. Audio-Multimedia Fax :+49 9131 776-398 Am Wolfsmantel 33 http://www.iis.fhg.de/amm D-91058 Erlangen, Germany ================================================================== ______________________________________________________________________ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
