I'm using CryptoPP's AES-256 encryption. It's working for 99% of people just fine. So far, 2 separate people are experiencing segfaults. The seg fault seems to happen after successfully encrypting thousands of blocks, so even on their machines, it doesn't always fail.
Program terminated with signal 11, Segmentation fault. #0 CryptoPP::Rijndael::Enc::AdvancedProcessBlocks (this=Cannot access memory at address 0x8 ) at rijndael.cpp:1233 1233 return length % BLOCKSIZE; (gdb) (gdb) bt #0 CryptoPP::Rijndael::Enc::AdvancedProcessBlocks (this=Cannot access memory at address 0x8 ) at rijndael.cpp:1233 Cannot access memory at address 0x4 (gdb) info registers eax 0x7639370 123966320 ecx 0x0 0 edx 0xac64 44132 ebx 0x0 0 esp 0x76391f0 0x76391f0 ebp 0x0 0x0 esi 0x64 100 edi 0x8643434e -2042412210 eip 0x83d45e8 0x83d45e8 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2024> eflags 0x10246 [ PF ZF IF RF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0xc040007b -1069547397 fs 0x0 0 gs 0x33 51 Since the ebp register is 0x0, I can't get a good stack trace. It's important to note that both of these people are running the x86 version of this library (on an x64 machine), and their CPUs do not support AES-NI. This means that they're executing they're executing the SSE2 codepath. (gdb) print g_hasAESNI $1 = false (gdb) print g_hasSSE2 $2 = true If you look at the source, just before the seg fault, it executes an all-assembly function called Rijndael_Enc_AdvancedProcessBlocks. 01232 Rijndael_Enc_AdvancedProcessBlocks(&locals, m_key); 01233 return length % BLOCKSIZE; That function sets up and manages its own stack space. On x86, one of the first things that it does is push ebx and ebp on the stack. One of the last things it does is pop them both off of the stack. This matches up perfectly with the assembly code that I'm seeing. |0x83d45cd <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+1997> movaps %xmm0,0x30(%eax) |0x83d45d1 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2001> movaps %xmm0,0x40(%eax) |0x83d45d5 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2005> movaps %xmm0,0x50(%eax) |0x83d45d9 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2009> movaps %xmm0,0x60(%eax) |0x83d45dd <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2013> mov 0x300(%esp),%esp |0x83d45e4 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2020> emms |0x83d45e6 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2022> pop %ebp |0x83d45e7 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2023> pop %ebx >|0x83d45e8 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2024> andl $0xf,0x18(%ebp) Apparently my compiler has inlined the function. It seg faults when it tries to access the variable "length" to perform %BLOCKSIZE (which is equivalent to bitwise AND of 0xf). "length" should be 0x18 bytes after the ebp register. But my ebp register is 0x0, meaning that somewhere inbetween pushing it to the stack and popping it off the stack, something has probably overwritten it with 0x0. It certainly looks like a buffer overflow in the assembly code. All of my attempts to reproduce this problem or analyze the asm function Rijndael_Enc_AdvancedProcessBlocks have failed. I haven't tried 5.6.2 yet, because getting someone else to reproduce this problem is hard. Also, there is only one change in 5.6.2 that could possibly be related to this, and it supposedly only fixes a valgrind false-positive warning. http://cryptopp.svn.sourceforge.net/viewvc/cryptopp?view=revision&revision=525 1. Interestingly, valgrind will report an error on the exact same assembly instruction, when attempting to access "length", saying that it's uninitialized. 2. Valgrind will report that error, even when "length" is perfectly initialized, supporting the claim that it really is a false-positive. 3. I have no idea why the change in 5.6.2 (increasing the assembly function's stack space from 512 to 768) would fix the valgrind false-positive. I don't have any good reason to believe this change in 5.6.2 will fix my problem. Can anyone help? Thanks -- -- You received this message because you are subscribed to the "Crypto++ Users" Google Group. To unsubscribe, send an email to [email protected]. More information about Crypto++ and this group is available at http://www.cryptopp.com. --- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
