On Thu, Mar 7, 2013 at 6:41 AM, Brian Vincent <[email protected]> wrote:
> I'm using CryptoPP's AES-256 encryption. It's working for 99% of people > just fine. So far, 2 separate people are experiencing segfaults. The seg > fault seems to happen after successfully encrypting thousands of blocks, so > even on their machines, it doesn't always fail. > > Program terminated with signal 11, Segmentation fault. > #0 CryptoPP::Rijndael::Enc::AdvancedProcessBlocks (this=Cannot access > memory at address 0x8 > ) at rijndael.cpp:1233 > 1233 return length % BLOCKSIZE; > (gdb) > > (gdb) bt > #0 CryptoPP::Rijndael::Enc::AdvancedProcessBlocks (this=Cannot access > memory at address 0x8 > ) at rijndael.cpp:1233 > Cannot access memory at address 0x4 > > (gdb) info registers > eax 0x7639370 123966320 > ecx 0x0 0 > edx 0xac64 44132 > ebx 0x0 0 > esp 0x76391f0 0x76391f0 > ebp 0x0 0x0 > esi 0x64 100 > edi 0x8643434e -2042412210 > eip 0x83d45e8 0x83d45e8 > <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, byte const*, > byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2024> > eflags 0x10246 [ PF ZF IF RF ] > cs 0x73 115 > ss 0x7b 123 > ds 0x7b 123 > es 0xc040007b -1069547397 > fs 0x0 0 > gs 0x33 51 > > Since the ebp register is 0x0, I can't get a good stack trace. > > It's important to note that both of these people are running the x86 > version of this library (on an x64 machine), and their CPUs do not support > AES-NI. This means that they're executing they're executing the SSE2 > codepath. > > (gdb) print g_hasAESNI > $1 = false > (gdb) print g_hasSSE2 > $2 = true > > If you look at the source, just before the seg fault, it executes an > all-assembly function called Rijndael_Enc_AdvancedProcessBlocks. > > 01232 Rijndael_Enc_AdvancedProcessBlocks(&locals, m_key); > 01233 return length % BLOCKSIZE; > > That function sets up and manages its own stack space. On x86, one of the > first things that it does is push ebx and ebp on the stack. One of the > last things it does is pop them both off of the stack. This matches up > perfectly with the assembly code that I'm seeing. > > |0x83d45cd <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+1997> > movaps %xmm0,0x30(%eax) > |0x83d45d1 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2001> > movaps %xmm0,0x40(%eax) > |0x83d45d5 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2005> > movaps %xmm0,0x50(%eax) > |0x83d45d9 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2009> > movaps %xmm0,0x60(%eax) > |0x83d45dd <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2013> > mov 0x300(%esp),%esp > |0x83d45e4 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2020> > emms > |0x83d45e6 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2022> > pop %ebp > |0x83d45e7 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2023> > pop %ebx > >|0x83d45e8 <CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(byte const*, > byte const*, byte*, size_t, CryptoPP::Rijndael::Dec::word32) const+2024> > andl $0xf,0x18(%ebp) > > Apparently my compiler has inlined the function. It seg faults when it > tries to access the variable "length" to perform %BLOCKSIZE (which is > equivalent to bitwise AND of 0xf). "length" should be 0x18 bytes after the > ebp register. But my ebp register is 0x0, meaning that somewhere inbetween > pushing it to the stack and popping it off the stack, something has > probably overwritten it with 0x0. It certainly looks like a buffer > overflow in the assembly code. > > All of my attempts to reproduce this problem or analyze the asm function > Rijndael_Enc_AdvancedProcessBlocks have failed. > > I haven't tried 5.6.2 yet, because getting someone else to reproduce this > problem is hard. Also, there is only one change in 5.6.2 that could > possibly be related to this, and it supposedly only fixes a valgrind > false-positive warning. > > > http://cryptopp.svn.sourceforge.net/viewvc/cryptopp?view=revision&revision=525 > > 1. Interestingly, valgrind will report an error on the exact same > assembly instruction, when attempting to access "length", saying that it's > uninitialized. > 2. Valgrind will report that error, even when "length" is perfectly > initialized, supporting the claim that it really is a false-positive. > 3. I have no idea why the change in 5.6.2 (increasing the assembly > function's stack space from 512 to 768) would fix the valgrind > false-positive. > > I don't have any good reason to believe this change in 5.6.2 will fix my > problem. > > Can anyone help? > > Thanks > > -- > -- > > First off, what a great posting and very detailed. A couple of questions (I am aware this is horrible as fault is so random). Is it possible you can supply a minimal test case that will show this error (I appreciate it's random so possibly forcing many threads to concurrently run the test to speed up fail ) ? 2nd can you please give a description of the machines/compiler + switches etc. I am very interested and can possibly set this up and test on several platforms. David -- -- You received this message because you are subscribed to the "Crypto++ Users" Google Group. To unsubscribe, send an email to [email protected]. More information about Crypto++ and this group is available at http://www.cryptopp.com. --- You received this message because you are subscribed to the Google Groups "Crypto++ Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
