>> I am seeing a core dump in RSA_check_key() function. The backtrace is
>> as below.
What OS? Which OpenSSL version? Is it correct assumption that it's
custom/own program? Can you reproduce the problem with openssl utility,
with 'openssl rsa -in file.pem -check -noout')? Can you reproduce it
with 'openssl genrsa 2048 | openssl -check -noout'? Then there is big
question if sha1-x86_64.s:2240 is reliable. To answer that question you
have run 'disassemble' at gdb prompt and proceed till you see failed
instruction (marked with => or something) and write it down. What's
'info reg' at that point? If we are talking about this segment:
je .Ldone_ssse3
movdqa 64(%r11),%xmm6
movdqa 0(%r11),%xmm9
=> movdqu 0(%r9),%xmm0
movdqu 16(%r9),%xmm1
movdqu 32(%r9),%xmm2
movdqu 48(%r9),%xmm3
then there is only one possibility: corrupted input. I mean there is no
room for sha1_block_data_order_ssse3 to screw input parameters, its
caller has to do it...
>> I am unable to understand what is causing the issue. The core dump is
>> intermittent (not always reproduced).
>> Could someone please provide any pointers if I am missing any check?
>>
>> --
>> Thanks,
>> Nilesh
>>
>> P.S.
>>
>> The code generating crash is:
>> // fp is a valid pointer to the private key file in
>> PEM format.
>> RSA *rsa = (RSA *) PEM_read_RSAPrivateKey(fp, NULL,
>> NULL, NULL);
>> if (!rsa) {
>> return -1;
>> }
>>
>> if (RSA_check_key(rsa) != 1) {
>> RSA_free(rsa);
>> rsa = NULL;
>> }
>>
>> Back trace:
>> Program terminated with signal 11, Segmentation fault.
>> #0 sha1_block_data_order_ssse3 () at sha1-x86_64.s:2240
>> #0 sha1_block_data_order_ssse3 () at sha1-x86_64.s:2240
>> #1 0xca62c1d6ca62c1d6 in ?? ()
>> #2 0xca62c1d6ca62c1d6 in ?? ()
>> #3 0xca62c1d6ca62c1d6 in ?? ()
>> #4 0xca62c1d6ca62c1d6 in ?? ()
>> #5 0xca62c1d6ca62c1d6 in ?? ()
>> #6 0xca62c1d6ca62c1d6 in ?? ()
>> #7 0xca62c1d6ca62c1d6 in ?? ()
>> #8 0xca62c1d6ca62c1d6 in ?? ()
>> #9 0x00000032543be076 in state () from /lib64/libcrypto.so.10
>> #10 0x00007fd508008210 in ?? ()
>> #11 0x0000000000000029 in ?? ()
>> #12 0x000000325406acf7 in SHA1_Update (c=0xe992ba8d, data_=<value
>> optimized out>, len=<value optimized out>) at ../md32_common.h:325
>> #13 0x00000032540ce30f in ssleay_rand_bytes (buf=0x7fd5080081c0 "",
>> num=54, pseudo=1) at md_rand.c:498
>> #14 0x00000032540a2535 in bnrand (pseudorand=1, rnd=0x7fd5080075f0,
>> bits=512, top=-1, bottom=0) at bn_rand.c:147
>> #15 0x00000032540a2358 in bn_rand_range (pseudo=<value optimized out>,
>> r=0x7fd5080075f0, range=0x7fd5080075c0) at bn_rand.c:281
>> #16 0x00000032540a533b in BN_is_prime_fasttest_ex (a=0x7fd508007be0,
>> checks=6, ctx_passed=0x0, do_trial_division=<value optimized out>,
>> cb=0x0)
>> at bn_prime.c:322
>> #17 0x00000032540b2775 in RSA_check_key (key=0x7fd508007940) at
>> rsa_chk.c:88
>
> ______________________________________________________________________
> OpenSSL Project http://www.openssl.org
> Development Mailing List openssl-dev@openssl.org
> Automated List Manager majord...@openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
