Sorry for the imprecisions on the original message. The cipher suite with which I see the problem is indeed the TLS_RSA_WITH_3DES_EDE_CBC_SHA (0,10).
The problem is the verification by the client of the finished message sent by the server. This finished message contains the verify_data that is 12 bytes. Now, here is the relevant part of the code in OpenSSL in ssl/t1_enc.c: =========== int tls1_mac(SSL *ssl, unsigned char *md, int send) { SSL3_RECORD *rec; unsigned char *mac_sec,*seq; const EVP_MD *hash; unsigned int md_size; int i; HMAC_CTX hmac; unsigned char buf[5]; if (send) { rec= &(ssl->s3->wrec); mac_sec= &(ssl->s3->write_mac_secret[0]); seq= &(ssl->s3->write_sequence[0]); hash=ssl->write_hash; } else { rec= &(ssl->s3->rrec); mac_sec= &(ssl->s3->read_mac_secret[0]); seq= &(ssl->s3->read_sequence[0]); hash=ssl->read_hash; } md_size=EVP_MD_size(hash); buf[0]=rec->type; buf[1]=TLS1_VERSION_MAJOR; buf[2]=TLS1_VERSION_MINOR; buf[3]=rec->length>>8; buf[4]=rec->length&0xff; /* I should fix this up TLS TLS TLS TLS TLS XXXXXXXX */ HMAC_CTX_init(&hmac); HMAC_Init_ex(&hmac,mac_sec,EVP_MD_size(hash),hash,NULL); HMAC_Update(&hmac,seq,8); HMAC_Update(&hmac,buf,5); HMAC_Update(&hmac,rec->input,rec->length); HMAC_Final(&hmac,md,&md_size); HMAC_CTX_cleanup(&hmac); #ifdef TLS_DEBUG printf("sec="); {unsigned int z; for (z=0; z<md_size; z++) printf("%02X ",mac_sec[z]); printf("\n"); } printf("seq="); {int z; for (z=0; z<8; z++) printf("%02X ",seq[z]); printf("\n"); } printf("buf="); {int z; for (z=0; z<5; z++) printf("%02X ",buf[z]); printf("\n"); } printf("rec="); {unsigned int z; for (z=0; z<rec->length; z++) printf("%02X ",rec->input[z]); printf("\n"); } #endif for (i=7; i>=0; i--) { ++seq[i]; if (seq[i] != 0) break; } #ifdef TLS_DEBUG {unsigned int z; for (z=0; z<md_size; z++) printf("%02X ",md[z]); printf("\n"); } #endif return(md_size); } ================= Note that in the code above, I replaced the original line {unsigned int z; for (z=0; z<rec->length; z++) printf("%02X ",buf[z]); printf("\n"); } which was printing misleading debug information(!) by {unsigned int z; for (z=0; z<rec->length; z++) printf("%02X ",rec->input[z]); printf("\n"); } which now really prints the content of the record (and the input to the HMAC computation). If you run the s_client program (compiled with the change above and TLS_DEBUG #define'd) using the following arguments: -msg -tls1 and connecting to a SSL server that will select the TLS_RSA_WITH_3DES_EDE_CBC_SHA cipher suite, the TLS handshake will work properly until the OpenSSL client receives the Finished message sent by the server. This is the output of the s_client executable at this point: ---------- tls1_enc(0) EVP_Cipher(ds=00918B90,rec->data=009F55F5,rec->input=009F55F5,l=40) ==> EVP_CIPHER_CTX: 0 buf_len, 24 key_len [8 128], 8 iv_len IV: 1D44982C1B69A80E rec->input= 0a b6 20 43 0c 28 c3 04 a9 73 fe eb c3 74 38 d1 e9 60 da f3 0b 0a 26 25 fc af 2c 51 b2 db 0a 8b bb a4 0f 78 29 e5 5d 06 rec->data= 14 00 00 0c 77 f3 c1 07 53 b8 5f 4f 9c 15 b2 fe 96 af 5e f3 5e 32 a7 29 a1 0f 5b 88 8d 6b 67 d7 dd 6b b9 0f 00 01 02 03 dec 40 14 00 00 0C 77 F3 C1 07 53 B8 5F 4F 9C 15 B2 FE 96 AF 5E F3 5E 32 A7 29 A1 0F 5B 88 8D 6B 67 D7 DD 6B B9 0F 00 01 02 03 sec=4E 0E 1E 13 E9 F6 98 69 12 A5 65 79 0C 37 B5 CB 5B 2B 12 D5 seq=00 00 00 00 00 00 00 00 buf=16 03 01 00 14 rec=14 00 00 0C 77 F3 C1 07 53 B8 5F 4F 9C 15 B2 FE 96 AF 5E F3 --------------- Now, you can see that the message is correctly decrypted ( rec->data) rec->data= 14 00 00 0c 77 f3 c1 07 53 b8 5f 4f 9c 15 b2 fe 96 af 5e f3 5e 32 a7 29 a1 0f 5b 88 8d 6b 67 d7 dd 6b b9 0f 00 01 02 03 We see the handshake header: 14 00 00 0c (type = 0x14: finished, length = 0x0c (12) bytes) the verify data (12 bytes): 77 f3 c1 07 53 b8 5f 4f 9c 15 b2 fe the SHA1 hmac ( 20 bytes SHA-1 hash): 96 af 5e f3 5e 32 a7 29 a1 0f 5b 88 8d 6b 67 d7 dd 6b b9 0f and finally the padding ( 4 bytes) to make the whole message a multiple of the Triple DES block size (= 8 bytes). 4 + 12 + 20 = 36 padded with 4 to make 40 ( 5 blocks of 8) 00 01 02 03 Now, the parameters used for the HMAC verification are printed inside the #ifdef TLS_DEBUG block: sec=4E 0E 1E 13 E9 F6 98 69 12 A5 65 79 0C 37 B5 CB 5B 2B 12 D5 seq=00 00 00 00 00 00 00 00 buf=16 03 01 00 14 rec=14 00 00 0C 77 F3 C1 07 53 B8 5F 4F 9C 15 B2 FE 96 AF 5E F3 The secret (sec) is correct, and so is the sequence number (seq). But the last byte of the buf value should be 0x10 (16 = 4 bytes header + 12 bytes verify_data) and not 0x14 (20)). This leads to the problem seen on the rec line where the last four bytes of the HMAC ( 96 AF 5E F3) are incorporated in the HMAC computation. ======= HMAC_Update(&hmac,seq,8); HMAC_Update(&hmac,buf,5); HMAC_Update(&hmac,rec->input,rec->length); ====== The problem is that the value of rec->length is not correct: it is 4 units off, maybe because there is a 4 bytes padding. But it's clear that the HMAC should not be computed using the first 4 bytes of the final result as the input! Based on the arguments sent to the function, it seems that the ssl->s3->rrec->length value is incorrect. I hope this makes the things a bit more clear. Let me know if you need more information: it's difficult to put this in writing. The problem is quite easy to reproduce if you have a server that will select the (0, 10) cipher suite. Fabrice Ferino. P.S.: It might be possible to reproduce the problem using openssl s_server -tls1 -cipher EDH-RSA-DES-CBC3-SHA as the server and openssl s_client -tls1 as the client. Can somebody with more OpenSSL expertise try it? ===== __________________________________ Do you Yahoo!? Free Pop-Up Blocker - Get it now http://companion.yahoo.com/ ______________________________________________________________________ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]