RFC 5246 allows a TLS 1.2 client to omit the Signature Algorithm extension. See section 7.4.1.4.1 for details. This creates a problem for OpenSSL 1.0.1 when acting as a server and either a DSA or ECDSA certificate is used. Because the ClientHello does not contain the Signature Algorithm extension, tls1_process_sigalgs() is never invoked on the server side to set the digest used for the session context. Later during the handshake when the server sends the ServerKeyExchange, the call to tls12_get_sigandhash() fails, which causes the server to abort the handshake.
The attached patch to tls12_get_sigandhash() resolves the problem. However, the OpenSSL maintainers may have a better solution.
Index: t1_lib.c
===================================================================
--- t1_lib.c (revision 585)
+++ t1_lib.c (working copy)
@@ -2830,6 +2830,20 @@
return 0;
md_id = tls12_find_id(EVP_MD_type(md), tls12_md,
sizeof(tls12_md)/sizeof(tls12_lookup));
+
+ /*
+ * There's a special case that we need to check. If
+ * the client didn't send a signature algorithm extension, which
+ * is possible under RFC5246 guidelines, the above check to
+ * find the hash algorithm will fail if the server's cert is
+ * either DSA or ECDSA. In this case, we default to SHA1
+ * as the hash algorithm
+ */
+ if (md_id == -1 &&
+ (EVP_MD_type(md) == NID_ecdsa_with_SHA1 ||
+ EVP_MD_type(md) == NID_dsa)) {
+ md_id = TLSEXT_hash_sha1;
+ }
if (md_id == -1)
return 0;
sig_id = tls12_get_sigid(pk);
<<inline: foleyj.vcf>>
