It looks like the client sets the hash to use for each key type to the first matching algorithm sent by the server in the certificateRequest. These values are stored in a private structure with no apparent way for client code to modify this, and it seems both the client and server always send the full list regardless.
So in the case where CAPI is utilized for certificate access, if OpenSSL is built to support SHA512, then that is the hash algorithm that will always be chosen, even if the CSP for the selected certificate cannot perform that function. I don't see any solution other than to manually manipulate the data in cert_st, even though that is in ssl_locl.h. Anyone have any better ideas? .................................... Erik Tkal Juniper OAC/UAC/Pulse Development From: [email protected] [mailto:[email protected]] On Behalf Of Erik Tkal Sent: Monday, October 22, 2012 3:12 PM To: John Foley; [email protected] Subject: RE: OpenSSL choosing inappropriate signature algorithm It appears that the clientHello only indicates overall what the client supports. The server sends another such list in its certificateRequest message. I'm trying to find where in the client code the determination is made as to which algorithm to use... .................................... Erik Tkal Juniper OAC/UAC/Pulse Development From: John Foley [mailto:[email protected]] Sent: Monday, October 22, 2012 3:02 PM To: [email protected]<mailto:[email protected]> Cc: Erik Tkal Subject: Re: OpenSSL choosing inappropriate signature algorithm Agreed. This does raise an interesting design dilemma. The client has no way to know in advance whether the server will be sending the CertificateRequest message. If the server is not doing client auth, then the client certificate will never be needed. Hence, offering all the signature algorithms in the ClientHello isn't a problem. However, as you indicated, when the server does implement client authentication, then limiting the supported signature algorithms in the ClientHello would make sense. The TLS protocol simply doesn't provide any mechanism to make the optimal decision when sending the ClientHello. It'll be interesting to hear the comments from the OpenSSL experts. On 10/22/2012 02:27 PM, Erik Tkal wrote: Using OpenSSL 1.0.1c I notice that the client always sends the full set of supported signature algorithms in the clientHello, with no option to limit this at runtime. However, if using callbacks to choose a certificate and perform the private key operation via callback (the certificate is accessed via CAPI and may be on a smart card), a certificate may be chosen whose CSP does not allow signing using the selected algorithm. Is this a fundamental problem with the protocol? It seems the signature algorithm should be negotiated along with the client certificate selection, as opposed to forcing it up front. Or am I missing some option here? .................................... Erik Tkal Juniper OAC/UAC/Pulse Development
