Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Do you have session caching disabled in the server configuration, either accidentally or deliberately? That seems to me to be the only thing that fits the tcpdump you sent. If so please turn session caching on and try the SSL_OP_NO_TICKET patch again. Steve. Yes SSLSessionCache was not enabled(by default it is 'none'). Yes I could see it 'working' with SSL_OP_NO_TICKET patch and 'SSLSessionCache shm:/tmp/sessioncache' directive. IIUC, Proper fix for this should be in openssl client. Reasonable fix for this on the server side is to apply SSL_OP_NO_TICKET patch and enable SSLSessionCache. Thanks With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
On 10/28/2009 04:17 AM, Dr Stephen Henson wrote: Kamesh Jayachandran wrote: Hi Kaspar, I applied your 'mod_ssl-disable_tls_tickets.diff' and 'mod_ssl-log_ssloptions.diff' to apache-2.2.12 and initiated the 'failing svn import operation'. [Mon Oct 26 15:48:21 2009] [warn] [client 10.2.0.88] ssl_init_ssl_connection: options=0x1114fff [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] ssl_init_ssl_connection: options=0x1114fff [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] ssl_init_ssl_connection: options=0x1114fff The tcpdump for this failure is at, http://www.livecipher.com/tlsext_dump/tlsext.dmp.4 Sorry for the delay. As I mentioned something strange is going on there. The server is sending back empty session IDs which shouldn't be happening if tickets are disabled properly. With OpenSSL 0.9.8k client, can you try this connecting to that server: openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 Does any value appear after "Session-ID"? Hit Q to exit. Yes it appears. Also try: openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 -no_ticket again do you get anything after "Session-ID"? Yes I get. Finally this pair of commands: openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 -sess_out foo.pem openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 -sess_in foo.pem Do you still get the error when you call the command with the server including SSL_OP_NO_TICKET? Yes I get the error with Server running SSL_OP_NO_TICKET patch. [kam...@kamesh httpd-2.2.12]$ openssl s_client -connect kamesh:443 -servername kamesh -tls1 -sess_in foo.pem CONNECTED(0003) 4155:error:140920DF:SSL routines:SSL3_GET_SERVER_HELLO:parse tlsext:s3_clnt.c:880: FYI I used this openssl client on my linux for this test which is openssl0.9.8-k(while original issue was posted against the win32 svn client built with openssl-0.9.8j. With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote: > >>Do you have session caching disabled in the server configuration, either >>accidentally or deliberately? That seems to me to be the only thing > that fits >>the tcpdump you sent. > >>If so please turn session caching on and try the SSL_OP_NO_TICKET patch > again. > > I am away from the test setup. Should be able to do this only on coming > Monday. > OK, will be interested to know if that works. Based on my analysis the latest snapshots of OpenSSL should fix this properly but that is a client side fix. Server side disabling tickets and making sure the session cache is enabled should be a usable workaround. Steve. -- Dr Stephen N. Henson. Senior Technical/Cryptography Advisor, Open Source Software Institute: www.oss-institute.org OpenSSL Core team: www.openssl.org
RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
>Do you have session caching disabled in the server configuration, either >accidentally or deliberately? That seems to me to be the only thing that fits >the tcpdump you sent. >If so please turn session caching on and try the SSL_OP_NO_TICKET patch again. I am away from the test setup. Should be able to do this only on coming Monday. Thanks With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote: > >>That is most peculiar. The server is sending back a zero length session > ID in >>the server hello which it shouldn't be doing if tickets are disabled. > >>Is the server somehow using an older version of OpenSSL? There has been > a bug >in >>the past which might do that but it was fixed well before 0.9.8k. > > Server uses openssl-0.9.8k. > Do you have session caching disabled in the server configuration, either accidentally or deliberately? That seems to me to be the only thing that fits the tcpdump you sent. If so please turn session caching on and try the SSL_OP_NO_TICKET patch again. Steve. -- Dr Stephen N. Henson. Senior Technical/Cryptography Advisor, Open Source Software Institute: www.oss-institute.org OpenSSL Core team: www.openssl.org
RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
>That is most peculiar. The server is sending back a zero length session ID in >the server hello which it shouldn't be doing if tickets are disabled. >Is the server somehow using an older version of OpenSSL? There has been a bug >>in >the past which might do that but it was fixed well before 0.9.8k. Server uses openssl-0.9.8k. With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote: > Hi Kaspar, > > I applied your 'mod_ssl-disable_tls_tickets.diff' and > 'mod_ssl-log_ssloptions.diff' to apache-2.2.12 > > and initiated the 'failing svn import operation'. > > > [Mon Oct 26 15:48:21 2009] [warn] [client 10.2.0.88] > ssl_init_ssl_connection: options=0x1114fff > [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] > ssl_init_ssl_connection: options=0x1114fff > [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] > ssl_init_ssl_connection: options=0x1114fff > > > The tcpdump for this failure is at, > > http://www.livecipher.com/tlsext_dump/tlsext.dmp.4 > As I mentioned something strange is going on there. The server is sending back empty session IDs which shouldn't be happening if tickets are disabled properly. With OpenSSL 0.9.8k client, can you try this connecting to that server: openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 Does any value appear after "Session-ID"? Hit Q to exit. Also try: openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 -no_ticket again do you get anything after "Session-ID"? Finally this pair of commands: openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 -sess_out foo.pem openssl s_client -connect hostname.whatever.com:443 -servername hostname.whatever.com -tls1 -sess_in foo.pem Do you still get the error when you call the command with the server including SSL_OP_NO_TICKET? Steve. -- Dr Stephen N. Henson. Senior Technical/Cryptography Advisor, Open Source Software Institute: www.oss-institute.org OpenSSL Core team: www.openssl.org
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Dr Stephen Henson wrote: > Kaspar Brand wrote: >> Kamesh Jayachandran, >>> [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] >>> ssl_init_ssl_connection: options=0x1114fff >>> >>> >>> The tcpdump for this failure is at, >>> >>> http://www.livecipher.com/tlsext_dump/tlsext.dmp.4 >> Thanks - this shows that session tickets are indeed disabled on the >> server side (0x...4fff means SSL_OP_ALL | SSL_OP_NO_TICKET), and from >> the packet dump it can be seen that the server does not send any session >> ticket extensions in its ServerHellos. >> >> With the client, the difference to the case where tickets are enabled is >> the fact that for the third connection, it includes an empty session >> ticket extension in the ClientHello (indicating that a NewSessionTicket >> message would be accepted). The server then again replies with a >> ServerHello with a server_name extension - which is immediately rejected >> by the client with SSL_AD_DECODE_ERROR. I therefore assume that it still >> ends up in the "if (!s->hit && tlsext_servername == 1)" code block, >> irrespective of whether the server uses SSL_OP_NO_TICKET or not. >> > > That's very strange. It seems like that shouldn't happen because a stateful > session resumption should occur and s->hit == 1. > > Will analyse it more when I have access to my main test setup later today. > That is most peculiar. The server is sending back a zero length session ID in the server hello which it shouldn't be doing if tickets are disabled. Is the server somehow using an older version of OpenSSL? There has been a bug in the past which might do that but it was fixed well before 0.9.8k. Steve. -- Dr Stephen N. Henson. Senior Technical/Cryptography Advisor, Open Source Software Institute: www.oss-institute.org OpenSSL Core team: www.openssl.org
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kaspar Brand wrote: > Kamesh Jayachandran, >> [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] >> ssl_init_ssl_connection: options=0x1114fff >> >> >> The tcpdump for this failure is at, >> >> http://www.livecipher.com/tlsext_dump/tlsext.dmp.4 > > Thanks - this shows that session tickets are indeed disabled on the > server side (0x...4fff means SSL_OP_ALL | SSL_OP_NO_TICKET), and from > the packet dump it can be seen that the server does not send any session > ticket extensions in its ServerHellos. > > With the client, the difference to the case where tickets are enabled is > the fact that for the third connection, it includes an empty session > ticket extension in the ClientHello (indicating that a NewSessionTicket > message would be accepted). The server then again replies with a > ServerHello with a server_name extension - which is immediately rejected > by the client with SSL_AD_DECODE_ERROR. I therefore assume that it still > ends up in the "if (!s->hit && tlsext_servername == 1)" code block, > irrespective of whether the server uses SSL_OP_NO_TICKET or not. > That's very strange. It seems like that shouldn't happen because a stateful session resumption should occur and s->hit == 1. Will analyse it more when I have access to my main test setup later today. Steve. -- Dr Stephen N. Henson. Senior Technical/Cryptography Advisor, Open Source Software Institute: www.oss-institute.org OpenSSL Core team: www.openssl.org
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran, > [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] > ssl_init_ssl_connection: options=0x1114fff > > > The tcpdump for this failure is at, > > http://www.livecipher.com/tlsext_dump/tlsext.dmp.4 Thanks - this shows that session tickets are indeed disabled on the server side (0x...4fff means SSL_OP_ALL | SSL_OP_NO_TICKET), and from the packet dump it can be seen that the server does not send any session ticket extensions in its ServerHellos. With the client, the difference to the case where tickets are enabled is the fact that for the third connection, it includes an empty session ticket extension in the ClientHello (indicating that a NewSessionTicket message would be accepted). The server then again replies with a ServerHello with a server_name extension - which is immediately rejected by the client with SSL_AD_DECODE_ERROR. I therefore assume that it still ends up in the "if (!s->hit && tlsext_servername == 1)" code block, irrespective of whether the server uses SSL_OP_NO_TICKET or not. Kaspar
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Hi Kaspar, I applied your 'mod_ssl-disable_tls_tickets.diff' and 'mod_ssl-log_ssloptions.diff' to apache-2.2.12 and initiated the 'failing svn import operation'. [Mon Oct 26 15:48:21 2009] [warn] [client 10.2.0.88] ssl_init_ssl_connection: options=0x1114fff [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] ssl_init_ssl_connection: options=0x1114fff [Mon Oct 26 15:48:22 2009] [warn] [client 10.2.0.88] ssl_init_ssl_connection: options=0x1114fff The tcpdump for this failure is at, http://www.livecipher.com/tlsext_dump/tlsext.dmp.4 With regards Kamesh Jayachandran On 10/25/2009 09:21 PM, Kaspar Brand wrote: Dr Stephen Henson wrote: Disabling tickets using SSL_OP_NO_TICKET server side SHOULD work too (does in my tests) so I've no idea why that wouldn't in the OPs setup unless the patch doesn't set it in all contexts. Try placing it right after any call to SSL_CTX_new(). I'm still a bit puzzled as to why my previously posted patch does not turn off TLS session tickets... there's only one place in mod_ssl where a new context is created, and in my tests, SSL_OP_NO_TICKET was reliably applied (i.e., I didn't see any session tickets on the wire). Maybe there's another issue if tickets are turned off? Kamesh, could you apply the attached patch, for diagnostic purposes (in addition to mod_ssl-disable_tls_tickets.diff), and let us know what "options=" values you see in your ErrorLog? Note that you don't have to increase Apache's LogLevel, the options for any new SSL connection will be logged with "warn" already. Also, it would be helpful to have another capture (with mod_ssl patched like this) where the svn client still fails with a "parse tlsext" error. Thanks. Kaspar
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kaspar Brand wrote: Joe Orton wrote: the OpenSSL client (SNI extensions should never contain literal IPv4 addresses). Good point - I've changed neon for future releases to only enable SNI if the hostname is not a numeric IP address. This logic should go into OpenSSL, I think... Making openssl "intelligent" like "you have requested some value that I don't think is a valid hostname, so I will ignore you sni request" is not exactly a nice thing. You must reject everything that is not a DNS hostname. Looks ugly. If you have just a "raw" IP address an application may probably already know this case.
RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Hi Kaspar, I am away from the test environment right now, Will get back in next 13 hours. With regards Kamesh Jayachandran -Original Message- From: Kaspar Brand [mailto:httpd-dev.2...@velox.ch] Sent: Sun 10/25/2009 9:21 PM To: dev@httpd.apache.org Subject: Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 Dr Stephen Henson wrote: > Disabling tickets using SSL_OP_NO_TICKET server side SHOULD work too (does in > my > tests) so I've no idea why that wouldn't in the OPs setup unless the patch > doesn't set it in all contexts. Try placing it right after any call to > SSL_CTX_new(). I'm still a bit puzzled as to why my previously posted patch does not turn off TLS session tickets... there's only one place in mod_ssl where a new context is created, and in my tests, SSL_OP_NO_TICKET was reliably applied (i.e., I didn't see any session tickets on the wire). Maybe there's another issue if tickets are turned off? Kamesh, could you apply the attached patch, for diagnostic purposes (in addition to mod_ssl-disable_tls_tickets.diff), and let us know what "options=" values you see in your ErrorLog? Note that you don't have to increase Apache's LogLevel, the options for any new SSL connection will be logged with "warn" already. Also, it would be helpful to have another capture (with mod_ssl patched like this) where the svn client still fails with a "parse tlsext" error. Thanks. Kaspar
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Joe Orton wrote:
>> the OpenSSL client (SNI extensions should never contain literal IPv4
>> addresses).
>
> Good point - I've changed neon for future releases to only enable SNI if
> the hostname is not a numeric IP address.
This logic should go into OpenSSL, I think... I know that this is
httpd-dev (not openssl-dev), but since Steve is listening anyway:
something like the attached patch? For the client side (i.e. in
ssl3_ctrl()), depending on how schoolmasterish OpenSSL should be towards
its users, the check could also be moved further down / be rejected with
INVALID_SERVERNAME.
Kaspar
Index: ssl/s3_lib.c
===
RCS file: /openssl-cvs/openssl/ssl/s3_lib.c,v
retrieving revision 1.130
diff -p -U 8 -r1.130 s3_lib.c
--- ssl/s3_lib.c16 Oct 2009 15:24:07 - 1.130
+++ ssl/s3_lib.c25 Oct 2009 16:04:06 -
@@ -148,16 +148,17 @@
* OTHERWISE.
*/
#include
#include
#include "ssl_locl.h"
#include "kssl_lcl.h"
#ifndef OPENSSL_NO_TLSEXT
+#include
#ifndef OPENSSL_NO_EC
#include "../crypto/ec/ec_lcl.h"
#endif /* OPENSSL_NO_EC */
#endif /* OPENSSL_NO_TLSEXT */
#include
#ifndef OPENSSL_NO_DH
#include
#endif
@@ -2382,22 +2383,25 @@ long ssl3_ctrl(SSL *s, int cmd, long lar
return(ret);
}
break;
#endif /* !OPENSSL_NO_ECDH */
#ifndef OPENSSL_NO_TLSEXT
case SSL_CTRL_SET_TLSEXT_HOSTNAME:
if (larg == TLSEXT_NAMETYPE_host_name)
{
+ unsigned char tmp[16];
if (s->tlsext_hostname != NULL)
OPENSSL_free(s->tlsext_hostname);
s->tlsext_hostname = NULL;
ret = 1;
- if (parg == NULL)
+ if (parg == NULL ||
+ /* silently ignore literal IP addresses */
+ a2i_ipadd(tmp, (const char *)parg))
break;
if (strlen((char *)parg) > TLSEXT_MAXLEN_host_name)
{
SSLerr(SSL_F_SSL3_CTRL,
SSL_R_SSL3_EXT_INVALID_SERVERNAME);
return 0;
}
if ((s->tlsext_hostname = BUF_strdup((char *)parg)) ==
NULL)
{
Index: ssl/t1_lib.c
===
RCS file: /openssl-cvs/openssl/ssl/t1_lib.c,v
retrieving revision 1.66
diff -p -U 8 -r1.66 t1_lib.c
--- ssl/t1_lib.c4 Sep 2009 17:42:53 - 1.66
+++ ssl/t1_lib.c25 Oct 2009 12:32:39 -
@@ -109,16 +109,19 @@
*
*/
#include
#include
#include
#include
#include
+#ifndef OPENSSL_NO_TLSEXT
+#include
+#endif
#include "ssl_locl.h"
const char tls1_version_str[]="TLSv1" OPENSSL_VERSION_PTEXT;
#ifndef OPENSSL_NO_TLSEXT
static int tls_decrypt_ticket(SSL *s, const unsigned char *tick, int ticklen,
const unsigned char *sess_id, int sesslen,
SSL_SESSION **psess);
@@ -618,16 +621,17 @@ int ssl_parse_clienthello_tlsext(SSL *s,
i.e. when the session has been established with a servername extension.
- On session reconnect, the servername extension may be absent.
*/
if (type == TLSEXT_TYPE_server_name)
{
unsigned char *sdata;
+ unsigned char tmp[16];
int servname_type;
int dsize;
if (size < 2)
{
*al = SSL_AD_DECODE_ERROR;
return 0;
}
@@ -650,16 +654,22 @@ int ssl_parse_clienthello_tlsext(SSL *s,
{
*al = SSL_AD_DECODE_ERROR;
return 0;
}
if (s->servername_done == 0)
switch (servname_type)
{
case TLSEXT_NAMETYPE_host_name:
+ if (a2i_ipadd(tmp, (const char *)sdata))
+ {
+ /* literal IP addresses are not
permitted */
+ *al = SSL3_AD_ILLEGAL_PARAMETER;
+ return 0;
+ }
if (s->session->tlsext_hostname == NULL)
{
if (len
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Dr Stephen Henson wrote: > Disabling tickets using SSL_OP_NO_TICKET server side SHOULD work too (does in > my > tests) so I've no idea why that wouldn't in the OPs setup unless the patch > doesn't set it in all contexts. Try placing it right after any call to > SSL_CTX_new(). I'm still a bit puzzled as to why my previously posted patch does not turn off TLS session tickets... there's only one place in mod_ssl where a new context is created, and in my tests, SSL_OP_NO_TICKET was reliably applied (i.e., I didn't see any session tickets on the wire). Maybe there's another issue if tickets are turned off? Kamesh, could you apply the attached patch, for diagnostic purposes (in addition to mod_ssl-disable_tls_tickets.diff), and let us know what "options=" values you see in your ErrorLog? Note that you don't have to increase Apache's LogLevel, the options for any new SSL connection will be logged with "warn" already. Also, it would be helpful to have another capture (with mod_ssl patched like this) where the svn client still fails with a "parse tlsext" error. Thanks. Kaspar --- httpd-2.2.14/modules/ssl/mod_ssl.c.orig 2009-05-19 13:44:59.0 +0200 +++ httpd-2.2.14/modules/ssl/mod_ssl.c 2009-10-24 17:58:20.0 +0200 @@ -394,6 +394,9 @@ int ssl_init_ssl_connection(conn_rec *c) return DECLINED; /* XXX */ } +ap_log_cerror(APLOG_MARK, APLOG_WARNING, 0, c, + "ssl_init_ssl_connection: options=0x%lx", SSL_get_options(ssl)); + vhost_md5 = ap_md5_binary(c->pool, (unsigned char *)sc->vhost_id, sc->vhost_id_len);
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Dr Stephen Henson wrote:
> Kaspar Brand wrote:
>> As Joe observed in an earlier message, there are only two places in
>> t1_lib.c:ssl_parse_serverhello_tlsext() which set SSL_AD_DECODE_ERROR,
>> and it seems very likely that the following code is hit:
>>
>>> if (!s->hit && tlsext_servername == 1)
>>> {
>>> if (s->tlsext_hostname)
>>> {
>>> if (s->session->tlsext_hostname == NULL)
>>> {
>>> s->session->tlsext_hostname =
>>> BUF_strdup(s->tlsext_hostname);
>>> if (!s->session->tlsext_hostname)
>>> {
>>> *al = SSL_AD_UNRECOGNIZED_NAME;
>>> return 0;
>>> }
>>> }
>>> else
>>> {
>>> *al = SSL_AD_DECODE_ERROR;
>>> return 0;
>>> }
>>> }
>>> }
>
> OK that makes sense. I can now reproduce this with OpenSSL s_server and
> s_client and stateless session resumption. A fix for this would have to be in
> the client code, I'll look into that.
>
I've looked into this further now. The cause is an interaction between session
tickets and the server name extension.
When a stateless session resumption is attempted OpenSSL sends (as permitted by
RFC5077) an empty session ID. This has the side effect that it is not clear
whether the server has accepted the ticket until later in the handshake whereas
in a stateful resumption this is apparent after received the server hello by
comparing session IDs.
As a result the value of s->hit above cannot be trusted to check for a resumed
session.
I'm looking into the best way to fix this. For now changing the above to:
else if (!s->session->tlsext_tick)
{
*al = SSL_AD_DECODE_ERROR;
return 0;
}
client side is one way but it doesn't retain all the original logic.
Disabling tickets using SSL_OP_NO_TICKET server side SHOULD work too (does in my
tests) so I've no idea why that wouldn't in the OPs setup unless the patch
doesn't set it in all contexts. Try placing it right after any call to
SSL_CTX_new().
Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
On Fri, Oct 23, 2009 at 06:08:52PM +0200, Kaspar Brand wrote: > Kamesh Jayachandran wrote: > > Find the tcpdump while this failure occurs at > > http://www.livecipher.com/tlsext_dump/tlsext.dmp > > It seems that you used a URI with an IP address (https://10.2.1.97/...), > is that correct? This actually uncovers a - probably unrelated - bug in > the OpenSSL client (SNI extensions should never contain literal IPv4 > addresses). Good point - I've changed neon for future releases to only enable SNI if the hostname is not a numeric IP address. Regards, Joe
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kaspar Brand wrote:
>
> As Joe observed in an earlier message, there are only two places in
> t1_lib.c:ssl_parse_serverhello_tlsext() which set SSL_AD_DECODE_ERROR,
> and it seems very likely that the following code is hit:
>
>> if (!s->hit && tlsext_servername == 1)
>> {
>> if (s->tlsext_hostname)
>> {
>> if (s->session->tlsext_hostname == NULL)
>> {
>> s->session->tlsext_hostname =
>> BUF_strdup(s->tlsext_hostname);
>> if (!s->session->tlsext_hostname)
>> {
>> *al = SSL_AD_UNRECOGNIZED_NAME;
>> return 0;
>> }
>> }
>> else
>> {
>> *al = SSL_AD_DECODE_ERROR;
>> return 0;
>> }
>> }
>> }
>
> I'll defer to the OpenSSL gurus for further analysis, but apparently
> it has to do with the fact that the client reuses an SSL session
> (s->session->tlsext_hostname is non-null), but this case is not
> expected by ssl_parse_serverhello_tlsext(). It would also explain,
> however, why the issue does not show up immediately, but only after
> a couple of minutes (i.e., when a session is really being reused).
>
OK that makes sense. I can now reproduce this with OpenSSL s_server and
s_client and stateless session resumption. A fix for this would have to be in
the client code, I'll look into that.
It doesn't occur when tickets are disabled so setting the SSL_OP_NO_TICKET
should be a workaround on the server as long as it is set in all relevant
SSL_CTX structures.
Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org
RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
>Ok, thanks. So, for the sake of reference, your setup for this capture >was: >- (Windows) client with OpenSSL 0.9.8k, compiled with defaults >- server with OpenSSL 0.9.8j, compiled with defaults >- httpd 2.2.14, w/o the OP_NO_TICKET patch >Is that correct? No. My setup is, - win32 client with openssl 0.9.8j compiled with defaults. - server with OpenSSL 0.9.8k, compiled with defaults - httpd 2.2.12, w/o the OP_NO_TICKET patch With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote:
> Yes done, find the dump at http://www.livecipher.com/tlsext_dump/tlsext.dmp.2
Ok, thanks. So, for the sake of reference, your setup for this capture
was:
- (Windows) client with OpenSSL 0.9.8k, compiled with defaults
- server with OpenSSL 0.9.8j, compiled with defaults
- httpd 2.2.14, w/o the OP_NO_TICKET patch
Is that correct?
I have extracted the two Hello messages into a separate pcap file (attached).
It's the ServerHello in the second packet to which the client then immediately
replies to with SSL_AD_DECODE_ERROR. I'm also attaching Wireshark's
interpretation of the bytes in question, followed by their hex dumps.
As Joe observed in an earlier message, there are only two places in
t1_lib.c:ssl_parse_serverhello_tlsext() which set SSL_AD_DECODE_ERROR,
and it seems very likely that the following code is hit:
> if (!s->hit && tlsext_servername == 1)
> {
> if (s->tlsext_hostname)
> {
> if (s->session->tlsext_hostname == NULL)
> {
> s->session->tlsext_hostname =
> BUF_strdup(s->tlsext_hostname);
> if (!s->session->tlsext_hostname)
> {
> *al = SSL_AD_UNRECOGNIZED_NAME;
> return 0;
> }
> }
> else
> {
> *al = SSL_AD_DECODE_ERROR;
> return 0;
> }
> }
> }
I'll defer to the OpenSSL gurus for further analysis, but apparently
it has to do with the fact that the client reuses an SSL session
(s->session->tlsext_hostname is non-null), but this case is not
expected by ssl_parse_serverhello_tlsext(). It would also explain,
however, why the issue does not show up immediately, but only after
a couple of minutes (i.e., when a session is really being reused).
Kaspar
ClientHello: includes both an SNI and a SessionTicket extension
(i.e., client tries a stateless resume)
TLSv1 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 316
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 312
Version: TLS 1.0 (0x0301)
Random
gmt_unix_time: Sep 1, 2009 18:24:07.0
random_bytes:
DC61CBA8386F040540B164CA43C99C0A25618A34DC84B0CF...
Session ID Length: 0
Cipher Suites Length: 40
Cipher Suites (20 suites)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_IDEA_CBC_SHA (0x0007)
Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
Cipher Suite: TLS_DHE_RSA_WITH_DES_CBC_SHA (0x0015)
Cipher Suite: TLS_DHE_DSS_WITH_DES_CBC_SHA (0x0012)
Cipher Suite: TLS_RSA_WITH_DES_CBC_SHA (0x0009)
Cipher Suite: TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA (0x0014)
Cipher Suite: TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA (0x0011)
Cipher Suite: TLS_RSA_EXPORT_WITH_DES40_CBC_SHA (0x0008)
Cipher Suite: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 (0x0006)
Cipher Suite: TLS_RSA_EXPORT_WITH_RC4_40_MD5 (0x0003)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 231
Extension: server_name
Type: server_name (0x)
Length: 15
Data (15 bytes)
Extension: SessionTicket TLS
Type: SessionTicket TLS (0x0023)
Length: 208
Data (208 bytes)
0030 01 00 01 38 03 ...8.
0040 01 4a 9d 4a a7 dc 61 cb a8 38 6f 04 05 40 b1 64 .j.j..a..8...@.d
0050 ca 43 c9 9c 0a 25 61 8a 34 dc 84 b0 cf e2 f2 0c .C...%a.4...
0060 ff 00 00 28 00 39 00 38 00 35 00 16 00 13 00 0a ...(.9.8.5.
RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
>It seems that you used a URI with an IP address (https://10.2.1.97/...), >is that correct? Yes. >Could you retry the test and make sure that you use an FQDN in the URI >you specify for the client (through an entry in the hosts file or so)? Yes done, find the dump at http://www.livecipher.com/tlsext_dump/tlsext.dmp.2 Thanks With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote: > Find the tcpdump while this failure occurs at > http://www.livecipher.com/tlsext_dump/tlsext.dmp It seems that you used a URI with an IP address (https://10.2.1.97/...), is that correct? This actually uncovers a - probably unrelated - bug in the OpenSSL client (SNI extensions should never contain literal IPv4 addresses). Could you retry the test and make sure that you use an FQDN in the URI you specify for the client (through an entry in the hosts file or so)? Kaspar
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
On 10/22/2009 02:09 PM, Joe Orton wrote: On Thu, Oct 22, 2009 at 12:49:10PM +0530, Kamesh Jayachandran wrote: I tried your patch. It does *not* fix the issue. One difference it makes is , triggers failure early at 20/30 files(PUT requests) instead of 20k files earlier. Can you get a packet dump/trace from the client side? Is there anything between client and server which is intercepting the SSL traffic? (physical/software firewall?) It would be good whether this problem is due to the traffic becoming corrupted. Find the tcpdump while this failure occurs at http://www.livecipher.com/tlsext_dump/tlsext.dmp I could not suspect the firewall as this occurs only with httpd-2.2.12+openssl-with-tls-ext *not* with httpd-2.2.11 or httpd-2.2.13+openssl-without-tls-ext. Thanks There seem to be two places in OpenSSL's ssl_parse_serverhello_tlsext() which can send a "decode error" alert, if I am reading the code and following the error handling correctly. It would be useful if you could use a custom OpenSSL build with an fprintf(stderr, ... ) or similar added before each of the "*al = SSL_AD_DECODE_ERROR;" lines in that function (in ssl/t1_lib.c), if you're able to try that? Regards, Joe Will try this next week as it involves building in win32 which I am not used to. With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Did you say what version of OpenSSL the failing client was using on Windows? It happens with openssl-0.9.8j on client openssl-0.9.8k on server Hmm... could be 0.9.8j sending bad data with invalid extension syntax under rare circumstances. A packet sniffer or logging the errant extensions received by OpenSSL could help trace this further. Find the tcpdump while this failure occurs at http://www.livecipher.com/tlsext_dump/tlsext.dmp Thanks with regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
On 10/22/2009 10:37 PM, Dr Stephen Henson wrote: Kamesh Jayachandran wrote: On 10/22/2009 05:24 PM, Dr Stephen Henson wrote: That's due to the function pointer issues which gcc 4.2 and later doesn't like: this was fixed in newer versions of OpenSSL. Is there any switch we can pass to gcc 4.2 to compile and make it work properly. No. If you really want to use 0.9.8b it needs an older version of gcc or you can backport the fixes. They are rather extensive but mainly contained in: http://cvs.openssl.org/chngview?cn=16526 and http://cvs.openssl.org/chngview?cn=16528 OpenSSL 0.9.8b doesn't use TLS extensions at all. Do you need TLS extensions on the client/server? If not try compiling OpenSSL with no-tlsext. May not be possible as *client* builds are not in our control. I believe no-tlsext does *not* disable TLS functionality itself. The no-tlsext option disables TLS extension functionality. If that works on the server side then an alternative workaround could be found. Yes, building 'openssl-0.9.8k' on the server side with no-tlsext works around this issue irrespective of the client openssl. Did you say what version of OpenSSL the failing client was using on Windows? It happens with openssl-0.9.8j on client openssl-0.9.8k on server Hmm... could be 0.9.8j sending bad data with invalid extension syntax under rare circumstances. A packet sniffer or logging the errant extensions received by OpenSSL could help trace this further. Will post tcpdump once I set it up on win32. Thanks. With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote: > On 10/22/2009 05:24 PM, Dr Stephen Henson wrote: >> That's due to the function pointer issues which gcc 4.2 and later >> doesn't like: >> this was fixed in newer versions of OpenSSL. >> >> > > Is there any switch we can pass to gcc 4.2 to compile and make it work > properly. > No. If you really want to use 0.9.8b it needs an older version of gcc or you can backport the fixes. They are rather extensive but mainly contained in: http://cvs.openssl.org/chngview?cn=16526 and http://cvs.openssl.org/chngview?cn=16528 OpenSSL 0.9.8b doesn't use TLS extensions at all. >> Do you need TLS extensions on the client/server? If not try compiling >> OpenSSL >> with no-tlsext. >> > > May not be possible as *client* builds are not in our control. > > I believe no-tlsext does *not* disable TLS functionality itself. > The no-tlsext option disables TLS extension functionality. If that works on the server side then an alternative workaround could be found. >> Did you say what version of OpenSSL the failing client was using on >> Windows? >> >> > > It happens with openssl-0.9.8j on client openssl-0.9.8k on server > Hmm... could be 0.9.8j sending bad data with invalid extension syntax under rare circumstances. A packet sniffer or logging the errant extensions received by OpenSSL could help trace this further. Steve. -- Dr Stephen N. Henson. Senior Technical/Cryptography Advisor, Open Source Software Institute: www.oss-institute.org OpenSSL Core team: www.openssl.org
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
On 10/22/2009 05:24 PM, Dr Stephen Henson wrote: Kamesh Jayachandran wrote: I need to double check it by myself(One of the internal tester was saying that this happens with openssl-0.9.8b). I vaguely remember this happening with openssl-0.9.8g. Unfortunately I could not build/use openssl-0.9.8b and openssl-0.9.8e on my box. I get the following error for which I do not know *solution* right now, [kam...@kamesh openssl-0.9.8e]$ /home/kamesh/openssl-0.9.8e/bin/openssl genrsa 1024 -out /tmp/server.key Generating RSA private key, 1024 bit long modulus ++ ++ e is 65537 (0x10001) Illegal instruction That's due to the function pointer issues which gcc 4.2 and later doesn't like: this was fixed in newer versions of OpenSSL. Is there any switch we can pass to gcc 4.2 to compile and make it work properly. Do you need TLS extensions on the client/server? If not try compiling OpenSSL with no-tlsext. May not be possible as *client* builds are not in our control. I believe no-tlsext does *not* disable TLS functionality itself. Did you say what version of OpenSSL the failing client was using on Windows? It happens with openssl-0.9.8j on client openssl-0.9.8k on server With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote: > >> I need to double check it by myself(One of the internal tester was >> saying that this happens with openssl-0.9.8b). >> I vaguely remember this happening with openssl-0.9.8g. >> >> > > Unfortunately I could not build/use openssl-0.9.8b and openssl-0.9.8e on > my box. > > I get the following error for which I do not know *solution* right now, > > > [kam...@kamesh openssl-0.9.8e]$ /home/kamesh/openssl-0.9.8e/bin/openssl > genrsa 1024 -out /tmp/server.key > Generating RSA private key, 1024 bit long modulus > ++ > ++ > e is 65537 (0x10001) > Illegal instruction > > That's due to the function pointer issues which gcc 4.2 and later doesn't like: this was fixed in newer versions of OpenSSL. Do you need TLS extensions on the client/server? If not try compiling OpenSSL with no-tlsext. Did you say what version of OpenSSL the failing client was using on Windows? It is strange that this doesn't happen straight away. Steve. -- Dr Stephen N. Henson. Senior Technical/Cryptography Advisor, Open Source Software Institute: www.oss-institute.org OpenSSL Core team: www.openssl.org
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
I need to double check it by myself(One of the internal tester was saying that this happens with openssl-0.9.8b). I vaguely remember this happening with openssl-0.9.8g. Unfortunately I could not build/use openssl-0.9.8b and openssl-0.9.8e on my box. I get the following error for which I do not know *solution* right now, [kam...@kamesh openssl-0.9.8e]$ /home/kamesh/openssl-0.9.8e/bin/openssl genrsa 1024 -out /tmp/server.key Generating RSA private key, 1024 bit long modulus ++ ++ e is 65537 (0x10001) Illegal instruction >Also since you post that the problem is with the client - did you also >rebuild the client with newer OpenSSL 0.8.8k, or even with 1.0.0.b3? Will experiment and get back. I do not have win32 build system to build svn clients against openssl 1.0.0.b3 With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
On Thu, Oct 22, 2009 at 12:49:10PM +0530, Kamesh Jayachandran wrote: > I tried your patch. It does *not* fix the issue. > One difference it makes is , triggers failure early at 20/30 files(PUT > requests) instead of 20k files earlier. Can you get a packet dump/trace from the client side? Is there anything between client and server which is intercepting the SSL traffic? (physical/software firewall?) It would be good whether this problem is due to the traffic becoming corrupted. There seem to be two places in OpenSSL's ssl_parse_serverhello_tlsext() which can send a "decode error" alert, if I am reading the code and following the error handling correctly. It would be useful if you could use a custom OpenSSL build with an fprintf(stderr, ... ) or similar added before each of the "*al = SSL_AD_DECODE_ERROR;" lines in that function (in ssl/t1_lib.c), if you're able to try that? Regards, Joe
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
On 10/21/2009 10:29 PM, Kaspar Brand wrote: Kamesh Jayachandran wrote: When I built the server against openssl-1.0.0-beta3, I could *not* access svn at all using svn client while I could access the same via browser. Any clues? The TLS session ticket extension might be the culprit here (or more precisely, OpenSSL's implementation of that extension). Can you try the attached patch and see whether it makes a difference? Hi Kaspar, I tried your patch. It does *not* fix the issue. One difference it makes is , triggers failure early at 20/30 files(PUT requests) instead of 20k files earlier. With regards Kamesh Jayachandran
RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Thanks Kaspar, will try that tomorrow(Right now away from my dev box) and let you know. With regards Kamesh Jayachandran -Original Message- From: Kaspar Brand [mailto:httpd-dev.2...@velox.ch] Sent: Wed 10/21/2009 10:29 PM To: dev@httpd.apache.org Subject: Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 Kamesh Jayachandran wrote: > When I built the server against openssl-1.0.0-beta3, I could *not* > access svn at all using svn client while I could access the same via > browser. > > Any clues? The TLS session ticket extension might be the culprit here (or more precisely, OpenSSL's implementation of that extension). Can you try the attached patch and see whether it makes a difference? Kaspar
RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Hi Gunter, Nice to meet you after a long time. >sounds all strange. I would say since we have SNI support since 2.2.12 >that there is the problem, and from the bug report it seems that the OP >used already 2 SSL virtual hosts with same IP before 2.2.12 which was >neither supported feature nor it was working properly at all; so >probably his configuration is the problem? In my setup where this fails has only *one* SSL virtual host(_default_). >On the other side the needed support in OpenSSL started with 0.9.8j >IIRC, and with 0.9.8k it started to be enabled by default. So I would >assume that builds with 0.9.8g and 0.9.8b are not affected ... I need to double check it by myself(One of the internal tester was saying that this happens with openssl-0.9.8b). I vaguely remember this happening with openssl-0.9.8g. >Also since you post that the problem is with the client - did you also >rebuild the client with newer OpenSSL 0.8.8k, or even with 1.0.0.b3? Will experiment and get back. With regards Kamesh Jayachandran
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Kamesh Jayachandran wrote: > When I built the server against openssl-1.0.0-beta3, I could *not* > access svn at all using svn client while I could access the same via > browser. > > Any clues? The TLS session ticket extension might be the culprit here (or more precisely, OpenSSL's implementation of that extension). Can you try the attached patch and see whether it makes a difference? Kaspar --- httpd-2.2.14/modules/ssl/ssl_engine_init.c.orig 2009-08-16 17:53:12.0 +0200 +++ httpd-2.2.14/modules/ssl/ssl_engine_init.c 2009-10-21 18:49:05.0 +0200 @@ -382,6 +382,8 @@ static void ssl_init_ctx_tls_extensions( ssl_log_ssl_error(APLOG_MARK, APLOG_ERR, s); ssl_die(); } + +SSL_CTX_set_options(mctx->ssl_ctx, SSL_OP_NO_TICKET); } #endif
Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12
Hi Kamesh, nice to meet you here again! Kamesh Jayachandran schrieb: > I could not isolate this issue to openssl versions as it happens with > openssl-0.9.8k, openssl-0.9.8g, openssl-0.9.8-b > > When I built the server against openssl-1.0.0-beta3, I could *not* > access svn at all using svn client while I could access the same via > browser. > > Any clues? sounds all strange. I would say since we have SNI support since 2.2.12 that there is the problem, and from the bug report it seems that the OP used already 2 SSL virtual hosts with same IP before 2.2.12 which was neither supported feature nor it was working properly at all; so probably his configuration is the problem? On the other side the needed support in OpenSSL started with 0.9.8j IIRC, and with 0.9.8k it started to be enabled by default. So I would assume that builds with 0.9.8g and 0.9.8b are not affected ... Also since you post that the problem is with the client - did you also rebuild the client with newer OpenSSL 0.8.8k, or even with 1.0.0.b3? Günter.
