[openssl.org #1929] DTLS MTU bug

[Robin Seggelmann via RT]

On May 12, 2009, at 8:24 PM, Daniel Mentz wrote:

> I've got the impression that the DTLS part of OpenSSL is based on an  
> incorrect understanding of the term MTU (Maximum Transmission Unit).
>
> My understanding is that the MTU refers to the size of the IP packet  
> including the IP header (usually 20 bytes) and the UDP header  
> (usually 8 bytes) in case UDP is used. This means that I can  
> transfer 1472 bytes of payload if the MTU is 1500 bytes.
>
> Now, if I start openssl s_server with the following command line
>
> ./openssl s_server -dtls1 -no_ecdhe -timeout -cert large.pem -mtu 1400
>
> and monitor packets with wireshark I can see IP packets with a total  
> length of 1428 bytes.
> From looking at this I infere that OpenSSL interprets the MTU as the  
> maximum payload size of an UDP packet.
>
> If I get rid of the -mtu parameter, rely on Path MTU discovery and  
> set the MTU of the outgoing interface to 1400 I don't get any  
> communication going. But I do see an avalanche of "Destination  
> unreachable (Fragmentation needed)" ICMP messages. I guess that this  
> is due to the incorrect understanding of the MTU. OpenSSL appears to  
> try sending larger packets than allowed by the PMTU.
>
> Can anyone confirm this problem?


In the BIO object the PMTU is either discovered or set by the user. In  
both cases the subtraction of the IP and UDP headers are missing,  
while everywhere else is assumed that the set MTU represents the  
available message size for DTLS. The MTU discovery is also specific to  
Linux and is not enclosed by defines if Linux is actually used. On  
other platforms this mechanism either doesn't work at all or result in  
strange values. Here is a patch addressing these issues.


--- crypto/bio/bss_dgram.c      2009-05-05 10:45:33.000000000 +0200
+++ crypto/bio/bss_dgram.c      2009-05-15 12:46:39.000000000 +0200
@@ -70,7 +70,9 @@
  #include <sys/timeb.h>
  #endif

+#ifdef OPENSSL_SYS_LINUX
  #define IP_MTU      14 /* linux is lame */
+#endif

  #ifdef WATT32
  #define sock_write SockWrite  /* Watt-32 uses same names */
@@ -334,22 +336,26 @@
                break;
  #endif
        case BIO_CTRL_DGRAM_QUERY_MTU:
-         sockopt_len = sizeof(sockopt_val);
+#ifdef IP_MTU
+        sockopt_len = sizeof(sockopt_val);
                if ((ret = getsockopt(b->num, IPPROTO_IP, IP_MTU, (void  
*)&sockopt_val,
                        &sockopt_len)) < 0 || sockopt_val < 0)
                        { ret = 0; }
                else
                        {
-                       data->mtu = sockopt_val;
+                       data->mtu = sockopt_val - 20 - 8; /* Subtract IP and 
UDP header */
                        ret = data->mtu;
                        }
+#else
+               ret = 0;
+#endif
                break;
        case BIO_CTRL_DGRAM_GET_MTU:
                return data->mtu;
                break;
        case BIO_CTRL_DGRAM_SET_MTU:
-               data->mtu = num;
-               ret = num;
+               data->mtu = num - 20 - 8; /* Subtract IP and UDP header */
+               ret = data->mtu;
                break;
        case BIO_CTRL_DGRAM_SET_CONNECTED:
                to = (struct sockaddr *)ptr;

dtls-mtu-bug-1.0.0.patch
Description: Binary data