I have an application which needs to protect datagram traffic, and
also directly control the socket I/O. Using DTLS over a BIO pair
appears to work for my purposes except for one problem when handling timeouts.
In dtls1_check_timeout_num(), after 2 unsuccessful retransmission
attempts, the code calls BIO_ctrl() with the BIO_CTRL_DGRAM_GET_FALLBACK_MTU
option to adjust the MTU. This operation is not defined for a BIO
pair, and results in the MTU being set to zero. That eventually
causes an OpenSSL_assert() to fail in dtls1_do_write().
It would make sense to recognize that zero can't be a valid fallback MTU
value, and avoid resetting the MTU. A patch with a possible fix is attached.
--- Gary
diff --git a/ssl/d1_lib.c b/ssl/d1_lib.c
index db180f2..371199d 100644
--- a/ssl/d1_lib.c
+++ b/ssl/d1_lib.c
@@ -401,12 +401,17 @@ void dtls1_stop_timer(SSL *s)
int dtls1_check_timeout_num(SSL *s)
{
+ unsigned int mtu;
s->d1->timeout.num_alerts++;
/* Reduce MTU after 2 unsuccessful retransmissions */
if (s->d1->timeout.num_alerts > 2)
{
- s->d1->mtu = BIO_ctrl(SSL_get_wbio(s), BIO_CTRL_DGRAM_GET_FALLBACK_MTU, 0, NULL);
+ mtu = BIO_ctrl(SSL_get_wbio(s), BIO_CTRL_DGRAM_GET_FALLBACK_MTU, 0, NULL);
+ if (mtu > 0)
+ {
+ s->d1->mtu = mtu;
+ }
}
if (s->d1->timeout.num_alerts > DTLS1_TMO_ALERT_COUNT)
