Re: [Openvpn-devel] MTU auto-discovery
> I am developing this patch on the 2.3 branch, as that is what I use for > my networking needs, but the files I am touching have not changed in > incompatible ways on the master. At the moment, the "Great > Reformatting" commit is the biggest stumbling block to rebasing this > patch to the master. > > Also, I don't have the resources to test it on anything other than > Linux, but I am hoping that the existing continuous integration magic > will help overcome this limitation. > > At the moment, I expect it will take another two weeks until the patch > is ready to be released on 2.3 > > Any comments or offers to assist with testing are welcome. Specifically, > how does this patch fit in with the release plans for 2.5 or 2.6+? The idea sounds good. Whether it can be integrated in 2.5 or not depends on the patch quality, how intrusive it is and some other factors. We will need to see the patch before we can make a decision on that. Arne signature.asc Description: OpenPGP digital signature ___ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel
Re: [Openvpn-devel] MTU auto-discovery
I'm thinking that this feature would need to be able to be toggled on/off, because it will be noisy on networks that are trying to circumvent censorship systems, making for easy discovery of openvpn tunnels in Egypt / China / India / etc. It does look like a significant improvement for normal uses of OpenVPN, though. Derek Zimmer Chief Executive Officer Open Source Technology Improvement Fund On Mon, Feb 24, 2020 at 1:26 PM Radu Hociung wrote: > > Hello, > > I would like to gauge interest in including my upcoming MTU patch into > the 2.5 release, and also to notify you that I am working on this patch, > to avoid effort duplication, if any. > > Description of the "MTU auto-negotiation" patch: > > MTU settings will be fully auto-negotiated between server and > client, making most if not all related options unnecessary, > such as --tun-mtu, --link-mtu, --mssfix, --fragment, etc. > Interoperation with old (pre-patch) servers and clients will be > guaranteed by design and tested with the included test suite. > > Summary of operation: > Peers will use the P_CONTROL_HARD_RESET_* messages to do Path > MTU discovery, and configure their own send/receive buffers > accordingly. The "reliable" layer's ability to ACK packets it > can receive, including repeated packets, is leveraged. > The algorithm will re-send one or more HARD_RESET_* messsages, > padded to a larger size, after the initial reset packet is > acked by the remote, in order to discover the largest packet > the remote can receive (and thus ack). Regular TLS handshake > will start after one padded HARD_RESET message is acked, with > frame buffer parameters set appropriately. > > After the connection is established, Path MTU will be monitored > with a new PING message over the TLS channel, configurable with > an option like "--ping-pmtud " that will complement or > replace the regular "--ping" which is sent over the data > channel. Each peer will adjust the max size of packet > transmitted over the link when/if the reliable --ping-pmtud > indicates the path MTU has decreased. > > The TUN interface MTU setting will be adjusted dynamically > as a result of link PMTUD results, if possible. > > The patch will generate ICMP messages "fragmentation needed" > on the TUN interface to handle the case where interface MTU > cannot be changed, and also the client-to-client communication > case where a client with a larger link-mtu sends traffic to a > client with a smaller link-mtu (as described in RFC1191) > > The user-facing configuration like --link-mtu etc, are > deprecated, and only indicate ceiling values, while the PMTUD > algo will set buffers to smaller sizes if needed. In the future, > these options should be completely ignored, and/or obsoleted. > > Testing: > The patch will include a comprehensive test suite that can be > run (unattended) to verify inter-operation with other versions > of ovpn, and with various combinations of settings like cipher, > compression, link MTU, etc. The test suite requires two hosts > (server and client), with root capability, as to vary the link's > MTU configuration. The host running the test suite runs > the server side of the test, and it starts a client with the > appropriate configuration on the other host, via ssh. It > requires both test host to have access to a shared filesystem, > and installed older versions of ovpn in addition to the patched > version. > > I am developing this patch on the 2.3 branch, as that is what I use for > my networking needs, but the files I am touching have not changed in > incompatible ways on the master. At the moment, the "Great > Reformatting" commit is the biggest stumbling block to rebasing this > patch to the master. > > Also, I don't have the resources to test it on anything other than > Linux, but I am hoping that the existing continuous integration magic > will help overcome this limitation. > > At the moment, I expect it will take another two weeks until the patch > is ready to be released on 2.3 > > Any comments or offers to assist with testing are welcome. Specifically, > how does this patch fit in with the release plans for 2.5 or 2.6+? > > Thank you > Radu Hociung. > > > > ___ > Openvpn-devel mailing list > Openvpn-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/openvpn-devel ___ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel
[Openvpn-devel] MTU auto-discovery
Hello, I would like to gauge interest in including my upcoming MTU patch into the 2.5 release, and also to notify you that I am working on this patch, to avoid effort duplication, if any. Description of the "MTU auto-negotiation" patch: MTU settings will be fully auto-negotiated between server and client, making most if not all related options unnecessary, such as --tun-mtu, --link-mtu, --mssfix, --fragment, etc. Interoperation with old (pre-patch) servers and clients will be guaranteed by design and tested with the included test suite. Summary of operation: Peers will use the P_CONTROL_HARD_RESET_* messages to do Path MTU discovery, and configure their own send/receive buffers accordingly. The "reliable" layer's ability to ACK packets it can receive, including repeated packets, is leveraged. The algorithm will re-send one or more HARD_RESET_* messsages, padded to a larger size, after the initial reset packet is acked by the remote, in order to discover the largest packet the remote can receive (and thus ack). Regular TLS handshake will start after one padded HARD_RESET message is acked, with frame buffer parameters set appropriately. After the connection is established, Path MTU will be monitored with a new PING message over the TLS channel, configurable with an option like "--ping-pmtud " that will complement or replace the regular "--ping" which is sent over the data channel. Each peer will adjust the max size of packet transmitted over the link when/if the reliable --ping-pmtud indicates the path MTU has decreased. The TUN interface MTU setting will be adjusted dynamically as a result of link PMTUD results, if possible. The patch will generate ICMP messages "fragmentation needed" on the TUN interface to handle the case where interface MTU cannot be changed, and also the client-to-client communication case where a client with a larger link-mtu sends traffic to a client with a smaller link-mtu (as described in RFC1191) The user-facing configuration like --link-mtu etc, are deprecated, and only indicate ceiling values, while the PMTUD algo will set buffers to smaller sizes if needed. In the future, these options should be completely ignored, and/or obsoleted. Testing: The patch will include a comprehensive test suite that can be run (unattended) to verify inter-operation with other versions of ovpn, and with various combinations of settings like cipher, compression, link MTU, etc. The test suite requires two hosts (server and client), with root capability, as to vary the link's MTU configuration. The host running the test suite runs the server side of the test, and it starts a client with the appropriate configuration on the other host, via ssh. It requires both test host to have access to a shared filesystem, and installed older versions of ovpn in addition to the patched version. I am developing this patch on the 2.3 branch, as that is what I use for my networking needs, but the files I am touching have not changed in incompatible ways on the master. At the moment, the "Great Reformatting" commit is the biggest stumbling block to rebasing this patch to the master. Also, I don't have the resources to test it on anything other than Linux, but I am hoping that the existing continuous integration magic will help overcome this limitation. At the moment, I expect it will take another two weeks until the patch is ready to be released on 2.3 Any comments or offers to assist with testing are welcome. Specifically, how does this patch fit in with the release plans for 2.5 or 2.6+? Thank you Radu Hociung. ___ Openvpn-devel mailing list Openvpn-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openvpn-devel
[Openvpn-devel] MTU handling on Windows
On Windows 7 at least you can set the MTU with netsh: netsh interface $PROTO set subinterface $TUNDEV mtu=$MTU store=active Where PROTO is 'ipv4' or 'ipv6' and the others are even more obvious. OpenVPN doesn't appear to do this; it seems to leave the MTU untouched and instead query the TAP driver for what *it* wants to use as an MTU. Isn't that backwards? -- dwmw2 smime.p7s Description: S/MIME cryptographic signature
Re: [Openvpn-devel] MTU
Jan Johanssonsaid: > On Sun, 2003-02-23 at 17:10, James Yonan wrote: > > Russ, > > > > Have you tried the tracepath utility to attempt to measure the Path MTU? > > > > Are the routers in the path properly forwarding ICMP code 3 (destination > > unreachable), and ICMP code 4 (fragmentation needed but don't-fragment bit set)? > > > > Most MTU problems are caused by routers and firewalls which do not handle > > these ICMP codes correctly, causing path MTU discovery to break. > > > > The symptoms of this problem are as you show below... Running an app over > > the > > tunnel works fine initially until the first large packet is sent, then the > > app hangs. > > I have a different theory here. > I am experiencing what I believe to be MTU problems, which have made me > switch from RSA-auth to shared-key, just because the RSA auth stuff > seems to trigger it also. That's interesting, though it makes sense that if OpenVPN thinks the MTU is higher than it really is, that this would occur. That's because the SSL/TLS handshake involves some relatively large exchanges (several K octets) that are fragmented by OpenVPN to the udp-mtu packet size. In this case, if OpenVPN thinks that the udp-mtu is 1300 then it will attempt to send packets of exactly 1300 octets in size, until it reaches the last fragment which will probably be smaller. So essentially, OpenVPN's current method of fragmenting SSL/TLS handshakes guarantees that it will push the MTU parameter it's given to the limit (udp_mtu in this case). > My theory is that part of the problem lies in how the Linux kernel > handles udp. I already got it into the README-file that openbsd firewall > scrub rules will "kill" openvpn the same way it will kill Linux-nfs, but > somehow other routers along the route probably do similar things. > > One of the problems is that Linux sometimes (this is more a vague > feeling and guesswork than the next part) sends fragments in reverse, > meaning that the second half of the packet will be sent before the first > part of a split packet. This is why, for instance, you can't netboot Sun > boxes from Linux tftp-servers. The prom-code (bios sort of) in Sun > machines can't reassemble fragged packets, at least not packets that > arrive in reverse. This might make routers and firewalls along the way > to behave badly, especially with the next part in mind: > > The second part is that linux will tag all UDP (or at least all UDP from > OpenVPN and NFS) as "Dont fragment". This is sort of ok. The problem is, > they do get fragmented anyway. I can't prove that it isn't related to > the transport in between me and the peers I have set up, but I find it > likely that someone is fragmenting them anyway, perhaps even by the > sending linux box itself. Since there is a *BSD box in between in my > case, it will regard any fragmented UDP (or ip) packet with DF set as a > bug, it silently and without mercy kills them. It may also mean that if > the second half of a fragment comes first, with the DF flag on, it's > even more suspicious. Well actually what you are observing here is RFC 1191 ("Path MTU Discovery") in action (or inaction as the case may be :) http://www.faqs.org/rfcs/rfc1191.html This process causes the DF bit to always be set on outgoing packets, but assumes that other routers and firewalls in the path will play nice and return an "ICMP need fragmentation but DF set message" if the packet is too big. For linux at least, Path MTU discovery can be turned off for a given UDP socket, causing all packets to be sent with DF cleared and no attempt by the kernel to offer fragmentation or defragmentation services. I already have a feature in the CVS under branch MTU_EXPERIMENTAL which gives command line control over the Path MTU discovery settings on the UDP socket. > As far as I can guess, this probably goes for lots of ISP's and other > hardware involved in the transport of ip packets between me and my > peers. > For some reason, I seem to get the feeling that RSA negotiation doesn't > follow the udp-mtu (or tun-mtu-derived udp-mtu) settings, and send too > large packets, and them getting fragmented leads to me having to revert > back to precalced secrets. I will check this, but I doubt that SSL/TLS could go over the udp-mtu (I would expect lots of errors and/or assertion crashes in OpenVPN if this was the case). It seems more likely to me that the udp-mtu might be too large. > It also means that I get the same error as you've described in the last > mails. I have a remote-backup-over-rsync that is protected by both > OpenVPN and ssh inside of that. It goes some 256k and then dies. It dies > reliably every time, and almost always on the same byte, some 240506 > bytes into the RSYNC-stream. (whatever, the number isn't important). > If I forego the openvpn and ssh straight into the machine, the rsync > doesn't die. I am considering adding a fragmenting feature to OpenVPN where it would
Re: [Openvpn-devel] MTU
On Sun, 2003-02-23 at 17:10, James Yonan wrote: > Russ, > > Have you tried the tracepath utility to attempt to measure the Path MTU? > > Are the routers in the path properly forwarding ICMP code 3 (destination > unreachable), and ICMP code 4 (fragmentation needed but don't-fragment bit > set)? > > Most MTU problems are caused by routers and firewalls which do not handle > these ICMP codes correctly, causing path MTU discovery to break. > > The symptoms of this problem are as you show below... Running an app over the > tunnel works fine initially until the first large packet is sent, then the > app hangs. I have a different theory here. I am experiencing what I believe to be MTU problems, which have made me switch from RSA-auth to shared-key, just because the RSA auth stuff seems to trigger it also. My theory is that part of the problem lies in how the Linux kernel handles udp. I already got it into the README-file that openbsd firewall scrub rules will "kill" openvpn the same way it will kill Linux-nfs, but somehow other routers along the route probably do similar things. One of the problems is that Linux sometimes (this is more a vague feeling and guesswork than the next part) sends fragments in reverse, meaning that the second half of the packet will be sent before the first part of a split packet. This is why, for instance, you can't netboot Sun boxes from Linux tftp-servers. The prom-code (bios sort of) in Sun machines can't reassemble fragged packets, at least not packets that arrive in reverse. This might make routers and firewalls along the way to behave badly, especially with the next part in mind: The second part is that linux will tag all UDP (or at least all UDP from OpenVPN and NFS) as "Dont fragment". This is sort of ok. The problem is, they do get fragmented anyway. I can't prove that it isn't related to the transport in between me and the peers I have set up, but I find it likely that someone is fragmenting them anyway, perhaps even by the sending linux box itself. Since there is a *BSD box in between in my case, it will regard any fragmented UDP (or ip) packet with DF set as a bug, it silently and without mercy kills them. It may also mean that if the second half of a fragment comes first, with the DF flag on, it's even more suspicious. As far as I can guess, this probably goes for lots of ISP's and other hardware involved in the transport of ip packets between me and my peers. For some reason, I seem to get the feeling that RSA negotiation doesn't follow the udp-mtu (or tun-mtu-derived udp-mtu) settings, and send too large packets, and them getting fragmented leads to me having to revert back to precalced secrets. It also means that I get the same error as you've described in the last mails. I have a remote-backup-over-rsync that is protected by both OpenVPN and ssh inside of that. It goes some 256k and then dies. It dies reliably every time, and almost always on the same byte, some 240506 bytes into the RSYNC-stream. (whatever, the number isn't important). If I forego the openvpn and ssh straight into the machine, the rsync doesn't die. So there definately exist some sort of issue with MTU's, which I can't seem to solve correctly. The question is, does the SSL-stuff really check the mtu's, and can you make the linux kernel not put DF's on UDP packets? (since they are ip packets, they can be fragmented and reassembled like any other packets I guess) "man 7 udp" on RedHat says: UDP fragments a packet when its total length exceeds the interface MTU (Maximum Transmission Unit). A more network friendly alternative is to use path MTU discovery as described in the IP_PMTU_DISCOVER section of ip(7). From this text, I can't figure out why Linux puts DF on the packets, which tcpdump clearly shows it does. man 7 ip, udp, socket and friends have never shown me a way to stop DF, also none of them mention why Linux puts it there in the first place. 8-( -- Jan Johansson IT Coordinator jan.johans...@biomatsys.com - BioMat System AB Klarabergsgatan 37, III S-111 21 STOCKHOLM SWEDEN Phone: +46-(0)8-23 35 00 Fax: +46-(0)70-387 39 52 - THIS COMMUNICATION IS ONLY INTENDED FOR THE USE OF THE INDIVIDUAL, OR ENTITY, TO WHICH IT IS DIRECTED AND MAY CONTAIN INFORMATION THAT IS PRIVILIGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF RECEIVED IN ERROR: PLEASE NOTIFY US IMMEDIATELY THROUGH i...@biomatsys.com - signature.asc Description: This is a digitally signed message part
Re: [Openvpn-devel] MTU
Aaron Sethmansaid: > > On Sat, 22 Feb 2003, James Yonan wrote: > > This might be handled in a way similar to --ping-restart or SIGHUP/SIGUSR1, > > where the openvpn daemon would essentially restart if the MTU size changed. > > This would be effective if path MTU changes are infrequent but still creates > > problems when --user/--group nobody is used, since the openvpn daemon will > > lack sufficient privilege to reopen and re-ifconfig the TUN/TAP device. > > > > One option to deal with the --user/--group stuff is to keep a parent > process running as root with the real work being done by the child with > dropped privledges. The parent just waits around waiting for a signal > regarding the child, and then let the parent start a new child. Of course > if you are going to go down the IPC route, there is other fun things you > can do like passing sockets via AF_UNIX sockets. It really depends on how > you want to do it. > > Regards, > > Aaron Aaron, Actually OpenVPN has some of this right now. If built with pthread support, it will negotiate SSL/TLS keys in a background thread and communicate with the foreground via a pair of AF_UNIX sockets (this helps tunnel latency a lot). The split-privilege model is a good idea, and has been implemented with some success in OpenSSH, but is arguably overkill for OpenVPN given that up till now, OpenVPN has been able to do all of its privileged work at startup, then downgrade to nobody with little loss of flexibility. IMHO, the alternative is more complex and prone to becoming a bug-magnet. James
Re: [Openvpn-devel] MTU
Russ, Have you tried the tracepath utility to attempt to measure the Path MTU? Are the routers in the path properly forwarding ICMP code 3 (destination unreachable), and ICMP code 4 (fragmentation needed but don't-fragment bit set)? Most MTU problems are caused by routers and firewalls which do not handle these ICMP codes correctly, causing path MTU discovery to break. The symptoms of this problem are as you show below... Running an app over the tunnel works fine initially until the first large packet is sent, then the app hangs. Right now OpenVPN relies on Path MTU Discovery (a service provided by the kernel) to get the MTU right, so in the short run the best thing you can do is to try to get Path MTU working. You could also try to lower --udp-mtu by trial and error until you find something that works (essentially a manual approach to MTU discovery). If you are running tunnels over tunnels, the lower-level (more nested) tunnels will need lower --udp-mtu settings. Longer-term, I hope to put some intelligence in OpenVPN to do this automatically. James R P Herroldsaid: > On Sat, 22 Feb 2003, James Yonan wrote: > > > Recently I've been taking a new look at MTU issues. It seems that some > > people > > are having problems lately that are related to MTU or PMTU discovery. > > > > I was trying to think up some ways to make OpenVPN's handling of MTU a bit > > less opaque, more automatic, and also give more manual control to those who > > know what they're doing. > > If it is any help, I can replicate the error on demand. Coing > frm my host 'couch', through two openvpn legs, once of which > is running encryption, the other not, I get: > > [root@couch ORC]# scp ftp.victim.lan:/etc/ORC/* . > r...@ftp.victim.lan's password: > Permission denied, please try again. > r...@ftp.victim.lan's password: > ORCsetPXE 0% | | > 0 --:-- ETAKilled by signal 2. > > it is locked up and hung here. > > [root@couch ORC]# ssh ftp.victim.lan > r...@ftp.victim.lan's password: > Last login: Sat Feb 22 15:21:51 2003 from router.victim.lan > [root@ftp root]# cd /etc/ORC > [root@ftp ORC]# scp * couch.basement.lan:/etc/ORC > The authenticity of host 'couch.basement.lan (10.16.33.101)' can't be established. > RSA1 key fingerprint is 38:d4:69:9e:cb:96:81:68:d2:2b:6b:6f:b9:33:c2:39. > Are you sure you want to continue connecting (yes/no)? yes > Warning: Permanently added 'couch.basement.lan,10.16.33.101' > (RSA1) to the list of known hosts. > r...@couch.basement.lan's password: > ORCsetPXE100% |*| 3032 00:00 > ORCsetPXE~ 100% |*| 2955 00:00 > installCDrc 100% |*| 234 00:00 > installCDrc-80 100% |*| 227 00:00 > installCDrc-8094 100% |*| 234 00:00 > installCDrc-LTSP 100% |*| 188 00:00 > installCDrc-template 100% |*| 565 00:00 > [root@ftp ORC]# > [root@ftp ORC]# dmesg > Linux version 2.4.18-17.8.0 (bhcomp...@daffy.perf.redhat.com) > (gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)) #1 Tue > Oct 8 13:51:08 EDT 2002 > > it is locked up and hung here. > > > going to another host in that same remote subnet, but behind > the router I also get: > > (from 10.16.33.101 - host: couch ) > > bash-2.05b$ ssh ftp.victim.lan > herr...@ftp.victim.lan's password: > [herrold@ftp herrold]$ dmesg > Linux version 2.4.18-17.8.0 (bhcomp...@daffy.perf.redhat.com) > (gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)) #1 Tue Oct 8 13:51:08 EDT 2002 > > it is locked up and hung here. > > > lots of these on ftp.victim.lan: > > ISO 9660 Extensions: RRIP_1991A > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > ICMP: 10.16.33.101: fragmentation needed and DF set. > > but no 'noise; attributable to the tun device or openvpn in > the router: > > > eth0: Promiscuous mode enabled. > device eth0 entered promiscuous mode > Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky > divert: not allocating divert_blk for non-ethernet device tun0ip_conntrack (2047 buckets, 16376 max) > TCP: Treason uncloaked! Peer 10.1.2.30:515/692 shrinks window 2141696489:2141697513. Repaired. > TCP: Treason uncloaked! Peer 10.1.2.30:515/692 shrinks window 2141696489:2141697513. Repaired. > TCP: Treason uncloaked!
Re: [Openvpn-devel] MTU
On Sat, 22 Feb 2003, James Yonan wrote: > This might be handled in a way similar to --ping-restart or SIGHUP/SIGUSR1, > where the openvpn daemon would essentially restart if the MTU size changed. > This would be effective if path MTU changes are infrequent but still creates > problems when --user/--group nobody is used, since the openvpn daemon will > lack sufficient privilege to reopen and re-ifconfig the TUN/TAP device. > One option to deal with the --user/--group stuff is to keep a parent process running as root with the real work being done by the child with dropped privledges. The parent just waits around waiting for a signal regarding the child, and then let the parent start a new child. Of course if you are going to go down the IPC route, there is other fun things you can do like passing sockets via AF_UNIX sockets. It really depends on how you want to do it. Regards, Aaron