Re: [Openvpn-devel] openvpn-2.0_beta1: tunnel MTU a bit too large
On Sunday 16 May 2004 23:49, James Yonan wrote: > Denis, > > There are two ways of setting the MTU in OpenVPN, one is to use --tun-mtu > which doesn't include any encapsulation overhead, the other is to use > --link-mtu which sets the maximum encrypted UDP datagram size sent between > OpenVPN daemons after encapsulation. A bit misleading name. Both ifconfig and ip tools report MTU as maximum size of whole IP packet, not UDP packet encapsulated in IP packet. Do you say that --link-mtu means UDP packet size, not IP? Can it be the case that openvpn code itself use different meanings of --link-mtu in different parts of code? > 2.0 takes a different approach to earlier version of OpenVPN on this. 1.x > tries to use a lower MTU to get around fragmentation problems. But since > MTU=1500 is a kind of industry standard for ethernet networks, lowering it > can cause problems. So the 2.0 approach is to set the MTU to 1500 and then > use --mssfix 1450 by default. It's not a proper fix. IP traffic shall work through path of any valid MTU. Artificially keeping tunnel MTU=1500 is a hack. Think about multi-gigabit nets of the future: do you still want MTU=1500 everywhere? Think about lossy links with small 'real' MTU: with tunnel MTU=1500 UDP/ICMP/other being packets fragmented and some fragments lost, you get horrible performance of e.g. NFS over UDP. > This will keep TCP connections over the > tunnel from using the full size 1500 MTU by advertising lower MSS values to > TCP senders. Instead TCP senders will use a segment size that will result > in total encrypted, encapsulated OpenVPN packet sizes of 1450 or less. And > then when you add on the IP header and UDP header, you are still under > 1500. Yes. There is similar thing in iptables (--TCP-MSS). Also, PMTU discovery will do this in properly configured network. > So if you're getting fragmentation on an HTTP download, I would try > reducing --mssfix to something below 1450 and see if that that fixes the > problem. I don't get fragmentation on HTTP. I get fragmentation of carrier packets whan I send large UDP over tunnel link. Anyway, regarding --mssfix: I dont want to work around the problem. I want to set correct tunnel MTU. An MTU such that: 1) large UDP packets (say, 7000 bytes) sent over tunnel link are cut into fragments of such size that, when encapsulated and sent over carrier link, do not exceed MTU of carrier path (in my case, 1500). Ideally, they should be equal to it. 2) TCP streams should use MSS which, again, results in sending full-sized packets over carrier link. Currently, openvpn, when configured with --link-mtu 1500, sets --tun-mtu which is too large and (1) does not hold. "UDP doublefrag" bug. TCP works according to (2). If I force smaller --tun-mtu 1434, UDP doublefrag does not happen. But now TCP does not use full-sized packets. Look at the sizes of IP packets carrying UDP carrier packets (sniffed over carrier link). UDP is almost 1500, TCP is _not_: > > --tun-mtu 1434: > > > > UDP flood (dd if=/dev/zero bs=1M | nc -nuvvv -w1 1.1.5.6 34564): > > ... > > 20:44:23.376801 IP (tos 0x0, ttl 64, id 5657, offset 0, flags [DF], > > length: 1496) > > TCP flood (e.g. HTTP download): > > ... > > 20:41:55.514646 IP (tos 0x0, ttl 64, id 5293, offset 0, flags [DF], > > length: 1472) Hope this clarifies. -- vda
Re: [Openvpn-devel] openvpn-2.0_beta1: tunnel MTU a bit too large
Denis Vlasenkosaid: > On Sunday 16 May 2004 23:49, James Yonan wrote: > > Denis, > > > > There are two ways of setting the MTU in OpenVPN, one is to use --tun-mtu > > which doesn't include any encapsulation overhead, the other is to use > > --link-mtu which sets the maximum encrypted UDP datagram size sent between > > OpenVPN daemons after encapsulation. > > A bit misleading name. Both ifconfig and ip tools report MTU > as maximum size of whole IP packet, not UDP packet encapsulated > in IP packet. Do you say that --link-mtu means UDP packet size, > not IP? --link-mtu doesn't include the IP header size or ethernet header size of the carrier protocol. > Can it be the case that openvpn code itself use different > meanings of --link-mtu in different parts of code? No, link MTU should always mean --tun-mtu + encapsulation overhead. > > 2.0 takes a different approach to earlier version of OpenVPN on this. 1.x > > tries to use a lower MTU to get around fragmentation problems. But since > > MTU=1500 is a kind of industry standard for ethernet networks, lowering it > > can cause problems. So the 2.0 approach is to set the MTU to 1500 and then > > use --mssfix 1450 by default. > > It's not a proper fix. IP traffic shall work through path of any valid MTU. > Artificially keeping tunnel MTU=1500 is a hack. > Think about multi-gigabit nets of the future: do you still want MTU=1500 > everywhere? No, the MTU can be set to whatever you want. There is no limitation. The MTU=1500, MSSFIX=1450 is just a default that works well across a large range of OSes and routers. > Think about lossy links with small 'real' MTU: with tunnel MTU=1500 > UDP/ICMP/other being packets fragmented and some fragments lost, you > get horrible performance of e.g. NFS over UDP. There are a lot of ways you can deal with fragmentation. OpenVPN gives you --fragment, --mssfix, --tun-mtu, and --link-mtu. For UDP application protocols, like NFS, you can also tell the app directly what kind of packet size to use to avoid fragmentation. This is the best approach when using NFS. I don't have any trouble running NFS over OpenVPN. > > This will keep TCP connections over the > > tunnel from using the full size 1500 MTU by advertising lower MSS values to > > TCP senders. Instead TCP senders will use a segment size that will result > > in total encrypted, encapsulated OpenVPN packet sizes of 1450 or less. And > > then when you add on the IP header and UDP header, you are still under > > 1500. > > Yes. There is similar thing in iptables (--TCP-MSS). > Also, PMTU discovery will do this in properly configured network. No, PMTU discovery doesn't work very well on the real internet, because of the way that firewalls tend to be configured. If you don't believe me, google for "PMTU black holes". > > So if you're getting fragmentation on an HTTP download, I would try > > reducing --mssfix to something below 1450 and see if that that fixes the > > problem. > > I don't get fragmentation on HTTP. I get fragmentation of carrier packets > whan I send large UDP over tunnel link. Right, the purpose of --mssfix is to avoid the fragmentation that occurs at the carrier level. > Anyway, regarding --mssfix: I dont want to work around the problem. > I want to set correct tunnel MTU. An MTU such that: > 1) large UDP packets (say, 7000 bytes) sent over tunnel link are cut into > fragments of such size that, when encapsulated and sent over > carrier link, do not exceed MTU of carrier path (in my case, 1500). > Ideally, they should be equal to it. > 2) TCP streams should use MSS which, again, results in sending full-sized > packets over carrier link. This is possible with correct settings of --tun-mtu/--link-mtu, --fragment, and --mssfix. James > Currently, openvpn, when configured with --link-mtu 1500, sets --tun-mtu > which is too large and (1) does not hold. "UDP doublefrag" bug. > TCP works according to (2). > > If I force smaller --tun-mtu 1434, UDP doublefrag does not happen. > But now TCP does not use full-sized packets. > > Look at the sizes of IP packets carrying UDP carrier packets > (sniffed over carrier link). UDP is almost 1500, TCP is _not_: > > > > --tun-mtu 1434: > > > > > > UDP flood (dd if=/dev/zero bs=1M | nc -nuvvv -w1 1.1.5.6 34564): > > > ... > > > 20:44:23.376801 IP (tos 0x0, ttl 64, id 5657, offset 0, flags [DF], length: 1496) > > > > TCP flood (e.g. HTTP download): > > > ... > > > 20:41:55.514646 IP (tos 0x0, ttl 64, id 5293, offset 0, flags [DF], length: 1472) > > > Hope this clarifies. > -- > vda > --
Re: [Openvpn-devel] openvpn-2.0_beta1: tunnel MTU a bit too large
> Or maybe I'm mistaken and I shall set --link-mtu not to the value > reported by 'ip a l dev eth0' (i.e. max IP packet size), but > to the max *UDP* packet size? 1500-28=1472, then. Not every user > knows IP overhead size. I don't. I looked at tcpdump to figure out. "Double frag" bug does not happen for UDP if I use --tun-mtu 1435 or lower. With openvpn \ --secret "$PWD/key" \ --dev tun \ --proto udp \ --port 8002 \ --local 1.1.4.1 \ --remote 1.1.4.2 \ --ifconfig 1.1.5.1 1.1.5.2 \ --tun-mtu 1434 \ --ping 30 \ --ping-exit 66 \ --verb 3 \ --mute 20: UDP flood (dd if=/dev/zero bs=1M | nc -nuvvv -w1 1.1.5.6 34564): ... 20:44:23.376801 IP (tos 0x0, ttl 64, id 5657, offset 0, flags [DF], length: 1496) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1468 20:44:23.376907 IP (tos 0x0, ttl 64, id 5658, offset 0, flags [DF], length: 1248) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1220 20:44:23.378234 IP (tos 0x0, ttl 64, id 5659, offset 0, flags [DF], length: 1496) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1468 20:44:23.378380 IP (tos 0x0, ttl 64, id 5660, offset 0, flags [DF], length: 1496) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1468 20:44:23.378520 IP (tos 0x0, ttl 64, id 5661, offset 0, flags [DF], length: 1496) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1468 20:44:23.378659 IP (tos 0x0, ttl 64, id 5662, offset 0, flags [DF], length: 1496) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1468 20:44:23.378798 IP (tos 0x0, ttl 64, id 5663, offset 0, flags [DF], length: 1496) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1468 20:44:23.37 IP (tos 0x0, ttl 64, id 5664, offset 0, flags [DF], length: 1248) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1220 ... Ok. TCP flood (e.g. HTTP download): ... 20:41:55.514646 IP (tos 0x0, ttl 64, id 5293, offset 0, flags [DF], length: 1472) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1444 20:41:55.514786 IP (tos 0x0, ttl 64, id 5294, offset 0, flags [DF], length: 1472) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1444 20:41:55.514923 IP (tos 0x0, ttl 64, id 5295, offset 0, flags [DF], length: 1472) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1444 20:41:55.515049 IP (tos 0x0, ttl 64, id 5296, offset 0, flags [DF], length: 1472) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1444 20:41:55.515207 IP (tos 0x0, ttl 64, id 5297, offset 0, flags [DF], length: 1472) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1444 20:41:55.515341 IP (tos 0x0, ttl 64, id 5298, offset 0, flags [DF], length: 1472) 1.1.4.1.8006 > 1.1.4.6.8006: [udp sum ok] UDP, length: 1444 ... Something is wrong here. #ip a l eth0 tun0 2: eth0:mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:fc:b0:e1:17 brd ff:ff:ff:ff:ff:ff inet 1.1.4.1/24 brd 1.1.4.255 scope global ifi 49: tun0: mtu 1434 qdisc pfifo_fast qlen 100 link/ppp inet 1.1.5.5 peer 1.1.5.6/32 scope global tun0 openvpn log: Sun May 16 20:55:53 2004 OpenVPN 2.0_beta1 i386-pc-linux-gnu [SSL] [LZO] built on May 12 2004 Sun May 16 20:55:53 2004 Static Encrypt: Cipher 'BF-CBC' initialized with 128 bit key Sun May 16 20:55:53 2004 Static Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Sun May 16 20:55:53 2004 Static Decrypt: Cipher 'BF-CBC' initialized with 128 bit key Sun May 16 20:55:53 2004 Static Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication Sun May 16 20:55:53 2004 WARNING: normally if you use --mssfix and/or --fragment, you should also set --tun-mtu 1500 (currently it is 1434) Sun May 16 20:55:53 2004 TUN/TAP device tun1 opened Sun May 16 20:55:53 2004 /bin/ifconfig tun1 1.1.5.1 pointopoint 1.1.5.2 mtu 1434 Sun May 16 20:55:53 2004 Data Channel MTU parms [ L:1478 D:1450 EF:44 EB:0 ET:0 EL:0 ] Sun May 16 20:55:53 2004 Local Options hash (VER=V3): '5194cd41' Sun May 16 20:55:53 2004 Expected Remote Options hash (VER=V3): 'e341fa03' Sun May 16 20:55:53 2004 UDPv4 link local (bound): 1.1.4.1:8002 Sun May 16 20:55:53 2004 UDPv4 link remote: 1.1.4.2:8002 Sun May 16 20:55:58 2004 Peer Connection Initiated with 1.1.4.2:8002 -- vda