Re: Unusual TCP/IP Packet Size
On Thu, Feb 14, 2013 at 10:37:23AM +0900, YongHyeon PYUN wrote: > On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: > > On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: > > > 13.02.2013 17:25, Doug Hardie ??: > > > > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > > > > following interface: > > > > > > > > msk0: flags=8843 metric 0 mtu > > > > 1500 > > > > > > > > options=c011b > > > > ether 00:11:2f:2a:c7:03 > > > > inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > > > > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > > > > nd6 options=29 > > > > media: Ethernet autoselect (100baseTX > > > > ) > > > > status: active > > > > > > > > > > > > It sent the following packet: (data content abbreviated) > > > > > > > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq > > > > 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr > > > > 920110183], length 3946 > > > > 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 > > > > E.@.@.*. > > > > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc > > > > ...J..h..7.. > > > > 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 > > > > 4...??. > > > > > > > > > > > > The indicated packet length is 3946 and the load of data shown is that > > > > size. The MTU on both interfaces is 1500. The receiving system > > > > received 3 packets. There is a router and switch between them. One of > > > > them fragmented that packet. This is part of a SSL/TLS exchange and one > > > > side or the other is hanging on this and just dropping the connection. > > > > I suspect the packet size is the issue. ssldump complains about the > > > > packet too and stops monitoring. Could this possibly be related to the > > > > hardware checksums? > > > > > > You have TSO enabled on the interface, so large outgoing TCP packet is > > > pretty normal. > > > It will be split by the NIC. Disable TSO with ifconfig if it interferes > > > with your ssldump. > > > > This is not the behaviour I see with em(4) on a 82573E with all defaults > > used (which includes TSO4). Note that Doug is using msk(4). > > > > I can provide packet captures on both ends of a LAN segment using both > > tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that > > show a difference in behaviour compared to what Doug sees. > > This is strange. tcpdump sees a (big) TCP segment right before > controller actually transmits it. So if TSO is active for the TCP > segment, you should see a series of small TCP packets on receiver > side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP > segment with tcpdump on TX path, probably TSO was not used for the > TCP segment. > It's possible for controller to corrupt the TCP segment during > segmentation but Doug's tcpdump looks completely normal to me since > tcpdump sees the segment before TCP segmentation. Let me explain what I'm referring to. In the below tcpdump on the server, you'll find 5 "bad-len 0" messages. These correlate directly with TCP payloads that exceed MTU -- this is verified by comparing to the number of lines labelled "TCP segment of reassembled PDU" **that exceed MTU (>1500)** in Wireshark (when analysing the same server capture). Thus, the "bad-len 0" messages in tcpdump are indicators of TSO being used. Doug's capture with msk(4) + TSO shows a TCP length that exceeds MTU, yet my capture with em(4) + TSO shows "bad-len 0". Wireshark seems to decode server capture correctly, so I'm inclined to think this is a libpcap/tcpdump quirk/bug of sorts. I just find it very strange that NIC difference manifests itself like this (and in some regards, concerns me). I'm happy to provide the pcap files from both server and client if someone wants to do the analysis, as well as a "verbose" output from Wireshark (vs. just the summary lines) if all packet fields are desired. Server = 192.168.1.51 (FreeBSD, Intel 82573E, em(4), with TSO) Client = 192.168.1.50 (Windows XP SP3) Server capture (tcpdump -p -i em0 -n -s 0 -w server.pcap): 12:14:54.512542 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [S], seq 375412185, win 65535, options [mss 1460,nop,wscale 3,nop,nop,sackOK], length 0 12:14:54.512576 IP 192.168.1.51.80 > 192.168.1.50.1300: Flags [S.], seq 339580135, ack 375412186, win 65535, options [mss 1460,nop,wscale 6,sackOK,eol], length 0 12:14:54.512659 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 1, win 64240, length 0 12:14:54.512784 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [P.], seq 1:330, ack 1, win 64240, length 329 12:14:54.515194 IP 192.168.1.51.80 > 192.168.1.50.1300: Flags [P.], seq 1:279, ack 330, win 1026, length 278 12:14:54.515230 IP bad-len 0 12:14:54.515410 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 1739, win 64240, length 0 12:14:54.515423 IP bad-len 0 12:1
Re: Unusual TCP/IP Packet Size
On Wed, Feb 13, 2013 at 11:54:24PM -0800, Doug Hardie wrote: > > On 13 February 2013, at 22:45, YongHyeon PYUN wrote: > > > On Wed, Feb 13, 2013 at 09:10:36PM -0800, Kevin Oberman wrote: > >> On Wed, Feb 13, 2013 at 5:37 PM, YongHyeon PYUN wrote: > >>> On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: > On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: > > 13.02.2013 17:25, Doug Hardie ??: > >> Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > >> following interface: > >> > >> msk0: flags=8843 metric 0 mtu > >> 1500 > >> > >> options=c011b > >> ether 00:11:2f:2a:c7:03 > >> inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > >> inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > >> nd6 options=29 > >> media: Ethernet autoselect (100baseTX > >> ) > >> status: active > >> > >> > >> It sent the following packet: (data content abbreviated) > >> > >> 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq > >> 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr > >> 920110183], length 3946 > >> 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. > >> 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. > >> 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. > >> > >> > >> The indicated packet length is 3946 and the load of data shown is that > >> size. The MTU on both interfaces is 1500. The receiving system > >> received 3 packets. There is a router and switch between them. One > >> of them fragmented that packet. This is part of a SSL/TLS exchange and > >> one side or the other is hanging on this and just dropping the > >> connection. I suspect the packet size is the issue. ssldump complains > >> about the packet too and stops monitoring. Could this possibly be > >> related to the hardware checksums? > > > > You have TSO enabled on the interface, so large outgoing TCP packet is > > pretty normal. > > It will be split by the NIC. Disable TSO with ifconfig if it interferes > > with your ssldump. > > This is not the behaviour I see with em(4) on a 82573E with all defaults > used (which includes TSO4). Note that Doug is using msk(4). > > I can provide packet captures on both ends of a LAN segment using both > tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that > show a difference in behaviour compared to what Doug sees. > >>> > >>> This is strange. tcpdump sees a (big) TCP segment right before > >>> controller actually transmits it. So if TSO is active for the TCP > >>> segment, you should see a series of small TCP packets on receiver > >>> side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP > >>> segment with tcpdump on TX path, probably TSO was not used for the > >>> TCP segment. > >>> It's possible for controller to corrupt the TCP segment during > >>> segmentation but Doug's tcpdump looks completely normal to me since > >>> tcpdump sees the segment before TCP segmentation. > >>> > > What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" > messages for payloads which are chunked or segmented as a result of TSO. > I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I > only see one "bad-len" entry for all chunks (up until the next ACK or > PSH+ACK of course). > > >>> > >>> I vaguely recall that some users reported similar TSO issues on > >>> various drivers. The root cause of the issue was not identified > >>> though. Personally I couldn't reproduce the issue at that time. > >>> It could be a driver or network stack bug. > >> > >> Beware TSO. It can significantly improve throughput on high speed > >> networks, but it really has issues. > >> > >> TSO segments the data and transmits all of them back-to-back with no > >> delay beyond IFG (the 802.3 mandated space between frames) TSO does > >> not understand congestion control. If there is congestion and TSO > >> sends several frames in a row, it is entirely possible that a queue is > >> full or getting close enough to full to start dropping packets and > >> these segmented frames are excellent candidates. > > > > I'm not saying the drawback of TSO. Sometimes segmented packets > > have malformed IP header length under certain circumstances such > > that these packets were dropped on receiver side. > > How do I configure the msk0 interface in rc.conf to disable tso4? I can > easily do it with ifconfig, but don't see how to make sure its disabled after > a boot. Adjust your ifconfig_msk0 line in rc.conf to contain "-tso", i.e.: ifconfig_msk0="inet 10.0.1.199 netmask 255.255.255.0 -tso" -- | Jeremy Chadwick j...@koitsu.org | | UNIX Sy
Re: Unusual TCP/IP Packet Size
14.02.2013 14:54, Doug Hardie пишет: > > How do I configure the msk0 interface in rc.conf to disable tso4? I can > easily do it with ifconfig, but don't see how to make sure its disabled after > a boot. Just add corresponding flag to ifconfig_msk0 line in rc.conf: ifconfig_msk0="inet 10.0.1.199 netmask 0xff00 -tso -vlanhwtso" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On 13 February 2013, at 22:45, YongHyeon PYUN wrote: > On Wed, Feb 13, 2013 at 09:10:36PM -0800, Kevin Oberman wrote: >> On Wed, Feb 13, 2013 at 5:37 PM, YongHyeon PYUN wrote: >>> On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: > 13.02.2013 17:25, Doug Hardie ??: >> Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the >> following interface: >> >> msk0: flags=8843 metric 0 mtu >> 1500 >> >> options=c011b >> ether 00:11:2f:2a:c7:03 >> inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 >> inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 >> nd6 options=29 >> media: Ethernet autoselect (100baseTX >> ) >> status: active >> >> >> It sent the following packet: (data content abbreviated) >> >> 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq >> 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr >> 920110183], length 3946 >> 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. >> 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. >> 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. >> >> >> The indicated packet length is 3946 and the load of data shown is that >> size. The MTU on both interfaces is 1500. The receiving system >> received 3 packets. There is a router and switch between them. One of >> them fragmented that packet. This is part of a SSL/TLS exchange and one >> side or the other is hanging on this and just dropping the connection. >> I suspect the packet size is the issue. ssldump complains about the >> packet too and stops monitoring. Could this possibly be related to the >> hardware checksums? > > You have TSO enabled on the interface, so large outgoing TCP packet is > pretty normal. > It will be split by the NIC. Disable TSO with ifconfig if it interferes > with your ssldump. This is not the behaviour I see with em(4) on a 82573E with all defaults used (which includes TSO4). Note that Doug is using msk(4). I can provide packet captures on both ends of a LAN segment using both tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that show a difference in behaviour compared to what Doug sees. >>> >>> This is strange. tcpdump sees a (big) TCP segment right before >>> controller actually transmits it. So if TSO is active for the TCP >>> segment, you should see a series of small TCP packets on receiver >>> side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP >>> segment with tcpdump on TX path, probably TSO was not used for the >>> TCP segment. >>> It's possible for controller to corrupt the TCP segment during >>> segmentation but Doug's tcpdump looks completely normal to me since >>> tcpdump sees the segment before TCP segmentation. >>> What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" messages for payloads which are chunked or segmented as a result of TSO. I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I only see one "bad-len" entry for all chunks (up until the next ACK or PSH+ACK of course). >>> >>> I vaguely recall that some users reported similar TSO issues on >>> various drivers. The root cause of the issue was not identified >>> though. Personally I couldn't reproduce the issue at that time. >>> It could be a driver or network stack bug. >> >> Beware TSO. It can significantly improve throughput on high speed >> networks, but it really has issues. >> >> TSO segments the data and transmits all of them back-to-back with no >> delay beyond IFG (the 802.3 mandated space between frames) TSO does >> not understand congestion control. If there is congestion and TSO >> sends several frames in a row, it is entirely possible that a queue is >> full or getting close enough to full to start dropping packets and >> these segmented frames are excellent candidates. > > I'm not saying the drawback of TSO. Sometimes segmented packets > have malformed IP header length under certain circumstances such > that these packets were dropped on receiver side. How do I configure the msk0 interface in rc.conf to disable tso4? I can easily do it with ifconfig, but don't see how to make sure its disabled after a boot. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On Wed, Feb 13, 2013 at 09:10:36PM -0800, Kevin Oberman wrote: > On Wed, Feb 13, 2013 at 5:37 PM, YongHyeon PYUN wrote: > > On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: > >> On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: > >> > 13.02.2013 17:25, Doug Hardie ??: > >> > > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > >> > > following interface: > >> > > > >> > > msk0: flags=8843 metric 0 mtu > >> > > 1500 > >> > > > >> > > options=c011b > >> > > ether 00:11:2f:2a:c7:03 > >> > > inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > >> > > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > >> > > nd6 options=29 > >> > > media: Ethernet autoselect (100baseTX > >> > > ) > >> > > status: active > >> > > > >> > > > >> > > It sent the following packet: (data content abbreviated) > >> > > > >> > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq > >> > > 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr > >> > > 920110183], length 3946 > >> > > 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. > >> > > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. > >> > > 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. > >> > > > >> > > > >> > > The indicated packet length is 3946 and the load of data shown is that > >> > > size. The MTU on both interfaces is 1500. The receiving system > >> > > received 3 packets. There is a router and switch between them. One > >> > > of them fragmented that packet. This is part of a SSL/TLS exchange and > >> > > one side or the other is hanging on this and just dropping the > >> > > connection. I suspect the packet size is the issue. ssldump complains > >> > > about the packet too and stops monitoring. Could this possibly be > >> > > related to the hardware checksums? > >> > > >> > You have TSO enabled on the interface, so large outgoing TCP packet is > >> > pretty normal. > >> > It will be split by the NIC. Disable TSO with ifconfig if it interferes > >> > with your ssldump. > >> > >> This is not the behaviour I see with em(4) on a 82573E with all defaults > >> used (which includes TSO4). Note that Doug is using msk(4). > >> > >> I can provide packet captures on both ends of a LAN segment using both > >> tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that > >> show a difference in behaviour compared to what Doug sees. > > > > This is strange. tcpdump sees a (big) TCP segment right before > > controller actually transmits it. So if TSO is active for the TCP > > segment, you should see a series of small TCP packets on receiver > > side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP > > segment with tcpdump on TX path, probably TSO was not used for the > > TCP segment. > > It's possible for controller to corrupt the TCP segment during > > segmentation but Doug's tcpdump looks completely normal to me since > > tcpdump sees the segment before TCP segmentation. > > > >> > >> What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" > >> messages for payloads which are chunked or segmented as a result of TSO. > >> I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I > >> only see one "bad-len" entry for all chunks (up until the next ACK or > >> PSH+ACK of course). > >> > > > > I vaguely recall that some users reported similar TSO issues on > > various drivers. The root cause of the issue was not identified > > though. Personally I couldn't reproduce the issue at that time. > > It could be a driver or network stack bug. > > Beware TSO. It can significantly improve throughput on high speed > networks, but it really has issues. > > TSO segments the data and transmits all of them back-to-back with no > delay beyond IFG (the 802.3 mandated space between frames) TSO does > not understand congestion control. If there is congestion and TSO > sends several frames in a row, it is entirely possible that a queue is > full or getting close enough to full to start dropping packets and > these segmented frames are excellent candidates. I'm not saying the drawback of TSO. Sometimes segmented packets have malformed IP header length under certain circumstances such that these packets were dropped on receiver side. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On Wed, Feb 13, 2013 at 5:37 PM, YongHyeon PYUN wrote: > On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: >> On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: >> > 13.02.2013 17:25, Doug Hardie ??: >> > > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the >> > > following interface: >> > > >> > > msk0: flags=8843 metric 0 mtu >> > > 1500 >> > > >> > > options=c011b >> > > ether 00:11:2f:2a:c7:03 >> > > inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 >> > > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 >> > > nd6 options=29 >> > > media: Ethernet autoselect (100baseTX >> > > ) >> > > status: active >> > > >> > > >> > > It sent the following packet: (data content abbreviated) >> > > >> > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq >> > > 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr >> > > 920110183], length 3946 >> > > 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. >> > > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. >> > > 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. >> > > >> > > >> > > The indicated packet length is 3946 and the load of data shown is that >> > > size. The MTU on both interfaces is 1500. The receiving system >> > > received 3 packets. There is a router and switch between them. One of >> > > them fragmented that packet. This is part of a SSL/TLS exchange and one >> > > side or the other is hanging on this and just dropping the connection. >> > > I suspect the packet size is the issue. ssldump complains about the >> > > packet too and stops monitoring. Could this possibly be related to the >> > > hardware checksums? >> > >> > You have TSO enabled on the interface, so large outgoing TCP packet is >> > pretty normal. >> > It will be split by the NIC. Disable TSO with ifconfig if it interferes >> > with your ssldump. >> >> This is not the behaviour I see with em(4) on a 82573E with all defaults >> used (which includes TSO4). Note that Doug is using msk(4). >> >> I can provide packet captures on both ends of a LAN segment using both >> tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that >> show a difference in behaviour compared to what Doug sees. > > This is strange. tcpdump sees a (big) TCP segment right before > controller actually transmits it. So if TSO is active for the TCP > segment, you should see a series of small TCP packets on receiver > side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP > segment with tcpdump on TX path, probably TSO was not used for the > TCP segment. > It's possible for controller to corrupt the TCP segment during > segmentation but Doug's tcpdump looks completely normal to me since > tcpdump sees the segment before TCP segmentation. > >> >> What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" >> messages for payloads which are chunked or segmented as a result of TSO. >> I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I >> only see one "bad-len" entry for all chunks (up until the next ACK or >> PSH+ACK of course). >> > > I vaguely recall that some users reported similar TSO issues on > various drivers. The root cause of the issue was not identified > though. Personally I couldn't reproduce the issue at that time. > It could be a driver or network stack bug. Beware TSO. It can significantly improve throughput on high speed networks, but it really has issues. TSO segments the data and transmits all of them back-to-back with no delay beyond IFG (the 802.3 mandated space between frames) TSO does not understand congestion control. If there is congestion and TSO sends several frames in a row, it is entirely possible that a queue is full or getting close enough to full to start dropping packets and these segmented frames are excellent candidates. -- R. Kevin Oberman, Network Engineer E-mail: kob6...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: > On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: > > 13.02.2013 17:25, Doug Hardie ??: > > > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > > > following interface: > > > > > > msk0: flags=8843 metric 0 mtu 1500 > > > > > > options=c011b > > > ether 00:11:2f:2a:c7:03 > > > inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > > > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > > > nd6 options=29 > > > media: Ethernet autoselect (100baseTX > > > ) > > > status: active > > > > > > > > > It sent the following packet: (data content abbreviated) > > > > > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq > > > 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr > > > 920110183], length 3946 > > > 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. > > > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. > > > 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. > > > > > > > > > The indicated packet length is 3946 and the load of data shown is that > > > size. The MTU on both interfaces is 1500. The receiving system received > > > 3 packets. There is a router and switch between them. One of them > > > fragmented that packet. This is part of a SSL/TLS exchange and one side > > > or the other is hanging on this and just dropping the connection. I > > > suspect the packet size is the issue. ssldump complains about the packet > > > too and stops monitoring. Could this possibly be related to the hardware > > > checksums? > > > > You have TSO enabled on the interface, so large outgoing TCP packet is > > pretty normal. > > It will be split by the NIC. Disable TSO with ifconfig if it interferes > > with your ssldump. > > This is not the behaviour I see with em(4) on a 82573E with all defaults > used (which includes TSO4). Note that Doug is using msk(4). > > I can provide packet captures on both ends of a LAN segment using both > tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that > show a difference in behaviour compared to what Doug sees. This is strange. tcpdump sees a (big) TCP segment right before controller actually transmits it. So if TSO is active for the TCP segment, you should see a series of small TCP packets on receiver side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP segment with tcpdump on TX path, probably TSO was not used for the TCP segment. It's possible for controller to corrupt the TCP segment during segmentation but Doug's tcpdump looks completely normal to me since tcpdump sees the segment before TCP segmentation. > > What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" > messages for payloads which are chunked or segmented as a result of TSO. > I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I > only see one "bad-len" entry for all chunks (up until the next ACK or > PSH+ACK of course). > I vaguely recall that some users reported similar TSO issues on various drivers. The root cause of the issue was not identified though. Personally I couldn't reproduce the issue at that time. It could be a driver or network stack bug. > The important part: I do not see captured TCP packets reporting a length > greater than MTU (or MSS for that matter (remember: IP header + TCP > header + MSS <= MTU). > > Also note for Doug: remember that if you're doing packet captures > between two devices that have NAT involved, you may see different > behaviour. Example case: > > 03:58:47.907582 IP 67.180.84.87.2983 > 206.125.172.42.80: Flags [.], ack > 13419, win 64240, length 0 > 03:58:47.907649 IP 206.125.172.42.80 > 67.180.84.87.2983: Flags [.], seq > 17799:19259, ack 292, win 1026, length 1460 > 03:58:47.907679 IP 206.125.172.42.80 > 67.180.84.87.2983: Flags [.], seq > 19259:20719, ack 292, win 1026, length 1460 > 03:58:47.912546 IP 67.180.84.87.2983 > 206.125.172.42.80: Flags [.], ack > 16339, win 64240, length 0 > > In the above example there's a Linux NAT router (67.180.84.87) with a > client (192.168.1.50) behind it, talking to 206.125.172.42. MTU is 1500 > (I obviously didn't include the initial SYN :-) ). > > -- > | Jeremy Chadwick j...@koitsu.org | > | UNIX Systems Administratorhttp://jdc.koitsu.org/ | > | Mountain View, CA, US| > | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On Wed, Feb 13, 2013 at 02:01:32PM -0800, Jeremy Chadwick wrote: > On Wed, Feb 13, 2013 at 01:57:38PM -0800, Doug Hardie wrote: > > > > On 13 February 2013, at 02:29, Eugene Grosbein wrote: > > > > > 13.02.2013 17:25, Doug Hardie ??: > > >> Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > > >> following interface: > > >> > > >> msk0: flags=8843 metric 0 mtu > > >> 1500 > > >> > > >> options=c011b > > >> ether 00:11:2f:2a:c7:03 > > >> inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > > >> inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > > >> nd6 options=29 > > >> media: Ethernet autoselect (100baseTX > > >> ) > > >> status: active > > >> > > >> > > >> It sent the following packet: (data content abbreviated) > > >> > > >> 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq > > >> 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr > > >> 920110183], length 3946 > > >> 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. > > >> 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. > > >> 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. > > >> > > >> > > >> The indicated packet length is 3946 and the load of data shown is that > > >> size. The MTU on both interfaces is 1500. The receiving system > > >> received 3 packets. There is a router and switch between them. One of > > >> them fragmented that packet. This is part of a SSL/TLS exchange and one > > >> side or the other is hanging on this and just dropping the connection. > > >> I suspect the packet size is the issue. ssldump complains about the > > >> packet too and stops monitoring. Could this possibly be related to the > > >> hardware checksums? > > > > > > You have TSO enabled on the interface, so large outgoing TCP packet is > > > pretty normal. > > > It will be split by the NIC. Disable TSO with ifconfig if it interferes > > > with your ssldump. > > > > Thanks. Now all the packets are 1500 or under. They all are received with > > a SSL header. > > If disabling TSO on msk(4) fixed the issue of the remote end > dropping/ignoring the packet, that sounds like a bug in msk(4). > > Yong-Hyeon, do you have any recent msk(4) patches relating to TSO? No, I'm not aware of msk(4) related TSO issues. For some controllers, msk(4) used to touch reserved registers which in turn resulted in unexpected results. This was fixed long time ago but it would be good idea to cold-boot the box and see whether that makes any difference. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On Wed, Feb 13, 2013 at 01:57:38PM -0800, Doug Hardie wrote: > > On 13 February 2013, at 02:29, Eugene Grosbein wrote: > > > 13.02.2013 17:25, Doug Hardie ??: > >> Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > >> following interface: > >> > >> msk0: flags=8843 metric 0 mtu 1500 > >> > >> options=c011b > >>ether 00:11:2f:2a:c7:03 > >>inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > >>inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > >>nd6 options=29 > >>media: Ethernet autoselect (100baseTX > >> ) > >>status: active > >> > >> > >> It sent the following packet: (data content abbreviated) > >> > >> 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq > >> 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr > >> 920110183], length 3946 > >>0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. > >>0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. > >>0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. > >> > >> > >> The indicated packet length is 3946 and the load of data shown is that > >> size. The MTU on both interfaces is 1500. The receiving system received > >> 3 packets. There is a router and switch between them. One of them > >> fragmented that packet. This is part of a SSL/TLS exchange and one side or > >> the other is hanging on this and just dropping the connection. I suspect > >> the packet size is the issue. ssldump complains about the packet too and > >> stops monitoring. Could this possibly be related to the hardware > >> checksums? > > > > You have TSO enabled on the interface, so large outgoing TCP packet is > > pretty normal. > > It will be split by the NIC. Disable TSO with ifconfig if it interferes > > with your ssldump. > > Thanks. Now all the packets are 1500 or under. They all are received with a > SSL header. If disabling TSO on msk(4) fixed the issue of the remote end dropping/ignoring the packet, that sounds like a bug in msk(4). Yong-Hyeon, do you have any recent msk(4) patches relating to TSO? -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Mountain View, CA, US| | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On 13 February 2013, at 02:29, Eugene Grosbein wrote: > 13.02.2013 17:25, Doug Hardie пишет: >> Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the >> following interface: >> >> msk0: flags=8843 metric 0 mtu 1500 >> >> options=c011b >> ether 00:11:2f:2a:c7:03 >> inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 >> inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 >> nd6 options=29 >> media: Ethernet autoselect (100baseTX >> ) >> status: active >> >> >> It sent the following packet: (data content abbreviated) >> >> 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq >> 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr >> 920110183], length 3946 >> 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. >> 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. >> 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...……. >> >> >> The indicated packet length is 3946 and the load of data shown is that size. >> The MTU on both interfaces is 1500. The receiving system received 3 >> packets. There is a router and switch between them. One of them fragmented >> that packet. This is part of a SSL/TLS exchange and one side or the other is >> hanging on this and just dropping the connection. I suspect the packet size >> is the issue. ssldump complains about the packet too and stops monitoring. >> Could this possibly be related to the hardware checksums? > > You have TSO enabled on the interface, so large outgoing TCP packet is pretty > normal. > It will be split by the NIC. Disable TSO with ifconfig if it interferes with > your ssldump. Thanks. Now all the packets are 1500 or under. They all are received with a SSL header. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: > 13.02.2013 17:25, Doug Hardie ??: > > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > > following interface: > > > > msk0: flags=8843 metric 0 mtu 1500 > > > > options=c011b > > ether 00:11:2f:2a:c7:03 > > inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > > nd6 options=29 > > media: Ethernet autoselect (100baseTX > > ) > > status: active > > > > > > It sent the following packet: (data content abbreviated) > > > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq > > 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr > > 920110183], length 3946 > > 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. > > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. > > 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...??. > > > > > > The indicated packet length is 3946 and the load of data shown is that > > size. The MTU on both interfaces is 1500. The receiving system received 3 > > packets. There is a router and switch between them. One of them > > fragmented that packet. This is part of a SSL/TLS exchange and one side or > > the other is hanging on this and just dropping the connection. I suspect > > the packet size is the issue. ssldump complains about the packet too and > > stops monitoring. Could this possibly be related to the hardware checksums? > > You have TSO enabled on the interface, so large outgoing TCP packet is pretty > normal. > It will be split by the NIC. Disable TSO with ifconfig if it interferes with > your ssldump. This is not the behaviour I see with em(4) on a 82573E with all defaults used (which includes TSO4). Note that Doug is using msk(4). I can provide packet captures on both ends of a LAN segment using both tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that show a difference in behaviour compared to what Doug sees. What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" messages for payloads which are chunked or segmented as a result of TSO. I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I only see one "bad-len" entry for all chunks (up until the next ACK or PSH+ACK of course). The important part: I do not see captured TCP packets reporting a length greater than MTU (or MSS for that matter (remember: IP header + TCP header + MSS <= MTU). Also note for Doug: remember that if you're doing packet captures between two devices that have NAT involved, you may see different behaviour. Example case: 03:58:47.907582 IP 67.180.84.87.2983 > 206.125.172.42.80: Flags [.], ack 13419, win 64240, length 0 03:58:47.907649 IP 206.125.172.42.80 > 67.180.84.87.2983: Flags [.], seq 17799:19259, ack 292, win 1026, length 1460 03:58:47.907679 IP 206.125.172.42.80 > 67.180.84.87.2983: Flags [.], seq 19259:20719, ack 292, win 1026, length 1460 03:58:47.912546 IP 67.180.84.87.2983 > 206.125.172.42.80: Flags [.], ack 16339, win 64240, length 0 In the above example there's a Linux NAT router (67.180.84.87) with a client (192.168.1.50) behind it, talking to 206.125.172.42. MTU is 1500 (I obviously didn't include the initial SYN :-) ). -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Mountain View, CA, US| | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unusual TCP/IP Packet Size
13.02.2013 17:25, Doug Hardie пишет: > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the > following interface: > > msk0: flags=8843 metric 0 mtu 1500 > > options=c011b > ether 00:11:2f:2a:c7:03 > inet 10.0.1.199 netmask 0xff00 broadcast 10.0.1.255 > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet autoselect (100baseTX > ) > status: active > > > It sent the following packet: (data content abbreviated) > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq 930:4876, > ack 846, win 1040, options [nop,nop,TS val 401838072 ecr 920110183], length > 3946 > 0x: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.@.@.*. > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc ...J..h..7.. > 0x0020: 8018 0410 3407 0101 080a 17f3 8ff8 4...……. > > > The indicated packet length is 3946 and the load of data shown is that size. > The MTU on both interfaces is 1500. The receiving system received 3 packets. > There is a router and switch between them. One of them fragmented that > packet. This is part of a SSL/TLS exchange and one side or the other is > hanging on this and just dropping the connection. I suspect the packet size > is the issue. ssldump complains about the packet too and stops monitoring. > Could this possibly be related to the hardware checksums? You have TSO enabled on the interface, so large outgoing TCP packet is pretty normal. It will be split by the NIC. Disable TSO with ifconfig if it interferes with your ssldump. Eugene Grosbein ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
