Re: [ofa-general] Re: IPoIB forwarding
I have been able to install and use the 1.3.0 myricom driver and everything works as I expected and performance is pretty decent. Interesting little side tour through various drivers...The router node sees almost no load which is really encouraging. Thanks, bryan At 03:05 PM 5/1/2007, Loic Prylli wrote: On 4/30/2007 2:12 PM, Rick Jones wrote: Speaking of defaults, it would seem that the external 1.2.0 driver comes with 9000 bytes as the default MTU? At least I think that is what I am seeing now that I've started looking more closely. rick jones That's the same for the in-kernel-tree code (9K MTU by default). Assuming this is not wanted, I will submit a patch for that. Loic - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ofa-general] Re: IPoIB forwarding
I hit the IP NIC over the head with a hammer and turned off all offload features and I no longer get the super jumbo packet and I have symmetric performance. This NIC supported ethtool -K ethx tso/tx/rx/sg on/off and I am not sure at this time which one I needed to whack but all off solved the problem. Thanks for listening and re enforcing my search process. bryan At 01:32 PM 4/27/2007, Rick Jones wrote: Bryan Lawver wrote: Your right about the ipoib module not combining packets (I believed you without checking) but I did never the less. The ipoib_start_xmit routine is definitely handed a double packet which means that the IP NIC driver or the kernel is combining two packets into a single super jumbo packet. This issue is irrespective of the IP MTU setting because I have set all interfaces to 9000k yet ipoib accepts and forwards this 17964 packet to the next IB node and onto the TCP stack where it is never acknowledged. This may not have come up in prior testing because I am using some of the fastest IP NICs which have no trouble keeping up with or exceeding the bandwidth of the IB side. This issue arises exactly every 8 packets...(ring buffer overrun??) I will be at Sonoma for the next few days as many on this list will be. Some NICs (esp 10G) support large receive offload - they coalesce TCP segments from the wire/fiber into larger ones they pass up the stack. Perhaps that is happening here? I'm going to go out a bit on a limb, cross the streams, and include netdev, because I suspect that if a system is acting as an IP router, one doesn't want large receive offload enabled. That may need some discussion in netdev - it may then require some changes to default settings or some documentation enhancements. That or I'll learn that the stack is already dealing with the issue... rick jones bryan At 11:06 AM 4/26/2007, Michael S. Tsirkin wrote: Quoting Bryan Lawver [EMAIL PROTECTED]: Subject: Re: IPoIB forwarding Here's a tcpdump of the same sequence. The TCP MSS is 8960 and it appears that two payloads are queued at ipoib which combines them into a single 17920 payload with assumingly correct IP header (40) and IB header (4). The application or TCP stack does not acknowledge this double packet ie. it does not ACK until each of the 8960 packets are resent individually. Being an IB newbie, I am guessing this combining is allowable but may violate TCP protocol. IPoIB does nothing like this - it's just a network device so it sends all packets out as is. -- MST ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ofa-general] Re: IPoIB forwarding
I had so much debugging turned on that it was not the slowing of the traffic but the non-coelescencing that was the remedy. The NIC is a MyriCom NIC and these are easy options to set. At 03:32 PM 4/27/2007, Rick Jones wrote: Bryan Lawver wrote: I hit the IP NIC over the head with a hammer and turned off all offload features and I no longer get the super jumbo packet and I have symmetric performance. This NIC supported ethtool -K ethx tso/tx/rx/sg on/off and I am not sure at this time which one I needed to whack but all off solved the problem. Yeah, that does seem like a rather broad remedy, but I guess if it works... :) And I suppose most of those offloads don't matter for a NIC being used in a router. Only problem is we don't know if it worked because it slowed-down the 10G side or because it had LRO disabling as a side-effect. If I were to guess, of those things listed, I'd guess that receive cko would have that as a side effect. Just what sort of 10G NIC was this anyway? With that knowledge we could probably narrow things down to a more specific modprobe setting, or maybe even an ethtool command, for some suitable revision of ethtool. rick jones Thanks for listening and re enforcing my search process. bryan At 01:32 PM 4/27/2007, Rick Jones wrote: Bryan Lawver wrote: Your right about the ipoib module not combining packets (I believed you without checking) but I did never the less. The ipoib_start_xmit routine is definitely handed a double packet which means that the IP NIC driver or the kernel is combining two packets into a single super jumbo packet. This issue is irrespective of the IP MTU setting because I have set all interfaces to 9000k yet ipoib accepts and forwards this 17964 packet to the next IB node and onto the TCP stack where it is never acknowledged. This may not have come up in prior testing because I am using some of the fastest IP NICs which have no trouble keeping up with or exceeding the bandwidth of the IB side. This issue arises exactly every 8 packets...(ring buffer overrun??) I will be at Sonoma for the next few days as many on this list will be. Some NICs (esp 10G) support large receive offload - they coalesce TCP segments from the wire/fiber into larger ones they pass up the stack. Perhaps that is happening here? I'm going to go out a bit on a limb, cross the streams, and include netdev, because I suspect that if a system is acting as an IP router, one doesn't want large receive offload enabled. That may need some discussion in netdev - it may then require some changes to default settings or some documentation enhancements. That or I'll learn that the stack is already dealing with the issue... rick jones bryan At 11:06 AM 4/26/2007, Michael S. Tsirkin wrote: Quoting Bryan Lawver [EMAIL PROTECTED]: Subject: Re: IPoIB forwarding Here's a tcpdump of the same sequence. The TCP MSS is 8960 and it appears that two payloads are queued at ipoib which combines them into a single 17920 payload with assumingly correct IP header (40) and IB header (4). The application or TCP stack does not acknowledge this double packet ie. it does not ACK until each of the 8960 packets are resent individually. Being an IB newbie, I am guessing this combining is allowable but may violate TCP protocol. IPoIB does nothing like this - it's just a network device so it sends all packets out as is. -- MST ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
