Evidently, we're seeing performance problems when a card on one side is capable of scatter gather IO but the mellanox driver is not able to handle it. Then a whole lot of time is spent gathering the data into one skbuff.
This is currently being observed on RHEL4.5 with OFED 1.2. Is the Mellanox card capable of doing scatter gather IO and this just a driver limitation or is the mellanox card just not able to do scatter gather IO? Here is the original description of the problem in case the original synopsis was oversimplified: A second issue arises in how current kernels and specifically NETDEV handle scatter/gather and the related SKBs. Kernel.org is currently in the middle of a massive re-write of this section but existing ports may incur an extra buffer copy. Of the existing backports there are two skbuff variations. One is simply a pointer to the SKB while the other is a list of SKBs. If a driver supports the list and forwards a packet to a driver which supports only a pointer then NETDEV is stuck copying the list of buffers into a new single SKB. At 10gbs this copy can sap a processor's capacity to move data. The identified case where this conflict arises is the Myri10GE driver which supports lists of SKBs and Mellanox IB (IPoIB) driver which does not. Initial search through the web suggests that the IB verbs layer does support scatter gather IO and that it has a limit of 60 segments. However, we've as of yet been unable to determine if that is implemented in the verbs layer or if it is done in hardware. If anyone has any info that might help here, it will be appreciated. Thanks. _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
