Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
On 03/04/2013 23:12, Hefty, Sean wrote: Hi Sean, Ping. You had concerns on the suggested concept, we want to know if we addressed them, can you comment? I'm in meetings this week until tomorrow. I'll try to take a look at the updated patches then or Friday. any feedback? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
any feedback? I have no issue with RSS/TSS. But the 'qp group' interface to using this seems kludgy. On a node, this is multiple send/receive queues grouped together to form a larger construct. On the wire, this is a single QP - maybe? I'm still not clear on that. From what's written, all the send queues appear as a single QPN. The receive queues appear as different QPNs. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
On Tue, Apr 9, 2013 at 8:06 PM, Hefty, Sean sean.he...@intel.com wrote: I have no issue with RSS/TSS. But the 'qp group' interface to using this seems kludgy. OK, so lets take it over the patch that has the QP group description On a node, this is multiple send/receive queues grouped together to form a larger construct. On the wire, this is a single QP - maybe? I'm still not clear on that. From what's written, all the send queues appear as a single QPN. The receive queues appear as different QPNs. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
On Tue, Apr 2, 2013 at 12:50 AM, Or Gerlitz or.gerl...@gmail.com wrote: On Sun, Mar 24, 2013 at 2:44 PM, Or Gerlitz ogerl...@mellanox.com wrote: On 19/03/2013 20:57, Hefty, Sean wrote: I have not had a chance to look at v3 yet. will love it if you do so... we've just posted V4 which is respin of V3 over an ipoib change Hi Sean, we have posted the TSS/RSS V0 patches on May 2012 and so far attempted to address all the feedback / questions you provided/had. Could you comment how you see things w.r.t to these patches? specifically the QP groups concept on which you had raised some concerns which we believe were addressed with V3. Hi Sean, Ping. You had concerns on the suggested concept, we want to know if we addressed them, can you comment? Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
Hi Sean, Ping. You had concerns on the suggested concept, we want to know if we addressed them, can you comment? I'm in meetings this week until tomorrow. I'll try to take a look at the updated patches then or Friday. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
On Wed, Apr 3, 2013 at 11:12 PM, Hefty, Sean sean.he...@intel.com wrote: Hi Sean, Ping. You had concerns on the suggested concept, we want to know if we addressed them, can you comment? I'm in meetings this week until tomorrow. I'll try to take a look at the updated patches then or Friday. OK, thanks, the 3.10 merge window is coming closer and I want to know where are we in that respect. Almost every Ethernet NIC you use has RSS, there's no reason for IPoIB not to support that too. Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
On Sun, Mar 24, 2013 at 2:44 PM, Or Gerlitz ogerl...@mellanox.com wrote: On 19/03/2013 20:57, Hefty, Sean wrote: I have not had a chance to look at v3 yet. will love it if you do so... we've just posted V4 which is respin of V3 over an ipoib change Hi Sean, we have posted the TSS/RSS V0 patches on May 2012 and so far attempted to address all the feedback / questions you provided/had. Could you comment how you see things w.r.t to these patches? specifically the QP groups concept on which you had raised some concerns which we believe were addressed with V3. thanks, Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
On 19/03/2013 20:57, Hefty, Sean wrote: I have not had a chance to look at v3 yet. will love it if you do so... we've just posted V4 which is respin of V3 over an ipoib change Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
Here's V3 of the IPoIB TSS/RSS patch series, basically its very similar to V2, with fix to for one issue we stepped over while testing V2 and addressing of feedback provided by Sean on the QP groups concept. Hi Sean, Re your feedback on V2, do you feel the concept has been flushed out deep enough now? I have not had a chance to look at v3 yet. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support
From: Shlomo Pongratz shlo...@mellanox.com Here's V3 of the IPoIB TSS/RSS patch series, basically its very similar to V2, with fix to for one issue we stepped over while testing V2 and addressing of feedback provided by Sean on the QP groups concept. The concept of QP groups for TSS/RSS was introduced in the 2012 OFA conference, you can take a look on the user mode ethernet session slides 10-14, the author didn't use the terms RSS/TSS but that's the intention... see https://openfabrics.org/resources/document-downloads/presentations/cat_view/57-ofa-documents/23-presentations/81-openfabrics-international-workshops/104-2012-ofa-international-workshop/107-2012-ofa-intl-workshop-wednesday.html V2 http://marc.info/?l=linux-rdmam=136007935605406w=2 V1 http://marc.info/?l=linux-rdmam=133881081520248w=2 V0 http://marc.info/?l=linux-rdmam=133649429821312w=2 V3 changes: - rebased to 3.9-rc1 - fixed few sparse errors on patch on patch #3 - Implement Sean's Hefty suggestion, that is don't allow to modify parent QP state before all RSS/TSS children were created. Also disallow to destroy the parent QP unless all RSS/TSS children were destroyed. - solved a race condition when creation of an ipoib_neigh was attempted from more than one TX context, the change was merged into patch #3 V2 changes: - added pre-patch correcting the ipoib_neigh hash function - ported to infiniband tree / for-next branch - following commit b63b70d877 IPoIB: Use a private hash table for path lookup in xmit path from kernel 3.6, the TX select queue logic for UD neighbours was changed to be based on full hashing ala skb_tx_hash that covers L4 too wheres in V1 the queue selection was in the neighbours level. This means that different sessions (TCP/UDP five-tuples) would map to different TX rings subject to hashing. - for CM neighbours, the queue selection uses the destination IPoIB HW addr as the base for hashing. Previously each ipoib_neigh was assigned a running index upon creation and that neighbour was accessed during select queue. Now, we want to issue only ONE ipoib_neigh lookup in the xmit path and do that in start_xmit. - added patch #6 to allow for the number of TX and RX rings to be changed at runtime. By supporting ethtool directives to get/set the number of channels. move code which is common to device cleanup and device reinit from ipoib_dev_cleanup to ipoib_dev_uninit. - CM TX completions are spreaded among CQs (for NAPI) using hash of the destination IPoIB HW address. - use netif_tx bh locking in ipoib_cm_handle_tx_wc and drain_tx_cq. Also, in drain_tx_cq revert from subqueue locking to full locking, did it since __netif_tx_lock doesn't set __QUEUE_STATE_FROZEN_BIT. - handle the rare case were the device CM state ipoib_cm_admin_enabled() status changes between the time select queue was done to when the transmit routine was called. - fixed a race in the CM RX drain/reap logic caused by the change to multiple rings, added detailed comment in ipoib_cm_start_rx_drain to explain the fix. - changed the CM code that posts receive buffers (both srq and non-srq flows) to use per ring WR and SGE objects, since now buffer re-fill may happen from different NAPI contexts V1 changes: - removed accepted patches, the first three on the V0 series - fixed crash in the driver EQ teardown flow - merged by commit 3aac6ff IB/mlx4: Fix EQ deallocation in legacy mode - removed wrong setting done in the ehca driver in ehca_create_srq - fixed user space QP creation to specify QPG_NONE - fixed usage of wrong API for netif queues stopping in patch 3/4 (V0 6/7) - fixed use-after-free of device attr pointer in patch 4/4 (V0 7/7) * Add support for for RSS and TSS for UD. The number of RSS and TSS queues is a function of the number of cores and HW capability. * Utilize multi core CPU and NIC's multi queuing in order to increase throughput. It utilize a new QP Group concept. A QP group is a set of QP consists of a parent QP and two disjoint subsets of RSS and TSS QP. * If RSS is supported by HW then the number of RSS queues is highest power of two greater than or equal to the number of cores. Otherwise the number is one. * If TSS is supported by HW then the number of TSS queues is highest power of two greater than or equal to the number of cores. Otherwise the number is highest power of two greater than or equal to the number of cores plus one. * Transmission and receiving in CM mode uses a send and receive queue assigned to each CM instance at creation time. * Advertise that packets sent from set of QPs will be received. That is, A received packets with a source QPN different from the QPN advertised with ARP will be accepted. * The advertising is done by setting a third bit in the flags part of
