Re: tune ib stack

2013-04-09 Thread Vasiliy Tolstov
2013/4/9 Sebastian Riemer sebastian.rie...@profitbricks.com: Because 2048 is the default and 4096 is the max. supported MTU by the hardware. How can i set active mtu? Something like this: echo 4096 /sys/class/infiniband/mlx4_0/device/mlx4_port1_mtu After doing this all srp connections

Re: tune ib stack

2013-04-09 Thread Sebastian Riemer
On 09.04.2013 13:51, Vasiliy Tolstov wrote: Something like this: echo 4096 /sys/class/infiniband/mlx4_0/device/mlx4_port1_mtu After doing this all srp connections down and port is down. I need to restart openibd Sorry for that! It's much easier to set the IP MTU. Managed switches support

Re: tune ib stack

2013-04-09 Thread Hal Rosenstock
On 4/9/2013 8:15 AM, Sebastian Riemer wrote: On 09.04.2013 13:51, Vasiliy Tolstov wrote: Something like this: echo 4096 /sys/class/infiniband/mlx4_0/device/mlx4_port1_mtu After doing this all srp connections down and port is down. I need to restart openibd Sorry for that! It's much

RE: [RFC/PATCH v3] IPoIB: Leave space in skb linear buffer for IP headers

2013-04-09 Thread Luick, Dean
From: Roland Dreier rol...@purestorage.com + if (wc-byte_len IPOIB_UD_HEAD_SIZE) { + page = priv-rx_ring[wr_id].page; + priv-rx_ring[wr_id].page = NULL; + } else { + page = NULL; + } + /* * If we can't allocate a new RX

Re: tune ib stack

2013-04-09 Thread Sebastian Riemer
On 09.04.2013 14:49, Hal Rosenstock wrote: On 4/9/2013 7:12 AM, Vasiliy Tolstov wrote: Hello. I have some servers, with mellanox ConnectX-3 and have some questions: Why max_mtu differs with active_mtu? What does peer port say for max MTU ? How can i set active mtu? SM sets active MTU

Re: tune ib stack

2013-04-09 Thread Hal Rosenstock
On 4/9/2013 9:16 AM, Sebastian Riemer wrote: On 09.04.2013 14:49, Hal Rosenstock wrote: On 4/9/2013 7:12 AM, Vasiliy Tolstov wrote: Hello. I have some servers, with mellanox ConnectX-3 and have some questions: Why max_mtu differs with active_mtu? What does peer port say for max MTU ?

Re: tune ib stack

2013-04-09 Thread Sebastian Riemer
On 09.04.2013 15:34, Hal Rosenstock wrote: On 4/9/2013 9:16 AM, Sebastian Riemer wrote: On 09.04.2013 14:49, Hal Rosenstock wrote: On 4/9/2013 7:12 AM, Vasiliy Tolstov wrote: Hello. I have some servers, with mellanox ConnectX-3 and have some questions: Why max_mtu differs with active_mtu?

Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support

2013-04-09 Thread Or Gerlitz
On 03/04/2013 23:12, Hefty, Sean wrote: Hi Sean, Ping. You had concerns on the suggested concept, we want to know if we addressed them, can you comment? I'm in meetings this week until tomorrow. I'll try to take a look at the updated patches then or Friday. any feedback? -- To unsubscribe

Re: tune ib stack

2013-04-09 Thread Hal Rosenstock
On 4/9/2013 9:56 AM, Sebastian Riemer wrote: On 09.04.2013 15:34, Hal Rosenstock wrote: On 4/9/2013 9:16 AM, Sebastian Riemer wrote: On 09.04.2013 14:49, Hal Rosenstock wrote: On 4/9/2013 7:12 AM, Vasiliy Tolstov wrote: Hello. I have some servers, with mellanox ConnectX-3 and have some

Re: tune ib stack

2013-04-09 Thread Sebastian Riemer
On 09.04.2013 16:23, Hal Rosenstock wrote: So these values are exactly the same as in ibv_devinfo and can be set in /sys/class/infiniband/mlx4_0/device/mlx4_port1_mtu. I've found the PortInfo with the command smpquery portinfo -C mlx4_0 3 1 where I'm using the first HCA to contact the SM. I

Re: [RFC/PATCH v3] IPoIB: Leave space in skb linear buffer for IP headers

2013-04-09 Thread Roland Dreier
On Tue, Apr 9, 2013 at 6:13 AM, Luick, Dean dean.lu...@intel.com wrote: Can you go through the else of the first if (page is NULL), then enter the second if? If so, isn't the page lost? Thanks, good catch. I'll fix that up. -- To unsubscribe from this list: send the line unsubscribe

RE: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support

2013-04-09 Thread Hefty, Sean
any feedback? I have no issue with RSS/TSS. But the 'qp group' interface to using this seems kludgy. On a node, this is multiple send/receive queues grouped together to form a larger construct. On the wire, this is a single QP - maybe? I'm still not clear on that. From what's written,

Re: [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag

2013-04-09 Thread Michael S. Tsirkin
On Fri, Apr 05, 2013 at 04:54:39PM -0400, Michael R. Hines wrote: To be more specific, here's what I did: 1. apply kernel module patch - re-insert module 1. QEMU does: ibv_reg_mr(IBV_ACCESS_GIFT | IBV_ACCESS_REMOTE_READ) 2. Start the RDMA migration 3. Migration completes without any

Re: [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag

2013-04-09 Thread Michael R. Hines
On 04/09/2013 12:39 PM, Michael S. Tsirkin wrote: On Fri, Apr 05, 2013 at 04:54:39PM -0400, Michael R. Hines wrote: To be more specific, here's what I did: 1. apply kernel module patch - re-insert module 1. QEMU does: ibv_reg_mr(IBV_ACCESS_GIFT | IBV_ACCESS_REMOTE_READ) 2. Start the

Re: [RFC/PATCH v3] IPoIB: Leave space in skb linear buffer for IP headers

2013-04-09 Thread Markus Stockhausen
-IPOIB_UD_HEAD_SIZE = IB_GRH_BYTES + IPOIB_ENCAP_LEN, +/* add 128 bytes of tailroom for IP/TCP headers */ +IPOIB_UD_HEAD_SIZE = IB_GRH_BYTES + IPOIB_ENCAP_LEN + 128, Hello, the version 3 of the patch finally works. I can see the performance gains but I cannot feel them

Re: [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag

2013-04-09 Thread Michael S. Tsirkin
presumably is_dup_page reads the page, so should not break COW ... I'm not sure about the cgroups swap limit - you might have too many non COW pages so attempting to fault them all in makes you exceed the limit. You really should look at what is going on in the pagemap, to see if there's

RE: [PATCH 2/2] Ad IB_MTU_1500|9000 enums.

2013-04-09 Thread Weiny, Ira
-Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma- Subject: Re: [PATCH 2/2] Ad IB_MTU_1500|9000 enums. On Apr 4, 2013, at 1:57 PM, Weiny, Ira ira.we...@intel.com wrote: In hindsight, the user space API never should have exposed the mtu as an enum...

Re: [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag

2013-04-09 Thread Michael S. Tsirkin
On Fri, Apr 05, 2013 at 01:43:49PM -0700, Roland Dreier wrote: On Fri, Apr 5, 2013 at 1:17 PM, Michael R. Hines mrhi...@linux.vnet.ibm.com wrote: I also removed the IBV_*_WRITE flags on the sender-side and activated cgroups with the memory.memsw.limit_in_bytes activated and the migration

Re: [PATCH V3 for-next 0/5] IB/IPoIB: Add multi-queue TSS and RSS support

2013-04-09 Thread Or Gerlitz
On Tue, Apr 9, 2013 at 8:06 PM, Hefty, Sean sean.he...@intel.com wrote: I have no issue with RSS/TSS. But the 'qp group' interface to using this seems kludgy. OK, so lets take it over the patch that has the QP group description On a node, this is multiple send/receive queues grouped

Re: [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag

2013-04-09 Thread Michael S. Tsirkin
On Fri, Apr 05, 2013 at 04:17:36PM -0400, Michael R. Hines wrote: The userland part of the patch was missing (IBV_ACCESS_GIFT). I added flag that to /usr/include in addition to this patch and did a test RDMA migrate and it seems to work without any problems. I also removed the IBV_*_WRITE

Re: [PATCH V4 for-next 1/5] IB/core: Add RSS and TSS QP groups

2013-04-09 Thread Or Gerlitz
This patch introduces the concept of RSS and TSS QP groups which allows for implementing them by low level drivers and using it by IPoIB and later also by user space ULPs. A QP group is a set of QPs consists of a parent QP and two disjoint sets of RSS and TSS QPs. The creation of a QP group

Re: [PATCH 2/2] Ad IB_MTU_1500|9000 enums.

2013-04-09 Thread Jeff Squyres (jsquyres)
On Apr 8, 2013, at 6:16 PM, Hefty, Sean sean.he...@intel.com wrote: Why can't IB_MTU_1500 = 1500? It certainly could. Additionally, since Roland was a little concerned about the IB prefix (since 1500 and 9000 are not IBTA-sanctioned MTUs), they could have a different prefix -- perhaps

RE: [PATCH 2/2] Ad IB_MTU_1500|9000 enums.

2013-04-09 Thread Weiny, Ira
-Original Message- From: Jeff Squyres (jsquyres) [mailto:jsquy...@cisco.com] Subject: Re: [PATCH 2/2] Ad IB_MTU_1500|9000 enums. On Apr 8, 2013, at 6:16 PM, Hefty, Sean sean.he...@intel.com wrote: Why can't IB_MTU_1500 = 1500? Sean, If the IBTA were to release new MTU

Re: [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag

2013-04-09 Thread Michael R. Hines
With respect, I'm going to offload testing this patch back to the author =) because I'm trying to address all of Paolo's other minor issues with the RDMA patch before we can merge. Since dynamic page registration (as you requested) is now fully implemented, this patch is less urgent since we now

RE: [PATCH 2/2] Ad IB_MTU_1500|9000 enums.

2013-04-09 Thread Weiny, Ira
-Original Message- From: Hefty, Sean Sent: Tuesday, April 09, 2013 6:30 PM To: Weiny, Ira; Jeff Squyres (jsquyres) Cc: Hal Rosenstock; Roland Dreier; linux-rdma@vger.kernel.org; Upinder Malhi (umalhi) Subject: RE: [PATCH 2/2] Ad IB_MTU_1500|9000 enums. If the IBTA were to

Re: [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag

2013-04-09 Thread Michael S. Tsirkin
On Tue, Apr 09, 2013 at 09:26:59PM -0400, Michael R. Hines wrote: With respect, I'm going to offload testing this patch back to the author =) because I'm trying to address all of Paolo's other minor issues with the RDMA patch before we can merge. Fair enough, this likely means it won't happen