Re: [PATCH v3 5/6] svcrdma: Add infrastructure to receive backwards direction RPC/RDMA replies

Three comments. On 12/7/2015 12:43 PM, Chuck Lever wrote: To support the NFSv4.1 backchannel on RDMA connections, add a capability for receiving an RPC/RDMA reply on a connection established by a client. (snip) +/* By convention, backchannel calls arrive via rdma_msg type "By convention"

Re: [PATCH v3 1/6] svcrdma: Do not send XDR roundup bytes for a write chunk

Two small comments. On 12/7/2015 12:42 PM, Chuck Lever wrote: Minor optimization: when dealing with write chunk XDR roundup, do not post a Write WR for the zero bytes in the pad. Simply update the write segment in the RPC-over-RDMA header to reflect the extra pad bytes. The Reply chunk is also

Re: [PATCH v2 6/7] xprtrdma: Add class for RDMA backwards direction transport

On 11/30/2015 5:25 PM, Chuck Lever wrote: To support the server-side of an NFSv4.1 backchannel on RDMA connections, add a transport class that enables backward direction messages on an existing forward channel connection. +static void * +xprt_rdma_bc_allocate(struct rpc_task *task, size_t

Re: Future of FMR support, was: Re: [PATCH v1 5/9] xprtrdma: Add ro_unmap_sync method for FMR

On 11/24/2015 1:52 AM, Christoph Hellwig wrote: On Mon, Nov 23, 2015 at 07:57:42PM -0500, Tom Talpey wrote: On 11/23/2015 5:14 PM, Chuck Lever wrote: FMR's ro_unmap method is already synchronous because ib_unmap_fmr() is a synchronous verb. However, some improvements can be made here. I

Re: [PATCH 2/9] IB: add a proper completion queue abstraction

On 11/24/2015 2:03 AM, Jason Gunthorpe wrote: On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote: Are there actual HCAs that make this mistake? All IB HCAs have this behavior and require apps to see a send CQ completion before making any statements about the state of the send Q

Re: [PATCH v1 3/9] xprtrdma: Introduce ro_unmap_sync method

On 11/24/2015 5:59 AM, Sagi Grimberg wrote: As I see it, if we don't wait for local-invalidate to complete before unmap and IO completion (and no one does) For the record, that is false. Windows quite strictly performs these steps, and deliver millions of real 4K IOPS over SMB Direct.

Re: [PATCH v1 2/8] svcrdma: Define maximum number of backchannel requests

On 11/23/2015 5:20 PM, Chuck Lever wrote: Extra resources for handling backchannel requests have to be pre-allocated when a transport instance is created. Set a limit. Signed-off-by: Chuck Lever --- include/linux/sunrpc/svc_rdma.h |5 +

Re: [PATCH v1 2/8] svcrdma: Define maximum number of backchannel requests

On 11/23/2015 8:09 PM, Chuck Lever wrote: On Nov 23, 2015, at 7:39 PM, Tom Talpey <t...@talpey.com> wrote: On 11/23/2015 5:20 PM, Chuck Lever wrote: Extra resources for handling backchannel requests have to be pre-allocated when a transport instance is created. Set a limit. Sign

Re: [PATCH v1 6/8] xprtrdma: Add class for RDMA backwards direction transport

On 11/23/2015 5:21 PM, Chuck Lever wrote: To support the server-side of an NFSv4.1 backchannel on RDMA connections, add a transport class for backwards direction operation. So, what's special here is that it re-uses an existing forward channel's connection? If not, it would seem unnecessary to

Re: [PATCH v1 1/9] xprtrdma: Add a safety margin for receive buffers

On 11/23/2015 5:13 PM, Chuck Lever wrote: Rarely, senders post a Send that is larger than the client's inline threshold. That can be due to a bug, or the client and server may not have communicated about their inline limits. RPC-over-RDMA currently doesn't specify any particular limit on inline

Re: [PATCH v1 1/9] xprtrdma: Add a safety margin for receive buffers

On 11/23/2015 8:16 PM, Chuck Lever wrote: On Nov 23, 2015, at 7:55 PM, Tom Talpey <t...@talpey.com> wrote: On 11/23/2015 5:13 PM, Chuck Lever wrote: Rarely, senders post a Send that is larger than the client's inline threshold. That can be due to a bug, or the client and server may no

Re: [PATCH 2/9] IB: add a proper completion queue abstraction

On 11/23/2015 7:00 PM, Jason Gunthorpe wrote: On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote: The receive completion can be safely assumed to indicate transmit completion over a reliable connection unless your peer has gone completely bonkers and is replying to a

Re: [PATCH v1 8/9] xprtrdma: Invalidate in the RPC reply handler

On 11/23/2015 5:14 PM, Chuck Lever wrote: There is a window between the time the RPC reply handler wakes the waiting RPC task and when xprt_release() invokes ops->buf_free. During this time, memory regions containing the data payload may still be accessed by a broken or malicious server, but the

Re: [PATCH v1 5/8] svcrdma: Add infrastructure to receive backwards direction RPC/RDMA replies

On 11/23/2015 5:20 PM, Chuck Lever wrote: To support the NFSv4.1 backchannel on RDMA connections, add a capability for receiving an RPC/RDMA reply on a connection established by a client. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 76

Re: [PATCH v1 8/8] svcrdma: Remove svc_rdma_fastreg_mr::access_flags field

On 11/23/2015 5:21 PM, Chuck Lever wrote: Clean up: The access_flags field is not used outside of rdma_read_chunk_frmr() and is always set to the same value. Signed-off-by: Chuck Lever --- include/linux/sunrpc/svc_rdma.h |1 -

Re: [PATCH v1 5/9] xprtrdma: Add ro_unmap_sync method for FMR

On 11/23/2015 5:14 PM, Chuck Lever wrote: FMR's ro_unmap method is already synchronous because ib_unmap_fmr() is a synchronous verb. However, some improvements can be made here. I thought FMR support was about to be removed in the core. 1. Gather all the MRs for the RPC request onto a

Re: [PATCH v1 5/8] svcrdma: Add infrastructure to receive backwards direction RPC/RDMA replies

an upper layer question, not an RPC one. On 11/23/2015 8:47 PM, Chuck Lever wrote: On Nov 23, 2015, at 7:44 PM, Tom Talpey <t...@talpey.com> wrote: On 11/23/2015 5:20 PM, Chuck Lever wrote: To support the NFSv4.1 backchannel on RDMA connections, add a capability for receiving an RPC/RDMA

Re: [PATCH v1 2/8] svcrdma: Define maximum number of backchannel requests

On 11/23/2015 8:36 PM, Chuck Lever wrote: On Nov 23, 2015, at 8:19 PM, Tom Talpey <t...@talpey.com> wrote: On 11/23/2015 8:09 PM, Chuck Lever wrote: On Nov 23, 2015, at 7:39 PM, Tom Talpey <t...@talpey.com> wrote: On 11/23/2015 5:20 PM, Chuck Lever wrote: Extra resources

Re: [PATCH libmlx5 0/7] Completion timestamping

On 11/15/2015 7:30 AM, Matan Barak wrote: This series adds support for completion timestamp. In order to support this feature, several extended verbs were implemented (as instructed in libibverbs). Can you describe what these timestamps are actually for? It's not clear at all from the

Re: [PATCH RFC 2/3] svcrdma: Use device rdma_read_access_flags

On 11/10/2015 4:17 PM, Jason Gunthorpe wrote: On Tue, Nov 10, 2015 at 04:00:43PM -0500, Tom Talpey wrote: Hmm, agreed, but it must still be acceptable to perform a registration instead of using the local_dma_lkey. As mentioned earlier, there are scatter limits and other implications for all

Re: [PATCH RFC 2/3] svcrdma: Use device rdma_read_access_flags

On 11/10/2015 1:25 PM, Jason Gunthorpe wrote: On Tue, Nov 10, 2015 at 04:04:32AM -0800, Christoph Hellwig wrote: On Tue, Nov 10, 2015 at 01:46:40PM +0200, Sagi Grimberg wrote: On 10/11/2015 13:41, Christoph Hellwig wrote: Oh, and while we're at it. Can someone explain why we're even using

Re: [PATCH RFC 1/3] IB/core: Expose a device attribute for rdma_read access flags

On 11/10/2015 6:51 AM, Yann Droneaud wrote: Le mardi 10 novembre 2015 à 12:44 +0200, Sagi Grimberg a écrit : Also, set rdma_read_access_flags in the relevant device drivers: mlx4, mlx5, qib, ocrdma, nes: IB_ACCESS_LOCAL_WRITE cxgb3, cxgb4, hfi: IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_WRITE

Re: RDMA/CM and multiple QPs

On 9/6/2015 11:15 AM, Bart Van Assche wrote: On 09/06/15 00:50, Christoph Hellwig wrote: Note that the SRP driver already in tree is a good example for this, although it doesn't use RDMA/CM and thus already operates on a per-ib_device level. The challenges with regard to adding RDMA/CM

Re: [PATCH for-next 00/10] Add RoCE support to the mlx5 driver

On 8/25/2015 4:35 AM, Achiad Shochat wrote: On 8/24/2015 11:40 PM, Tom Talpey wrote: On 8/20/2015 12:46 PM, Achiad Shochat wrote: Hi Doug, This patchset adds RoCE V1 and RoCE V2 support to the mlx5 device driver. Question - assuming I read them correctly, these patches add the RoCE v1

Re: [PATCH for-next 00/10] Add RoCE support to the mlx5 driver

On 8/20/2015 12:46 PM, Achiad Shochat wrote: Hi Doug, This patchset adds RoCE V1 and RoCE V2 support to the mlx5 device driver. Question - assuming I read them correctly, these patches add the RoCE v1 and RoCE v2 support on a per-port basis. That is, a port can be either IB, RoCE v1 or RoCE

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/21/2015 7:33 AM, Steve Wise wrote: -Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Tom Talpey Sent: Monday, July 20, 2015 7:15 PM To: Steve Wise; 'Jason Gunthorpe' Cc: 'Chuck Lever'; linux-rdma@vger.kernel.org

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/20/2015 1:55 PM, Chuck Lever wrote: On Jul 20, 2015, at 4:34 PM, Tom Talpey t...@talpey.com wrote: On 7/20/2015 12:03 PM, Chuck Lever wrote: All HCA providers have an ib_get_dma_mr() verb. Thus rpcrdma_ia_open() will either grab the device's local_dma_key if one is available

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/20/2015 2:16 PM, Steve Wise wrote: -Original Message- From: linux-nfs-ow...@vger.kernel.org [mailto:linux-nfs-ow...@vger.kernel.org] On Behalf Of Jason Gunthorpe Sent: Monday, July 20, 2015 4:06 PM To: Tom Talpey; Steve Wise Cc: Chuck Lever; linux-rdma@vger.kernel.org; linux

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/20/2015 12:03 PM, Chuck Lever wrote: All HCA providers have an ib_get_dma_mr() verb. Thus rpcrdma_ia_open() will either grab the device's local_dma_key if one is available, or it will call ib_get_dma_mr() which is a 100% guaranteed fallback. I recall that in the past, some providers did

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/20/2015 3:17 PM, Jason Gunthorpe wrote: On Mon, Jul 20, 2015 at 03:04:21PM -0700, Tom Talpey wrote: B) why bother to check? Are machines with 4GB interesting, and worth supporting a special optimization? mainline drivers shouldn't silently malfunction. I meant why bother to check, just

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/20/2015 5:34 PM, Chuck Lever wrote: On Jul 20, 2015, at 8:11 PM, Tom Talpey t...@talpey.com wrote: On 7/20/2015 4:36 PM, Chuck Lever wrote: On Jul 20, 2015, at 6:41 PM, Jason Gunthorpe jguntho...@obsidianresearch.com wrote: On Mon, Jul 20, 2015 at 06:31:11PM -0400, Chuck Lever wrote

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/20/2015 3:21 PM, Chuck Lever wrote: On Jul 20, 2015, at 5:55 PM, Tom Talpey t...@talpey.com wrote: On 7/20/2015 1:55 PM, Chuck Lever wrote: On Jul 20, 2015, at 4:34 PM, Tom Talpey t...@talpey.com wrote: On 7/20/2015 12:03 PM, Chuck Lever wrote: All HCA providers have an ib_get_dma_mr

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

know. Tom Talpey or Tom Tucker? I don't think I used a dma_lkey at all in the original work, but there were some changes later. I thought not all rnic's supported a dma_lkey. So requiring it seems like a bad idea, to me. Just use the code you have for fmr with frwr to get the lkey, probably move

Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

On 7/20/2015 3:41 PM, Steve Wise wrote: -Original Message- From: Tom Talpey [mailto:t...@talpey.com] Sent: Monday, July 20, 2015 5:04 PM To: Steve Wise; 'Jason Gunthorpe' Cc: 'Chuck Lever'; linux-rdma@vger.kernel.org; linux-...@vger.kernel.org Subject: Re: [PATCH v3 05/15] xprtrdma

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/14/2015 5:22 AM, Sagi Grimberg wrote: On 7/14/2015 10:37 AM, 'Christoph Hellwig' wrote: On Mon, Jul 13, 2015 at 03:36:44PM -0400, Tom Talpey wrote: On 7/11/2015 6:25 AM, 'Christoph Hellwig' wrote: I think what we need to support for now is FRMR as the primary target, and FMR

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/14/2015 11:36 AM, 'Christoph Hellwig' wrote: On Tue, Jul 14, 2015 at 12:10:36PM +0300, Sagi Grimberg wrote: Having an API that does FRMR/FMR/PHYS_MR is even worse from the ULP PoV. If you expose an API that might schedule (PHYS_MR) it limits the context that the caller is allowed to call

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/14/2015 3:32 PM, Steve Wise wrote: On Tue, Jul 14, 2015 at 02:25:50PM -0500, Steve Wise wrote: The benefit is that we don't have to check for iWARP protocol in the ULP. IB should have to pay the cost of FRMR for lkey on RDMA READ, I'm pretty sure you have to check for iWarp at

Re: [PATCH v2 07/14] xprtrdma: Remove logic that constructs RDMA_MSGP type calls

On 7/13/2015 12:30 PM, Chuck Lever wrote: RDMA_MSGP type calls insert a zero pad in the middle of the RPC message to align the RPC request's data payload to the server's alignment preferences. A server can then page flip the payload into place to avoid a data copy in certain circumstances.

Re: Kernel fast memory registration API proposal [RFC]

Assche; Liran Liss; Hefty, Sean; Doug Ledford; Tom Talpey Subject: Re: Kernel fast memory registration API proposal [RFC] On 7/14/2015 6:33 PM, Christoph Hellwig wrote: On Tue, Jul 14, 2015 at 11:39:24AM +0300, Sagi Grimberg wrote: This is exactly what I don't want to do. I don't think

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/11/2015 6:25 AM, 'Christoph Hellwig' wrote: I think what we need to support for now is FRMR as the primary target, and FMR as a secondar[y]. FMR is a *very* bad choice, for several reasons. 1) It's not supported by very many devices, in fact it might even be thought to be obsolete. 2)

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/13/2015 1:18 PM, Jason Gunthorpe wrote: On Fri, Jul 10, 2015 at 11:10:23PM -0400, Doug Ledford wrote: Black hat server is beyond the scope of this discussion. We cannot assume an all-trusted model here, there are many configurations to deploy NFS/iSCSI that don't assume that. Even if you

Re: [PATCH v1 09/12] xprtrdma: Prepare rpcrdma_ep_post() for RDMA_NOMSG calls

On 7/10/2015 7:29 AM, Devesh Sharma wrote: we need to honor the max limits of device by checking dev_attr.max_sge? a vendor may not support 4 sges. iWARP requires a minimum of 4 send SGEs (draft-hilland-verbs 8.1.3.2) An RI MUST support at least four Scatter/Gather Elements per

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/9/2015 6:53 PM, Jason Gunthorpe wrote: On Thu, Jul 09, 2015 at 06:18:26PM -0400, Doug Ledford wrote: A lot of this discussion is interesting and has gone off in an area that I think is worth pursuing in the long run. However, in the short run, this patch was a minor

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/10/2015 1:56 PM, Doug Ledford wrote: On 07/10/2015 12:11 PM, Jason Gunthorpe wrote: On Fri, Jul 10, 2015 at 09:22:24AM -0400, Tom Talpey wrote: and it is enabled only when the RDMA Read is active. ??? How is that done? ib_get_dma_mr is defined to return a remote usable rkey

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/9/2015 1:01 PM, Jason Gunthorpe wrote: Laid out like this, I think it even means we can nuke the IB DMA API for these cases. rdma_post_read and rdma_post_complete_read are the two points that need dma api calls (cache flushes), and they can just do them internally. This also tells me that

Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

On 7/8/2015 3:08 PM, Jason Gunthorpe wrote: The MR stuff was never really designed, the HW people provided some capability and the SW side just raw exposed it, thoughtlessly. Jason, I don't disagree that the API can be improved. I have some responses to your statements below though. Why is

Re: [PATCH v6 01/26] IB/Verbs: Implement new callback query_transport()

On 4/27/2015 2:52 PM, ira.weiny wrote: On Mon, Apr 27, 2015 at 09:39:05AM +0200, Michael Wang wrote: On 04/24/2015 05:12 PM, Liran Liss wrote: From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma- [snip] a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 65994a1..d54f91e

Re: [PATCH v6 01/26] IB/Verbs: Implement new callback query_transport()

On 4/27/2015 5:36 PM, Doug Ledford wrote: On Mon, 2015-04-27 at 17:16 -0700, Tom Talpey wrote: On 4/27/2015 2:52 PM, ira.weiny wrote: On Mon, Apr 27, 2015 at 09:39:05AM +0200, Michael Wang wrote: On 04/24/2015 05:12 PM, Liran Liss wrote: [snip] Like: enum rdma_protocol

Re: [PATCH v6 01/26] IB/Verbs: Implement new callback query_transport()

On 4/27/2015 6:24 PM, Doug Ledford wrote: On Mon, 2015-04-27 at 17:53 -0700, Tom Talpey wrote: On 4/27/2015 5:36 PM, Doug Ledford wrote: On Mon, 2015-04-27 at 17:16 -0700, Tom Talpey wrote: On 4/27/2015 2:52 PM, ira.weiny wrote: On Mon, Apr 27, 2015 at 09:39:05AM +0200, Michael Wang wrote

Re: [PATCH v6 01/26] IB/Verbs: Implement new callback query_transport()

On 4/24/2015 8:23 AM, Michael Wang wrote: Add new callback query_transport() and implement for each HW. Mapping List: node-type link-layer old-transport new-transport ... mlx4IB_CA IB/ETH IB IB/IBOE mlx5IB_CA

Re: [PATCH v6 01/26] IB/Verbs: Implement new callback query_transport()

On 4/24/2015 10:35 AM, Michael Wang wrote: On 04/24/2015 04:29 PM, Tom Talpey wrote: On 4/24/2015 8:23 AM, Michael Wang wrote: [snip] +static enum rdma_transport_type +mlx5_ib_query_transport(struct ib_device *device, u8 port_num) +{ +return RDMA_TRANSPORT_IB; +} + Just noticed

Re: [PATCH v4 10/27] IB/Verbs: Reform cm related part in IB-core cma/ucm

On 4/16/2015 11:22 AM, Michael Wang wrote: On 04/16/2015 04:31 PM, Hefty, Sean wrote: This is equivalent to today where the checks are per node rather than per port. Should all checks here be port 1 based or only certain ones like listen ? For example, in connect/reject/disconnect, don't we

Re: [PATCH v2 01/17] IB/Verbs: Implement new callback query_transport() for each HW

On 4/10/2015 5:06 PM, ira.weiny wrote: On Fri, Apr 10, 2015 at 01:17:23PM -0600, Jason Gunthorpe wrote: ... So trying to sum up. Have we settled on the following capabilities? Helper function names aside. /* legacy to communicate to userspace */ RDMA_LINK_LAYER_IB = 0x0001,

Re: [PATCH v2 00/17] IB/Verbs: IB Management Helpers

On 4/7/2015 8:25 AM, Michael Wang wrote: Mapping List: node-type link-layer old-transport new-transport nes RNICETH IWARP IWARP amso1100RNICETH IWARP IWARP cxgb3 RNIC

Re: [PATCH v2 01/17] IB/Verbs: Implement new callback query_transport() for each HW

On 4/8/2015 4:10 PM, Jason Gunthorpe wrote: On Wed, Apr 08, 2015 at 02:29:46PM -0400, Doug Ledford wrote: ... rdma_port_get_read_sge(dev, port) { if (rdma_transport_is_iwarp) return 1; return dev-port[port]-max_sge; } Hum, that is nice, but it doesn't quite

Re: [PATCH v2 09/17] IB/Verbs: Use helper cap_read_multi_sge() and reform svc_rdma_accept()

On 4/7/2015 8:34 AM, Michael Wang wrote: /** + * cap_read_multi_sge - Check if the port of device has the capability + * RDMA Read Multiple Scatter-Gather Entries. + * + * @device: Device to be checked + * @port_num: Port number of the device + * + * Return 0 when port of the device don't

Re: [RFC PATCH 08/11] IB/Verbs: Use management helper has_iwarp() for, iwarp-check

On 3/31/2015 3:39 AM, Michael Wang wrote: On 03/31/2015 12:35 AM, Jason Gunthorpe wrote: On Mon, Mar 30, 2015 at 05:10:12PM +0200, Michael Wang wrote: I found that actually we don't have to touch this one which only used by HW driver currently. I'm having a hard time understanding this, the

Re: [RFC PATCH 08/11] IB/Verbs: Use management helper has_iwarp() for, iwarp-check

On 3/31/2015 7:41 AM, Michael Wang wrote: Hi, Tom Thanks for the comments :-) On 03/31/2015 01:19 PM, Tom Talpey wrote: [oops - repeating my reply with full cc's] [snip] Actually I'm thinking about Doug's idea to use rdma_transport_is_XX() instead of the current basic helper, thus may

Re: Proposal for simplifying NFS/RDMA client memory registration

On 2/26/2014 8:44 AM, Chuck Lever wrote: Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. One complication is the number of memory registration modes

Re: Helps to Decode rpc_debug Output

On 8/21/2013 11:55 AM, Wendy Cheng wrote: On Thu, Aug 15, 2013 at 11:08 AM, Wendy Cheng s.wendy.ch...@gmail.com wrote: On Thu, Aug 15, 2013 at 5:46 AM, Tom Talpey t...@talpey.com wrote: On 8/14/2013 8:14 PM, Wendy Cheng wrote: Longer version of the question: I'm trying to enable NFS-RDMA

Re: Helps to Decode rpc_debug Output

On 8/14/2013 8:14 PM, Wendy Cheng wrote: Longer version of the question: I'm trying to enable NFS-RDMA on an embedded system (based on 2.6.38 kernel) as a client. The IB stacks are taken from OFED 1.5.4. NFS server is a RHEL 6.3 Xeon box. The connection uses mellox-4 driver. Memory registration

Re: NFS over RDMA benchmark

On 4/30/2013 1:09 AM, Yan Burman wrote: -Original Message- From: J. Bruce Fields [mailto:bfie...@fieldses.org] Sent: Sunday, April 28, 2013 17:43 To: Yan Burman Cc: Wendy Cheng; Atchley, Scott; Tom Tucker; linux-rdma@vger.kernel.org; linux-...@vger.kernel.org; Or Gerlitz Subject: Re:

Re: NFS over RDMA benchmark

On 4/30/2013 1:09 AM, Yan Burman wrote: I now get up to ~95K IOPS and 4.1GB/sec bandwidth. ... ib_send_bw with intel iommu enabled did get up to 4.5GB/sec BTW, you may want to verify that these are the same GB. Many benchmarks say KB/MB/GB when they really mean KiB/MiB/GiB. At GB/GiB, the

Re: NFS over RDMA benchmark

On 4/30/2013 10:23 AM, Yan Burman wrote: -Original Message- From: Tom Talpey [mailto:t...@talpey.com] On Sun, Apr 28, 2013 at 06:28:16AM +, Yan Burman wrote: I finally got up to 4.1GB/sec bandwidth with RDMA (ipoib-CM bandwidth is also way higher now). For some reason when I had

Re: NFS over RDMA benchmark

On 4/25/2013 1:18 PM, Wendy Cheng wrote: On Wed, Apr 24, 2013 at 11:26 AM, Tom Talpey t...@talpey.com wrote: On Wed, Apr 24, 2013 at 9:27 AM, Wendy Cheng s.wendy.ch...@gmail.com wrote: So I did a quick read on sunrpc/xprtrdma source (based on OFA 1.5.4.1 tar ball) ... Here is a random

Re: NFS over RDMA benchmark

On 4/25/2013 3:01 PM, Phil Pishioneri wrote: On 4/25/13 1:18 PM, Wendy Cheng wrote: On Wed, Apr 24, 2013 at 11:26 AM, Tom Talpey t...@talpey.com wrote: 1) The client slot count is not hard-coded, it can easily be changed by writing a value to /proc and initiating a new mount. But I doubt

Re: NFS over RDMA benchmark

On 4/24/2013 2:04 PM, Wendy Cheng wrote: On Wed, Apr 24, 2013 at 9:27 AM, Wendy Cheng s.wendy.ch...@gmail.com wrote: On Wed, Apr 24, 2013 at 8:26 AM, J. Bruce Fields bfie...@fieldses.org wrote: On Wed, Apr 24, 2013 at 11:05:40AM -0400, J. Bruce Fields wrote: On Wed, Apr 24, 2013 at 12:35:03PM