Re: [PATCH v3 09/11] SUNRPC: Introduce xprt_commit_rqst()
Hi Chuck,
Sorry for the last minute comment.
On 12/14/2015 04:19 PM, Chuck Lever wrote:
> I'm about to add code in the RPC/RDMA reply handler between the
> xprt_lookup_rqst() and xprt_complete_rqst() call site that needs
> to execute outside of spinlock critical sections.
>
> Add a hook to remove an rpc_rqst from the pending list once
> the transport knows its going to invoke xprt_complete_rqst().
>
> Signed-off-by: Chuck Lever
> ---
> include/linux/sunrpc/xprt.h|1 +
> net/sunrpc/xprt.c | 14 ++
> net/sunrpc/xprtrdma/rpc_rdma.c |4
> 3 files changed, 19 insertions(+)
>
> diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
> index 69ef5b3..ab6c3a5 100644
> --- a/include/linux/sunrpc/xprt.h
> +++ b/include/linux/sunrpc/xprt.h
> @@ -366,6 +366,7 @@ void
> xprt_wait_for_buffer_space(struct rpc_task *task, rpc_action action);
> void xprt_write_space(struct rpc_xprt *xprt);
> void xprt_adjust_cwnd(struct rpc_xprt *xprt, struct rpc_task
> *task, int result);
> struct rpc_rqst *xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid);
> +void xprt_commit_rqst(struct rpc_task *task);
> void xprt_complete_rqst(struct rpc_task *task, int copied);
> void xprt_release_rqst_cong(struct rpc_task *task);
> void xprt_disconnect_done(struct rpc_xprt *xprt);
> diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
> index 2e98f4a..a5be4ab 100644
> --- a/net/sunrpc/xprt.c
> +++ b/net/sunrpc/xprt.c
> @@ -837,6 +837,20 @@ static void xprt_update_rtt(struct rpc_task *task)
> }
>
> /**
> + * xprt_commit_rqst - remove rqst from pending list early
> + * @task: RPC request to remove
> + *
> + * Caller holds transport lock.
> + */
> +void xprt_commit_rqst(struct rpc_task *task)
> +{
> + struct rpc_rqst *req = task->tk_rqstp;
> +
> + list_del_init(&req->rq_list);
> +}
> +EXPORT_SYMBOL_GPL(xprt_commit_rqst);
Can you move this function into the xprtrdma code, since it's not called
outside of there?
Thanks,
Anna
> +
> +/**
> * xprt_complete_rqst - called when reply processing is complete
> * @task: RPC request that recently completed
> * @copied: actual number of bytes received from the transport
> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
> index c10d969..0bc8c39 100644
> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
> @@ -804,6 +804,9 @@ rpcrdma_reply_handler(struct rpcrdma_rep *rep)
> if (req->rl_reply)
> goto out_duplicate;
>
> + xprt_commit_rqst(rqst->rq_task);
> + spin_unlock_bh(&xprt->transport_lock);
> +
> dprintk("RPC: %s: reply 0x%p completes request 0x%p\n"
> " RPC request 0x%p xid 0x%08x\n",
> __func__, rep, req, rqst,
> @@ -894,6 +897,7 @@ badheader:
> else if (credits > r_xprt->rx_buf.rb_max_requests)
> credits = r_xprt->rx_buf.rb_max_requests;
>
> + spin_lock_bh(&xprt->transport_lock);
> cwnd = xprt->cwnd;
> xprt->cwnd = credits << RPC_CWNDSHIFT;
> if (xprt->cwnd > cwnd)
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/11] NFS/RDMA client patches for 4.5
Thanks, Chuck!
Everything looks okay to me, so I'll apply these patches and send them to Trond
before the holiday.
On 12/14/2015 04:17 PM, Chuck Lever wrote:
> For 4.5, I'd like to address the send queue accounting and
> invalidation/unmap ordering issues Jason brought up a couple of
> months ago.
>
> In preparation for Doug's final topic branch, Anna, I've rebased
> these on Christoph's ib_device_attr branch, but there were no merge
> conflicts or other changes needed. Could you begin preparing these
> for linux-next and other final testing and review?
No merge conflicts is nice, and we might not need to worry about ordering the
pull request.
Thanks,
Anna
>
> Also available in the "nfs-rdma-for-4.5" topic branch of this git repo:
>
> git://git.linux-nfs.org/projects/cel/cel-2.6.git
>
> Or for browsing:
>
> http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=log;h=refs/heads/nfs-rdma-for-4.5
>
>
> Changes since v2:
> - Rebased on Christoph's ib_device_attr branch
>
>
> Changes since v1:
>
> - Rebased on v4.4-rc3
> - Receive buffer safety margin patch dropped
> - Backchannel pr_err and pr_info converted to dprintk
> - Backchannel spin locks converted to work queue-safe locks
> - Fixed premature release of backchannel request buffer
> - NFSv4.1 callbacks tested with for-4.5 server
>
> ---
>
> Chuck Lever (11):
> xprtrdma: Fix additional uses of spin_lock_irqsave(rb_lock)
> xprtrdma: xprt_rdma_free() must not release backchannel reqs
> xprtrdma: Disable RPC/RDMA backchannel debugging messages
> xprtrdma: Move struct ib_send_wr off the stack
> xprtrdma: Introduce ro_unmap_sync method
> xprtrdma: Add ro_unmap_sync method for FRWR
> xprtrdma: Add ro_unmap_sync method for FMR
> xprtrdma: Add ro_unmap_sync method for all-physical registration
> SUNRPC: Introduce xprt_commit_rqst()
> xprtrdma: Invalidate in the RPC reply handler
> xprtrdma: Revert commit e7104a2a9606 ('xprtrdma: Cap req_cqinit').
>
>
> include/linux/sunrpc/xprt.h|1
> net/sunrpc/xprt.c | 14 +++
> net/sunrpc/xprtrdma/backchannel.c | 22 ++---
> net/sunrpc/xprtrdma/fmr_ops.c | 64 +
> net/sunrpc/xprtrdma/frwr_ops.c | 175
> +++-
> net/sunrpc/xprtrdma/physical_ops.c | 13 +++
> net/sunrpc/xprtrdma/rpc_rdma.c | 14 +++
> net/sunrpc/xprtrdma/transport.c|3 +
> net/sunrpc/xprtrdma/verbs.c| 13 +--
> net/sunrpc/xprtrdma/xprt_rdma.h| 12 +-
> 10 files changed, 283 insertions(+), 48 deletions(-)
>
> --
> Chuck Lever
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: device attr cleanup (was: Handle mlx4 max_sge_rd correctly)
On Thu, Dec 10, 2015 at 7:49 PM, Chuck Lever wrote: > >> On Dec 10, 2015, at 6:30 PM, Christoph Hellwig wrote: >> >> On Thu, Dec 10, 2015 at 11:07:03AM -0700, Jason Gunthorpe wrote: >>> The ARM folks do this sort of stuff on a regular basis.. Very early on >>> Doug prepares a topic branch with only the big change, NFS folks pull >>> it and then pull your work. Then Doug would send the topic branch to >>> Linus as soon as the merge window opens, then NFS would send theirs. >>> >>> This is alot less work overall than trying to sequence multiple >>> patches over multiple releases.. >> >> Agreed. Staging has alaways been a giant pain and things tend to never >> finish moving over that way if they are non-trivial enough. > > In that case: > > You need to make sure you have all the right Acks. I've added > Anna and Bruce to Ack the NFS-related portions. Santosh should > Ack the RDS part. The NFS client parts look okay to me: Acked-by: Anna Schumaker > > http://git.infradead.org/users/hch/rdma.git/shortlog/refs/heads/ib_device_attr > > Given the proximity to the holidays and the next merge window, > Doug will need to get a properly-acked topic branch published > in the next day or two so the rest of us can rebase and start > testing before the relevant parties disappear for the holiday. > > > -- > Chuck Lever > > > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1 7/9] SUNRPC: Introduct xprt_commit_rqst()
Hi Chuck,
On 11/23/2015 05:14 PM, Chuck Lever wrote:
> I'm about to add code in the RPC/RDMA reply handler between the
> xprt_lookup_rqst() and xprt_complete_rqst() call site that needs
> to execute outside of spinlock critical sections.
>
> Add a hook to remove an rpc_rqst from the pending list once
> the transport knows its going to invoke xprt_complete_rqst().
>
> Signed-off-by: Chuck Lever
> ---
> include/linux/sunrpc/xprt.h|1 +
> net/sunrpc/xprt.c | 14 ++
> net/sunrpc/xprtrdma/rpc_rdma.c |4
> 3 files changed, 19 insertions(+)
>
> diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
> index 69ef5b3..ab6c3a5 100644
> --- a/include/linux/sunrpc/xprt.h
> +++ b/include/linux/sunrpc/xprt.h
> @@ -366,6 +366,7 @@ void
> xprt_wait_for_buffer_space(struct rpc_task *task, rpc_action action);
> void xprt_write_space(struct rpc_xprt *xprt);
> void xprt_adjust_cwnd(struct rpc_xprt *xprt, struct rpc_task
> *task, int result);
> struct rpc_rqst *xprt_lookup_rqst(struct rpc_xprt *xprt, __be32 xid);
> +void xprt_commit_rqst(struct rpc_task *task);
> void xprt_complete_rqst(struct rpc_task *task, int copied);
> void xprt_release_rqst_cong(struct rpc_task *task);
> void xprt_disconnect_done(struct rpc_xprt *xprt);
> diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
> index 2e98f4a..a5be4ab 100644
> --- a/net/sunrpc/xprt.c
> +++ b/net/sunrpc/xprt.c
> @@ -837,6 +837,20 @@ static void xprt_update_rtt(struct rpc_task *task)
> }
>
> /**
> + * xprt_commit_rqst - remove rqst from pending list early
> + * @task: RPC request to remove
Is xprt_commit_rqst() the right name for this function? Removing a request
from a list isn't how I would expect a commit to work.
Anna
> + *
> + * Caller holds transport lock.
> + */
> +void xprt_commit_rqst(struct rpc_task *task)
> +{
> + struct rpc_rqst *req = task->tk_rqstp;
> +
> + list_del_init(&req->rq_list);
> +}
> +EXPORT_SYMBOL_GPL(xprt_commit_rqst);
> +
> +/**
> * xprt_complete_rqst - called when reply processing is complete
> * @task: RPC request that recently completed
> * @copied: actual number of bytes received from the transport
> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
> index a169252..d7b9156 100644
> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
> @@ -811,6 +811,9 @@ rpcrdma_reply_handler(struct rpcrdma_rep *rep)
> if (req->rl_reply)
> goto out_duplicate;
>
> + xprt_commit_rqst(rqst->rq_task);
> + spin_unlock_bh(&xprt->transport_lock);
> +
> dprintk("RPC: %s: reply 0x%p completes request 0x%p\n"
> " RPC request 0x%p xid 0x%08x\n",
> __func__, rep, req, rqst,
> @@ -901,6 +904,7 @@ badheader:
> else if (credits > r_xprt->rx_buf.rb_max_requests)
> credits = r_xprt->rx_buf.rb_max_requests;
>
> + spin_lock_bh(&xprt->transport_lock);
> cwnd = xprt->cwnd;
> xprt->cwnd = credits << RPC_CWNDSHIFT;
> if (xprt->cwnd > cwnd)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 16/16] NFS: Enable client side NFSv4.1 backchannel to use other transports
On 11/02/2015 04:16 PM, Chuck Lever wrote:
>
>> On Nov 2, 2015, at 3:43 PM, Anna Schumaker wrote:
>>
>> Hi Chuck,
>>
>> Sorry for the delay in reviewing.
>>
>> On 10/24/2015 05:28 PM, Chuck Lever wrote:
>>> Forechannel transports get their own "bc_up" method to create an
>>> endpoint for the backchannel service.
>>>
>>> Signed-off-by: Chuck Lever
>>> ---
>>> fs/nfs/callback.c | 40
>>> +++--
>>> include/linux/sunrpc/xprt.h |2 ++
>>> net/sunrpc/xprtrdma/backchannel.c | 21 +++
>>> net/sunrpc/xprtrdma/transport.c |1 +
>>> net/sunrpc/xprtrdma/xprt_rdma.h |1 +
>>> net/sunrpc/xprtsock.c | 12 +++
>>> 6 files changed, 45 insertions(+), 32 deletions(-)
>>>
>>> diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
>>> index 75f7c0a..a7f2e6e 100644
>>> --- a/fs/nfs/callback.c
>>> +++ b/fs/nfs/callback.c
>>> @@ -99,17 +99,6 @@ nfs4_callback_up(struct svc_serv *serv)
>>> }
>>>
>>> #if defined(CONFIG_NFS_V4_1)
>>> -static int nfs41_callback_up_net(struct svc_serv *serv, struct net *net)
>>> -{
>>> - /*
>>> -* Create an svc_sock for the back channel service that shares the
>>> -* fore channel connection.
>>> -* Returns the input port (0) and sets the svc_serv bc_xprt on success
>>> -*/
>>> - return svc_create_xprt(serv, "tcp-bc", net, PF_INET, 0,
>>> - SVC_SOCK_ANONYMOUS);
>>> -}
>>> -
>>> /*
>>> * The callback service for NFSv4.1 callbacks
>>> */
>>> @@ -184,11 +173,6 @@ static inline void nfs_callback_bc_serv(u32
>>> minorversion, struct rpc_xprt *xprt,
>>> xprt->bc_serv = serv;
>>> }
>>> #else
>>> -static int nfs41_callback_up_net(struct svc_serv *serv, struct net *net)
>>> -{
>>> - return 0;
>>> -}
>>> -
>>> static void nfs_minorversion_callback_svc_setup(struct svc_serv *serv,
>>> struct svc_rqst **rqstpp, int (**callback_svc)(void *vrqstp))
>>> {
>>> @@ -259,7 +243,8 @@ static void nfs_callback_down_net(u32 minorversion,
>>> struct svc_serv *serv, struc
>>> svc_shutdown_net(serv, net);
>>> }
>>>
>>> -static int nfs_callback_up_net(int minorversion, struct svc_serv *serv,
>>> struct net *net)
>>> +static int nfs_callback_up_net(int minorversion, struct svc_serv *serv,
>>> + struct net *net, struct rpc_xprt *xprt)
>>> {
>>> struct nfs_net *nn = net_generic(net, nfs_net_id);
>>> int ret;
>>> @@ -275,20 +260,11 @@ static int nfs_callback_up_net(int minorversion,
>>> struct svc_serv *serv, struct n
>>> goto err_bind;
>>> }
>>>
>>> - switch (minorversion) {
>>> - case 0:
>>> - ret = nfs4_callback_up_net(serv, net);
>>> - break;
>>> - case 1:
>>> - case 2:
>>> - ret = nfs41_callback_up_net(serv, net);
>>> - break;
>>> - default:
>>> - printk(KERN_ERR "NFS: unknown callback version: %d\n",
>>> - minorversion);
>>> - ret = -EINVAL;
>>> - break;
>>> - }
>>> + ret = -EPROTONOSUPPORT;
>>> + if (minorversion == 0)
>>> + ret = nfs4_callback_up_net(serv, net);
>>> + else if (xprt->ops->bc_up)
>>> + ret = xprt->ops->bc_up(serv, net);
>>>
>>> if (ret < 0) {
>>> printk(KERN_ERR "NFS: callback service start failed\n");
>>> @@ -364,7 +340,7 @@ int nfs_callback_up(u32 minorversion, struct rpc_xprt
>>> *xprt)
>>> goto err_create;
>>> }
>>>
>>> - ret = nfs_callback_up_net(minorversion, serv, net);
>>> + ret = nfs_callback_up_net(minorversion, serv, net, xprt);
>>> if (ret < 0)
>>> goto err_net;
>>>
>>> diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
>>> index 82c0839..9d22f43 100644
>>> --- a/include/linux/sunrpc/xprt.h
>>> +++ b/include/linux/sunrpc/xpr
Re: [PATCH v4 16/16] NFS: Enable client side NFSv4.1 backchannel to use other transports
Hi Chuck,
Sorry for the delay in reviewing.
On 10/24/2015 05:28 PM, Chuck Lever wrote:
> Forechannel transports get their own "bc_up" method to create an
> endpoint for the backchannel service.
>
> Signed-off-by: Chuck Lever
> ---
> fs/nfs/callback.c | 40
> +++--
> include/linux/sunrpc/xprt.h |2 ++
> net/sunrpc/xprtrdma/backchannel.c | 21 +++
> net/sunrpc/xprtrdma/transport.c |1 +
> net/sunrpc/xprtrdma/xprt_rdma.h |1 +
> net/sunrpc/xprtsock.c | 12 +++
> 6 files changed, 45 insertions(+), 32 deletions(-)
>
> diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
> index 75f7c0a..a7f2e6e 100644
> --- a/fs/nfs/callback.c
> +++ b/fs/nfs/callback.c
> @@ -99,17 +99,6 @@ nfs4_callback_up(struct svc_serv *serv)
> }
>
> #if defined(CONFIG_NFS_V4_1)
> -static int nfs41_callback_up_net(struct svc_serv *serv, struct net *net)
> -{
> - /*
> - * Create an svc_sock for the back channel service that shares the
> - * fore channel connection.
> - * Returns the input port (0) and sets the svc_serv bc_xprt on success
> - */
> - return svc_create_xprt(serv, "tcp-bc", net, PF_INET, 0,
> - SVC_SOCK_ANONYMOUS);
> -}
> -
> /*
> * The callback service for NFSv4.1 callbacks
> */
> @@ -184,11 +173,6 @@ static inline void nfs_callback_bc_serv(u32
> minorversion, struct rpc_xprt *xprt,
> xprt->bc_serv = serv;
> }
> #else
> -static int nfs41_callback_up_net(struct svc_serv *serv, struct net *net)
> -{
> - return 0;
> -}
> -
> static void nfs_minorversion_callback_svc_setup(struct svc_serv *serv,
> struct svc_rqst **rqstpp, int (**callback_svc)(void *vrqstp))
> {
> @@ -259,7 +243,8 @@ static void nfs_callback_down_net(u32 minorversion,
> struct svc_serv *serv, struc
> svc_shutdown_net(serv, net);
> }
>
> -static int nfs_callback_up_net(int minorversion, struct svc_serv *serv,
> struct net *net)
> +static int nfs_callback_up_net(int minorversion, struct svc_serv *serv,
> +struct net *net, struct rpc_xprt *xprt)
> {
> struct nfs_net *nn = net_generic(net, nfs_net_id);
> int ret;
> @@ -275,20 +260,11 @@ static int nfs_callback_up_net(int minorversion, struct
> svc_serv *serv, struct n
> goto err_bind;
> }
>
> - switch (minorversion) {
> - case 0:
> - ret = nfs4_callback_up_net(serv, net);
> - break;
> - case 1:
> - case 2:
> - ret = nfs41_callback_up_net(serv, net);
> - break;
> - default:
> - printk(KERN_ERR "NFS: unknown callback version: %d\n",
> - minorversion);
> - ret = -EINVAL;
> - break;
> - }
> + ret = -EPROTONOSUPPORT;
> + if (minorversion == 0)
> + ret = nfs4_callback_up_net(serv, net);
> + else if (xprt->ops->bc_up)
> + ret = xprt->ops->bc_up(serv, net);
>
> if (ret < 0) {
> printk(KERN_ERR "NFS: callback service start failed\n");
> @@ -364,7 +340,7 @@ int nfs_callback_up(u32 minorversion, struct rpc_xprt
> *xprt)
> goto err_create;
> }
>
> - ret = nfs_callback_up_net(minorversion, serv, net);
> + ret = nfs_callback_up_net(minorversion, serv, net, xprt);
> if (ret < 0)
> goto err_net;
>
> diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
> index 82c0839..9d22f43 100644
> --- a/include/linux/sunrpc/xprt.h
> +++ b/include/linux/sunrpc/xprt.h
> @@ -54,6 +54,7 @@ enum rpc_display_format_t {
> struct rpc_task;
> struct rpc_xprt;
> struct seq_file;
> +struct svc_serv;
>
> /*
> * This describes a complete RPC request
> @@ -138,6 +139,7 @@ struct rpc_xprt_ops {
> void(*inject_disconnect)(struct rpc_xprt *xprt);
> int (*bc_setup)(struct rpc_xprt *xprt,
> unsigned int min_reqs);
> + int (*bc_up)(struct svc_serv *serv, struct net *net);
Disabling CONFIG_SUNRPC_DEBUG gives me several:
CC [M] fs/nfsd/nfs4acl.o
CC [M] fs/nfsd/nfs4callback.o
In file included from include/linux/sunrpc/clnt.h:19:0,
from fs/nfsd/nfs4callback.c:34:
include/linux/sunrpc/xprt.h:142:46: error: 'struct net' declared inside
parameter list [-Werror]
int (*bc_up)(struct svc_serv *serv, struct net *net);
^
include/linux/sunrpc/xprt.h:142:46: error: its scope is only this definition or
declaration, which is probably not what you want [-Werror]
I'm guessing an include file is missing somewhere?
Anna
> void(*bc_free_rqst)(struct rpc_rqst *rqst);
> void(*bc_destroy)(struct rpc_xprt *xprt,
>
Re: [GIT PULL] Please pull NFSoRDMA changes for 4.4
On 10/24/2015 02:41 PM, Chuck Lever wrote: > >> On Oct 23, 2015, at 4:44 PM, Anna Schumaker >> wrote: >> >> Hi Trond, >> >> The following changes since commit 7379047d5585187d1288486d4627873170d0005a: >> >> Linux 4.3-rc6 (2015-10-18 16:08:42 -0700) >> >> are available in the git repository at: >> >> git://git.linux-nfs.org/projects/anna/nfs-rdma.git tags/nfs-rdma-4.4 >> >> for you to fetch changes up to 50add18fd4c82cc1eaedc6ce879d998a66f20e1f: >> >> NFS: Enable client side NFSv4.1 backchannel to use other transports >> (2015-10-19 14:48:18 -0400) > > Anna, did you already send Steve Wise’s fix for 4.4? Oops! I think I forgot that patch. I'll include it with the v4 pull request today. Anna > > http://marc.info/?l=linux-nfs&m=144285627709672&w=2 > > >> Thanks, >> Anna >> >> >> Chuck Lever (16): >> xprtrdma: Enable swap-on-NFS/RDMA >> xprtrdma: Re-arm after missed events >> xprtrdma: Prevent loss of completion signals >> xprtrdma: Refactor reply handler error handling >> xprtrdma: Replace send and receive arrays >> xprtrdma: Use workqueue to process RPC/RDMA replies >> xprtrdma: Remove reply tasklet >> xprtrdma: Saving IRQs no longer needed for rb_lock >> SUNRPC: Abstract backchannel operations >> xprtrdma: Pre-allocate backward rpc_rqst and send/receive buffers >> xprtrdma: Pre-allocate Work Requests for backchannel >> xprtrdma: Add support for sending backward direction RPC replies >> xprtrdma: Handle incoming backward direction RPC calls >> svcrdma: Add backward direction service for RPC/RDMA transport >> SUNRPC: Remove the TCP-only restriction in bc_svc_process() >> NFS: Enable client side NFSv4.1 backchannel to use other transports >> >> fs/nfs/callback.c| 33 +-- >> include/linux/sunrpc/bc_xprt.h | 5 +++ >> include/linux/sunrpc/svc_rdma.h | 6 ++- >> include/linux/sunrpc/xprt.h | 7 >> net/sunrpc/backchannel_rqst.c| 24 ++- >> net/sunrpc/svc.c | 5 --- >> net/sunrpc/xprtrdma/Makefile | 1 + >> net/sunrpc/xprtrdma/backchannel.c| 373 >> >> net/sunrpc/xprtrdma/rpc_rdma.c | 148 >> + >> net/sunrpc/xprtrdma/svc_rdma.c | 6 +++ >> net/sunrpc/xprtrdma/svc_rdma_transport.c | 58 ++ >> net/sunrpc/xprtrdma/transport.c | 18 +++- >> net/sunrpc/xprtrdma/verbs.c | 479 >> ++- >> net/sunrpc/xprtrdma/xprt_rdma.h | 53 +++- >> net/sunrpc/xprtsock.c| 16 >> 15 files changed, 916 insertions(+), 316 deletions(-) >> create mode 100644 net/sunrpc/xprtrdma/backchannel.c >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > — > Chuck Lever > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] Please pull NFSoRDMA changes for 4.4
Hi Trond, The following changes since commit 7379047d5585187d1288486d4627873170d0005a: Linux 4.3-rc6 (2015-10-18 16:08:42 -0700) are available in the git repository at: git://git.linux-nfs.org/projects/anna/nfs-rdma.git tags/nfs-rdma-4.4 for you to fetch changes up to 50add18fd4c82cc1eaedc6ce879d998a66f20e1f: NFS: Enable client side NFSv4.1 backchannel to use other transports (2015-10-19 14:48:18 -0400) Thanks, Anna Chuck Lever (16): xprtrdma: Enable swap-on-NFS/RDMA xprtrdma: Re-arm after missed events xprtrdma: Prevent loss of completion signals xprtrdma: Refactor reply handler error handling xprtrdma: Replace send and receive arrays xprtrdma: Use workqueue to process RPC/RDMA replies xprtrdma: Remove reply tasklet xprtrdma: Saving IRQs no longer needed for rb_lock SUNRPC: Abstract backchannel operations xprtrdma: Pre-allocate backward rpc_rqst and send/receive buffers xprtrdma: Pre-allocate Work Requests for backchannel xprtrdma: Add support for sending backward direction RPC replies xprtrdma: Handle incoming backward direction RPC calls svcrdma: Add backward direction service for RPC/RDMA transport SUNRPC: Remove the TCP-only restriction in bc_svc_process() NFS: Enable client side NFSv4.1 backchannel to use other transports fs/nfs/callback.c| 33 +-- include/linux/sunrpc/bc_xprt.h | 5 +++ include/linux/sunrpc/svc_rdma.h | 6 ++- include/linux/sunrpc/xprt.h | 7 net/sunrpc/backchannel_rqst.c| 24 ++- net/sunrpc/svc.c | 5 --- net/sunrpc/xprtrdma/Makefile | 1 + net/sunrpc/xprtrdma/backchannel.c| 373 net/sunrpc/xprtrdma/rpc_rdma.c | 148 + net/sunrpc/xprtrdma/svc_rdma.c | 6 +++ net/sunrpc/xprtrdma/svc_rdma_transport.c | 58 ++ net/sunrpc/xprtrdma/transport.c | 18 +++- net/sunrpc/xprtrdma/verbs.c | 479 ++- net/sunrpc/xprtrdma/xprt_rdma.h | 53 +++- net/sunrpc/xprtsock.c| 16 15 files changed, 916 insertions(+), 316 deletions(-) create mode 100644 net/sunrpc/xprtrdma/backchannel.c -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/16] NFS/RDMA patches for merging into v4.4
Thanks! I'll push these out to Trond by the end of the week. Anna On 10/16/2015 09:23 AM, Chuck Lever wrote: > Introduce client-side support for bi-directional RPC/RDMA. > Bi-directional RPC/RDMA is a pre-requisite for NFSv4.1 on RDMA > transports. > > Anna, these are ready for you to consider merging. > > Also available in the "nfs-rdma-for-4.4" topic branch of this git repo: > > git://git.linux-nfs.org/projects/cel/cel-2.6.git > > Or for browsing: > > http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=log;h=refs/heads/nfs-rdma-for-4.4 > > Changes since v2: > - Rebased on v4.3-rc5 > - Updated Reviewed-by: and Tested-by: tags > - Some interop testing at NFS Bake-a-thon > > > Changes since v1: > - Dropped "RFC" in Subject: line > - Rebased on v4.3-rc4 + Steve W's recent fixes > - NFS Server-side backchannel support postponed > - "xprtrdma: Replace global lkey" dropped, already merged > - Addressed Sagi's comments on "Replace send and receive arrays" > - Addressed Jason's comment regarding ib_req_notify_cq return code > - Moved RPC/RDMA reply handling into a work queue > > --- > > Chuck Lever (16): > xprtrdma: Enable swap-on-NFS/RDMA > xprtrdma: Re-arm after missed events > xprtrdma: Prevent loss of completion signals > xprtrdma: Refactor reply handler error handling > xprtrdma: Replace send and receive arrays > xprtrdma: Use workqueue to process RPC/RDMA replies > xprtrdma: Remove reply tasklet > xprtrdma: Saving IRQs no longer needed for rb_lock > SUNRPC: Abstract backchannel operations > xprtrdma: Pre-allocate backward rpc_rqst and send/receive buffers > xprtrdma: Pre-allocate Work Requests for backchannel > xprtrdma: Add support for sending backward direction RPC replies > xprtrdma: Handle incoming backward direction RPC calls > svcrdma: Add backward direction service for RPC/RDMA transport > SUNRPC: Remove the TCP-only restriction in bc_svc_process() > NFS: Enable client side NFSv4.1 backchannel to use other transports > > > fs/nfs/callback.c| 33 +- > include/linux/sunrpc/bc_xprt.h |5 > include/linux/sunrpc/svc_rdma.h |6 > include/linux/sunrpc/xprt.h |7 > net/sunrpc/backchannel_rqst.c| 24 +- > net/sunrpc/svc.c |5 > net/sunrpc/xprtrdma/Makefile |1 > net/sunrpc/xprtrdma/backchannel.c| 373 +++ > net/sunrpc/xprtrdma/rpc_rdma.c | 148 ++--- > net/sunrpc/xprtrdma/svc_rdma.c |6 > net/sunrpc/xprtrdma/svc_rdma_transport.c | 58 > net/sunrpc/xprtrdma/transport.c | 18 + > net/sunrpc/xprtrdma/verbs.c | 479 > +++--- > net/sunrpc/xprtrdma/xprt_rdma.h | 53 +++ > net/sunrpc/xprtsock.c| 16 + > 15 files changed, 916 insertions(+), 316 deletions(-) > create mode 100644 net/sunrpc/xprtrdma/backchannel.c > > -- > Chuck Lever > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 12/26] xprtrdma: Port to new memory registration API
Hi Sagi,
On 10/08/2015 03:50 AM, Sagi Grimberg wrote:
> Instead of maintaining a fastreg page list, keep an sg table
> and convert an array of pages to a sg list. Then call ib_map_mr_sg
> and construct ib_reg_wr.
>
> Note that the next step would be to have NFS work with sg lists
> as it maps well to sk_frags (see comment from hch
> http://marc.info/?l=linux-rdma&m=143677002622296&w=2).
Looks good to me!
Acked-by: Anna Schumaker
>
> Signed-off-by: Sagi Grimberg
> Acked-by: Christoph Hellwig
> Tested-by: Steve Wise
> Tested-by: Selvin Xavier
> ---
> net/sunrpc/xprtrdma/frwr_ops.c | 112
> +++-
> net/sunrpc/xprtrdma/xprt_rdma.h | 3 +-
> 2 files changed, 67 insertions(+), 48 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
> index 0d2f46f600b6..3c89f8052786 100644
> --- a/net/sunrpc/xprtrdma/frwr_ops.c
> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
> @@ -151,9 +151,13 @@ __frwr_init(struct rpcrdma_mw *r, struct ib_pd *pd,
> struct ib_device *device,
> f->fr_mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG, depth);
> if (IS_ERR(f->fr_mr))
> goto out_mr_err;
> - f->fr_pgl = ib_alloc_fast_reg_page_list(device, depth);
> - if (IS_ERR(f->fr_pgl))
> +
> + f->sg = kcalloc(depth, sizeof(*f->sg), GFP_KERNEL);
> + if (!f->sg)
> goto out_list_err;
> +
> + sg_init_table(f->sg, depth);
> +
> return 0;
>
> out_mr_err:
> @@ -163,7 +167,7 @@ out_mr_err:
> return rc;
>
> out_list_err:
> - rc = PTR_ERR(f->fr_pgl);
> + rc = -ENOMEM;
> dprintk("RPC: %s: ib_alloc_fast_reg_page_list status %i\n",
> __func__, rc);
> ib_dereg_mr(f->fr_mr);
> @@ -179,7 +183,7 @@ __frwr_release(struct rpcrdma_mw *r)
> if (rc)
> dprintk("RPC: %s: ib_dereg_mr status %i\n",
> __func__, rc);
> - ib_free_fast_reg_page_list(r->r.frmr.fr_pgl);
> + kfree(r->r.frmr.sg);
> }
>
> static int
> @@ -312,14 +316,11 @@ frwr_op_map(struct rpcrdma_xprt *r_xprt, struct
> rpcrdma_mr_seg *seg,
> struct rpcrdma_mw *mw;
> struct rpcrdma_frmr *frmr;
> struct ib_mr *mr;
> - struct ib_fast_reg_wr fastreg_wr;
> + struct ib_reg_wr reg_wr;
> struct ib_send_wr *bad_wr;
> + unsigned int dma_nents;
> u8 key;
> - int len, pageoff;
> - int i, rc;
> - int seg_len;
> - u64 pa;
> - int page_no;
> + int i, rc, len, n;
>
> mw = seg1->rl_mw;
> seg1->rl_mw = NULL;
> @@ -332,64 +333,81 @@ frwr_op_map(struct rpcrdma_xprt *r_xprt, struct
> rpcrdma_mr_seg *seg,
> } while (mw->r.frmr.fr_state != FRMR_IS_INVALID);
> frmr = &mw->r.frmr;
> frmr->fr_state = FRMR_IS_VALID;
> + mr = frmr->fr_mr;
>
> - pageoff = offset_in_page(seg1->mr_offset);
> - seg1->mr_offset -= pageoff; /* start of page */
> - seg1->mr_len += pageoff;
> - len = -pageoff;
> if (nsegs > ia->ri_max_frmr_depth)
> nsegs = ia->ri_max_frmr_depth;
>
> - for (page_no = i = 0; i < nsegs;) {
> - rpcrdma_map_one(device, seg, direction);
> - pa = seg->mr_dma;
> - for (seg_len = seg->mr_len; seg_len > 0; seg_len -= PAGE_SIZE) {
> - frmr->fr_pgl->page_list[page_no++] = pa;
> - pa += PAGE_SIZE;
> - }
> + for (len = 0, i = 0; i < nsegs;) {
> + if (seg->mr_page)
> + sg_set_page(&frmr->sg[i],
> + seg->mr_page,
> + seg->mr_len,
> + offset_in_page(seg->mr_offset));
> + else
> + sg_set_buf(&frmr->sg[i], seg->mr_offset,
> +seg->mr_len);
> +
> len += seg->mr_len;
> ++seg;
> ++i;
> +
> /* Check for holes */
> if ((i < nsegs && offset_in_page(seg->mr_offset)) ||
> offset_in_page((seg-1)->mr_offset + (seg-1)->mr_len))
> break;
> }
> + frmr->sg_nents = i;
> +
> + dma_nents = ib_dma_map_sg(device, frmr->sg, frmr->sg_nents, direction);
> + if (!dma_nents) {
> + pr_err("RPC: %s: failed to dma map sg %p sg_nents %d\n",
> +__func__, frmr->sg, frmr-
Re: [PATCH rdma-rc v1] xprtrdma: Don't require LOCAL_DMA_LKEY support for fasterg
On 10/06/2015 01:36 PM, Doug Ledford wrote:
> On 10/06/2015 01:25 PM, Anna Schumaker wrote:
>> Hi Sagi,
>>
>> On 10/06/2015 12:52 PM, Sagi Grimberg wrote:
>>> There is no need to require LOCAL_DMA_LKEY support as the
>>> PD allocation makes sure that there is a local_dma_lkey. Also
>>> correctly set a return value in error path.
>>>
>>> This caused a NULL pointer dereference in mlx5 which removed
>>> the support for LOCAL_DMA_LKEY.
>>
>> Looks good to me! This is another patch going directly to an RDMA tree
>> somewhere, right?
>
> Yeah, I'll pick this one up. Thanks!
Cool! In that case you can add:
Acked-by: Anna Schumaker
>
>> Anna
>>
>>>
>>> Fixes: bb6c96d72879 ("xprtrdma: Replace global lkey with lkey local to PD")
>>> Signed-off-by: Sagi Grimberg
>>> Reviewed-by: Chuck Lever
>>> ---
>>> Changes from v0:
>>> - Added error path rc assignment.
>>>
>>> net/sunrpc/xprtrdma/verbs.c |8 +++-
>>> 1 files changed, 3 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
>>> index eb081ad..81e8d31 100644
>>> --- a/net/sunrpc/xprtrdma/verbs.c
>>> +++ b/net/sunrpc/xprtrdma/verbs.c
>>> @@ -543,11 +543,8 @@ rpcrdma_ia_open(struct rpcrdma_xprt *xprt, struct
>>> sockaddr *addr, int memreg)
>>> }
>>>
>>> if (memreg == RPCRDMA_FRMR) {
>>> - /* Requires both frmr reg and local dma lkey */
>>> - if (((devattr->device_cap_flags &
>>> -(IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) !=
>>> - (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) ||
>>> - (devattr->max_fast_reg_page_list_len == 0)) {
>>> + if (!(devattr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS)
>>> ||
>>> + (devattr->max_fast_reg_page_list_len == 0)) {
>>> dprintk("RPC: %s: FRMR registration "
>>> "not supported by HCA\n", __func__);
>>> memreg = RPCRDMA_MTHCAFMR;
>>> @@ -557,6 +554,7 @@ rpcrdma_ia_open(struct rpcrdma_xprt *xprt, struct
>>> sockaddr *addr, int memreg)
>>> if (!ia->ri_device->alloc_fmr) {
>>> dprintk("RPC: %s: MTHCAFMR registration "
>>> "not supported by HCA\n", __func__);
>>> + rc = -EINVAL;
>>> goto out3;
>>> }
>>> }
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH rdma-rc v1] xprtrdma: Don't require LOCAL_DMA_LKEY support for fasterg
Hi Sagi,
On 10/06/2015 12:52 PM, Sagi Grimberg wrote:
> There is no need to require LOCAL_DMA_LKEY support as the
> PD allocation makes sure that there is a local_dma_lkey. Also
> correctly set a return value in error path.
>
> This caused a NULL pointer dereference in mlx5 which removed
> the support for LOCAL_DMA_LKEY.
Looks good to me! This is another patch going directly to an RDMA tree
somewhere, right?
Anna
>
> Fixes: bb6c96d72879 ("xprtrdma: Replace global lkey with lkey local to PD")
> Signed-off-by: Sagi Grimberg
> Reviewed-by: Chuck Lever
> ---
> Changes from v0:
> - Added error path rc assignment.
>
> net/sunrpc/xprtrdma/verbs.c |8 +++-
> 1 files changed, 3 insertions(+), 5 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index eb081ad..81e8d31 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -543,11 +543,8 @@ rpcrdma_ia_open(struct rpcrdma_xprt *xprt, struct
> sockaddr *addr, int memreg)
> }
>
> if (memreg == RPCRDMA_FRMR) {
> - /* Requires both frmr reg and local dma lkey */
> - if (((devattr->device_cap_flags &
> - (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) !=
> - (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) ||
> - (devattr->max_fast_reg_page_list_len == 0)) {
> + if (!(devattr->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS)
> ||
> + (devattr->max_fast_reg_page_list_len == 0)) {
> dprintk("RPC: %s: FRMR registration "
> "not supported by HCA\n", __func__);
> memreg = RPCRDMA_MTHCAFMR;
> @@ -557,6 +554,7 @@ rpcrdma_ia_open(struct rpcrdma_xprt *xprt, struct
> sockaddr *addr, int memreg)
> if (!ia->ri_device->alloc_fmr) {
> dprintk("RPC: %s: MTHCAFMR registration "
> "not supported by HCA\n", __func__);
> + rc = -EINVAL;
> goto out3;
> }
> }
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3] xprtrdma: disconnect and flush cqs before freeing buffers
On 09/28/2015 10:50 AM, Steve Wise wrote: > > >> -Original Message----- >> From: Anna Schumaker [mailto:anna.schuma...@netapp.com] >> Sent: Monday, September 28, 2015 9:45 AM >> To: Steve Wise; trond.mykleb...@primarydata.com; bfie...@fieldses.org >> Cc: linux-...@vger.kernel.org; linux-rdma@vger.kernel.org >> Subject: Re: [PATCH 1/3] xprtrdma: disconnect and flush cqs before freeing >> buffers >> >> Hi Steve, >> >> On 09/28/2015 10:30 AM, Steve Wise wrote: >>> On 9/21/2015 12:24 PM, Steve Wise wrote: >>>> Otherwise a FRMR completion can cause a touch-after-free crash. >>>> >>>> In xprt_rdma_destroy(), call rpcrdma_buffer_destroy() only after calling >>>> rpcrdma_ep_destroy(). >>>> >>>> In rpcrdma_ep_destroy(), disconnect the cm_id first which should flush the >>>> qp, then drain the cqs, then destroy the qp, and finally destroy the cqs. >>>> >>>> Signed-off-by: Steve Wise >>>> Tested-by: Chuck Lever >>>> --- >>> >>> Hey Trond, I'm hoping this can make 4.3-rc (and stable if you agree). >> >> This patch looks fine to me. I'll pass it on to Trond! >> >> I'll save patch 3/3 for the Linux 4.4 merge. >> >> Thanks, >> Anna >> > > Thanks. Going forward I'll make sure you are CC'd for client patches too! I > wasn't sure if you are formally taking all xprtrdma patches and sending them > to Trond... Yeah, I'm taking all the client RDMA patches. I'm glad that's cleared up now! :) Anna > > Steve. > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3] xprtrdma: disconnect and flush cqs before freeing buffers
Hi Steve, On 09/28/2015 10:30 AM, Steve Wise wrote: > On 9/21/2015 12:24 PM, Steve Wise wrote: >> Otherwise a FRMR completion can cause a touch-after-free crash. >> >> In xprt_rdma_destroy(), call rpcrdma_buffer_destroy() only after calling >> rpcrdma_ep_destroy(). >> >> In rpcrdma_ep_destroy(), disconnect the cm_id first which should flush the >> qp, then drain the cqs, then destroy the qp, and finally destroy the cqs. >> >> Signed-off-by: Steve Wise >> Tested-by: Chuck Lever >> --- > > Hey Trond, I'm hoping this can make 4.3-rc (and stable if you agree). This patch looks fine to me. I'll pass it on to Trond! I'll save patch 3/3 for the Linux 4.4 merge. Thanks, Anna > > Thanks, > > Steve. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH rdma-rc v2 1/4] xprtrdma: Replace global lkey with lkey local to PD
Hi Sagi,
On 09/24/2015 03:34 AM, Sagi Grimberg wrote:
> From: Chuck Lever
>
> The core API has changed so that devices that do not have a global
> DMA lkey automatically create an mr, per-PD, and make that lkey
> available. The global DMA lkey interface is going away in favor of
> the per-PD DMA lkey.
>
> The per-PD DMA lkey is always available. Convert xprtrdma to use the
> device's per-PD DMA lkey for regbufs, no matter which memory
> registration scheme is in use.
This is going through an RDMA tree somewhere, correct? You can add my:
Acked-by: Anna Schumaker
>
> Signed-off-by: Chuck Lever
> Signed-off-by: Sagi Grimberg
> Cc: linux-nfs
> ---
> net/sunrpc/xprtrdma/fmr_ops.c | 19 ---
> net/sunrpc/xprtrdma/frwr_ops.c | 5 -
> net/sunrpc/xprtrdma/physical_ops.c | 10 +-
> net/sunrpc/xprtrdma/verbs.c| 2 +-
> net/sunrpc/xprtrdma/xprt_rdma.h| 1 -
> 5 files changed, 2 insertions(+), 35 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/fmr_ops.c b/net/sunrpc/xprtrdma/fmr_ops.c
> index cb25c89da623..f1e8dafbd507 100644
> --- a/net/sunrpc/xprtrdma/fmr_ops.c
> +++ b/net/sunrpc/xprtrdma/fmr_ops.c
> @@ -39,25 +39,6 @@ static int
> fmr_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep,
> struct rpcrdma_create_data_internal *cdata)
> {
> - struct ib_device_attr *devattr = &ia->ri_devattr;
> - struct ib_mr *mr;
> -
> - /* Obtain an lkey to use for the regbufs, which are
> - * protected from remote access.
> - */
> - if (devattr->device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY) {
> - ia->ri_dma_lkey = ia->ri_device->local_dma_lkey;
> - } else {
> - mr = ib_get_dma_mr(ia->ri_pd, IB_ACCESS_LOCAL_WRITE);
> - if (IS_ERR(mr)) {
> - pr_err("%s: ib_get_dma_mr for failed with %lX\n",
> -__func__, PTR_ERR(mr));
> - return -ENOMEM;
> - }
> - ia->ri_dma_lkey = ia->ri_dma_mr->lkey;
> - ia->ri_dma_mr = mr;
> - }
> -
> return 0;
> }
>
> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
> index d6653f5d0830..5318951b3b53 100644
> --- a/net/sunrpc/xprtrdma/frwr_ops.c
> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
> @@ -189,11 +189,6 @@ frwr_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep
> *ep,
> struct ib_device_attr *devattr = &ia->ri_devattr;
> int depth, delta;
>
> - /* Obtain an lkey to use for the regbufs, which are
> - * protected from remote access.
> - */
> - ia->ri_dma_lkey = ia->ri_device->local_dma_lkey;
> -
> ia->ri_max_frmr_depth =
> min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS,
> devattr->max_fast_reg_page_list_len);
> diff --git a/net/sunrpc/xprtrdma/physical_ops.c
> b/net/sunrpc/xprtrdma/physical_ops.c
> index 72cf8b15bbb4..617b76f22154 100644
> --- a/net/sunrpc/xprtrdma/physical_ops.c
> +++ b/net/sunrpc/xprtrdma/physical_ops.c
> @@ -23,7 +23,6 @@ static int
> physical_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep,
>struct rpcrdma_create_data_internal *cdata)
> {
> - struct ib_device_attr *devattr = &ia->ri_devattr;
> struct ib_mr *mr;
>
> /* Obtain an rkey to use for RPC data payloads.
> @@ -37,15 +36,8 @@ physical_op_open(struct rpcrdma_ia *ia, struct rpcrdma_ep
> *ep,
> __func__, PTR_ERR(mr));
> return -ENOMEM;
> }
> - ia->ri_dma_mr = mr;
> -
> - /* Obtain an lkey to use for regbufs.
> - */
> - if (devattr->device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY)
> - ia->ri_dma_lkey = ia->ri_device->local_dma_lkey;
> - else
> - ia->ri_dma_lkey = ia->ri_dma_mr->lkey;
>
> + ia->ri_dma_mr = mr;
> return 0;
> }
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 682996779970..eb081ad05e33 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1252,7 +1252,7 @@ rpcrdma_alloc_regbuf(struct rpcrdma_ia *ia, size_t
> size, gfp_t flags)
> goto out_free;
>
> iov->length = size;
> - iov->lkey = ia->ri_dma_lkey;
> + iov->lkey = ia->ri_pd->local_dma_lkey;
> rb->rg_size = size;
> rb->rg_owner = NULL;
> return rb;
> diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h
> index d252457ff21a..df5ad4e15702 100644
> --- a/net/sunrp
Re: [PATCH v2] xprtrdma: take HCA driver refcount at client
Hi Doug,
On 09/03/2015 04:11 PM, Doug Ledford wrote:
> On 07/30/2015 07:00 AM, Devesh Sharma wrote:
>> Thanks Chuck Lever for the valuable feedback and suggestions.
>>
>> This is a rework of the following patch sent almost a year back:
>> http://www.mail-archive.com/linux-rdma%40vger.kernel.org/msg20730.html
>>
>> In presence of active mount if someone tries to rmmod vendor-driver, the
>> command remains stuck forever waiting for destruction of all rdma-cm-id.
>> in worst case client can crash during shutdown with active mounts.
>>
>> The existing code assumes that ia->ri_id->device cannot change during
>> the lifetime of a transport. xprtrdma do not have support for
>> DEVICE_REMOVAL event either. Lifting that assumption and adding support
>> for DEVICE_REMOVAL event is a long chain of work, and is in plan.
>>
>> The community decided that preventing the hang right now is more
>> important than waiting for architectural changes.
>>
>> Thus, this patch introduces a temporary workaround to acquire HCA driver
>> module reference count during the mount of a nfs-rdma mount point.
>>
>> Cc: chuck.le...@oracle.com
>> Cc: linux-...@vger.kernel.org
>> Signed-off-by: Devesh Sharma
>> Reviewed-by: Sagi Grimberg
>
> Chuck, was this given final approval, and if so, who's tree is it
> expected to go through? Just trying to make sure I don't need to do
> anything here as I don't see a rejection in my linux-rdma folder, but I
> also didn't see it in the initial 4.3 nfs merge.
This was approved and went through my tree to Trond. I don't think he's sent
out the v4.3 pull request yet, but hopefully it won't be too much longer!
Thanks,
Anna
>
>> ---
>> net/sunrpc/xprtrdma/verbs.c | 38 ++
>> 1 files changed, 30 insertions(+), 8 deletions(-)
>>
>> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
>> index 891c4ed..1c3c420 100644
>> --- a/net/sunrpc/xprtrdma/verbs.c
>> +++ b/net/sunrpc/xprtrdma/verbs.c
>> @@ -52,6 +52,7 @@
>> #include
>> #include
>> #include
>> +#include /* try_module_get()/module_put() */
>>
>> #include "xprt_rdma.h"
>>
>> @@ -414,6 +415,14 @@ connected:
>> return 0;
>> }
>>
>> +static void rpcrdma_destroy_id(struct rdma_cm_id *id)
>> +{
>> +if (id) {
>> +module_put(id->device->owner);
>> +rdma_destroy_id(id);
>> +}
>> +}
>> +
>> static struct rdma_cm_id *
>> rpcrdma_create_id(struct rpcrdma_xprt *xprt,
>> struct rpcrdma_ia *ia, struct sockaddr *addr)
>> @@ -440,6 +449,18 @@ rpcrdma_create_id(struct rpcrdma_xprt *xprt,
>> }
>> wait_for_completion_interruptible_timeout(&ia->ri_done,
>> msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT) + 1);
>> +
>> +/* FIXME: We hate to break the notion of ULP<-->Core<-->Provider
>> + * by calling try_module_get() on HCA driver. This is to prevent a
>> + * system hang or a possible crash during reboot with active nfs-rdma
>> + * mount. We will keep this workaround until xprtrdma comes back with a
>> + * massive architectural changes to have proper fix.
>> + */
>> +if (!ia->ri_async_rc && !try_module_get(id->device->owner)) {
>> +dprintk("RPC: %s: Failed to get device module\n",
>> +__func__);
>> +ia->ri_async_rc = -ENODEV;
>> +}
>> rc = ia->ri_async_rc;
>> if (rc)
>> goto out;
>> @@ -449,16 +470,17 @@ rpcrdma_create_id(struct rpcrdma_xprt *xprt,
>> if (rc) {
>> dprintk("RPC: %s: rdma_resolve_route() failed %i\n",
>> __func__, rc);
>> -goto out;
>> +goto put;
>> }
>> wait_for_completion_interruptible_timeout(&ia->ri_done,
>> msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT) + 1);
>> rc = ia->ri_async_rc;
>> if (rc)
>> -goto out;
>> +goto put;
>>
>> return id;
>> -
>> +put:
>> +module_put(id->device->owner);
>> out:
>> rdma_destroy_id(id);
>> return ERR_PTR(rc);
>> @@ -592,7 +614,7 @@ out3:
>> ib_dealloc_pd(ia->ri_pd);
>> ia->ri_pd = NULL;
>> out2:
>> -rdma_destroy_id(ia->ri_id);
>> +rpcrdma_destroy_id(ia->ri_id);
>> ia->ri_id = NULL;
>> out1:
>> return rc;
>> @@ -618,7 +640,7 @@ rpcrdma_ia_close(struct rpcrdma_ia *ia)
>> if (ia->ri_id != NULL && !IS_ERR(ia->ri_id)) {
>> if (ia->ri_id->qp)
>> rdma_destroy_qp(ia->ri_id);
>> -rdma_destroy_id(ia->ri_id);
>> +rpcrdma_destroy_id(ia->ri_id);
>> ia->ri_id = NULL;
>> }
>>
>> @@ -825,7 +847,7 @@ retry:
>> if (ia->ri_device != id->device) {
>> printk("RPC: %s: can't reconnect on "
>> "different device!\n", __func__);
>> -rdma_destroy_id(id);
>> +rpcrdma_destroy_id(id);
>> rc
Re: [PATCH v4 00/16] NFS/RDMA client side for Linux 4.3
Hi Chuck, I've applied these patches locally without a problem (and I made sure to remove the pr_warns() in the last patch). Are you planning to submit a v5, or are these ready for me now? Thanks, Anna On 08/03/2015 01:02 PM, Chuck Lever wrote: > Several important client-side performance and scalability > improvements are made in this series, proposed for the 4.3 > kernel, including: > > - Increase maximum RPC/RDMA credits to 128 > - Increase maximum NFS/RDMA r/wsize to one megabyte > - Prefer inline rather than reply chunk replies > > And these fixes: > > - Send NFSv4 WRITE compounds correctly > - Support RDMA_NOMSG calls > - Remove support for RDMA_MSGP calls > - Fix large NFS symlink operations > - Get inline threshold accounting right > > Also available in the "nfs-rdma-for-4.3" topic branch of this git repo: > > git://git.linux-nfs.org/projects/cel/cel-2.6.git > > Or for browsing: > > http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=log;h=refs/heads/nfs-rdma-for-4.3 > > > Changes since v3: > - Rebased on Linux v4.2-rc5 > - Added v2 of Devesh's reboot crash fix > - Clarified description of NFSv4 WRITE fix > > > Changes since v2: > - Rebased on Linux v4.2-rc3 > - Corrected RPCRDMA_MAX_IOVS macro > - Included patch to remove core ib_reg_phys_mr() API > > > Changes since v1: > - Rebased on Linux v4.2-rc2 > - PHYSICAL registration, being insecure, must now be explicitly selected > - Further clean-ups were done because ib_reg_phys_mr() is gone > - Support for RDMA_MSGP type calls has been removed > - Some patch descriptions have been clarified > > --- > > Chuck Lever (15): > xprtrdma: Make xprt_setup_rdma() agnostic to family of server address > xprtrdma: Raise maximum payload size to one megabyte > xprtrdma: Increase default credit limit > xprtrdma: Don't fall back to PHYSICAL memory registration > xprtrdma: Remove last ib_reg_phys_mr() call site > xprtrdma: Clean up rpcrdma_ia_open() > xprtrdma: Remove logic that constructs RDMA_MSGP type calls > xprtrdma: Account for RPC/RDMA header size when deciding to inline > xprtrdma: Always provide a write list when sending NFS READ > xprtrdma: Don't provide a reply chunk when expecting a short reply > xprtrdma: Fix XDR tail buffer marshalling > xprtrdma: Fix large NFS SYMLINK calls > xprtrdma: Clean up xprt_rdma_print_stats() > xprtrdma: Count RDMA_NOMSG type calls > core: Remove the ib_reg_phys_mr() and ib_rereg_phys_mr() verbs > > Devesh Sharma (1): > xprtrdma: take HCA driver refcount at client > > > drivers/infiniband/core/verbs.c| 67 -- > fs/nfs/nfs3xdr.c |1 > fs/nfs/nfs4xdr.c |4 - > include/linux/sunrpc/xprtrdma.h|2 > include/rdma/ib_verbs.h| 46 --- > net/sunrpc/xprtrdma/fmr_ops.c | 19 +++ > net/sunrpc/xprtrdma/frwr_ops.c |5 + > net/sunrpc/xprtrdma/physical_ops.c | 25 > net/sunrpc/xprtrdma/rpc_rdma.c | 197 +++--- > net/sunrpc/xprtrdma/transport.c| 77 +--- > net/sunrpc/xprtrdma/verbs.c| 238 > +--- > net/sunrpc/xprtrdma/xprt_rdma.h| 27 ++-- > 12 files changed, 285 insertions(+), 423 deletions(-) > > -- > Chuck Lever > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH for-4.3 06/15] xprtrdma, svcrdma: Convert to ib_alloc_mr
This looks good to me:
Signed-off-by: Anna Schumaker
Thanks,
Anna
On 07/30/2015 03:32 AM, Sagi Grimberg wrote:
> Signed-off-by: Sagi Grimberg
> ---
> net/sunrpc/xprtrdma/frwr_ops.c | 6 +++---
> net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +-
> 2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
> index 63f282e..d6653f5 100644
> --- a/net/sunrpc/xprtrdma/frwr_ops.c
> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
> @@ -117,7 +117,7 @@ __frwr_recovery_worker(struct work_struct *work)
> if (ib_dereg_mr(r->r.frmr.fr_mr))
> goto out_fail;
>
> - r->r.frmr.fr_mr = ib_alloc_fast_reg_mr(pd, depth);
> + r->r.frmr.fr_mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG, depth);
> if (IS_ERR(r->r.frmr.fr_mr))
> goto out_fail;
>
> @@ -148,7 +148,7 @@ __frwr_init(struct rpcrdma_mw *r, struct ib_pd *pd,
> struct ib_device *device,
> struct rpcrdma_frmr *f = &r->r.frmr;
> int rc;
>
> - f->fr_mr = ib_alloc_fast_reg_mr(pd, depth);
> + f->fr_mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG, depth);
> if (IS_ERR(f->fr_mr))
> goto out_mr_err;
> f->fr_pgl = ib_alloc_fast_reg_page_list(device, depth);
> @@ -158,7 +158,7 @@ __frwr_init(struct rpcrdma_mw *r, struct ib_pd *pd,
> struct ib_device *device,
>
> out_mr_err:
> rc = PTR_ERR(f->fr_mr);
> - dprintk("RPC: %s: ib_alloc_fast_reg_mr status %i\n",
> + dprintk("RPC: %s: ib_alloc_mr status %i\n",
> __func__, rc);
> return rc;
>
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c
> b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> index 6b36279..adcf904 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
> @@ -738,7 +738,7 @@ static struct svc_rdma_fastreg_mr *rdma_alloc_frmr(struct
> svcxprt_rdma *xprt)
> if (!frmr)
> goto err;
>
> - mr = ib_alloc_fast_reg_mr(xprt->sc_pd, RPCSVC_MAXPAGES);
> + mr = ib_alloc_mr(xprt->sc_pd, IB_MR_TYPE_MEM_REG, RPCSVC_MAXPAGES);
> if (IS_ERR(mr))
> goto err_free_frmr;
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] IB/core: Make ib_dealloc_pd return void
kabi
>> + requires the caller to guarantee we can't race here. */
>> +WARN_ON(atomic_read(&pd->usecnt));
>>
>> -return pd->device->dealloc_pd(pd);
>> +/* Making delalloc_pd a void return is a WIP, no driver should return
>> + an error here. */
>> +ret = pd->device->dealloc_pd(pd);
>> +WARN_ONCE(ret, "Infiniband HW driver failed dealloc_pd");
>> }
>> EXPORT_SYMBOL(ib_dealloc_pd);
>>
>> diff --git a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
>> b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
>> index 8c451983d8a5..78845b6e8b81 100644
>> --- a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
>> +++ b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
>> @@ -280,9 +280,7 @@ void ipoib_transport_dev_cleanup(struct net_device *dev)
>> priv->wq = NULL;
>> }
>>
>> -if (ib_dealloc_pd(priv->pd))
>> -ipoib_warn(priv, "ib_dealloc_pd failed\n");
>> -
>> +ib_dealloc_pd(priv->pd);
>> }
>>
>> void ipoib_event(struct ib_event_handler *handler,
>> diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c
>> b/drivers/infiniband/ulp/iser/iser_verbs.c
>> index 52268356c79e..6e984e4b553b 100644
>> --- a/drivers/infiniband/ulp/iser/iser_verbs.c
>> +++ b/drivers/infiniband/ulp/iser/iser_verbs.c
>> @@ -201,7 +201,7 @@ static void iser_free_device_ib_res(struct iser_device
>> *device)
>>
>> (void)ib_unregister_event_handler(&device->event_handler);
>> (void)ib_dereg_mr(device->mr);
>> -(void)ib_dealloc_pd(device->pd);
>> +ib_dealloc_pd(device->pd);
>>
>> kfree(device->comps);
>> device->comps = NULL;
>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>> index eaec3081fb87..4aaab4fe459c 100644
>> --- a/include/rdma/ib_verbs.h
>> +++ b/include/rdma/ib_verbs.h
>> @@ -2139,11 +2139,7 @@ int ib_find_pkey(struct ib_device *device,
>>
>> struct ib_pd *ib_alloc_pd(struct ib_device *device);
>>
>> -/**
>> - * ib_dealloc_pd - Deallocates a protection domain.
>> - * @pd: The protection domain to deallocate.
>> - */
>> -int ib_dealloc_pd(struct ib_pd *pd);
>> +void ib_dealloc_pd(struct ib_pd *pd);
>>
>> /**
>> * ib_create_ah - Creates an address handle for the given address vector.
>> diff --git a/net/rds/iw.c b/net/rds/iw.c
>> index 589935661d66..5f228e00ad09 100644
>> --- a/net/rds/iw.c
>> +++ b/net/rds/iw.c
>> @@ -149,10 +149,7 @@ static void rds_iw_remove_one(struct ib_device *device)
>> if (rds_iwdev->mr)
>> ib_dereg_mr(rds_iwdev->mr);
>>
>> -while (ib_dealloc_pd(rds_iwdev->pd)) {
>> -rdsdebug("Failed to dealloc pd %p\n", rds_iwdev->pd);
>> -msleep(1);
>> -}
>> +ib_dealloc_pd(rds_iwdev->pd);
>>
>> list_del(&rds_iwdev->list);
>> kfree(rds_iwdev);
>> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
>> index 891c4ede2c20..afd504375a9a 100644
>> --- a/net/sunrpc/xprtrdma/verbs.c
>> +++ b/net/sunrpc/xprtrdma/verbs.c
>> @@ -624,7 +624,7 @@ rpcrdma_ia_close(struct rpcrdma_ia *ia)
>>
>> /* If the pd is still busy, xprtrdma missed freeing a resource */
>> if (ia->ri_pd && !IS_ERR(ia->ri_pd))
>> -WARN_ON(ib_dealloc_pd(ia->ri_pd));
>> +ib_dealloc_pd(ia->ri_pd);
>> }
>>
>> /*
>
> Reviewed-by: Chuck Lever
>
> Does this hunk or the xprtrdma changes in the local DMA lkey
> series need an Acked-by: from Anna?
If it's a client side change, then yeah it'll need an Acked-by from me. I'm
not sure if I've seen the DMA lkey changes yet, can somebody point me to them?
This hunk looks fine to me:
Acked-by: Anna Schumaker
Thanks,
Anna
>
>
> --
> Chuck Lever
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 15/15] core: Remove the ib_reg_phys_mr() and ib_rereg_phys_mr() verbs
On 07/21/2015 04:16 PM, Chuck Lever wrote:
> Hi Anna-
>
>
> On Jul 21, 2015, at 4:08 PM, Anna Schumaker wrote:
>
>> Hey Chuck,
>>
>> Just a heads up that this patch should go through the infiniband tree, and
>> not NFS :).
>
> The IB maintainer has ACK’d it, so it can go through NFS. It depends
> on changes in an earlier patch in this series.
Fair enough! I didn't realize Doug was the IB maintainer.
Anna
>
>
>> Thanks,
>> Anna
>>
>> On 07/20/2015 03:04 PM, Chuck Lever wrote:
>>> The verbs are obsolete. The ib_rereg_phys_mr() verb is not used by
>>> kernel ULPs, and the last ib_reg_phys_mr() call site in the kernel
>>> tree has now been removed.
>>>
>>> Two staging tree call sites remain in the Lustre client. The Lustre
>>> team has been notified of the deprecation of reg_phys_mr.
>>>
>>> Signed-off-by: Chuck Lever
>>> Acked-by: Doug Ledford
>>> ---
>>> drivers/infiniband/core/verbs.c | 67
>>> ---
>>> include/rdma/ib_verbs.h | 46 ---
>>> 2 files changed, 113 deletions(-)
>>>
>>> diff --git a/drivers/infiniband/core/verbs.c
>>> b/drivers/infiniband/core/verbs.c
>>> index bac3fb4..30eb245 100644
>>> --- a/drivers/infiniband/core/verbs.c
>>> +++ b/drivers/infiniband/core/verbs.c
>>> @@ -1144,73 +1144,6 @@ struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int
>>> mr_access_flags)
>>> }
>>> EXPORT_SYMBOL(ib_get_dma_mr);
>>>
>>> -struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd,
>>> -struct ib_phys_buf *phys_buf_array,
>>> -int num_phys_buf,
>>> -int mr_access_flags,
>>> -u64 *iova_start)
>>> -{
>>> - struct ib_mr *mr;
>>> - int err;
>>> -
>>> - err = ib_check_mr_access(mr_access_flags);
>>> - if (err)
>>> - return ERR_PTR(err);
>>> -
>>> - if (!pd->device->reg_phys_mr)
>>> - return ERR_PTR(-ENOSYS);
>>> -
>>> - mr = pd->device->reg_phys_mr(pd, phys_buf_array, num_phys_buf,
>>> -mr_access_flags, iova_start);
>>> -
>>> - if (!IS_ERR(mr)) {
>>> - mr->device = pd->device;
>>> - mr->pd = pd;
>>> - mr->uobject = NULL;
>>> - atomic_inc(&pd->usecnt);
>>> - atomic_set(&mr->usecnt, 0);
>>> - }
>>> -
>>> - return mr;
>>> -}
>>> -EXPORT_SYMBOL(ib_reg_phys_mr);
>>> -
>>> -int ib_rereg_phys_mr(struct ib_mr *mr,
>>> -int mr_rereg_mask,
>>> -struct ib_pd *pd,
>>> -struct ib_phys_buf *phys_buf_array,
>>> -int num_phys_buf,
>>> -int mr_access_flags,
>>> -u64 *iova_start)
>>> -{
>>> - struct ib_pd *old_pd;
>>> - int ret;
>>> -
>>> - ret = ib_check_mr_access(mr_access_flags);
>>> - if (ret)
>>> - return ret;
>>> -
>>> - if (!mr->device->rereg_phys_mr)
>>> - return -ENOSYS;
>>> -
>>> - if (atomic_read(&mr->usecnt))
>>> - return -EBUSY;
>>> -
>>> - old_pd = mr->pd;
>>> -
>>> - ret = mr->device->rereg_phys_mr(mr, mr_rereg_mask, pd,
>>> - phys_buf_array, num_phys_buf,
>>> - mr_access_flags, iova_start);
>>> -
>>> - if (!ret && (mr_rereg_mask & IB_MR_REREG_PD)) {
>>> - atomic_dec(&old_pd->usecnt);
>>> - atomic_inc(&pd->usecnt);
>>> - }
>>> -
>>> - return ret;
>>> -}
>>> -EXPORT_SYMBOL(ib_rereg_phys_mr);
>>> -
>>> int ib_query_mr(struct ib_mr *mr, struct ib_mr_attr *mr_attr)
>>> {
>>> return mr->device->query_mr ?
>>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>>> index b0f898e..43c1cf0 100644
>>> --- a/include/rdma/ib_verbs.h
>>> +++ b/include/rdma/ib_verbs.h
>>> @@ -2760,52 +2760,6 @@ static inline void ib_dma_free_coherent(struct
>>> ib_device *dev,
>>> }
>>>
>>> /**
>>>
Re: [PATCH v3 15/15] core: Remove the ib_reg_phys_mr() and ib_rereg_phys_mr() verbs
Hey Chuck,
Just a heads up that this patch should go through the infiniband tree, and not
NFS :).
Thanks,
Anna
On 07/20/2015 03:04 PM, Chuck Lever wrote:
> The verbs are obsolete. The ib_rereg_phys_mr() verb is not used by
> kernel ULPs, and the last ib_reg_phys_mr() call site in the kernel
> tree has now been removed.
>
> Two staging tree call sites remain in the Lustre client. The Lustre
> team has been notified of the deprecation of reg_phys_mr.
>
> Signed-off-by: Chuck Lever
> Acked-by: Doug Ledford
> ---
> drivers/infiniband/core/verbs.c | 67
> ---
> include/rdma/ib_verbs.h | 46 ---
> 2 files changed, 113 deletions(-)
>
> diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
> index bac3fb4..30eb245 100644
> --- a/drivers/infiniband/core/verbs.c
> +++ b/drivers/infiniband/core/verbs.c
> @@ -1144,73 +1144,6 @@ struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int
> mr_access_flags)
> }
> EXPORT_SYMBOL(ib_get_dma_mr);
>
> -struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd,
> - struct ib_phys_buf *phys_buf_array,
> - int num_phys_buf,
> - int mr_access_flags,
> - u64 *iova_start)
> -{
> - struct ib_mr *mr;
> - int err;
> -
> - err = ib_check_mr_access(mr_access_flags);
> - if (err)
> - return ERR_PTR(err);
> -
> - if (!pd->device->reg_phys_mr)
> - return ERR_PTR(-ENOSYS);
> -
> - mr = pd->device->reg_phys_mr(pd, phys_buf_array, num_phys_buf,
> - mr_access_flags, iova_start);
> -
> - if (!IS_ERR(mr)) {
> - mr->device = pd->device;
> - mr->pd = pd;
> - mr->uobject = NULL;
> - atomic_inc(&pd->usecnt);
> - atomic_set(&mr->usecnt, 0);
> - }
> -
> - return mr;
> -}
> -EXPORT_SYMBOL(ib_reg_phys_mr);
> -
> -int ib_rereg_phys_mr(struct ib_mr *mr,
> - int mr_rereg_mask,
> - struct ib_pd *pd,
> - struct ib_phys_buf *phys_buf_array,
> - int num_phys_buf,
> - int mr_access_flags,
> - u64 *iova_start)
> -{
> - struct ib_pd *old_pd;
> - int ret;
> -
> - ret = ib_check_mr_access(mr_access_flags);
> - if (ret)
> - return ret;
> -
> - if (!mr->device->rereg_phys_mr)
> - return -ENOSYS;
> -
> - if (atomic_read(&mr->usecnt))
> - return -EBUSY;
> -
> - old_pd = mr->pd;
> -
> - ret = mr->device->rereg_phys_mr(mr, mr_rereg_mask, pd,
> - phys_buf_array, num_phys_buf,
> - mr_access_flags, iova_start);
> -
> - if (!ret && (mr_rereg_mask & IB_MR_REREG_PD)) {
> - atomic_dec(&old_pd->usecnt);
> - atomic_inc(&pd->usecnt);
> - }
> -
> - return ret;
> -}
> -EXPORT_SYMBOL(ib_rereg_phys_mr);
> -
> int ib_query_mr(struct ib_mr *mr, struct ib_mr_attr *mr_attr)
> {
> return mr->device->query_mr ?
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index b0f898e..43c1cf0 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -2760,52 +2760,6 @@ static inline void ib_dma_free_coherent(struct
> ib_device *dev,
> }
>
> /**
> - * ib_reg_phys_mr - Prepares a virtually addressed memory region for use
> - * by an HCA.
> - * @pd: The protection domain associated assigned to the registered region.
> - * @phys_buf_array: Specifies a list of physical buffers to use in the
> - * memory region.
> - * @num_phys_buf: Specifies the size of the phys_buf_array.
> - * @mr_access_flags: Specifies the memory access rights.
> - * @iova_start: The offset of the region's starting I/O virtual address.
> - */
> -struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd,
> - struct ib_phys_buf *phys_buf_array,
> - int num_phys_buf,
> - int mr_access_flags,
> - u64 *iova_start);
> -
> -/**
> - * ib_rereg_phys_mr - Modifies the attributes of an existing memory region.
> - * Conceptually, this call performs the functions deregister memory region
> - * followed by register physical memory region. Where possible,
> - * resources are reused instead of deallocated and reallocated.
> - * @mr: The memory region to modify.
> - * @mr_rereg_mask: A bit-mask used to indicate which of the following
> - * properties of the memory region are being modified.
> - * @pd: If %IB_MR_REREG_PD is set in mr_rereg_mask, this field specifies
> - * the new protection domain to associated with the memory region,
> - * otherwise, this parameter is ignored.
> - * @phys_buf_array: If %IB_MR_REREG_TRANS is set in mr_rereg_mask, this
> - * field specifies a list of physical buffers to use in the new
> - * translat
Re: [PATCH v1 10/12] xprtrdma: Fix large NFS SYMLINK calls
Hey Chuck,
On 07/09/2015 04:43 PM, Chuck Lever wrote:
> Repair how rpcrdma_marshal_req() chooses which RDMA message type
> to use for large non-WRITE operations so that it picks RDMA_NOMSG
> in the correct situations, and sets up the marshaling logic to
> SEND only the RPC/RDMA header.
>
> Large NFSv2 SYMLINK requests now use RDMA_NOMSG calls. The Linux NFS
> server XDR decoder for NFSv2 SYMLINK does not handle having the
> pathname argument arrive in a separate buffer. The decoder could be
> fixed, but this is simpler and RDMA_NOMSG can be used in a variety
> of other situations.
>
> Ensure that the Linux client continues to use "RDMA_MSG + read
> list" when sending large NFSv3 SYMLINK requests, which is more
> efficient than using RDMA_NOMSG.
>
> Large NFSv4 CREATE(NF4LNK) requests are changed to use "RDMA_MSG +
> read list" just like NFSv3 (see Section 5 of RFC 5667). Before,
> these did not work at all.
>
> Signed-off-by: Chuck Lever
> ---
> fs/nfs/nfs3xdr.c |1 +
> fs/nfs/nfs4xdr.c |1 +
> net/sunrpc/xprtrdma/rpc_rdma.c | 21 -
> 3 files changed, 14 insertions(+), 9 deletions(-)
It might be better to split this into separate patches for nfs and sunrpc,
since Trond might want to accept the nfs changes separately.
Anna
>
> diff --git a/fs/nfs/nfs3xdr.c b/fs/nfs/nfs3xdr.c
> index 9b04c2e..267126d 100644
> --- a/fs/nfs/nfs3xdr.c
> +++ b/fs/nfs/nfs3xdr.c
> @@ -1103,6 +1103,7 @@ static void nfs3_xdr_enc_symlink3args(struct rpc_rqst
> *req,
> {
> encode_diropargs3(xdr, args->fromfh, args->fromname, args->fromlen);
> encode_symlinkdata3(xdr, args);
> + xdr->buf->flags |= XDRBUF_WRITE;
> }
>
> /*
> diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
> index 558cd65d..03a20ec 100644
> --- a/fs/nfs/nfs4xdr.c
> +++ b/fs/nfs/nfs4xdr.c
> @@ -1155,6 +1155,7 @@ static void encode_create(struct xdr_stream *xdr, const
> struct nfs4_create_arg *
> p = reserve_space(xdr, 4);
> *p = cpu_to_be32(create->u.symlink.len);
> xdr_write_pages(xdr, create->u.symlink.pages, 0,
> create->u.symlink.len);
> + xdr->buf->flags |= XDRBUF_WRITE;
> break;
>
> case NF4BLK: case NF4CHR:
> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
> index 2e721f2..64fc4b4 100644
> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
> @@ -484,21 +484,24 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
>*
>* o If the total request is under the inline threshold, all ops
>* are sent as inline.
> - * o Large non-write ops are sent with the entire message as a
> - * single read chunk (protocol 0-position special case).
>* o Large write ops transmit data as read chunk(s), header as
>* inline.
> + * o Large non-write ops are sent with the entire message as a
> + * single read chunk (protocol 0-position special case).
>*
> - * Note: the NFS code sending down multiple argument segments
> - * implies the op is a write.
> - * TBD check NFSv4 setacl
> + * This assumes that the upper layer does not present a request
> + * that both has a data payload, and whose non-data arguments
> + * by themselves are larger than the inline threshold.
>*/
> - if (rpcrdma_args_inline(rqst))
> + if (rpcrdma_args_inline(rqst)) {
> rtype = rpcrdma_noch;
> - else if (rqst->rq_snd_buf.page_len == 0)
> - rtype = rpcrdma_areadch;
> - else
> + } else if (rqst->rq_snd_buf.flags & XDRBUF_WRITE) {
> rtype = rpcrdma_readch;
> + } else {
> + headerp->rm_type = htonl(RDMA_NOMSG);
> + rtype = rpcrdma_areadch;
> + rpclen = 0;
> + }
>
> /* The following simplification is not true forever */
> if (rtype != rpcrdma_noch && wtype == rpcrdma_replych)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1 09/12] xprtrdma: Prepare rpcrdma_ep_post() for RDMA_NOMSG calls
Hi Chuck,
On 07/09/2015 04:43 PM, Chuck Lever wrote:
> Only the RPC/RDMA header is sent when making an RDMA_NOMSG call.
> That header resides in the first element of the iovec array
> passed to rpcrdma_ep_post().
>
> Instead of special casing the iovec element with the pad, just
> sync all the elements in the send iovec. Syncing the zero pad is
> not strictly necessary, but the pad is rarely if ever used these
> days, and the extra cost in that case is small.
>
> Signed-off-by: Chuck Lever
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index cdf5220..9199436 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1316,22 +1318,15 @@ rpcrdma_ep_post(struct rpcrdma_ia *ia,
>
> send_wr.next = NULL;
> send_wr.wr_id = RPCRDMA_IGNORE_COMPLETION;
> - send_wr.sg_list = req->rl_send_iov;
> + send_wr.sg_list = iov;
> send_wr.num_sge = req->rl_niovs;
> send_wr.opcode = IB_WR_SEND;
> - if (send_wr.num_sge == 4) /* no need to sync any pad (constant) */
> - ib_dma_sync_single_for_device(ia->ri_device,
> - req->rl_send_iov[3].addr,
> - req->rl_send_iov[3].length,
> - DMA_TO_DEVICE);
> - ib_dma_sync_single_for_device(ia->ri_device,
> - req->rl_send_iov[1].addr,
> - req->rl_send_iov[1].length,
> - DMA_TO_DEVICE);
> - ib_dma_sync_single_for_device(ia->ri_device,
> - req->rl_send_iov[0].addr,
> - req->rl_send_iov[0].length,
> - DMA_TO_DEVICE);
> +
> + for (i = 0; i < send_wr.num_sge; i++)
> + ib_dma_sync_single_for_device(device, iov[i].addr,
> + iov[i].length, DMA_TO_DEVICE);
Two questions here:
1) Is syncing order important? The original code went 3 - 1 - 0, but now we're
going 0 - 1 - 2 - 3.
2) Is iov[2] the zero pad you mentioned in your commit message? If not, do you
know why it wasn't already syncing?
Thanks,
Anna
> + dprintk("RPC: %s: posting %d s/g entries\n",
> + __func__, send_wr.num_sge);
>
> if (DECR_CQCOUNT(ep) > 0)
> send_wr.send_flags = 0;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1 05/12] xprtrdma: Account for RPC/RDMA header size when deciding to inline
Hi Chuck,
On 07/09/2015 04:42 PM, Chuck Lever wrote:
> When marshaling RPC/RDMA requests, ensure the combined size of
> RPC/RDMA header and RPC header do not exceed the inline threshold.
> Endpoints typically reject RPC/RDMA messages that exceed the size
> of their receive buffers.
>
> Signed-off-by: Chuck Lever
> ---
> net/sunrpc/xprtrdma/rpc_rdma.c | 29 +++--
> 1 file changed, 27 insertions(+), 2 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
> index 84ea37d..8cf9402 100644
> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
> @@ -71,6 +71,31 @@ static const char transfertypes[][12] = {
> };
> #endif
>
> +/* The client can send a request inline as long as the RPCRDMA header
> + * plus the RPC call fit under the transport's inline limit. If the
> + * combined call message size exceeds that limit, the client must use
> + * the read chunk list for this operation.
> + */
> +static bool rpcrdma_args_inline(struct rpc_rqst *rqst)
> +{
> + unsigned int callsize = RPCRDMA_HDRLEN_MIN + rqst->rq_snd_buf.len;
> +
> + return callsize <= RPCRDMA_INLINE_WRITE_THRESHOLD(rqst);
> +}
> +
> +/* The client can’t know how large the actual reply will be. Thus it
^
This is showing up as "can<80><99>t" in git-show, leading me to think that the
apostrophe was replaced with a unicode-apostrophe. Google might have made the
switch, but can you double check the patch on your side just in case?
Thanks,
Anna
> + * plans for the largest possible reply for that particular ULP
> + * operation. If the maximum combined reply message size exceeds that
> + * limit, the client must provide a write list or a reply chunk for
> + * this request.
> + */
> +static bool rpcrdma_results_inline(struct rpc_rqst *rqst)
> +{
> + unsigned int repsize = RPCRDMA_HDRLEN_MIN + rqst->rq_rcv_buf.buflen;
> +
> + return repsize <= RPCRDMA_INLINE_READ_THRESHOLD(rqst);
> +}
> +
> /*
> * Chunk assembly from upper layer xdr_buf.
> *
> @@ -418,7 +443,7 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
>* a READ, then use write chunks to separate the file data
>* into pages; otherwise use reply chunks.
>*/
> - if (rqst->rq_rcv_buf.buflen <= RPCRDMA_INLINE_READ_THRESHOLD(rqst))
> + if (rpcrdma_results_inline(rqst))
> wtype = rpcrdma_noch;
> else if (rqst->rq_rcv_buf.page_len == 0)
> wtype = rpcrdma_replych;
> @@ -441,7 +466,7 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
>* implies the op is a write.
>* TBD check NFSv4 setacl
>*/
> - if (rqst->rq_snd_buf.len <= RPCRDMA_INLINE_WRITE_THRESHOLD(rqst))
> + if (rpcrdma_args_inline(rqst))
> rtype = rpcrdma_noch;
> else if (rqst->rq_snd_buf.page_len == 0)
> rtype = rpcrdma_areadch;
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1 02/12] xprtrdma: Raise maximum payload size to one megabyte
On 07/10/2015 03:33 PM, Chuck Lever wrote: > > On Jul 10, 2015, at 3:21 PM, Anna Schumaker wrote: > >> Hi Chuck, >> >> On 07/09/2015 04:41 PM, Chuck Lever wrote: >>> The point of larger rsize and wsize is to reduce the per-byte cost >>> of memory registration and deregistration. Modern HCAs can typically >>> handle a megabyte or more with a single registration operation. >>> >>> Signed-off-by: Chuck Lever >>> --- >>> net/sunrpc/xprtrdma/xprt_rdma.h |3 +-- >>> 1 file changed, 1 insertion(+), 2 deletions(-) >>> >>> diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h >>> b/net/sunrpc/xprtrdma/xprt_rdma.h >>> index f49dd8b..abee472 100644 >>> --- a/net/sunrpc/xprtrdma/xprt_rdma.h >>> +++ b/net/sunrpc/xprtrdma/xprt_rdma.h >>> @@ -165,8 +165,7 @@ rdmab_to_msg(struct rpcrdma_regbuf *rb) >>> * struct rpcrdma_buffer. N is the max number of outstanding requests. >>> */ >>> >>> -/* temporary static scatter/gather max */ >>> -#define RPCRDMA_MAX_DATA_SEGS (64)/* max scatter/gather */ >>> +#define RPCRDMA_MAX_DATA_SEGS ((1 * 1024 * 1024) / PAGE_SIZE) >> ^^^ >> What is the significance of the 1 here? > > This echoes the definition of RPCSVC_MAXPAYLOAD. > > “1” documents “one” megabyte, I assume. Makes sense. Thanks! > > >> Thanks, >> Anna >> >>> #define RPCRDMA_MAX_SEGS(RPCRDMA_MAX_DATA_SEGS + 2) /* head+tail = 2 */ >>> >>> struct rpcrdma_buffer; >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>> the body of a message to majord...@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > Chuck Lever > > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1 02/12] xprtrdma: Raise maximum payload size to one megabyte
Hi Chuck, On 07/09/2015 04:41 PM, Chuck Lever wrote: > The point of larger rsize and wsize is to reduce the per-byte cost > of memory registration and deregistration. Modern HCAs can typically > handle a megabyte or more with a single registration operation. > > Signed-off-by: Chuck Lever > --- > net/sunrpc/xprtrdma/xprt_rdma.h |3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) > > diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h > index f49dd8b..abee472 100644 > --- a/net/sunrpc/xprtrdma/xprt_rdma.h > +++ b/net/sunrpc/xprtrdma/xprt_rdma.h > @@ -165,8 +165,7 @@ rdmab_to_msg(struct rpcrdma_regbuf *rb) > * struct rpcrdma_buffer. N is the max number of outstanding requests. > */ > > -/* temporary static scatter/gather max */ > -#define RPCRDMA_MAX_DATA_SEGS(64)/* max scatter/gather */ > +#define RPCRDMA_MAX_DATA_SEGS((1 * 1024 * 1024) / PAGE_SIZE) ^^^ What is the significance of the 1 here? Thanks, Anna > #define RPCRDMA_MAX_SEGS (RPCRDMA_MAX_DATA_SEGS + 2) /* head+tail = 2 */ > > struct rpcrdma_buffer; > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] NFSoRDMA Client Changes for 4.2
Hi Trond, The following changes since commit d4a4f75cd8f29cd9464a5a32e9224a91571d6649: Linux 4.1-rc7 (2015-06-07 20:23:50 -0700) are available in the git repository at: git://git.linux-nfs.org/projects/anna/nfs-rdma.git tags/nfs-rdma-for-4.2 for you to fetch changes up to 40c6ed0c8a7f2c5d67f5d6774bbce230a42176db: xprtrdma: Reduce per-transport MR allocation (2015-06-12 13:10:37 -0400) Thanks, Anna NFS: NFSoRDMA Client Changes These patches continue to build up for improving the rsize and wsize that the NFS client uses when talking over RDMA. In addition, these patches also add in scalability enhancements and other bugfixes. Signed-off-by: Anna Schumaker Chuck Lever (14): xprtrdma: Warn when there are orphaned IB objects xprtrdma: Replace rpcrdma_rep::rr_buffer with rr_rxprt xprtrdma: Remove rr_func xprtrdma: Use ib_device pointer safely xprtrdma: Introduce helpers for allocating MWs xprtrdma: Acquire FMRs in rpcrdma_fmr_register_external() xprtrdma: Introduce an FRMR recovery workqueue xprtrdma: Acquire MRs in rpcrdma_register_external() xprtrdma: Remove unused LOCAL_INV recovery logic xprtrdma: Remove ->ro_reset xprtrdma: Remove rpcrdma_ia::ri_memreg_strategy xprtrdma: Split rb_lock xprtrdma: Stack relief in fmr_op_map() xprtrdma: Reduce per-transport MR allocation include/linux/sunrpc/xprtrdma.h| 3 ++- net/sunrpc/xprtrdma/fmr_ops.c | 120 +++-- net/sunrpc/xprtrdma/frwr_ops.c | 227 +- net/sunrpc/xprtrdma/physical_ops.c | 14 ++ net/sunrpc/xprtrdma/rpc_rdma.c | 8 ++-- net/sunrpc/xprtrdma/transport.c| 17 + net/sunrpc/xprtrdma/verbs.c| 257 +++ net/sunrpc/xprtrdma/xprt_rdma.h| 38 +++- 8 files changed, 345 insertions(+), 339 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] Please pull NFSoRDMA client changes
Hi Trond,
The following changes since commit e42391cd048809d903291d07f86ed3934ce138e9:
Linux 4.0-rc6 (2015-03-29 15:26:31 -0700)
are available in the git repository at:
git://git.linux-nfs.org/projects/anna/nfs-rdma.git tags/nfs-rdma-for-4.1-1
for you to fetch changes up to d654788e98f74f2df8dfc6079fa314938f739486:
xprtrdma: Make rpcrdma_{un}map_one() into inline functions (2015-03-31
09:52:53 -0400)
Thanks,
Anna
NFS: NFSoRDMA Client Changes
This patch series creates an operation vector for each of the different
memory registration modes. This should make it easier to one day increase
credit limit, rsize, and wsize.
Signed-off-by: Anna Schumaker
Chuck Lever (15):
SUNRPC: Introduce missing well-known netids
xprtrdma: Display IPv6 addresses and port numbers correctly
xprtrdma: Perform a full marshal on retransmit
xprtrdma: Byte-align FRWR registration
xprtrdma: Prevent infinite loop in rpcrdma_ep_create()
xprtrdma: Add vector of ops for each memory registration strategy
xprtrdma: Add a "max_payload" op for each memreg mode
xprtrdma: Add a "register_external" op for each memreg mode
xprtrdma: Add a "deregister_external" op for each memreg mode
xprtrdma: Add "init MRs" memreg op
xprtrdma: Add "reset MRs" memreg op
xprtrdma: Add "destroy MRs" memreg op
xprtrdma: Add "open" memreg op
xprtrdma: Handle non-SEND completions via a callout
xprtrdma: Make rpcrdma_{un}map_one() into inline functions
include/linux/sunrpc/msg_prot.h| 8 +-
include/linux/sunrpc/xprtrdma.h| 5 -
net/sunrpc/xprtrdma/Makefile | 3 +-
net/sunrpc/xprtrdma/fmr_ops.c | 208 +
net/sunrpc/xprtrdma/frwr_ops.c | 353
+
net/sunrpc/xprtrdma/physical_ops.c | 94 ++
net/sunrpc/xprtrdma/rpc_rdma.c | 87 ++---
net/sunrpc/xprtrdma/transport.c| 61 ++---
net/sunrpc/xprtrdma/verbs.c| 699
--
net/sunrpc/xprtrdma/xprt_rdma.h| 90 ++---
10 files changed, 882 insertions(+), 726 deletions(-)
create mode 100644 net/sunrpc/xprtrdma/fmr_ops.c
create mode 100644 net/sunrpc/xprtrdma/frwr_ops.c
create mode 100644 net/sunrpc/xprtrdma/physical_ops.c
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/15] NFS/RDMA patches proposed for 4.1
On 03/30/2015 02:33 PM, Chuck Lever wrote:
> This is a series of client-side patches for NFS/RDMA. In preparation
> for increasing the transport credit limit and maximum rsize/wsize,
> I've re-factored the memory registration logic into separate files,
> invoked via a method API.
>
> The series is available in the nfs-rdma-for-4.1 topic branch at
>
> git://linux-nfs.org/projects/cel/cel-2.6.git
Thanks! I've applied these and pushed them out to my nfs-rdma git tree.
Anna
>
> Changes since v2:
> - Rebased on 4.0-rc6
> - One minor fix squashed into 01/15
> - Tested-by tags added
>
> Changes since v1:
> - Rebased on 4.0-rc5
> - Main optimizations postponed to 4.2
> - Addressed review comments from Anna, Sagi, and Devesh
>
> ---
>
> Chuck Lever (15):
> SUNRPC: Introduce missing well-known netids
> xprtrdma: Display IPv6 addresses and port numbers correctly
> xprtrdma: Perform a full marshal on retransmit
> xprtrdma: Byte-align FRWR registration
> xprtrdma: Prevent infinite loop in rpcrdma_ep_create()
> xprtrdma: Add vector of ops for each memory registration strategy
> xprtrdma: Add a "max_payload" op for each memreg mode
> xprtrdma: Add a "register_external" op for each memreg mode
> xprtrdma: Add a "deregister_external" op for each memreg mode
> xprtrdma: Add "init MRs" memreg op
> xprtrdma: Add "reset MRs" memreg op
> xprtrdma: Add "destroy MRs" memreg op
> xprtrdma: Add "open" memreg op
> xprtrdma: Handle non-SEND completions via a callout
> xprtrdma: Make rpcrdma_{un}map_one() into inline functions
>
>
> include/linux/sunrpc/msg_prot.h|8
> include/linux/sunrpc/xprtrdma.h|5
> net/sunrpc/xprtrdma/Makefile |3
> net/sunrpc/xprtrdma/fmr_ops.c | 208 +++
> net/sunrpc/xprtrdma/frwr_ops.c | 353 ++
> net/sunrpc/xprtrdma/physical_ops.c | 94 +
> net/sunrpc/xprtrdma/rpc_rdma.c | 87 ++--
> net/sunrpc/xprtrdma/transport.c| 61 ++-
> net/sunrpc/xprtrdma/verbs.c| 699
> +++-
> net/sunrpc/xprtrdma/xprt_rdma.h| 90 -
> 10 files changed, 882 insertions(+), 726 deletions(-)
> create mode 100644 net/sunrpc/xprtrdma/fmr_ops.c
> create mode 100644 net/sunrpc/xprtrdma/frwr_ops.c
> create mode 100644 net/sunrpc/xprtrdma/physical_ops.c
>
> --
> Chuck Lever
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 00/15] NFS/RDMA patches proposed for 4.1
Hey Chuck,
I didn't see anything that needs to be fixed up in these patches. Are they
ready for me?
Anna
On 03/24/2015 04:30 PM, Chuck Lever wrote:
> This is a series of client-side patches for NFS/RDMA. In preparation
> for increasing the transport credit limit and maximum rsize/wsize,
> I've re-factored the memory registration logic into separate files,
> invoked via a method API.
>
> The two main optimizations in v1 of this series have been dropped.
> Sagi Grimberg didn't like the complexity of the solution, and there
> isn't enough time to rework it, test the new version, and get it
> reviewed before the 4.1 merge window opens. I'm going to prepare
> these for 4.2.
>
> Fixes suggested by reviewers have been included before the
> refactoring patches to make it easier to backport them to previous
> kernels.
>
> The series is available in the nfs-rdma-for-4.1 topic branch at
>
> git://linux-nfs.org/projects/cel/cel-2.6.git
>
> Changes since v1:
> - Rebased on 4.0-rc5
> - Main optimizations postponed to 4.2
> - Addressed review comments from Anna, Sagi, and Devesh
>
> ---
>
> Chuck Lever (15):
> SUNRPC: Introduce missing well-known netids
> xprtrdma: Display IPv6 addresses and port numbers correctly
> xprtrdma: Perform a full marshal on retransmit
> xprtrdma: Byte-align FRWR registration
> xprtrdma: Prevent infinite loop in rpcrdma_ep_create()
> xprtrdma: Add vector of ops for each memory registration strategy
> xprtrdma: Add a "max_payload" op for each memreg mode
> xprtrdma: Add a "register_external" op for each memreg mode
> xprtrdma: Add a "deregister_external" op for each memreg mode
> xprtrdma: Add "init MRs" memreg op
> xprtrdma: Add "reset MRs" memreg op
> xprtrdma: Add "destroy MRs" memreg op
> xprtrdma: Add "open" memreg op
> xprtrdma: Handle non-SEND completions via a callout
> xprtrdma: Make rpcrdma_{un}map_one() into inline functions
>
>
> include/linux/sunrpc/msg_prot.h|8
> net/sunrpc/xprtrdma/Makefile |3
> net/sunrpc/xprtrdma/fmr_ops.c | 208 +++
> net/sunrpc/xprtrdma/frwr_ops.c | 353 ++
> net/sunrpc/xprtrdma/physical_ops.c | 94 +
> net/sunrpc/xprtrdma/rpc_rdma.c | 87 ++--
> net/sunrpc/xprtrdma/transport.c| 61 ++-
> net/sunrpc/xprtrdma/verbs.c| 699
> +++-
> net/sunrpc/xprtrdma/xprt_rdma.h| 90 -
> 9 files changed, 882 insertions(+), 721 deletions(-)
> create mode 100644 net/sunrpc/xprtrdma/fmr_ops.c
> create mode 100644 net/sunrpc/xprtrdma/frwr_ops.c
> create mode 100644 net/sunrpc/xprtrdma/physical_ops.c
>
> --
> Chuck Lever
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] Please pull an NFSoRDMA fix
Hi Trond, The following changes since commit a0a1d50cd1e80652142af5cddcde500d06c71bdd: xprtrdma: Update the GFP flags used in xprt_rdma_allocate() (2015-01-30 12:18:48 -0500) are available in the git repository at: git://git.linux-nfs.org/projects/anna/nfs-rdma.git tags/nfs-rdma-for-3.20-part-2 for you to fetch changes up to b625a61698619c7af652de2701a2fb17c5c5d66e: xprtrdma: Address sparse complaint in rpcr_to_rdmar() (2015-02-05 15:38:29 -0500) Thanks, Anna NFS: RDMA Client Sparse Fixes This patch fixes a sparse warning in the initial submission. Signed-off-by: Anna Schumaker Chuck Lever (1): xprtrdma: Address sparse complaint in rpcr_to_rdmar() net/sunrpc/xprtrdma/xprt_rdma.h | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/20] NFS/RDMA client for 3.20
Hi Chuck, I've applied these patches and the GPF flags patch for 3.20. Let me know if you need anything else! Anna On 01/21/2015 11:01 AM, Chuck Lever wrote: > The following series of patches for the Linux NFS client breaks up > the per-transport buffer pool data structures. The goal is to allow > transport operation to scale better as the size of SEND/RECV buffers > (the inline thresholds), the maximum NFS r/wsize, and the number of > RDMA credits (concurrent RPC requests) is increased. > > The primary change is that the header send buffers have been split > from struct rpcrdma_req. Specific benefits are outlined in the patch > descriptions. > > More pre-requisites are required. Changes to raise the maximum > r/wsize and other limits are left for a future merge window. > > See the topic branch "nfs-rdma-for-3.20" at: > > git://git.linux-nfs.org/projects/cel/cel-2.6.git > > Anna, I think these are ready for you to merge. > > Changes since v2: > - Rebased on v3.19-rc5 > - Several patch descriptions clarified, based on review comments > - New size of client receive buffers was needlessly large > > Changes since v1: > - Rebased on v3.19-rc4 > - One short description fixed > - linux-rdma included this time (sorry for the noise) > > --- > > Chuck Lever (20): > xprtrdma: human-readable completion status > xprtrdma: Modernize htonl and ntohl > xprtrdma: Display XIDs in host byte order > xprtrdma: Clean up hdrlen > xprtrdma: Rename "xprt" and "rdma_connect" fields in struct rpcrdma_xprt > xprtrdma: Remove rpcrdma_ep::rep_ia > xprtrdma: Remove rl_mr field, and the mr_chunk union > xprtrdma: Move credit update to RPC reply handler > xprtrdma: Remove rpcrdma_ep::rep_func and ::rep_xprt > xprtrdma: Free the pd if ib_query_qp() fails > xprtrdma: Take struct ib_device_attr off the stack > xprtrdma: Take struct ib_qp_attr and ib_qp_init_attr off the stack > xprtrdma: Simplify synopsis of rpcrdma_buffer_create() > xprtrdma: Refactor rpcrdma_buffer_create() and rpcrdma_buffer_destroy() > xprtrdma: Add struct rpcrdma_regbuf and helpers > xprtrdma: Allocate RPC send buffer separately from struct rpcrdma_req > xprtrdma: Allocate RPC/RDMA send buffer separately from struct > rpcrdma_req > xprtrdma: Allocate RPC/RDMA receive buffer separately from struct > rpcrdma_rep > xprtrdma: Allocate zero pad separately from rpcrdma_buffer > xprtrdma: Clean up after adding regbuf management > > > include/linux/sunrpc/rpc_rdma.h | 14 + > include/linux/sunrpc/svc_rdma.h |2 > net/sunrpc/xprtrdma/rpc_rdma.c | 108 ++ > net/sunrpc/xprtrdma/transport.c | 179 +++-- > net/sunrpc/xprtrdma/verbs.c | 411 > --- > net/sunrpc/xprtrdma/xprt_rdma.h | 111 +++ > 6 files changed, 478 insertions(+), 347 deletions(-) > > -- > Chuck Lever > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 00/20] NFS/RDMA client for 3.20
Hi Chuck, You already answered my one question for this patch series, and I don't have any other comments. Are these ready for me to take now? Are you planning a v3? Thanks, Anna On 01/13/2015 11:24 AM, Chuck Lever wrote: > The following series of patches for the Linux NFS client breaks up > the per-transport buffer pool data structures to help them scale > better with the size of SEND/RECV buffers (the inline threshold), > the maximum NFS r/wsize, and the number of RDMA credits (concurrent > RPC requests). > > The primary change is that the header send buffers have been split > from struct rpcrdma_req. Specific benefits are outlined in the patch > descriptions. > > More pre-requisites are required. Changes to raise the maximum > r/wsize and other limits are left for a future merge window. > > See the topic branch "nfs-rdma-for-3.20" at: > > git://git.linux-nfs.org/projects/cel/cel-2.6.git > > Changes since v1: > - Rebased on v3.19-rc4 > - One short description fixed > - linux-rdma included this time (sorry for the noise) > > --- > > Chuck Lever (20): > xprtrdma: human-readable completion status > xprtrdma: Modernize htonl and ntohl > xprtrdma: Display XIDs in host byte order > xprtrdma: Clean up hdrlen > xprtrdma: Rename "xprt" and "rdma_connect" fields in struct rpcrdma_xprt > xprtrdma: Remove rpcrdma_ep::rep_ia > xprtrdma: Remove rl_mr field, and the mr_chunk union > xprtrdma: Move credit update to RPC reply handler > xprtrdma: Remove rpcrdma_ep::rep_func and ::rep_xprt > xprtrdma: Free the pd if ib_query_qp() fails > xprtrdma: Take struct ib_device_attr off the stack > xprtrdma: Take struct ib_qp_attr and ib_qp_init_attr off the stack > xprtrdma: Simplify synopsis of rpcrdma_buffer_create() > xprtrdma: Refactor rpcrdma_buffer_create() and rpcrdma_buffer_destroy() > xprtrdma: Add struct rpcrdma_regbuf and helpers > xprtrdma: Allocate RPC send buffer separately from struct rpcrdma_req > xprtrdma: Allocate RPC/RDMA send buffer separately from struct > rpcrdma_req > xprtrdma: Allocate RPC/RDMA receive buffer separately from struct > rpcrdma_rep > xprtrdma: Allocate zero pad separately from rpcrdma_buffer > xprtrdma: Clean up after adding regbuf management > > > include/linux/sunrpc/rpc_rdma.h | 14 + > include/linux/sunrpc/svc_rdma.h |2 > net/sunrpc/xprtrdma/rpc_rdma.c | 108 ++ > net/sunrpc/xprtrdma/transport.c | 179 +++-- > net/sunrpc/xprtrdma/verbs.c | 411 > --- > net/sunrpc/xprtrdma/xprt_rdma.h | 111 +++ > 6 files changed, 478 insertions(+), 347 deletions(-) > > -- > Chuck Lever > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 02/20] xprtrdma: Modernize htonl and ntohl
On 01/16/2015 01:56 PM, Chuck Lever wrote:
>
> On Jan 16, 2015, at 1:33 PM, Anna Schumaker wrote:
>
>> Hi Chuck,
>>
>> On 01/13/2015 11:25 AM, Chuck Lever wrote:
>>> Clean up: Replace htonl and ntohl with the be32 equivalents.
>>
>> After applying this patch I still see an ntohl() in both
>> xprtrdma/transport.c and xprtrdma/verbs.c. Should these be changed as well?
>
> Thanks for the review!
>
> The one in verbs.c is removed in 08/20.
I'll take a look in a sec, I'm up to 07/20 :)
>
> I was mostly interested in updating the XDR usage of the byte
> swapping macros. For sin_addr, transport.c uses ntohl() the same way
> as xprtsock.c. It’s typical for IP address manipulation to use ntohl().
> git grep shows only two or three places in the kernel where the new
> style byte-swap macros are used with sin_addr.
Fair enough. Thanks for answering!
Anna
>
> The code in xprt_rdma_format_addresses() should be fixed up to handle
> IPv6 too.
>
>> Thanks,
>> Anna
>>>
>>> Signed-off-by: Chuck Lever
>>> ---
>>> include/linux/sunrpc/rpc_rdma.h |9 +++
>>> include/linux/sunrpc/svc_rdma.h |2 --
>>> net/sunrpc/xprtrdma/rpc_rdma.c | 48
>>> +--
>>> 3 files changed, 35 insertions(+), 24 deletions(-)
>>>
>>> diff --git a/include/linux/sunrpc/rpc_rdma.h
>>> b/include/linux/sunrpc/rpc_rdma.h
>>> index b78f16b..1578ed2 100644
>>> --- a/include/linux/sunrpc/rpc_rdma.h
>>> +++ b/include/linux/sunrpc/rpc_rdma.h
>>> @@ -42,6 +42,9 @@
>>>
>>> #include
>>>
>>> +#define RPCRDMA_VERSION1
>>> +#define rpcrdma_versioncpu_to_be32(RPCRDMA_VERSION)
>>> +
>>> struct rpcrdma_segment {
>>> __be32 rs_handle; /* Registered memory handle */
>>> __be32 rs_length; /* Length of the chunk in bytes */
>>> @@ -115,4 +118,10 @@ enum rpcrdma_proc {
>>> RDMA_ERROR = 4 /* An RPC RDMA encoding error */
>>> };
>>>
>>> +#define rdma_msg cpu_to_be32(RDMA_MSG)
>>> +#define rdma_nomsg cpu_to_be32(RDMA_NOMSG)
>>> +#define rdma_msgp cpu_to_be32(RDMA_MSGP)
>>> +#define rdma_done cpu_to_be32(RDMA_DONE)
>>> +#define rdma_error cpu_to_be32(RDMA_ERROR)
>>> +
>>> #endif /* _LINUX_SUNRPC_RPC_RDMA_H */
>>> diff --git a/include/linux/sunrpc/svc_rdma.h
>>> b/include/linux/sunrpc/svc_rdma.h
>>> index 975da75..ddfe88f 100644
>>> --- a/include/linux/sunrpc/svc_rdma.h
>>> +++ b/include/linux/sunrpc/svc_rdma.h
>>> @@ -63,8 +63,6 @@ extern atomic_t rdma_stat_rq_prod;
>>> extern atomic_t rdma_stat_sq_poll;
>>> extern atomic_t rdma_stat_sq_prod;
>>>
>>> -#define RPCRDMA_VERSION 1
>>> -
>>> /*
>>> * Contexts are built when an RDMA request is created and are a
>>> * record of the resources that can be recovered when the request
>>> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
>>> index df01d12..a6fb30b 100644
>>> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
>>> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
>>> @@ -209,9 +209,11 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct
>>> xdr_buf *target,
>>> if (cur_rchunk) { /* read */
>>> cur_rchunk->rc_discrim = xdr_one;
>>> /* all read chunks have the same "position" */
>>> - cur_rchunk->rc_position = htonl(pos);
>>> - cur_rchunk->rc_target.rs_handle = htonl(seg->mr_rkey);
>>> - cur_rchunk->rc_target.rs_length = htonl(seg->mr_len);
>>> + cur_rchunk->rc_position = cpu_to_be32(pos);
>>> + cur_rchunk->rc_target.rs_handle =
>>> + cpu_to_be32(seg->mr_rkey);
>>> + cur_rchunk->rc_target.rs_length =
>>> + cpu_to_be32(seg->mr_len);
>>> xdr_encode_hyper(
>>> (__be32
>>> *)&cur_rchunk->rc_target.rs_offset,
>>> seg->mr_base);
>>> @@ -222,8 +224,10 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct
>>> xdr_buf *target,
>>> cur_rchunk++;
>>> r_xprt->rx_stats.read_chunk_c
Re: [PATCH v2 02/20] xprtrdma: Modernize htonl and ntohl
Hi Chuck,
On 01/13/2015 11:25 AM, Chuck Lever wrote:
> Clean up: Replace htonl and ntohl with the be32 equivalents.
After applying this patch I still see an ntohl() in both xprtrdma/transport.c
and xprtrdma/verbs.c. Should these be changed as well?
Thanks,
Anna
>
> Signed-off-by: Chuck Lever
> ---
> include/linux/sunrpc/rpc_rdma.h |9 +++
> include/linux/sunrpc/svc_rdma.h |2 --
> net/sunrpc/xprtrdma/rpc_rdma.c | 48
> +--
> 3 files changed, 35 insertions(+), 24 deletions(-)
>
> diff --git a/include/linux/sunrpc/rpc_rdma.h b/include/linux/sunrpc/rpc_rdma.h
> index b78f16b..1578ed2 100644
> --- a/include/linux/sunrpc/rpc_rdma.h
> +++ b/include/linux/sunrpc/rpc_rdma.h
> @@ -42,6 +42,9 @@
>
> #include
>
> +#define RPCRDMA_VERSION 1
> +#define rpcrdma_version cpu_to_be32(RPCRDMA_VERSION)
> +
> struct rpcrdma_segment {
> __be32 rs_handle; /* Registered memory handle */
> __be32 rs_length; /* Length of the chunk in bytes */
> @@ -115,4 +118,10 @@ enum rpcrdma_proc {
> RDMA_ERROR = 4 /* An RPC RDMA encoding error */
> };
>
> +#define rdma_msg cpu_to_be32(RDMA_MSG)
> +#define rdma_nomsg cpu_to_be32(RDMA_NOMSG)
> +#define rdma_msgpcpu_to_be32(RDMA_MSGP)
> +#define rdma_donecpu_to_be32(RDMA_DONE)
> +#define rdma_error cpu_to_be32(RDMA_ERROR)
> +
> #endif /* _LINUX_SUNRPC_RPC_RDMA_H */
> diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h
> index 975da75..ddfe88f 100644
> --- a/include/linux/sunrpc/svc_rdma.h
> +++ b/include/linux/sunrpc/svc_rdma.h
> @@ -63,8 +63,6 @@ extern atomic_t rdma_stat_rq_prod;
> extern atomic_t rdma_stat_sq_poll;
> extern atomic_t rdma_stat_sq_prod;
>
> -#define RPCRDMA_VERSION 1
> -
> /*
> * Contexts are built when an RDMA request is created and are a
> * record of the resources that can be recovered when the request
> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
> index df01d12..a6fb30b 100644
> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
> @@ -209,9 +209,11 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct
> xdr_buf *target,
> if (cur_rchunk) { /* read */
> cur_rchunk->rc_discrim = xdr_one;
> /* all read chunks have the same "position" */
> - cur_rchunk->rc_position = htonl(pos);
> - cur_rchunk->rc_target.rs_handle = htonl(seg->mr_rkey);
> - cur_rchunk->rc_target.rs_length = htonl(seg->mr_len);
> + cur_rchunk->rc_position = cpu_to_be32(pos);
> + cur_rchunk->rc_target.rs_handle =
> + cpu_to_be32(seg->mr_rkey);
> + cur_rchunk->rc_target.rs_length =
> + cpu_to_be32(seg->mr_len);
> xdr_encode_hyper(
> (__be32
> *)&cur_rchunk->rc_target.rs_offset,
> seg->mr_base);
> @@ -222,8 +224,10 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct
> xdr_buf *target,
> cur_rchunk++;
> r_xprt->rx_stats.read_chunk_count++;
> } else {/* write/reply */
> - cur_wchunk->wc_target.rs_handle = htonl(seg->mr_rkey);
> - cur_wchunk->wc_target.rs_length = htonl(seg->mr_len);
> + cur_wchunk->wc_target.rs_handle =
> + cpu_to_be32(seg->mr_rkey);
> + cur_wchunk->wc_target.rs_length =
> + cpu_to_be32(seg->mr_len);
> xdr_encode_hyper(
> (__be32
> *)&cur_wchunk->wc_target.rs_offset,
> seg->mr_base);
> @@ -257,7 +261,7 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct
> xdr_buf *target,
> *iptr++ = xdr_zero; /* encode a NULL reply chunk */
> } else {
> warray->wc_discrim = xdr_one;
> - warray->wc_nchunks = htonl(nchunks);
> + warray->wc_nchunks = cpu_to_be32(nchunks);
> iptr = (__be32 *) cur_wchunk;
> if (type == rpcrdma_writech) {
> *iptr++ = xdr_zero; /* finish the write chunk list */
> @@ -404,11 +408,11 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
>
> /* build RDMA header in private area at front */
> headerp = (struct rpcrdma_msg *) req->rl_base;
> - /* don't htonl XID, it's already done in request */
> + /* don't byte-swap XID, it's already done in request */
> headerp->rm_xid = rqst->rq_xid;
> - headerp->rm_vers = xdr_one;
> - headerp->rm_credit = htonl(r_xprt->rx_buf.rb_max_request
Re: [PATCH v2 00/10] NFS/RDMA patches for 3.19
Hi Chuck, I've applied these to two different branches - one for the RDMA changes and another for the generic client side changes. Both have been sent to Trond to be merged with 3.19. Anna On 11/08/2014 08:14 PM, Chuck Lever wrote: > Hi- > > This is a set of fixes and clean-ups for xprtrdma. Also available > in the nfs-rdma-for-3.19 topic branch at > > git://linux-nfs.org/projects/cel/cel-2.6.git > > I've run cthon04, iozone, dbench, multi-process kernel build and > xfstests with NFSv3 and NFSv4.0 against a Linux and a Solaris > NFS server via RDMA, with all three memory registration modes. > > Changes since v1: > > - Merged up to v3.18-rc3 > - Dropped patches that enable NFSv4.1 support for now > - CQ completion vector spinning dropped for now > - Addressed Anna's review comments > - Several bug fixes > > --- > > Chuck Lever (10): > xprtrdma: Return an errno from rpcrdma_register_external() > xprtrdma: Cap req_cqinit > xprtrdma: unmap all FMRs during transport disconnect > xprtrdma: Refactor tasklet scheduling > xprtrdma: Re-write rpcrdma_flush_cqs() > xprtrdma: Enable pad optimization > xprtrdma: Display async errors > SUNRPC: serialize iostats updates > NFS: SETCLIENTID XDR buffer sizes are incorrect > NFS: Clean up nfs4_init_callback() > > > fs/nfs/nfs4client.c | 31 +-- > fs/nfs/nfs4xdr.c| 10 ++- > include/linux/sunrpc/metrics.h |3 + > net/sunrpc/stats.c | 21 +-- > net/sunrpc/xprtrdma/transport.c |4 + > net/sunrpc/xprtrdma/verbs.c | 114 > ++- > net/sunrpc/xprtrdma/xprt_rdma.h |6 ++ > 7 files changed, 146 insertions(+), 43 deletions(-) > > -- > Chuck Lever > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 09/10] NFS: SETCLIENTID XDR buffer sizes are incorrect
Hi Chuck, It looks like this patch and the next one are general client changes, so they might have to be submitted to Trond directly rather than going through my tree. Trond, what do you think? Anna On 11/08/2014 08:15 PM, Chuck Lever wrote: > Use the correct calculation of the maximum size of a clientaddr4 > when encoding and decoding SETCLIENTID operations. clientaddr4 is > defined in section 2.2.10 of RFC3530bis-31. > > The usage in encode_setclientid_maxsz is missing the 4-byte length > in both strings, but is otherwise correct. decode_setclientid_maxsz > simply asks for a page of receive buffer space, which is > unnecessarily large (more than 4KB). > > Note that a SETCLIENTID reply is either clientid+verifier, or > clientaddr4, depending on the returned NFS status. It doesn't > hurt to allocate enough space for both. > > Signed-off-by: Chuck Lever > --- > fs/nfs/nfs4xdr.c | 10 ++ > 1 file changed, 6 insertions(+), 4 deletions(-) > > diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c > index 206c08a..f8afa67 100644 > --- a/fs/nfs/nfs4xdr.c > +++ b/fs/nfs/nfs4xdr.c > @@ -141,13 +141,15 @@ static int nfs4_stat_to_errno(int); > XDR_QUADLEN(NFS4_VERIFIER_SIZE) + \ > XDR_QUADLEN(NFS4_SETCLIENTID_NAMELEN) + \ > 1 /* sc_prog */ + \ > - XDR_QUADLEN(RPCBIND_MAXNETIDLEN) + \ > - XDR_QUADLEN(RPCBIND_MAXUADDRLEN) + \ > + 1 + XDR_QUADLEN(RPCBIND_MAXNETIDLEN) + \ > + 1 + XDR_QUADLEN(RPCBIND_MAXUADDRLEN) + \ > 1) /* sc_cb_ident */ > #define decode_setclientid_maxsz \ > (op_decode_hdr_maxsz + \ > - 2 + \ > - 1024) /* large value for CLID_INUSE */ > + 2 /* clientid */ + \ > + XDR_QUADLEN(NFS4_VERIFIER_SIZE) + \ > + 1 + XDR_QUADLEN(RPCBIND_MAXNETIDLEN) + \ > + 1 + XDR_QUADLEN(RPCBIND_MAXUADDRLEN)) > #define encode_setclientid_confirm_maxsz \ > (op_encode_hdr_maxsz + \ > 3 + (NFS4_VERIFIER_SIZE >> 2)) > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 06/10] xprtrdma: Enable pad optimization
Hey Chuck,
On 11/08/2014 08:14 PM, Chuck Lever wrote:
> The Linux NFS/RDMA server used to reject NFSv3 WRITE requests when
> pad optimization was enabled. That bug was fixed by commit
> e560e3b510d2 ("svcrdma: Add zero padding if the client doesn't send
> it").
Do we need to worry about backwards compatibility with servers that don't have
this patch?
Anna
>
> We can now enable pad optimization on the client, which helps
> performance and is supported now by both Linux and Solaris servers.
>
> Signed-off-by: Chuck Lever
> ---
> net/sunrpc/xprtrdma/transport.c |2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c
> index cfe9a81..8ed2576 100644
> --- a/net/sunrpc/xprtrdma/transport.c
> +++ b/net/sunrpc/xprtrdma/transport.c
> @@ -73,7 +73,7 @@ static unsigned int xprt_rdma_max_inline_read =
> RPCRDMA_DEF_INLINE;
> static unsigned int xprt_rdma_max_inline_write = RPCRDMA_DEF_INLINE;
> static unsigned int xprt_rdma_inline_write_padding;
> static unsigned int xprt_rdma_memreg_strategy = RPCRDMA_FRMR;
> -int xprt_rdma_pad_optimize = 0;
> + int xprt_rdma_pad_optimize = 1;
>
> #ifdef RPC_DEBUG
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] Please pull NFS client RDMA changes for 3.17
Hey Trond, The following changes since commit 9a3c4145af32125c5ee39c0272662b47307a8323: Linux 3.16-rc6 (2014-07-20 21:04:16 -0700) are available in the git repository at: git://git.linux-nfs.org/projects/anna/nfs-rdma.git nfs-rdma for you to fetch changes up to 8079fb785e34de6dff34bd846b8b79c212861edf: xprtrdma: Handle additional connection events (2014-07-31 16:22:59 -0400) Thanks! Anna Chuck Lever (21): xprtrdma: Fix panic in rpcrdma_register_frmr_external() xprtrdma: Protect ia->ri_id when unmapping/invalidating MRs xprtrdma: Limit data payload size for ALLPHYSICAL xprtrdma: Update rkeys after transport reconnect xprtrdma: On disconnect, don't ignore pending CQEs xprtrdma: Don't invalidate FRMRs if registration fails xprtrdma: Unclutter struct rpcrdma_mr_seg xprtrdma: Back off rkey when FAST_REG_MR fails xprtrdma: Chain together all MWs in same buffer pool xprtrdma: Properly handle exhaustion of the rb_mws list xprtrdma: Reset FRMRs when FAST_REG_MR is flushed by a disconnect xprtrdma: Reset FRMRs after a flushed LOCAL_INV Work Request xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() xprtrdma: Disable completions for FAST_REG_MR Work Requests xprtrdma: Disable completions for LOCAL_INV Work Requests xprtrdma: Rename frmr_wr xprtrdma: Allocate each struct rpcrdma_mw separately xprtrdma: Schedule reply tasklet once per upcall xprtrdma: Make rpcrdma_ep_disconnect() return void xprtrdma: Remove RPCRDMA_PERSISTENT_REGISTRATION macro xprtrdma: Handle additional connection events Yan Burman (1): xprtrdma: Fix DMA-API-DEBUG warning by checking dma_map result include/linux/sunrpc/xprtrdma.h | 2 - net/sunrpc/xprtrdma/rpc_rdma.c | 83 +-- net/sunrpc/xprtrdma/transport.c | 17 ++- net/sunrpc/xprtrdma/verbs.c | 739 ++--- net/sunrpc/xprtrdma/xprt_rdma.h | 61 ++-- 5 files changed, 604 insertions(+), 298 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 00/21] NFS/RDMA client patches for 3.17
I've applied v5 and updated my tree. Thanks, Chuck! Anna On 07/29/2014 05:23 PM, Chuck Lever wrote: > The main purpose of this series is to address connection drop > recovery issues by fixing FRMR re-use to make it less likely the > client will deadlock due to a memory management operation error. > Some clean-ups and other fixes are present as well. > > See topic branch nfs-rdma-for-3.17-plan-b in > > git://git.linux-nfs.org/projects/cel/cel-2.6.git > > Topic branch nfs-rdma-for-3.17 still contains v4 of this series. > You can choose whichever series is convenient for you and Roland. > > I tested with NFSv3 and NFSv4 on all three supported memory > registration modes. Used cthon04, iozone, vdbench, and dbench with > both Solaris and Linux NFS/RDMA servers. Used xfstests with Linux. > > > v5: > > - Rebased on v3.16-rc7 > > - Replaced patch 08/21 with an equivalent one-liner that does not > alter include/rdma/ib_verbs.h. A couple of subsequent patches were > updated to apply on the new 08/21. This allows the whole series to > be submitted through the NFS tree. > > > v4: > > - Rebased on v3.16-rc6 > > - Add Tested-by: from Shirley Ma and Devesh Sharma > > > v3: > Only two substantive changes: > > - Patch 08/21 now uses generic IB helpers for managing FRMR > rkeys > > - Add Tested-by: from Steve Wise > > > v2: > Many patches from v1 have been written or replaced. > > The MW ref counting approach in v1 is abandoned. Instead, I've > eliminated signaling FAST_REG_MR and LOCAL_INV, and added > appropriate recovery mechanisms after a transport reconnect that > should prevent rkey dis-synchrony entirely. > > A couple of optimizations have been added, including: > > - Allocating each MW separately rather than carving each out of a > large piece of contiguous memory > > - Now that the receive CQ upcall handler dequeues a bundle of CQEs > at once, fire off the reply handler tasklet just once per upcall > to reduce context switches and how often hard IRQs are disabled > > --- > > Chuck Lever (21): > xprtrdma: Fix panic in rpcrdma_register_frmr_external() > xprtrdma: Protect ia->ri_id when unmapping/invalidating MRs > xprtrdma: Limit data payload size for ALLPHYSICAL > xprtrdma: Update rkeys after transport reconnect > xprtrdma: On disconnect, don't ignore pending CQEs > xprtrdma: Don't invalidate FRMRs if registration fails > xprtrdma: Unclutter struct rpcrdma_mr_seg > xprtrdma: Back off rkey when FAST_REG_MR fails > xprtrdma: Chain together all MWs in same buffer pool > xprtrdma: Properly handle exhaustion of the rb_mws list > xprtrdma: Reset FRMRs when FAST_REG_MR is flushed by a disconnect > xprtrdma: Reset FRMRs after a flushed LOCAL_INV Work Request > xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() > xprtrdma: Disable completions for FAST_REG_MR Work Requests > xprtrdma: Disable completions for LOCAL_INV Work Requests > xprtrdma: Rename frmr_wr > xprtrdma: Allocate each struct rpcrdma_mw separately > xprtrdma: Schedule reply tasklet once per upcall > xprtrdma: Make rpcrdma_ep_disconnect() return void > xprtrdma: Remove RPCRDMA_PERSISTENT_REGISTRATION macro > xprtrdma: Handle additional connection events > > > include/linux/sunrpc/xprtrdma.h |2 > net/sunrpc/xprtrdma/rpc_rdma.c | 83 ++-- > net/sunrpc/xprtrdma/transport.c | 17 + > net/sunrpc/xprtrdma/verbs.c | 736 > ++- > net/sunrpc/xprtrdma/xprt_rdma.h | 61 +++ > 5 files changed, 601 insertions(+), 298 deletions(-) > > -- > Chuck Lever > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 08/21] xprtrdma: Back off rkey when FAST_REG_MR fails
Hey Roland, Sean and Hal.
We would like to submit this patch through the NFS tree, since the function
added to include/rdma/ib_verbs.h is only used for NFS over RDMA. Do any of you
have a problem with this patch, and can I have an acked-by if this is okay?
Thanks!
Anna
On 07/22/2014 10:52 AM, Chuck Lever wrote:
> If posting a FAST_REG_MR Work Reqeust fails, revert the rkey update
> to avoid subsequent IB_WC_MW_BIND_ERR completions.
>
> Suggested-by: Steve Wise
> Signed-off-by: Chuck Lever
> Tested-by: Shirley Ma
> Tested-by: Devesh Sharma
> ---
> include/rdma/ib_verbs.h | 11 +++
> net/sunrpc/xprtrdma/verbs.c | 12 +---
> 2 files changed, 16 insertions(+), 7 deletions(-)
>
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 7ccef34..c5c7ec6 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -2479,6 +2479,17 @@ static inline u32 ib_inc_rkey(u32 rkey)
> }
>
> /**
> + * ib_dec_rkey - decrements the key portion of the given rkey. Can be used
> + * when recovering from a local immediate error.
> + * @rkey - the rkey to decrement.
> + */
> +static inline u32 ib_dec_rkey(u32 rkey)
> +{
> + const u32 mask = 0x00ff;
> + return ((rkey - 1) & mask) | (rkey & ~mask);
> +}
> +
> +/**
> * ib_alloc_mw - Allocates a memory window.
> * @pd: The protection domain associated with the memory window.
> * @type: The type of the memory window (1 or 2).
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 8bb7945..394b13f 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1497,13 +1497,12 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> struct rpcrdma_frmr *frmr = &mw->r.frmr;
> struct ib_mr *mr = frmr->fr_mr;
> struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr;
> -
> - u8 key;
> int len, pageoff;
> int i, rc;
> int seg_len;
> u64 pa;
> int page_no;
> + u32 rkey;
>
> pageoff = offset_in_page(seg1->mr_offset);
> seg1->mr_offset -= pageoff; /* start of page */
> @@ -1558,14 +1557,11 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> rc = -EIO;
> goto out_err;
> }
> -
> - /* Bump the key */
> - key = (u8)(mr->rkey & 0x00FF);
> - ib_update_fast_reg_key(mr, ++key);
> -
> frmr_wr.wr.fast_reg.access_flags = (writing ?
> IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE :
> IB_ACCESS_REMOTE_READ);
> + rkey = ib_inc_rkey(mr->rkey);
> + ib_update_fast_reg_key(mr, rkey);
> frmr_wr.wr.fast_reg.rkey = mr->rkey;
> DECR_CQCOUNT(&r_xprt->rx_ep);
>
> @@ -1574,6 +1570,8 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> if (rc) {
> dprintk("RPC: %s: failed ib_post_send for register,"
> " status %i\n", __func__, rc);
> + rkey = ib_dec_rkey(mr->rkey);
> + ib_update_fast_reg_key(mr, rkey);
> goto out_err;
> } else {
> seg1->mr_rkey = mr->rkey;
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 08/21] xprtrdma: Back off rkey when FAST_REG_MR fails
Hi Roland,
I'm collecting changes for NFS over RDMA, and I wasn't sure if the change to
include/rdma/ib_verbs.h in this patch (adding ib_dec_rkey()) was something that
could be merged through the NFS tree. What do you think?
Thanks!
Anna
On 07/22/2014 10:52 AM, Chuck Lever wrote:
> If posting a FAST_REG_MR Work Reqeust fails, revert the rkey update
> to avoid subsequent IB_WC_MW_BIND_ERR completions.
>
> Suggested-by: Steve Wise
> Signed-off-by: Chuck Lever
> Tested-by: Shirley Ma
> Tested-by: Devesh Sharma
> ---
> include/rdma/ib_verbs.h | 11 +++
> net/sunrpc/xprtrdma/verbs.c | 12 +---
> 2 files changed, 16 insertions(+), 7 deletions(-)
>
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 7ccef34..c5c7ec6 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -2479,6 +2479,17 @@ static inline u32 ib_inc_rkey(u32 rkey)
> }
>
> /**
> + * ib_dec_rkey - decrements the key portion of the given rkey. Can be used
> + * when recovering from a local immediate error.
> + * @rkey - the rkey to decrement.
> + */
> +static inline u32 ib_dec_rkey(u32 rkey)
> +{
> + const u32 mask = 0x00ff;
> + return ((rkey - 1) & mask) | (rkey & ~mask);
> +}
> +
> +/**
> * ib_alloc_mw - Allocates a memory window.
> * @pd: The protection domain associated with the memory window.
> * @type: The type of the memory window (1 or 2).
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 8bb7945..394b13f 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1497,13 +1497,12 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> struct rpcrdma_frmr *frmr = &mw->r.frmr;
> struct ib_mr *mr = frmr->fr_mr;
> struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr;
> -
> - u8 key;
> int len, pageoff;
> int i, rc;
> int seg_len;
> u64 pa;
> int page_no;
> + u32 rkey;
>
> pageoff = offset_in_page(seg1->mr_offset);
> seg1->mr_offset -= pageoff; /* start of page */
> @@ -1558,14 +1557,11 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> rc = -EIO;
> goto out_err;
> }
> -
> - /* Bump the key */
> - key = (u8)(mr->rkey & 0x00FF);
> - ib_update_fast_reg_key(mr, ++key);
> -
> frmr_wr.wr.fast_reg.access_flags = (writing ?
> IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE :
> IB_ACCESS_REMOTE_READ);
> + rkey = ib_inc_rkey(mr->rkey);
> + ib_update_fast_reg_key(mr, rkey);
> frmr_wr.wr.fast_reg.rkey = mr->rkey;
> DECR_CQCOUNT(&r_xprt->rx_ep);
>
> @@ -1574,6 +1570,8 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> if (rc) {
> dprintk("RPC: %s: failed ib_post_send for register,"
> " status %i\n", __func__, rc);
> + rkey = ib_dec_rkey(mr->rkey);
> + ib_update_fast_reg_key(mr, rkey);
> goto out_err;
> } else {
> seg1->mr_rkey = mr->rkey;
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 00/21] NFS/RDMA client patches for 3.17
On 07/23/2014 05:23 PM, Chuck Lever wrote: > > >> On Jul 23, 2014, at 2:53 PM, Anna Schumaker >> wrote: >> >> I've applied these changes and pushed them out in my nfs-rdma branch. I'm >> planning to send a pull request to Trond on Friday so these changes can >> spend some time in linux-next before the next merge window opens. > > Thank you! Did you resolve your question about who will merge the ib_dec_rkey > patch, or do you want me to pursue that? I don't have an answer yet (fingers crossed that it's not a problem!). I think adding ib_dec_rkey() would need to go through Roland Dreier if it doesn't go through me. Anna > >> >> Anna >> >>> On 07/22/2014 10:50 AM, Chuck Lever wrote: >>> The main purpose of this series is to address connection drop >>> recovery issues by fixing FRMR re-use to make it less likely the >>> client will deadlock due to a memory management operation error. >>> Some clean-ups and other fixes are present as well. >>> >>> Anna, v4 of this series should be ready for merging. >>> >>> See topic branch nfs-rdma-for-3.17 in >>> >>> git://git.linux-nfs.org/projects/cel/cel-2.6.git >>> >>> I tested with NFSv3 and NFSv4 on all three supported memory >>> registration modes. Used cthon04, iozone, and dbench with both >>> Solaris and Linux NFS/RDMA servers. Used xfstests with Linux. >>> >>> v4: >>> >>> - Rebased on v3.16-rc6 >>> >>> - Add Tested-by: from Shirley Ma and Devesh Sharma >>> >>> >>> v3: >>> Only two substantive changes: >>> >>> - Patch 08/21 now uses generic IB helpers for managing FRMR >>> rkeys >>> >>> - Add Tested-by: from Steve Wise >>> >>> >>> v2: >>> Many patches from v1 have been written or replaced. >>> >>> The MW ref counting approach in v1 is abandoned. Instead, I've >>> eliminated signaling FAST_REG_MR and LOCAL_INV, and added >>> appropriate recovery mechanisms after a transport reconnect that >>> should prevent rkey dis-synchrony entirely. >>> >>> A couple of optimizations have been added, including: >>> >>> - Allocating each MW separately rather than carving each out of a >>> large piece of contiguous memory >>> >>> - Now that the receive CQ upcall handler dequeues a bundle of CQEs >>> at once, fire off the reply handler tasklet just once per upcall >>> to reduce context switches and how often hard IRQs are disabled >>> >>> --- >>> >>> Chuck Lever (21): >>> xprtrdma: Fix panic in rpcrdma_register_frmr_external() >>> xprtrdma: Protect ia->ri_id when unmapping/invalidating MRs >>> xprtrdma: Limit data payload size for ALLPHYSICAL >>> xprtrdma: Update rkeys after transport reconnect >>> xprtrdma: On disconnect, don't ignore pending CQEs >>> xprtrdma: Don't invalidate FRMRs if registration fails >>> xprtrdma: Unclutter struct rpcrdma_mr_seg >>> xprtrdma: Back off rkey when FAST_REG_MR fails >>> xprtrdma: Chain together all MWs in same buffer pool >>> xprtrdma: Properly handle exhaustion of the rb_mws list >>> xprtrdma: Reset FRMRs when FAST_REG_MR is flushed by a disconnect >>> xprtrdma: Reset FRMRs after a flushed LOCAL_INV Work Request >>> xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() >>> xprtrdma: Disable completions for FAST_REG_MR Work Requests >>> xprtrdma: Disable completions for LOCAL_INV Work Requests >>> xprtrdma: Rename frmr_wr >>> xprtrdma: Allocate each struct rpcrdma_mw separately >>> xprtrdma: Schedule reply tasklet once per upcall >>> xprtrdma: Make rpcrdma_ep_disconnect() return void >>> xprtrdma: Remove RPCRDMA_PERSISTENT_REGISTRATION macro >>> xprtrdma: Handle additional connection events >>> >>> >>> include/linux/sunrpc/xprtrdma.h |2 >>> include/rdma/ib_verbs.h | 11 + >>> net/sunrpc/xprtrdma/rpc_rdma.c | 83 ++-- >>> net/sunrpc/xprtrdma/transport.c | 17 + >>> net/sunrpc/xprtrdma/verbs.c | 744 >>> ++- >>> net/sunrpc/xprtrdma/xprt_rdma.h | 61 +++ >>> 6 files changed, 615 insertions(+), 303 deletions(-) >>> >>> -- >>> Chuck Lever >>> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 00/21] NFS/RDMA client patches for 3.17
I've applied these changes and pushed them out in my nfs-rdma branch. I'm planning to send a pull request to Trond on Friday so these changes can spend some time in linux-next before the next merge window opens. Anna On 07/22/2014 10:50 AM, Chuck Lever wrote: > The main purpose of this series is to address connection drop > recovery issues by fixing FRMR re-use to make it less likely the > client will deadlock due to a memory management operation error. > Some clean-ups and other fixes are present as well. > > Anna, v4 of this series should be ready for merging. > > See topic branch nfs-rdma-for-3.17 in > > git://git.linux-nfs.org/projects/cel/cel-2.6.git > > I tested with NFSv3 and NFSv4 on all three supported memory > registration modes. Used cthon04, iozone, and dbench with both > Solaris and Linux NFS/RDMA servers. Used xfstests with Linux. > > v4: > > - Rebased on v3.16-rc6 > > - Add Tested-by: from Shirley Ma and Devesh Sharma > > > v3: > Only two substantive changes: > > - Patch 08/21 now uses generic IB helpers for managing FRMR > rkeys > > - Add Tested-by: from Steve Wise > > > v2: > Many patches from v1 have been written or replaced. > > The MW ref counting approach in v1 is abandoned. Instead, I've > eliminated signaling FAST_REG_MR and LOCAL_INV, and added > appropriate recovery mechanisms after a transport reconnect that > should prevent rkey dis-synchrony entirely. > > A couple of optimizations have been added, including: > > - Allocating each MW separately rather than carving each out of a > large piece of contiguous memory > > - Now that the receive CQ upcall handler dequeues a bundle of CQEs > at once, fire off the reply handler tasklet just once per upcall > to reduce context switches and how often hard IRQs are disabled > > --- > > Chuck Lever (21): > xprtrdma: Fix panic in rpcrdma_register_frmr_external() > xprtrdma: Protect ia->ri_id when unmapping/invalidating MRs > xprtrdma: Limit data payload size for ALLPHYSICAL > xprtrdma: Update rkeys after transport reconnect > xprtrdma: On disconnect, don't ignore pending CQEs > xprtrdma: Don't invalidate FRMRs if registration fails > xprtrdma: Unclutter struct rpcrdma_mr_seg > xprtrdma: Back off rkey when FAST_REG_MR fails > xprtrdma: Chain together all MWs in same buffer pool > xprtrdma: Properly handle exhaustion of the rb_mws list > xprtrdma: Reset FRMRs when FAST_REG_MR is flushed by a disconnect > xprtrdma: Reset FRMRs after a flushed LOCAL_INV Work Request > xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() > xprtrdma: Disable completions for FAST_REG_MR Work Requests > xprtrdma: Disable completions for LOCAL_INV Work Requests > xprtrdma: Rename frmr_wr > xprtrdma: Allocate each struct rpcrdma_mw separately > xprtrdma: Schedule reply tasklet once per upcall > xprtrdma: Make rpcrdma_ep_disconnect() return void > xprtrdma: Remove RPCRDMA_PERSISTENT_REGISTRATION macro > xprtrdma: Handle additional connection events > > > include/linux/sunrpc/xprtrdma.h |2 > include/rdma/ib_verbs.h | 11 + > net/sunrpc/xprtrdma/rpc_rdma.c | 83 ++-- > net/sunrpc/xprtrdma/transport.c | 17 + > net/sunrpc/xprtrdma/verbs.c | 744 > ++- > net/sunrpc/xprtrdma/xprt_rdma.h | 61 +++ > 6 files changed, 615 insertions(+), 303 deletions(-) > > -- > Chuck Lever > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 08/21] xprtrdma: Back off rkey when FAST_REG_MR fails
On 07/22/2014 05:18 PM, Chuck Lever wrote:
>
> On Jul 22, 2014, at 4:03 PM, Anna Schumaker wrote:
>
>> On 07/22/2014 10:52 AM, Chuck Lever wrote:
>>> If posting a FAST_REG_MR Work Reqeust fails, revert the rkey update
>>> to avoid subsequent IB_WC_MW_BIND_ERR completions.
>>>
>>> Suggested-by: Steve Wise
>>> Signed-off-by: Chuck Lever
>>> Tested-by: Shirley Ma
>>> Tested-by: Devesh Sharma
>>> ---
>>> include/rdma/ib_verbs.h | 11 +++
>>> net/sunrpc/xprtrdma/verbs.c | 12 +---
>>> 2 files changed, 16 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>>> index 7ccef34..c5c7ec6 100644
>>> --- a/include/rdma/ib_verbs.h
>>> +++ b/include/rdma/ib_verbs.h
>>> @@ -2479,6 +2479,17 @@ static inline u32 ib_inc_rkey(u32 rkey)
>>> }
>>>
>>> /**
>>> + * ib_dec_rkey - decrements the key portion of the given rkey. Can be used
>>> + * when recovering from a local immediate error.
>>> + * @rkey - the rkey to decrement.
>>> + */
>>> +static inline u32 ib_dec_rkey(u32 rkey)
>>> +{
>>> + const u32 mask = 0x00ff;
>>> + return ((rkey - 1) & mask) | (rkey & ~mask);
>>
>> ib_inc_rkey() uses the same mask, and ib_update_fast_reg_key() uses !mask.
>> I think this should be a #define constant, rather than hardcoded in four
>> places.
>
> Such a clean up is not related to the change in this patch.
>
> However, it could go as a separate patch, sent from someone
> who can test the other current users of ib_inc_rkey() and
> ib_update_fast_reg_key().
I agree with separate patch sometime in the future. Is it appropriate to add
ib_dec_rkey() through the NFS tree, or is there an Infiniband tree that this
should go through instead?
Anna
>
>
>> Anna
>>
>>
>>> +}
>>> +
>>> +/**
>>> * ib_alloc_mw - Allocates a memory window.
>>> * @pd: The protection domain associated with the memory window.
>>> * @type: The type of the memory window (1 or 2).
>>> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
>>> index 8bb7945..394b13f 100644
>>> --- a/net/sunrpc/xprtrdma/verbs.c
>>> +++ b/net/sunrpc/xprtrdma/verbs.c
>>> @@ -1497,13 +1497,12 @@ rpcrdma_register_frmr_external(struct
>>> rpcrdma_mr_seg *seg,
>>> struct rpcrdma_frmr *frmr = &mw->r.frmr;
>>> struct ib_mr *mr = frmr->fr_mr;
>>> struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr;
>>> -
>>> - u8 key;
>>> int len, pageoff;
>>> int i, rc;
>>> int seg_len;
>>> u64 pa;
>>> int page_no;
>>> + u32 rkey;
>>>
>>> pageoff = offset_in_page(seg1->mr_offset);
>>> seg1->mr_offset -= pageoff; /* start of page */
>>> @@ -1558,14 +1557,11 @@ rpcrdma_register_frmr_external(struct
>>> rpcrdma_mr_seg *seg,
>>> rc = -EIO;
>>> goto out_err;
>>> }
>>> -
>>> - /* Bump the key */
>>> - key = (u8)(mr->rkey & 0x00FF);
>>> - ib_update_fast_reg_key(mr, ++key);
>>> -
>>> frmr_wr.wr.fast_reg.access_flags = (writing ?
>>> IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE :
>>> IB_ACCESS_REMOTE_READ);
>>> + rkey = ib_inc_rkey(mr->rkey);
>>> + ib_update_fast_reg_key(mr, rkey);
>>> frmr_wr.wr.fast_reg.rkey = mr->rkey;
>>> DECR_CQCOUNT(&r_xprt->rx_ep);
>>>
>>> @@ -1574,6 +1570,8 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
>>> *seg,
>>> if (rc) {
>>> dprintk("RPC: %s: failed ib_post_send for register,"
>>> " status %i\n", __func__, rc);
>>> + rkey = ib_dec_rkey(mr->rkey);
>>> + ib_update_fast_reg_key(mr, rkey);
>>> goto out_err;
>>> } else {
>>> seg1->mr_rkey = mr->rkey;
>>>
>>
>
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 08/21] xprtrdma: Back off rkey when FAST_REG_MR fails
On 07/22/2014 10:52 AM, Chuck Lever wrote:
> If posting a FAST_REG_MR Work Reqeust fails, revert the rkey update
> to avoid subsequent IB_WC_MW_BIND_ERR completions.
>
> Suggested-by: Steve Wise
> Signed-off-by: Chuck Lever
> Tested-by: Shirley Ma
> Tested-by: Devesh Sharma
> ---
> include/rdma/ib_verbs.h | 11 +++
> net/sunrpc/xprtrdma/verbs.c | 12 +---
> 2 files changed, 16 insertions(+), 7 deletions(-)
>
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 7ccef34..c5c7ec6 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -2479,6 +2479,17 @@ static inline u32 ib_inc_rkey(u32 rkey)
> }
>
> /**
> + * ib_dec_rkey - decrements the key portion of the given rkey. Can be used
> + * when recovering from a local immediate error.
> + * @rkey - the rkey to decrement.
> + */
> +static inline u32 ib_dec_rkey(u32 rkey)
> +{
> + const u32 mask = 0x00ff;
> + return ((rkey - 1) & mask) | (rkey & ~mask);
ib_inc_rkey() uses the same mask, and ib_update_fast_reg_key() uses !mask. I
think this should be a #define constant, rather than hardcoded in four places.
Anna
> +}
> +
> +/**
> * ib_alloc_mw - Allocates a memory window.
> * @pd: The protection domain associated with the memory window.
> * @type: The type of the memory window (1 or 2).
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 8bb7945..394b13f 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1497,13 +1497,12 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> struct rpcrdma_frmr *frmr = &mw->r.frmr;
> struct ib_mr *mr = frmr->fr_mr;
> struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr;
> -
> - u8 key;
> int len, pageoff;
> int i, rc;
> int seg_len;
> u64 pa;
> int page_no;
> + u32 rkey;
>
> pageoff = offset_in_page(seg1->mr_offset);
> seg1->mr_offset -= pageoff; /* start of page */
> @@ -1558,14 +1557,11 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> rc = -EIO;
> goto out_err;
> }
> -
> - /* Bump the key */
> - key = (u8)(mr->rkey & 0x00FF);
> - ib_update_fast_reg_key(mr, ++key);
> -
> frmr_wr.wr.fast_reg.access_flags = (writing ?
> IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE :
> IB_ACCESS_REMOTE_READ);
> + rkey = ib_inc_rkey(mr->rkey);
> + ib_update_fast_reg_key(mr, rkey);
> frmr_wr.wr.fast_reg.rkey = mr->rkey;
> DECR_CQCOUNT(&r_xprt->rx_ep);
>
> @@ -1574,6 +1570,8 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> if (rc) {
> dprintk("RPC: %s: failed ib_post_send for register,"
> " status %i\n", __func__, rc);
> + rkey = ib_dec_rkey(mr->rkey);
> + ib_update_fast_reg_key(mr, rkey);
> goto out_err;
> } else {
> seg1->mr_rkey = mr->rkey;
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 04/21] xprtrdma: Update rkeys after transport reconnect
On 07/22/2014 10:51 AM, Chuck Lever wrote:
> Various reports of:
>
> rpcrdma_qp_async_error_upcall: QP error 3 on device mlx4_0
> ep 8800bfd3e848
>
> Ensure that rkeys in already-marshalled RPC/RDMA headers are
> refreshed after the QP has been replaced by a reconnect.
>
> BugLink: https://bugzilla.linux-nfs.org/show_bug.cgi?id=249
> Suggested-by: Selvin Xavier
> Signed-off-by: Chuck Lever
> Tested-by: Steve Wise
> Tested-by: Shirley Ma
> Tested-by: Devesh Sharma
> ---
> net/sunrpc/xprtrdma/rpc_rdma.c | 75
> ---
> net/sunrpc/xprtrdma/transport.c | 11 +++---
> net/sunrpc/xprtrdma/xprt_rdma.h | 10 +
> 3 files changed, 54 insertions(+), 42 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
> index 693966d..54422f7 100644
> --- a/net/sunrpc/xprtrdma/rpc_rdma.c
> +++ b/net/sunrpc/xprtrdma/rpc_rdma.c
> @@ -53,14 +53,6 @@
> # define RPCDBG_FACILITY RPCDBG_TRANS
> #endif
>
> -enum rpcrdma_chunktype {
> - rpcrdma_noch = 0,
> - rpcrdma_readch,
> - rpcrdma_areadch,
> - rpcrdma_writech,
> - rpcrdma_replych
> -};
> -
> #ifdef RPC_DEBUG
> static const char transfertypes[][12] = {
> "pure inline", /* no chunks */
> @@ -286,6 +278,28 @@ out:
> }
>
> /*
> + * Marshal chunks. This routine returns the header length
> + * consumed by marshaling.
> + *
> + * Returns positive RPC/RDMA header size, or negative errno.
> + */
> +
> +ssize_t
> +rpcrdma_marshal_chunks(struct rpc_rqst *rqst, ssize_t result)
> +{
> + struct rpcrdma_req *req = rpcr_to_rdmar(rqst);
> + struct rpcrdma_msg *headerp = (struct rpcrdma_msg *)req->rl_base;
> +
> + if (req->rl_rtype != rpcrdma_noch)
> + result = rpcrdma_create_chunks(rqst, &rqst->rq_snd_buf,
> +headerp, req->rl_rtype);
> + else if (req->rl_wtype != rpcrdma_noch)
> + result = rpcrdma_create_chunks(rqst, &rqst->rq_rcv_buf,
> +headerp, req->rl_wtype);
> + return result;
> +}
> +
> +/*
> * Copy write data inline.
> * This function is used for "small" requests. Data which is passed
> * to RPC via iovecs (or page list) is copied directly into the
> @@ -377,7 +391,6 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
> char *base;
> size_t rpclen, padlen;
> ssize_t hdrlen;
> - enum rpcrdma_chunktype rtype, wtype;
> struct rpcrdma_msg *headerp;
>
> /*
> @@ -415,13 +428,13 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
>* into pages; otherwise use reply chunks.
>*/
> if (rqst->rq_rcv_buf.buflen <= RPCRDMA_INLINE_READ_THRESHOLD(rqst))
> - wtype = rpcrdma_noch;
> + req->rl_wtype = rpcrdma_noch;
> else if (rqst->rq_rcv_buf.page_len == 0)
> - wtype = rpcrdma_replych;
> + req->rl_wtype = rpcrdma_replych;
> else if (rqst->rq_rcv_buf.flags & XDRBUF_READ)
> - wtype = rpcrdma_writech;
> + req->rl_wtype = rpcrdma_writech;
> else
> - wtype = rpcrdma_replych;
> + req->rl_wtype = rpcrdma_replych;
>
> /*
>* Chunks needed for arguments?
> @@ -438,16 +451,16 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
>* TBD check NFSv4 setacl
>*/
> if (rqst->rq_snd_buf.len <= RPCRDMA_INLINE_WRITE_THRESHOLD(rqst))
> - rtype = rpcrdma_noch;
> + req->rl_rtype = rpcrdma_noch;
> else if (rqst->rq_snd_buf.page_len == 0)
> - rtype = rpcrdma_areadch;
> + req->rl_rtype = rpcrdma_areadch;
> else
> - rtype = rpcrdma_readch;
> + req->rl_rtype = rpcrdma_readch;
>
> /* The following simplification is not true forever */
> - if (rtype != rpcrdma_noch && wtype == rpcrdma_replych)
> - wtype = rpcrdma_noch;
> - if (rtype != rpcrdma_noch && wtype != rpcrdma_noch) {
> + if (req->rl_rtype != rpcrdma_noch && req->rl_wtype == rpcrdma_replych)
> + req->rl_wtype = rpcrdma_noch;
> + if (req->rl_rtype != rpcrdma_noch && req->rl_wtype != rpcrdma_noch) {
> dprintk("RPC: %s: cannot marshal multiple chunk lists\n",
> __func__);
Is this still a simplification, and is it still "not true forever"?
Anna
> return -EIO;
> @@ -461,7 +474,7 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
>* When padding is in use and applies to the transfer, insert
>* it and change the message type.
>*/
> - if (rtype == rpcrdma_noch) {
> + if (req->rl_rtype == rpcrdma_noch) {
>
> padlen = rpcrdma_inline_pullup(rqst,
> RPCRDMA_INLINE_PAD_VALUE(rqst));
> @@ -476,7 +489,7 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst)
> headerp->rm_body.rm_padded.rm_pempty[1] = xdr_zero;
>
Re: [PATCH v2 08/21] xprtrdma: Back off rkey when FAST_REG_MR fails
On 07/09/2014 12:57 PM, Chuck Lever wrote:
> If posting a FAST_REG_MR Work Reqeust fails, revert the rkey update
> to avoid subsequent IB_WC_MW_BIND_ERR completions.
>
> Suggested-by: Steve Wise
> Signed-off-by: Chuck Lever
> ---
> net/sunrpc/xprtrdma/verbs.c | 27 ---
> 1 file changed, 20 insertions(+), 7 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index ce847d2..14d24ec 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1487,6 +1487,24 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, struct
> rpcrdma_mr_seg *seg)
> seg->mr_dma, seg->mr_dmalen, seg->mr_dir);
> }
>
> +static void
> +rpcrdma_increment_frmr_rkey(struct rpcrdma_mw *mw)
> +{
> + struct ib_mr *mr = mw->r.frmr.fr_mr;
> + u8 key = mr->rkey & 0x00FF;
It's not super obvious to be how this key is being calculated. Can you change
0x00FF into a named constant to help?
It might also be useful to have an frmr_get_rkey() function rather than doing
this calculation in two different places.
Anna
> +
> + ib_update_fast_reg_key(mr, ++key);
> +}
> +
> +static void
> +rpcrdma_decrement_frmr_rkey(struct rpcrdma_mw *mw)
> +{
> + struct ib_mr *mr = mw->r.frmr.fr_mr;
> + u8 key = mr->rkey & 0x00FF;
> +
> + ib_update_fast_reg_key(mr, --key);
> +}
> +
> static int
> rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,
> int *nsegs, int writing, struct rpcrdma_ia *ia,
> @@ -1496,8 +1514,6 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> struct rpcrdma_mw *mw = seg1->mr_chunk.rl_mw;
> struct rpcrdma_frmr *frmr = &mw->r.frmr;
> struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr;
> -
> - u8 key;
> int len, pageoff;
> int i, rc;
> int seg_len;
> @@ -1557,14 +1573,10 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> rc = -EIO;
> goto out_err;
> }
> -
> - /* Bump the key */
> - key = (u8)(frmr->fr_mr->rkey & 0x00FF);
> - ib_update_fast_reg_key(frmr->fr_mr, ++key);
> -
> frmr_wr.wr.fast_reg.access_flags = (writing ?
> IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE :
> IB_ACCESS_REMOTE_READ);
> + rpcrdma_increment_frmr_rkey(mw);
> frmr_wr.wr.fast_reg.rkey = frmr->fr_mr->rkey;
> DECR_CQCOUNT(&r_xprt->rx_ep);
>
> @@ -1573,6 +1585,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> if (rc) {
> dprintk("RPC: %s: failed ib_post_send for register,"
> " status %i\n", __func__, rc);
> + rpcrdma_decrement_frmr_rkey(seg1->mr_chunk.rl_mw);
> goto out_err;
> } else {
> seg1->mr_rkey = frmr->fr_mr->rkey;
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No NFSoRDMA bi-weekly meeting from now to Aug.
I'll still be taking patches, so feel free to send me things now rather than waiting for the merge window to open! Anna On 07/08/2014 12:49 PM, Shirley Ma wrote: > Most of us will be on vacation in the summer so there will be no bi-weekly > meetings from now to Aug. We will resume the meeting in Sept. > > Let me know if you would like to have some discussion occasionally. I will be > around. > > Enjoy your summer! > > Shirley > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1 13/13] xprtrdma: Handle additional connection events
On 06/23/2014 06:40 PM, Chuck Lever wrote:
> Commit 38ca83a5 added RDMA_CM_EVENT_TIMEWAIT_EXIT. But that status
> is relevant only for consumers that re-use their QPs on new
> connections. xprtrdma creates a fresh QP on reconnection, so that
> event should be explicitly ignored.
>
> Squelch the alarming "unexpected CM event" message.
>
> Signed-off-by: Chuck Lever
> ---
> net/sunrpc/xprtrdma/verbs.c | 27 +--
> 1 file changed, 17 insertions(+), 10 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index ec98e48..dbd5f22 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -334,8 +334,16 @@ static const char * const conn[] = {
> "rejected",
> "established",
> "disconnected",
> - "device removal"
> + "device removal",
> + "multicast join",
> + "multicast error",
> + "address change",
> + "timewait exit",
> };
> +
> +#define CONNECTION_MSG(status)
> \
> + ((status) < ARRAY_SIZE(conn) ? \
> + conn[(status)] : "unrecognized connection error")
> #endif
>
> static int
> @@ -393,13 +401,10 @@ rpcrdma_conn_upcall(struct rdma_cm_id *id, struct
> rdma_cm_event *event)
> case RDMA_CM_EVENT_DEVICE_REMOVAL:
> connstate = -ENODEV;
> connected:
> - dprintk("RPC: %s: %s: %pI4:%u (ep 0x%p event 0x%x)\n",
> - __func__,
> - (event->event <= 11) ? conn[event->event] :
> - "unknown connection error",
> - &addr->sin_addr.s_addr,
> - ntohs(addr->sin_port),
> - ep, event->event);
> + dprintk("RPC: %s: %pI4:%u (ep 0x%p): %s\n",
> + __func__, &addr->sin_addr.s_addr,
> + ntohs(addr->sin_port), ep,
> + CONNECTION_MSG(event->event));
> atomic_set(&rpcx_to_rdmax(ep->rep_xprt)->rx_buf.rb_credits, 1);
> dprintk("RPC: %s: %sconnected\n",
> __func__, connstate > 0 ? "" : "dis");
> @@ -408,8 +413,10 @@ connected:
> wake_up_all(&ep->rep_connect_wait);
> break;
> default:
> - dprintk("RPC: %s: unexpected CM event %d\n",
> - __func__, event->event);
> + dprintk("RPC: %s: %pI4:%u (ep 0x%p): %s\n",
> + __func__, &addr->sin_addr.s_addr,
> + ntohs(addr->sin_port), ep,
> + CONNECTION_MSG(event->event));
These two dprintk()s are exactly the same, and only a few lines apart. Is
there some way to combine them? (I'm just curious, not saying that you need to!)
Anna
> break;
> }
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1 08/13] xprtrdma: Back off rkey when FAST_REG_MR fails
On 06/23/2014 06:40 PM, Chuck Lever wrote:
> If posting a FAST_REG_MR Work Reqeust fails, or the FAST_REG WR
> flushes, revert the rkey update to avoid subsequent
> IB_WC_MW_BIND_ERR completions.
>
> Suggested-by: Steve Wise
> Signed-off-by: Chuck Lever
> ---
> net/sunrpc/xprtrdma/verbs.c | 39 +--
> 1 file changed, 29 insertions(+), 10 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index cef67fd..3efc007 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -61,6 +61,8 @@
> # define RPCDBG_FACILITY RPCDBG_TRANS
> #endif
>
> +static void rpcrdma_decrement_frmr_rkey(struct rpcrdma_mw *);
> +
> /*
> * internal functions
> */
> @@ -154,13 +156,17 @@ rpcrdma_sendcq_process_wc(struct ib_wc *wc)
>
> if (wrid == 0)
> return;
> - if (wc->status != IB_WC_SUCCESS)
> - return;
>
> fastreg = test_and_clear_bit(RPCRDMA_BIT_FASTREG, &wrid);
> mw = (struct rpcrdma_mw *)wrid;
>
> - mw->r.frmr.fr_state = fastreg ? FRMR_IS_VALID : FRMR_IS_INVALID;
> + if (wc->status == IB_WC_SUCCESS) {
> + mw->r.frmr.fr_state = fastreg ?
> + FRMR_IS_VALID : FRMR_IS_INVALID;
> + } else {
> + if (fastreg)
> + rpcrdma_decrement_frmr_rkey(mw);
Isn't this the same as "else if (fastreg)"?
Anna
> + }
> }
>
> static int
> @@ -1480,6 +1486,24 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, struct
> rpcrdma_mr_seg *seg)
> seg->mr_dma, seg->mr_dmalen, seg->mr_dir);
> }
>
> +static void
> +rpcrdma_increment_frmr_rkey(struct rpcrdma_mw *mw)
> +{
> + struct ib_mr *frmr = mw->r.frmr.fr_mr;
> + u8 key = frmr->rkey & 0x00FF;
> +
> + ib_update_fast_reg_key(frmr, ++key);
> +}
> +
> +static void
> +rpcrdma_decrement_frmr_rkey(struct rpcrdma_mw *mw)
> +{
> + struct ib_mr *frmr = mw->r.frmr.fr_mr;
> + u8 key = frmr->rkey & 0x00FF;
> +
> + ib_update_fast_reg_key(frmr, --key);
> +}
> +
> static int
> rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,
> int *nsegs, int writing, struct rpcrdma_ia *ia,
> @@ -1487,8 +1511,6 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> {
> struct rpcrdma_mr_seg *seg1 = seg;
> struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr;
> -
> - u8 key;
> int len, pageoff;
> int i, rc;
> int seg_len;
> @@ -1552,14 +1574,10 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> rc = -EIO;
> goto out_err;
> }
> -
> - /* Bump the key */
> - key = (u8)(seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey & 0x00FF);
> - ib_update_fast_reg_key(seg1->mr_chunk.rl_mw->r.frmr.fr_mr, ++key);
> -
> frmr_wr.wr.fast_reg.access_flags = (writing ?
> IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE :
> IB_ACCESS_REMOTE_READ);
> + rpcrdma_increment_frmr_rkey(seg1->mr_chunk.rl_mw);
> frmr_wr.wr.fast_reg.rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey;
> DECR_CQCOUNT(&r_xprt->rx_ep);
>
> @@ -1568,6 +1586,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg
> *seg,
> if (rc) {
> dprintk("RPC: %s: failed ib_post_send for register,"
> " status %i\n", __func__, rc);
> + rpcrdma_decrement_frmr_rkey(seg1->mr_chunk.rl_mw);
> goto out_err;
> } else {
> seg1->mr_rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey;
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V2] xprtrdma: Fix DMA-API-DEBUG warning by checking dma_map result
On 06/19/2014 09:06 AM, Yan Burman wrote:
> Fix the following warning when DMA-API debug is enabled by checking
> ib_dma_map_single result:
> [ 1455.345548] [ cut here ]
> [ 1455.346863] WARNING: CPU: 3 PID: 3929 at
> /home/yanb/kernel/net-next/lib/dma-debug.c:1140 check_unmap+0x4e5/0x990()
> [ 1455.349350] mlx4_core :00:07.0: DMA-API: device driver failed to check
> map error[device address=0x7c9f2090] [size=2656 bytes] [mapped as
> single]
> [ 1455.349350] Modules linked in: xprtrdma netconsole configfs nfsv3 nfs_acl
> ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm autofs4
> auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc dm_mirror
> dm_region_hash dm_log microcode pcspkr mlx4_ib ib_sa ib_mad ib_core ib_addr
> mlx4_en ipv6 ptp pps_core vxlan mlx4_core virtio_balloon cirrus ttm
> drm_kms_helper drm sysimgblt sysfillrect syscopyarea i2c_piix4 i2c_core
> button ext3 jbd virtio_blk virtio_net virtio_pci virtio_ring virtio uhci_hcd
> ata_generic ata_piix libata
> [ 1455.349350] CPU: 3 PID: 3929 Comm: mount.nfs Not tainted 3.15.0-rc1-dbg+
> #13
> [ 1455.349350] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
> [ 1455.349350] 0474 880069dcf628 8151c341
> 817b69d8
> [ 1455.349350] 880069dcf678 880069dcf668 8105b5fc
> 69dcf658
> [ 1455.349350] 880069dcf778 88007b0c9f00 8255ec40
> 0a60
> [ 1455.349350] Call Trace:
> [ 1455.349350] [] dump_stack+0x52/0x81
> [ 1455.349350] [] warn_slowpath_common+0x8c/0xc0
> [ 1455.349350] [] warn_slowpath_fmt+0x46/0x50
> [ 1455.349350] [] check_unmap+0x4e5/0x990
> [ 1455.349350] [] ? _raw_spin_unlock_irq+0x30/0x60
> [ 1455.349350] [] debug_dma_unmap_page+0x5a/0x60
> [ 1455.349350] [] rpcrdma_deregister_internal+0xb3/0xd0
> [xprtrdma]
> [ 1455.349350] [] rpcrdma_buffer_destroy+0x69/0x170
> [xprtrdma]
> [ 1455.349350] [] xprt_rdma_destroy+0x3f/0xb0 [xprtrdma]
> [ 1455.349350] [] xprt_destroy+0x6f/0x80 [sunrpc]
> [ 1455.349350] [] xprt_put+0x15/0x20 [sunrpc]
> [ 1455.349350] [] rpc_free_client+0x8a/0xe0 [sunrpc]
> [ 1455.349350] [] rpc_release_client+0x68/0xa0 [sunrpc]
> [ 1455.349350] [] rpc_shutdown_client+0xb0/0xc0 [sunrpc]
> [ 1455.349350] [] ? rpc_ping+0x5d/0x70 [sunrpc]
> [ 1455.349350] [] rpc_create_xprt+0xbb/0xd0 [sunrpc]
> [ 1455.349350] [] rpc_create+0xb3/0x160 [sunrpc]
> [ 1455.349350] [] ? __probe_kernel_read+0x69/0xb0
> [ 1455.349350] [] nfs_create_rpc_client+0xdc/0x100 [nfs]
> [ 1455.349350] [] nfs_init_client+0x3a/0x90 [nfs]
> [ 1455.349350] [] nfs_get_client+0x478/0x5b0 [nfs]
> [ 1455.349350] [] ? nfs_get_client+0x100/0x5b0 [nfs]
> [ 1455.349350] [] ? kmem_cache_alloc_trace+0x24d/0x260
> [ 1455.349350] [] nfs_create_server+0xf3/0x4c0 [nfs]
> [ 1455.349350] [] ? nfs_request_mount+0xf0/0x1a0 [nfs]
> [ 1455.349350] [] nfs3_create_server+0x13/0x30 [nfsv3]
> [ 1455.349350] [] nfs_try_mount+0x1f3/0x230 [nfs]
> [ 1455.349350] [] ? get_parent_ip+0x11/0x50
> [ 1455.349350] [] ? __this_cpu_preempt_check+0x13/0x20
> [ 1455.349350] [] ? try_module_get+0x6b/0x190
> [ 1455.349350] [] nfs_fs_mount+0x187/0x9d0 [nfs]
> [ 1455.349350] [] ? nfs_clone_super+0x140/0x140 [nfs]
> [ 1455.349350] [] ? nfs_auth_info_match+0x40/0x40 [nfs]
> [ 1455.349350] [] mount_fs+0x20/0xe0
> [ 1455.349350] [] vfs_kern_mount+0x76/0x160
> [ 1455.349350] [] do_mount+0x428/0xae0
> [ 1455.349350] [] SyS_mount+0x90/0xe0
> [ 1455.349350] [] system_call_fastpath+0x16/0x1b
> [ 1455.349350] ---[ end trace f1f31572972e211d ]---
>
> Signed-off-by: Yan Burman
Thanks Yan! I've applied this to a bugfixes branch that I'll push out later
today.
Chuck - Should I add your "Reviewed By" from v1 to this version of the patch?
Anna
> ---
>
> Changes from V1:
> Removed curly braces
>
> net/sunrpc/xprtrdma/verbs.c |3 +++
> 1 files changed, 3 insertions(+), 0 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 13dbd1c..176dafc 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1388,6 +1388,9 @@ rpcrdma_register_internal(struct rpcrdma_ia *ia, void
> *va, int len,
>*/
> iov->addr = ib_dma_map_single(ia->ri_id->device,
> va, len, DMA_BIDIRECTIONAL);
> + if (ib_dma_mapping_error(ia->ri_id->device, iov->addr))
> + return -ENOMEM;
> +
> iov->length = len;
>
> if (ia->ri_have_dma_lkey) {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] xprtrdma: Fix DMA-API-DEBUG warning by checking dma_map result
On 06/17/2014 08:39 AM, Yan Burman wrote:
> Fix the following warning when DMA-API debug is enabled by checking
> ib_dma_map_single result:
> [ 1455.345548] [ cut here ]
> [ 1455.346863] WARNING: CPU: 3 PID: 3929 at
> /home/yanb/kernel/net-next/lib/dma-debug.c:1140 check_unmap+0x4e5/0x990()
> [ 1455.349350] mlx4_core :00:07.0: DMA-API: device driver failed to check
> map error[device address=0x7c9f2090] [size=2656 bytes] [mapped as
> single]
> [ 1455.349350] Modules linked in: xprtrdma netconsole configfs nfsv3 nfs_acl
> ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm autofs4
> auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc dm_mirror
> dm_region_hash dm_log microcode pcspkr mlx4_ib ib_sa ib_mad ib_core ib_addr
> mlx4_en ipv6 ptp pps_core vxlan mlx4_core virtio_balloon cirrus ttm
> drm_kms_helper drm sysimgblt sysfillrect syscopyarea i2c_piix4 i2c_core
> button ext3 jbd virtio_blk virtio_net virtio_pci virtio_ring virtio uhci_hcd
> ata_generic ata_piix libata
> [ 1455.349350] CPU: 3 PID: 3929 Comm: mount.nfs Not tainted 3.15.0-rc1-dbg+
> #13
> [ 1455.349350] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
> [ 1455.349350] 0474 880069dcf628 8151c341
> 817b69d8
> [ 1455.349350] 880069dcf678 880069dcf668 8105b5fc
> 69dcf658
> [ 1455.349350] 880069dcf778 88007b0c9f00 8255ec40
> 0a60
> [ 1455.349350] Call Trace:
> [ 1455.349350] [] dump_stack+0x52/0x81
> [ 1455.349350] [] warn_slowpath_common+0x8c/0xc0
> [ 1455.349350] [] warn_slowpath_fmt+0x46/0x50
> [ 1455.349350] [] check_unmap+0x4e5/0x990
> [ 1455.349350] [] ? _raw_spin_unlock_irq+0x30/0x60
> [ 1455.349350] [] debug_dma_unmap_page+0x5a/0x60
> [ 1455.349350] [] rpcrdma_deregister_internal+0xb3/0xd0
> [xprtrdma]
> [ 1455.349350] [] rpcrdma_buffer_destroy+0x69/0x170
> [xprtrdma]
> [ 1455.349350] [] xprt_rdma_destroy+0x3f/0xb0 [xprtrdma]
> [ 1455.349350] [] xprt_destroy+0x6f/0x80 [sunrpc]
> [ 1455.349350] [] xprt_put+0x15/0x20 [sunrpc]
> [ 1455.349350] [] rpc_free_client+0x8a/0xe0 [sunrpc]
> [ 1455.349350] [] rpc_release_client+0x68/0xa0 [sunrpc]
> [ 1455.349350] [] rpc_shutdown_client+0xb0/0xc0 [sunrpc]
> [ 1455.349350] [] ? rpc_ping+0x5d/0x70 [sunrpc]
> [ 1455.349350] [] rpc_create_xprt+0xbb/0xd0 [sunrpc]
> [ 1455.349350] [] rpc_create+0xb3/0x160 [sunrpc]
> [ 1455.349350] [] ? __probe_kernel_read+0x69/0xb0
> [ 1455.349350] [] nfs_create_rpc_client+0xdc/0x100 [nfs]
> [ 1455.349350] [] nfs_init_client+0x3a/0x90 [nfs]
> [ 1455.349350] [] nfs_get_client+0x478/0x5b0 [nfs]
> [ 1455.349350] [] ? nfs_get_client+0x100/0x5b0 [nfs]
> [ 1455.349350] [] ? kmem_cache_alloc_trace+0x24d/0x260
> [ 1455.349350] [] nfs_create_server+0xf3/0x4c0 [nfs]
> [ 1455.349350] [] ? nfs_request_mount+0xf0/0x1a0 [nfs]
> [ 1455.349350] [] nfs3_create_server+0x13/0x30 [nfsv3]
> [ 1455.349350] [] nfs_try_mount+0x1f3/0x230 [nfs]
> [ 1455.349350] [] ? get_parent_ip+0x11/0x50
> [ 1455.349350] [] ? __this_cpu_preempt_check+0x13/0x20
> [ 1455.349350] [] ? try_module_get+0x6b/0x190
> [ 1455.349350] [] nfs_fs_mount+0x187/0x9d0 [nfs]
> [ 1455.349350] [] ? nfs_clone_super+0x140/0x140 [nfs]
> [ 1455.349350] [] ? nfs_auth_info_match+0x40/0x40 [nfs]
> [ 1455.349350] [] mount_fs+0x20/0xe0
> [ 1455.349350] [] vfs_kern_mount+0x76/0x160
> [ 1455.349350] [] do_mount+0x428/0xae0
> [ 1455.349350] [] SyS_mount+0x90/0xe0
> [ 1455.349350] [] system_call_fastpath+0x16/0x1b
> [ 1455.349350] ---[ end trace f1f31572972e211d ]---
>
> Signed-off-by: Yan Burman
> ---
> net/sunrpc/xprtrdma/verbs.c |4
> 1 file changed, 4 insertions(+)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 13dbd1c..076c244 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -1388,6 +1388,10 @@ rpcrdma_register_internal(struct rpcrdma_ia *ia, void
> *va, int len,
>*/
> iov->addr = ib_dma_map_single(ia->ri_id->device,
> va, len, DMA_BIDIRECTIONAL);
> + if (ib_dma_mapping_error(ia->ri_id->device, iov->addr)) {
> + return -ENOMEM;
> + }
nit: you don't need the curly braces since this is a one-line "if".
Anna
> +
> iov->length = len;
>
> if (ia->ri_have_dma_lkey) {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 00/24] Series short description
On 06/09/2014 05:55 AM, Or Gerlitz wrote: > On Thu, May 29, 2014 at 7:59 PM, Steve Wise > wrote: >> I tested this series over cxgb4 and mlx4, running cthon04, fio read/rand-rw, >> and xfstests, and over mthca cthon04. No regressions. > > Steve, Chuck, all > > Where this series stands? I don't see it in Roland's tree for-next > branch... is this planned to go upstream through another maintainer > tree? if not, I think we're good to go, so Roland, are you picking it > up? > > Or. This is going through the NFS tree (Trond should already have it). Anna > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: NFSoRDMA developers bi-weekly meeting minutes (5/28)
On 05/29/2014 01:07 PM, Shirley Ma wrote: > Attendees: > > Jeff Beck (NASA) > Yan Burman (Mellanox) > Wendy Cheng (Intel) > Susan Coulter (LANL) > Rupert Dance (Soft Forge) > Chuck Lever (Oracle) > Doug Ledford (RedHat) > Shirley Ma (Oracle) > Devesh Sharma (Emulex) > Anna Schumaker (Net App) > Steve Wise (OpenGridComputing, Chelsio) > > Moderator: > Shirley Ma (Oracle) > > NFSoRDMA developers bi-weekly meeting is to help organizing NFSoRDMA > development and test effort from different resources to speed up NFSoRDMA > upstream kernel work and NFSoRDMA diagnosing/debugging tools development. > Hopefully the quality of NFSoRDMA upstream patches can be improved by being > tested by a quorum of HW vendors. > > Today's meeting notes: > 1. OFED release update from Rupert Dance through email: > > a. OFED 3.12 was released yesterday without any patch from Chuck's git tree, > one of the reason these patches haven't upstream yet. There were a number of > new bugs filed on NFSRDMA (2489 and 2490). > http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2489 > http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2490 > > b. Jeff Becker has pulled all of the patches from Chuck's git tree and made > backports in his local branch of OFED 3.12. He has begun testing and is > seeing good results so far. > > c. OFED next release will be OFED 3.12-1 and it will be including all these > updates along with support for RHEL 7.0 and Intel's OFED MIC. > > 2. NFSoRDMA support with soft RoCE and Soft iWARP > > There were some discussions regarding whether removing RPCRDMA_REGISTER > support (one of Chuck's patchset) would impact any other components in the > stack. So far soft RoCE hasn't been upstreamed yet.There was a broad > consensus not to support out-of-tree providers unless an issue also affects > in-tree providers. NFSoRDMA will follow kernel development policy, all work > should be based upon upstream kernel. However Chuck Level will check Soft > RoCE plan, Steve Wise will check soft iWARP plan to make sure nothing will be > broken in both kernel and OFED release. > > 3. Update on testing NFSoRDMA client patches: > > Devesh Sharma, Doug Ledford, Chuck Lever, Steve Wise all have tested Chuck > git tree (up to last weeks patchsets) on different platforms with various > vendor's HCAs. The test showed the stack pretty reliable for both NFSv3 and > NFSv4. However NFSv4.1 hit server crash. (NFSv4.1 hasn't support yet). > > Steve Wise's test covers iWARP Chelsio > Devesh Sharma's test covers Emulex > Chuck Lever/Shirley Ma's test covers Mellanox > Doug Ledford's test covers various platforms and combination of HCAs > (Interoperability test). > Jeff Beck's test covers OFED release (backport) > Rupert Dance's (IOL) team test covers various platforms and combination of > HCAs as well. > > The test coverage should be good enough for NFSoRDMA client patchsets to be > merged to upstream by Anna and Trond. > > Devesh is looking for performance benchmark tools. IOzone is recommended. > Anna is going to send more performance tools. > > A place to save test results as scratch sheets has been discussed so it will > be easy to track the test history for any regressions. Anna will help to > figure it out. > > A couple of new bugs have been filed to track existing issues. Devesh had hit > a bug in dbench test, which Steve Wise already worked on it. > > https://bugzilla.linux-nfs.org/show_bug.cgi?id=255 > > Klemens Senn has reported a soft lockup in unloading kernel module. Shirley > has tried to reproduce this problem with Linux server, Solaris client, it > didn't hit any issue. So it's a problem between Linux client and server. > > https://bugzilla.linux-nfs.org/show_bug.cgi?id=252 > > > 4. Steve shared his findings on some bug he has bee working on -- refactoring > patchset. > > 5. Followups update from last week > a. Linux server maintenance is still in unresolved status. > > b. NFSoRDMA debugging and diagnosis tools? > Yan has made some progress on NFSoRDMA wireshark dissector. Selecting > connection is not as simple as TCP, Yan has tried to use QP number/RDMA > establish status to build the connection. Chuck suggested to try RPC XID > field. > > c. NFSoRDMA virtualization validation: > Shirley has set up KVM guest with Mellanox Connect2 SRIOV. A panic occurred > right away during mount, the panic is different with XEN domU guest. > > Next meeting topics proposal: > 1. Follow up the work has been discussed from this meeting. > > 2. Walk through some of the stories on pivotal, link
Re: Soft lockup in unloading kernel modules
I haven't applied Chuck's recent (v3) patches to that kernel yet (I've been waiting to see if people have comments). I'll try to push something out today. On 05/08/2014 10:28 AM, Senn Klemens wrote: > Hi, > > I am getting a soft lockup on the NFS server on its reboot if at least > one client mount is established. I am using OpenSUSE 12.3 with the > nfs-rdma kernel from Anna Schumaker > (git://git.linux-nfs.org/projects/anna/nfs-rdma.git). > > The export on the server side is done with > /data *(fsid=0,crossmnt,rw,mp,no_root_squash,sync,no_subtree_check,insecure) > > Following command is used for mounting the NFSv4 share: > mount -t nfs -o port=20049,rdma,vers=4.0,timeo=900 172.16.100.19:/ /mnt > > The HCA is a Mellanox MT4099 on the server and the client. > > The soft lockup can be reproduced by following steps: > o server: Start the nfs server > o client: Mount the share > o client: Do a "ls" in the mounted directory > o server: Stop the nfs server > o server: Unload the nfs and mlx4 modules or reboot the server (I used > the openibd init script from the Mellanox driver without having the > Mellanox stack installed) > > The server reports a soft lockup > BUG: soft lockup - CPU#0 stuck for 22s! [modprobe:6146] > most times. > > Sometimes I get following kernel panic > BUG: unable to handle kernel NULL pointer dereference at 0003 > IP: [] _raw_spin_lock_bh+0x15/0x40 > PGD 82a820067 PUD 857832067 PMD 0 > Oops: 0002 [#1] SMP > Modules linked in: nfsd nfs_acl auth_rpcgss oid_registry nfnetlink_log > nfnetlink bluetooth rfkill nfsv4 svcrdma dm_mod cpuid nfs fscache lockd > sunrpc af_packet 8021q garp stp llc rdma_ucm ib_ucm rdma_cm iw_cm > ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en mlx4_ib(-) ib_sa ib_mad ib_core > ib_addr sr_mod cdrom usb_storage joydev mlx4_core usbhid > x86_pkg_temp_thermal coretemp kvm_intel kvm ghash_clmulni_intel > aesni_intel ablk_helper cryptd iTCO_wdt lrw igb gf128mul > iTCO_vendor_support ehci_pci glue_helper pcspkr i2c_algo_bit isci > ehci_hcd aes_x86_64 ptp libsas ioatdma lpc_ich microcode sb_edac sg > pps_core usbcore ipmi_si tpm_tis edac_core scsi_transport_sas i2c_i801 > mfd_core dca usb_common tpm ipmi_msghandler wmi acpi_cpufreq button edd > autofs4 xfs libcrc32c crc32c_intel processor thermal_sys scsi_dh_rdac > scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_dh [last unloaded: oid_registry] > CPU: 0 PID: 6603 Comm: modprobe Not tainted 3.15.0-rc2-anna-nfs-rdma+ #3 > Hardware name: Supermicro B9DRG-E/B9DRG-E, BIOS 3.0 09/04/2013 > task: 88105b8c6050 ti: 88105d814000 task.ti: 88105d814000 > RIP: 0010:[] [] > _raw_spin_lock_bh+0x15/0x40 > RSP: 0018:88105d815d18 EFLAGS: 00010286 > RAX: 0001 RBX: RCX: > RDX: 000b RSI: RDI: 0003 > RBP: 88105d815d18 R08: 88087c611f38 R09: 0001 > R10: R11: R12: 88087c3c9800 > R13: 88107b82ab00 R14: 0003 R15: 0007 > FS: 7fef64612700() GS:88087fc0() knlGS: > CS: 0010 DS: ES: CR0: 80050033 > CR2: 0003 CR3: 00087c2c7000 CR4: 000407f0 > Stack: > 88105d815d58 a05199f0 88105d815d88 88087c3c9800 > 88087c3c9400 88107b82ab00 88087c3c9660 88087c3c95c8 > 88105d815d78 a0421ce9 88087c3c9400 88107b82aac0 > Call Trace: > [] svc_xprt_enqueue+0x50/0x220 [sunrpc] > [] rdma_cma_handler+0x69/0x180 [svcrdma] > [] cma_remove_one+0x1f6/0x220 [rdma_cm] > [] ib_unregister_device+0x46/0x120 [ib_core] > [] mlx4_ib_remove+0x29/0x260 [mlx4_ib] > [] mlx4_remove_device+0xa0/0xc0 [mlx4_core] > [] mlx4_unregister_interface+0x3b/0xa0 [mlx4_core] > [] mlx4_ib_cleanup+0x10/0x23 [mlx4_ib] > [] SyS_delete_module+0x152/0x220 > [] ? vm_munmap+0x54/0x70 > [] system_call_fastpath+0x1a/0x1f > Code: 5d c3 0f b7 17 66 39 ca 74 f6 f3 90 0f b7 17 66 39 d1 75 f6 5d c3 > 55 65 81 04 25 20 b9 00 00 00 02 00 00 48 89 e5 b8 00 00 01 00 0f > c1 07 89 c2 c1 ea 10 66 39 c2 75 04 5d c3 f3 90 0f b7 07 > RIP [] _raw_spin_lock_bh+0x15/0x40 > RSP > CR2: 0003 > ---[ end trace 18e02ff413ac4b9b ]--- > Kernel panic - not syncing: Fatal exception in interrupt > Kernel Offset: 0x0 from 0x8100 (relocation range: > 0x8000-0x9fff) > ---[ end Kernel panic - not syncing: Fatal exception in interrupt > > Kind regards, > Klemens > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
