Re: [net-next][PATCH v2 11/13] RDS: IB: add Fastreg MR (FRMR) detection support

[santosh.shilim...@oracle.com]

On 2/28/16 1:08 AM, Christoph Hellwig wrote:

On Sat, Feb 27, 2016 at 06:19:48PM -0800, Santosh Shilimkar wrote:

Discovere Fast Memmory Registration support using IB device
IB_DEVICE_MEM_MGT_EXTENSIONS. Certain HCA might support just FRMR
or FMR or both FMR and FRWR. In case both mr type are supported,
default FMR is used.

Default MR is still kept as FMR against what everyone else
is following. Default will be changed to FRMR once the
RDS performance with FRMR is comparable with FMR. The
work is in progress for the same.

Signed-off-by: Santosh Shilimkar 
Signed-off-by: Santosh Shilimkar 
---
v2: Dropped the module parameter as suggested by David Miller


This means we only use the safer method if the HCA doesn't support
the other one.  All other RDMA ULP that support both methods have
a module_param so the veto from Dave is a bit unfortunate.  Anyway,
let's get the code in for now and figure out what to use later.


Indeed. It wasn't really deal breaker for RDS so I agreed to drop it.


Just curious:  where / how do you see worse peformance using FRs?


I wouldn't call it worse but its not comparable. Use case
is multi-threaded RDS RDMA perf test(s). Hopefully the follow
up series which am working on should minimise the gap. I am
leaving the details for later, but one of the main issue I
saw was contention on driver post_send() lock from send, MR reg
and MR invalidation.

Regards,
Santosh

Re: [net-next][PATCH v2 11/13] RDS: IB: add Fastreg MR (FRMR) detection support

[Christoph Hellwig]

On Sat, Feb 27, 2016 at 06:19:48PM -0800, Santosh Shilimkar wrote:
> Discovere Fast Memmory Registration support using IB device
> IB_DEVICE_MEM_MGT_EXTENSIONS. Certain HCA might support just FRMR
> or FMR or both FMR and FRWR. In case both mr type are supported,
> default FMR is used.
> 
> Default MR is still kept as FMR against what everyone else
> is following. Default will be changed to FRMR once the
> RDS performance with FRMR is comparable with FMR. The
> work is in progress for the same.
> 
> Signed-off-by: Santosh Shilimkar 
> Signed-off-by: Santosh Shilimkar 
> ---
> v2: Dropped the module parameter as suggested by David Miller

This means we only use the safer method if the HCA doesn't support
the other one.  All other RDMA ULP that support both methods have
a module_param so the veto from Dave is a bit unfortunate.  Anyway,
let's get the code in for now and figure out what to use later.

Just curious:  where / how do you see worse peformance using FRs?