Re: RFC: FMR support in SRP

2014-09-19 Thread Bart Van Assche
On 09/19/14 18:04, Jinpu Wang wrote: Another question, in srp_map_finish_fmr, the desc va is set to 0, could you point me how SRP protect multiple rdma operation write to same addr? or different fmr-rkey will protect this? Hello Jack, As you can see in srpt_map_sg_to_ib_sge() in the SRP targe

Re: [PATCH 7/8] IB/srp: Separate target and channel variables

2014-09-19 Thread Sagi Grimberg
On 9/19/2014 3:59 PM, Bart Van Assche wrote: > Changes in this patch: > - Move channel variables into a new structure (struct srp_rdma_ch). > - cm_id and completion handler context pointer are now of type >srp_rdma_ch * insteoad of srp_target_port *. > s/insteoad/instead > No functionality is

Re: [PATCH RFC 0/8] IB/srp: Add multichannel support

2014-09-19 Thread Jens Axboe
On 09/19/2014 06:55 AM, Bart Van Assche wrote: > Hello, > > Although the SRP protocol supports multichannel operation, although > since considerable time RDMA HCA's are available that support multiple > completion vectors and although multichannel operation yields better > performance than using a

Re: [PATCH 5/8] IB/srp: Remove stale connection retry mechanism

2014-09-19 Thread Sagi Grimberg
On 9/19/2014 3:58 PM, Bart Van Assche wrote: > Attempting to connect three times may be insufficient after an > initiator system that was using multiple RDMA channels tries to > relogin. Additionally, this login retry mechanism is a workaround > for particular behavior of the IB/CM. Since the srp_d

Re: [PATCH 2/8] scsi-mq: Add support for multiple hardware queues

2014-09-19 Thread Christoph Hellwig
On Fri, Sep 19, 2014 at 09:05:53PM +0300, Sagi Grimberg wrote: > I think this patch should be squashed with passing LLD hctx patch (in > whatever form it ends up). Agreed. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Christoph Hellwig
On Fri, Sep 19, 2014 at 11:33:15AM -0600, Jens Axboe wrote: > That'd be fine as well. The mapping is cheap, though, but it would make > sense to have an appropriate way to just pass it in like it happens for > ->queue_rq() for native blk-mq drivers. I think just passing the hw_ctx is fine. But I

Re: [PATCH 4/8] IB/srp: Move ib_destroy_cm_id() call into srp_free_ch_ib()

2014-09-19 Thread Sagi Grimberg
On 9/19/2014 3:58 PM, Bart Van Assche wrote: > The patch that adds multichannel support into the SRP initiator > driver introduces an additional call to srp_free_ch_ib(). This > patch helps to keep that later patch simple. > > Signed-off-by: Bart Van Assche > --- > drivers/infiniband/ulp/srp/ib_

Re: [PATCH 2/8] scsi-mq: Add support for multiple hardware queues

2014-09-19 Thread Sagi Grimberg
On 9/19/2014 3:57 PM, Bart Van Assche wrote: > Allow a SCSI LLD to declare how many hardware queues it supports > by setting Scsi_Host.nr_hw_queues before calling scsi_add_host(). > > Note: it is assumed that each hardware queue has a queue depth of > shost->can_queue. In other words, the total que

Re: [PATCH RFC 0/8] IB/srp: Add multichannel support

2014-09-19 Thread Sagi Grimberg
On 9/19/2014 3:56 PM, Bart Van Assche wrote: > [PATCH 1/8] blk-mq: Use all available hardware queues > > Suppose that a system has two CPU sockets, three cores per socket, > that it does not support hyperthreading and that four hardware > queues are provided by a block driver. With the current algo

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Jens Axboe
On 09/19/2014 11:30 AM, Sagi Grimberg wrote: > On 9/19/2014 6:38 PM, Jens Axboe wrote: >> On 09/19/2014 09:35 AM, Bart Van Assche wrote: >>> On 09/19/14 17:27, Ming Lei wrote: On Fri, Sep 19, 2014 at 11:21 PM, Bart Van Assche wrote: > On 09/19/14 16:28, Ming Lei wrote: >> >>

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Sagi Grimberg
On 9/19/2014 6:38 PM, Jens Axboe wrote: > On 09/19/2014 09:35 AM, Bart Van Assche wrote: >> On 09/19/14 17:27, Ming Lei wrote: >>> On Fri, Sep 19, 2014 at 11:21 PM, Bart Van Assche >>> wrote: On 09/19/14 16:28, Ming Lei wrote: > > On Fri, Sep 19, 2014 at 9:00 PM, Bart Van Assche >>>

Re: RFC: FMR support in SRP

2014-09-19 Thread Jinpu Wang
On Fri, Sep 19, 2014 at 5:35 PM, Jinpu Wang wrote: >> >> >> Hello Jack, >> >> Did you know that file descriptor 0 corresponds to stdin ? With command-line >> option -w the test program reads data from stdin and sends that data to a >> SCSI device. I think the test program is waiting for you to pro

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Jens Axboe
On 09/19/2014 09:35 AM, Bart Van Assche wrote: > On 09/19/14 17:27, Ming Lei wrote: >> On Fri, Sep 19, 2014 at 11:21 PM, Bart Van Assche wrote: >>> On 09/19/14 16:28, Ming Lei wrote: On Fri, Sep 19, 2014 at 9:00 PM, Bart Van Assche wrote: > > @@ -2643,7 +2754,8 @@ static st

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Bart Van Assche
On 09/19/14 17:27, Ming Lei wrote: > On Fri, Sep 19, 2014 at 11:21 PM, Bart Van Assche wrote: >> On 09/19/14 16:28, Ming Lei wrote: >>> >>> On Fri, Sep 19, 2014 at 9:00 PM, Bart Van Assche >>> wrote: @@ -2643,7 +2754,8 @@ static struct scsi_host_template srp_template = {

Re: RFC: FMR support in SRP

2014-09-19 Thread Jinpu Wang
> > > Hello Jack, > > Did you know that file descriptor 0 corresponds to stdin ? With command-line > option -w the test program reads data from stdin and sends that data to a > SCSI device. I think the test program is waiting for you to provide input > data :-) > > Bart. > Thanks Bart, Now I know

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Ming Lei
On Fri, Sep 19, 2014 at 11:21 PM, Bart Van Assche wrote: > On 09/19/14 16:28, Ming Lei wrote: >> >> On Fri, Sep 19, 2014 at 9:00 PM, Bart Van Assche >> wrote: >>> >>> @@ -2643,7 +2754,8 @@ static struct scsi_host_template srp_template = { >>> .proc_name = DRV_NAME, >

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Bart Van Assche
On 09/19/14 16:28, Ming Lei wrote: On Fri, Sep 19, 2014 at 9:00 PM, Bart Van Assche wrote: @@ -2643,7 +2754,8 @@ static struct scsi_host_template srp_template = { .proc_name = DRV_NAME, .slave_configure= srp_slave_configure, .info

Re: RFC: FMR support in SRP

2014-09-19 Thread Bart Van Assche
On 09/19/14 17:08, Jinpu Wang wrote: Thank you very much for quick reply. I tried you test program with: ./discontiguous-io -l 4096 -o 1024 -s -w /dev/sdb The program just wait there, and strace show it hang at read: munmap(0x7fb074c1, 24430) = 0 brk(0)

Re: RFC: FMR support in SRP

2014-09-19 Thread Jinpu Wang
Hello Bart, Thank you very much for quick reply. I tried you test program with: ./discontiguous-io -l 4096 -o 1024 -s -w /dev/sdb The program just wait there, and strace show it hang at read: munmap(0x7fb074c1, 24430) = 0 brk(0) = 0x1896000 brk(0x1

Re: [PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Ming Lei
On Fri, Sep 19, 2014 at 9:00 PM, Bart Van Assche wrote: > Improve performance by using multiple RDMA/RC channels per SCSI host > for communicating with an SRP target. > > Signed-off-by: Bart Van Assche > --- > Documentation/ABI/stable/sysfs-driver-ib_srp | 25 +- > drivers/infiniband/ulp/srp/ib

Re: RFC: FMR support in SRP

2014-09-19 Thread Bart Van Assche
On 09/19/14 15:11, Jinpu Wang wrote: During go through SRP FMR support, I found ib_srp pre-alloc fmr_list/map_page in each request, fmr_list are alloced as many as target->cmd_sg_cnt I add some debug message when run fio test with different settings. Result show, state.ndesc and state.nmdesc is

RFC: FMR support in SRP

2014-09-19 Thread Jinpu Wang
Hi Bart and all, During go through SRP FMR support, I found ib_srp pre-alloc fmr_list/map_page in each request, fmr_list are alloced as many as target->cmd_sg_cnt I add some debug message when run fio test with different settings. Result show, state.ndesc and state.nmdesc is 1, state.npages is 0,

[PATCH 8/8] IB/srp: Add multichannel support

2014-09-19 Thread Bart Van Assche
Improve performance by using multiple RDMA/RC channels per SCSI host for communicating with an SRP target. Signed-off-by: Bart Van Assche --- Documentation/ABI/stable/sysfs-driver-ib_srp | 25 +- drivers/infiniband/ulp/srp/ib_srp.c | 337 --- drivers/infiniband/

[PATCH 7/8] IB/srp: Separate target and channel variables

2014-09-19 Thread Bart Van Assche
Changes in this patch: - Move channel variables into a new structure (struct srp_rdma_ch). - cm_id and completion handler context pointer are now of type srp_rdma_ch * insteoad of srp_target_port *. No functionality is changed. Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srp/ib_

[PATCH 6/8] IB/srp: Avoid that I/O hangs due to a cable pull during LUN scanning

2014-09-19 Thread Bart Van Assche
If a cable is pulled during LUN scanning it can happen that the SRP rport and the SCSI host have been created but no LUNs have been added to the SCSI host. Since multipathd only sends SCSI commands to a SCSI target if one or more SCSI devices are present and since there is no keepalive mechanism fo

[PATCH 5/8] IB/srp: Remove stale connection retry mechanism

2014-09-19 Thread Bart Van Assche
Attempting to connect three times may be insufficient after an initiator system that was using multiple RDMA channels tries to relogin. Additionally, this login retry mechanism is a workaround for particular behavior of the IB/CM. Since the srp_daemon retries a failed login attempt anyway, remove t

[PATCH 4/8] IB/srp: Move ib_destroy_cm_id() call into srp_free_ch_ib()

2014-09-19 Thread Bart Van Assche
The patch that adds multichannel support into the SRP initiator driver introduces an additional call to srp_free_ch_ib(). This patch helps to keep that later patch simple. Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srp/ib_srp.c | 11 ++- 1 file changed, 6 insertions(+), 5

[PATCH 3/8] scsi-mq: Pass hctx to low-level SCSI drivers

2014-09-19 Thread Bart Van Assche
Low-level drivers (LLDs) need to know which hardware context has been selected by the block layer. Hence pass this information to SCSI LLDs. Signed-off-by: Bart Van Assche Cc: Christoph Hellwig --- drivers/scsi/scsi.c | 7 +-- drivers/scsi/scsi_lib.c | 4 ++-- drivers/scsi/scsi_priv

[PATCH 2/8] scsi-mq: Add support for multiple hardware queues

2014-09-19 Thread Bart Van Assche
Allow a SCSI LLD to declare how many hardware queues it supports by setting Scsi_Host.nr_hw_queues before calling scsi_add_host(). Note: it is assumed that each hardware queue has a queue depth of shost->can_queue. In other words, the total queue depth per host is (number of hardware queues) * (sh

Re: [PATCH RFC 0/8] IB/srp: Add multichannel support

2014-09-19 Thread Bart Van Assche
[PATCH 1/8] blk-mq: Use all available hardware queues Suppose that a system has two CPU sockets, three cores per socket, that it does not support hyperthreading and that four hardware queues are provided by a block driver. With the current algorithm this will lead to the following assignment of CP

[PATCH RFC 0/8] IB/srp: Add multichannel support

2014-09-19 Thread Bart Van Assche
Hello, Although the SRP protocol supports multichannel operation, although since considerable time RDMA HCA's are available that support multiple completion vectors and although multichannel operation yields better performance than using a single channel, the Linux SRP initiator does not yet

RE: [PATCH] qib: qib_qp: Replace rcu_assign_pointer() with RCU_INIT_POINTER()

2014-09-19 Thread Marciniszyn, Mike
> Subject: [PATCH] qib: qib_qp: Replace rcu_assign_pointer() with > RCU_INIT_POINTER() > Why not consolidate this with http://marc.info/?l=linux-rdma&m=140836578119485&w=2 so there is just one patch? Mike -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a m

RE: [PATCH] qib_keys: Replace rcu_assign_pointer() with RCU_INIT_POINTER()

2014-09-19 Thread Marciniszyn, Mike
> Subject: [PATCH] qib_keys: Replace rcu_assign_pointer() with > RCU_INIT_POINTER() > I would prefer the summary be: IB/qib: qib_remove_lkey() Replace rcu_assign_pointer() with > RCU_INIT_POINTER() Otherwise the patch looks ok and has been tested. -- To unsubscribe from this list: send the line

RE: [PATCH] qib_qp: Replace rcu_assign_pointer() with RCU_INIT_POINTER()

2014-09-19 Thread Marciniszyn, Mike
> Subject: [PATCH] qib_qp: Replace rcu_assign_pointer() with > RCU_INIT_POINTER() > I would prefer the summary line be: IB/qib: remove_qp() use RCU_INIT_POINTER() instead of rcu_assign_pointer() Otherwise I agree with the patch and I have tested it with large numbers of QPs. Mike -- To unsubscr

[PATCH] IB/qib: Correct reference counting in debugfs qp_stats

2014-09-19 Thread Mike Marciniszyn
This particular reference count is not needed with the rcu protection and the current code leaks a reference count, causing a hang in qib_qp_destroy(). Cc: Reviewed-by: Dennis Dalessandro Signed-off-by: Mike Marciniszyn --- drivers/infiniband/hw/qib/qib_debugfs.c |3 ++- drivers/infiniband

Re: [PATCH for-next 1/2] IB/uverbs: Add QP creation flags, allow blocking UD multicast loopback

2014-09-19 Thread Yann Droneaud
Hi, Le jeudi 18 septembre 2014 à 15:51 +0300, Or Gerlitz a écrit : > On 9/15/2014 7:52 PM, Yann Droneaud wrote: > > Le dimanche 14 septembre 2014 à 16:20 +0300, Or Gerlitz a écrit : > [...] > >> As a result, applications who send and recieve over the same QP to > >> the same multicast group get