date:20111020

Re: RDMA_CM over RoCE

2011-10-20 Thread Or Gerlitz

Kelly Burkhart  wrote:

>> I'm having a problem connecting using RDMA_CM over ethernet
> Actually not.  My network admin found some sort of misconfiguration on
> the switch that caused the problem  Everything works now.

Interesting, could you share the nature of  problem?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

2011-10-20 Thread David Miller

From: Steve Wise 
Date: Thu, 20 Oct 2011 12:28:07 -0500

> On 10/20/2011 12:17 PM, Roland Dreier wrote:
>>> I believe 5 and 7 have build dependencies.
>> Right, missed that one too.
>>
>> But it seems 4,6,8,9,10 are independent of the rest of the series?
>>
>> ie I can trivially apply them and then worry about working out
>> the drivers/net / drivers/infiniband interdependency a bit later?
>>
> 
> Some of these might be dependent on prior patches the series.  But if
> they aren't, yes, you could do that.

So, how do you guys want to do this?  If you give me a list of which
patches I should put into net-next and leave the rest to the infiniband
tree, that'd work fine for me as long as net-next is left in a working
state independent of the infiniband tree.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] IW_CXGB4: Fix iw_cxgb4 count_rcqes logic

2011-10-20 Thread Jonathan Lallinger

Fixes another place in the code where logic dealing with the
t4_cqe was using the wrong QID. This change fixes the counting
logic so that it tests against the SQ QID instead of the RQ QID
when counting RCQES

Signed-off by: Jonathan Lallinger 
Signed-off by: Steve Wise 
---

 drivers/infiniband/hw/cxgb4/cq.c |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/hw/cxgb4/cq.c b/drivers/infiniband/hw/cxgb4/cq.c
index 901c5fb..04ae97e 100644
--- a/drivers/infiniband/hw/cxgb4/cq.c
+++ b/drivers/infiniband/hw/cxgb4/cq.c
@@ -311,7 +311,7 @@ void c4iw_count_rcqes(struct t4_cq *cq, struct t4_wq *wq, 
int *count)
while (ptr != cq->sw_pidx) {
cqe = &cq->sw_queue[ptr];
if (RQ_TYPE(cqe) && (CQE_OPCODE(cqe) != FW_RI_READ_RESP) &&
-   (CQE_QPID(cqe) == wq->rq.qid) && cqe_completes_wr(cqe, wq))
+   (CQE_QPID(cqe) == wq->sq.qid) && cqe_completes_wr(cqe, wq))
(*count)++;
if (++ptr == cq->size)
ptr = 0;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib_srpt status

2011-10-20 Thread Bart Van Assche

On Thu, Oct 20, 2011 at 6:51 PM, Roland Dreier  wrote:
> On Wed, Oct 19, 2011 at 11:09 AM, Bart Van Assche  wrote:
>>> *) use_port_guid_in_session_name: This appears to be a legacy compat
>>>   item, can this be safetly removed for mainline..?
>>
>> As far as I know some people are still using this parameter.
>
> So what exactly does this parameter do?  How do I as a user
> know what to set this to?
>
> For mainline can we just decide what the right value of this is
> and get rid of the parameter?

If this parameter would be removed I would prefer to use the value 0
(false) - this is the only value that is compliant with the SRP r16a
draft.

If anyone is still using this parameter, I think it is a good time to speak up.

The most recent discussion I have been able to find about this subject
is: http://sourceforge.net/mailarchive/message.php?msg_id=23481074

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA_CM over RoCE

2011-10-20 Thread Kelly Burkhart

On Wed, Oct 19, 2011 at 8:28 AM, Kelly Burkhart
 wrote:
> Hello,
>
> I'm having a problem connecting using RDMA_CM over ethernet

Actually not.  My network admin found some sort of misconfiguration on
the switch that caused the problem  Everything works now.

-K
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

2011-10-20 Thread Steve Wise


On 10/20/2011 12:25 PM, parav.pan...@emulex.com wrote:

You are right, cq_lock will result into dead lock.
Should there be a additional compl_handler spin_lock?
I was measuring performance impact for adding it, and, irq_save() and 
irq_restore() variant showed additional 200 cycles, which I believe should be 
o.k.?

Parav



Yea, I think adding a comp_handler spin lock would do the trick.  I'll let 
Kumar come up with this change.




-Original Message-
From: Steve Wise [mailto:sw...@opengridcomputing.com]
Sent: Thursday, October 20, 2011 9:52 PM
To: Roland Dreier
Cc: Pandit, Parav; kuma...@chelsio.com; linux-rdma@vger.kernel.org; 
d...@chelsio.com
Subject: Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

On 10/20/2011 11:06 AM, Roland Dreier wrote:

On Thu, Oct 20, 2011 at 2:11 AM,   wrote:

http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt

Line no 66 to 97 states that - at a given point of time, there should be only 
one callback per CQ should be active.
Is this ensured?

compl_handler() is invoked from multiple places flush_qp() and post_qp_event() 
to my mind.

Yes, does look like an issue with this patch :(


And I think this bug exists in the T3 driver too.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

2011-10-20 Thread Steve Wise


On 10/20/2011 12:17 PM, Roland Dreier wrote:

I believe 5 and 7 have build dependencies.

Right, missed that one too.

But it seems 4,6,8,9,10 are independent of the rest of the series?

ie I can trivially apply them and then worry about working out
the drivers/net / drivers/infiniband interdependency a bit later?



Some of these might be dependent on prior patches the series.   But if they 
aren't, yes, you could do that.

Stevo
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

2011-10-20 Thread Parav.Pandit

You are right, cq_lock will result into dead lock.
Should there be a additional compl_handler spin_lock?
I was measuring performance impact for adding it, and, irq_save() and 
irq_restore() variant showed additional 200 cycles, which I believe should be 
o.k.?

Parav

-Original Message-
From: Steve Wise [mailto:sw...@opengridcomputing.com] 
Sent: Thursday, October 20, 2011 9:52 PM
To: Roland Dreier
Cc: Pandit, Parav; kuma...@chelsio.com; linux-rdma@vger.kernel.org; 
d...@chelsio.com
Subject: Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

On 10/20/2011 11:06 AM, Roland Dreier wrote:
> On Thu, Oct 20, 2011 at 2:11 AM,  wrote:
>> http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt
>>
>> Line no 66 to 97 states that - at a given point of time, there should be 
>> only one callback per CQ should be active.
>> Is this ensured?
>>
>> compl_handler() is invoked from multiple places flush_qp() and 
>> post_qp_event() to my mind.
> Yes, does look like an issue with this patch :(
>

And I think this bug exists in the T3 driver too.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH V2 02/10] cxgb4: Common platform specific changes for DB Drop Recovery

2011-10-20 Thread Felix Marti

Don't add stuff to the t4fw_interface.h, that is owned by firmware.

> -Original Message-
> From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-
> ow...@vger.kernel.org] On Behalf Of Vipul Pandya
> Sent: Thursday, October 20, 2011 10:10 AM
> To: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Cc: rol...@purestorage.com; da...@davemloft.net; Divy Le Ray;
Dimitrios
> Michailidis; Kumar A S; Steve Wise; Vipul Pandya
> Subject: [PATCH V2 02/10] cxgb4: Common platform specific changes for
DB
> Drop Recovery
> 
> - Add platform-specific callback functions for interrupts.  This
is
> needed to do a single read-clear of the CAUSE register and then
call
> out to platform specific functions for DB threshold interrupts and
DB
> drop interrupts.
> 
> - Add t4_mem_win_read_len() - mem-window reads for arbitrary
lengths.
> This is used to read the CIDX/PIDX values from EC contexts during
DB
> drop recovery.
> 
> - Add t4_fwaddrspace_write() - sends addrspace write cmds to the
fw.
> Needed to flush the sge eq context cache.
> 
> Signed-off-by: Vipul Pandya 
> Signed-off-by: Steve Wise 
> ---
> Changes:
> V2: Corrected the subject for patch.
> 
>  drivers/net/ethernet/chelsio/cxgb4/t4_hw.c|   69
> +
>  drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h |5 ++
>  2 files changed, 63 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
> b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
> index 13609bf..32e1dd5 100644
> --- a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
> +++ b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
> @@ -868,11 +868,14 @@ int t4_restart_aneg(struct adapter *adap,
unsigned
> int mbox, unsigned int port)
>   return t4_wr_mbox(adap, mbox, &c, sizeof(c), NULL);  }
> 
> +typedef void (*int_handler_t)(struct adapter *adap);
> +
>  struct intr_info {
>   unsigned int mask;   /* bits to check in interrupt status */
>   const char *msg; /* message to print or NULL */
>   short stat_idx;  /* stat counter to increment or -1 */
>   unsigned short fatal;/* whether the condition reported is
fatal */
> + int_handler_t int_handler; /* platform-specific int handler */
>  };
> 
>  /**
> @@ -905,6 +908,8 @@ static int t4_handle_intr_status(struct adapter
> *adapter, unsigned int reg,
>   } else if (acts->msg && printk_ratelimit())
>   dev_warn(adapter->pdev_dev, "%s (0x%x)\n", acts-
> >msg,
>status & acts->mask);
> + if (acts->int_handler)
> + acts->int_handler(adapter);
>   mask |= acts->mask;
>   }
>   status &= mask;
> @@ -1013,9 +1018,9 @@ static void sge_intr_handler(struct adapter
> *adapter)
>   { ERR_INVALID_CIDX_INC,
> "SGE GTS CIDX increment too large", -1, 0 },
>   { ERR_CPL_OPCODE_0, "SGE received 0-length CPL", -1, 0
},
> - { F_DBFIFO_LP_INT, NULL, -1, 0 },
> - { F_DBFIFO_HP_INT, NULL, -1, 0 },
> - { ERR_DROPPED_DB, "SGE doorbell dropped", -1, 0 },
> + { F_DBFIFO_LP_INT, NULL, -1, 0, t4_db_full },
> + { F_DBFIFO_HP_INT, NULL, -1, 0, t4_db_full },
> + { F_ERR_DROPPED_DB, NULL, -1, 0, t4_db_dropped },
>   { ERR_DATA_CPL_ON_HIGH_QID1 |
> ERR_DATA_CPL_ON_HIGH_QID0,
> "SGE IQID > 1023 received CPL for FL", -1, 0 },
>   { ERR_BAD_DB_PIDX3, "SGE DBP 3 pidx increment too
large",
> -1, @@ -1036,20 +1041,14 @@ static void sge_intr_handler(struct
adapter
> *adapter)
>   };
> 
>   v = (u64)t4_read_reg(adapter, SGE_INT_CAUSE1) |
> - ((u64)t4_read_reg(adapter, SGE_INT_CAUSE2) << 32);
> + ((u64)t4_read_reg(adapter, SGE_INT_CAUSE2) << 32);
>   if (v) {
>   dev_alert(adapter->pdev_dev, "SGE parity error
(%#llx)\n",
> -  (unsigned long long)v);
> + (unsigned long long)v);
>   t4_write_reg(adapter, SGE_INT_CAUSE1, v);
>   t4_write_reg(adapter, SGE_INT_CAUSE2, v >> 32);
>   }
> 
> - err = t4_read_reg(adapter, A_SGE_INT_CAUSE3);
> - if (err & (F_DBFIFO_HP_INT|F_DBFIFO_LP_INT))
> - t4_db_full(adapter);
> - if (err & F_ERR_DROPPED_DB)
> - t4_db_dropped(adapter);
> -
>   if (t4_handle_intr_status(adapter, SGE_INT_CAUSE3,
sge_intr_info)
> ||
>   v != 0)
>   t4_fatal_err(adapter);
> @@ -1995,6 +1994,54 @@ int t4_wol_pat_enable(struct adapter *adap,
> unsigned int port, unsigned int map,
>   (var).retval_len16 = htonl(FW_LEN16(var)); \  } while (0)
> 
> +int t4_fwaddrspace_write(struct adapter *adap, unsigned int mbox,
> +   u32 addr, u32 val)
> +{
> + struct fw_ldst_cmd c;
> +
> + memset(&c, 0, sizeof(c));
> + c.op_to_addrspace = htonl(V_FW_CMD_OP(FW_LDST_CMD) |
> F_FW_CMD_REQUEST |
> +

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

2011-10-20 Thread Roland Dreier

> I believe 5 and 7 have build dependencies.

Right, missed that one too.

But it seems 4,6,8,9,10 are independent of the rest of the series?

ie I can trivially apply them and then worry about working out
the drivers/net / drivers/infiniband interdependency a bit later?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH RESEND 03/10] cxgb4: DB Drop Recovery for RDMA and LLD queues.

2011-10-20 Thread Vipul Pandya

- recover LLD EQs for DB drop interrupts.  This includes adding a new
db_lock, a spin lock disabling BH too, used by the recovery thread and
the ring_tx_db() paths to allow db drop recovery.

- cleaned up initial db avoidance code.

- add read_eq_indices() - allows the LLD to use the pcie mw to efficiently
read hw eq contexts.

- add cxgb4_sync_txq_pidx() - called by iw_cxgb4 to sync up the sw/hw pidx
value.

- add flush_eq_cache() and cxgb4_flush_eq_cache().  This allows iw_cxgb4
to flush the sge eq context cache before beginning db drop recovery.

- add module parameter, dbfoifo_int_thresh, to allow tuning the db
interrupt threshold value.

- add dbfifo_int_thresh to cxgb4_lld_info so iw_cxgb4 knows the threshold.

- add module parameter, dbfoifo_drain_delay, to allow tuning the amount
of time delay between DB FULL and EMPTY upcalls to iw_cxgb4.

Signed-off-by: Vipul Pandya 
Signed-off-by: Steve Wise 
---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4.h  |7 +
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c |  214 +++
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h  |4 +
 drivers/net/ethernet/chelsio/cxgb4/sge.c|   20 ++-
 drivers/net/ethernet/chelsio/cxgb4/t4_regs.h|   53 ++
 drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h   |   23 +++
 6 files changed, 279 insertions(+), 42 deletions(-)

diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h 
b/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h
index e18b5ad..f202cb9 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h
@@ -51,6 +51,8 @@
 #define FW_VERSION_MINOR 1
 #define FW_VERSION_MICRO 0
 
+#define CH_WARN(adap, fmt, ...) dev_warn(adap->pdev_dev, fmt, ## __VA_ARGS__)
+
 enum {
MAX_NPORTS = 4, /* max # of ports */
SERNUM_LEN = 24,/* Serial # length */
@@ -403,6 +405,9 @@ struct sge_txq {
struct tx_sw_desc *sdesc;   /* address of SW Tx descriptor ring */
struct sge_qstat *stat; /* queue status entry */
dma_addr_tphys_addr;/* physical address of the ring */
+   spinlock_t db_lock;
+   int db_disabled;
+   unsigned short db_pidx;
 };
 
 struct sge_eth_txq {/* state for an SGE Ethernet Tx queue */
@@ -475,6 +480,7 @@ struct adapter {
void __iomem *regs;
struct pci_dev *pdev;
struct device *pdev_dev;
+   unsigned int mbox;
unsigned int fn;
unsigned int flags;
 
@@ -607,6 +613,7 @@ irqreturn_t t4_sge_intr_msix(int irq, void *cookie);
 void t4_sge_init(struct adapter *adap);
 void t4_sge_start(struct adapter *adap);
 void t4_sge_stop(struct adapter *adap);
+extern int dbfifo_int_thresh;
 
 #define for_each_port(adapter, iter) \
for (iter = 0; iter < (adapter)->params.nports; ++iter)
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c 
b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
index 870c320..64ad1c8 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
@@ -149,15 +149,6 @@ static unsigned int pfvfres_pmask(struct adapter *adapter,
 #endif
 
 enum {
-   MEMWIN0_APERTURE = 65536,
-   MEMWIN0_BASE = 0x3,
-   MEMWIN1_APERTURE = 32768,
-   MEMWIN1_BASE = 0x28000,
-   MEMWIN2_APERTURE = 2048,
-   MEMWIN2_BASE = 0x1b800,
-};
-
-enum {
MAX_TXQ_ENTRIES  = 16384,
MAX_CTRL_TXQ_ENTRIES = 1024,
MAX_RSPQ_ENTRIES = 16384,
@@ -369,6 +360,15 @@ static int set_addr_filters(const struct net_device *dev, 
bool sleep)
uhash | mhash, sleep);
 }
 
+int dbfifo_int_thresh = 10; /* 10 == 640 entry threshold */
+module_param(dbfifo_int_thresh, int, 0644);
+MODULE_PARM_DESC(dbfifo_int_thresh, "doorbell fifo interrupt threshold");
+
+int dbfifo_drain_delay = 1000; /* usecs to sleep while draining the dbfifo */
+module_param(dbfifo_drain_delay, int, 0644);
+MODULE_PARM_DESC(dbfifo_drain_delay,
+"usecs to sleep while draining the dbfifo");
+
 /*
  * Set Rx properties of a port, such as promiscruity, address filters, and MTU.
  * If @mtu is -1 it is left unchanged.
@@ -387,6 +387,8 @@ static int set_rxmode(struct net_device *dev, int mtu, bool 
sleep_ok)
return ret;
 }
 
+static struct workqueue_struct *workq;
+
 /**
  * link_start - enable a port
  * @dev: the port to enable
@@ -2201,7 +2203,7 @@ static void cxgb4_queue_tid_release(struct tid_info *t, 
unsigned int chan,
adap->tid_release_head = (void **)((uintptr_t)p | chan);
if (!adap->tid_release_task_busy) {
adap->tid_release_task_busy = true;
-   schedule_work(&adap->tid_release_task);
+   queue_work(workq, &adap->tid_release_task);
}
spin_unlock_bh(&adap->tid_release_lock);
 }
@@ -2428,6 +2430,59 @@ void cxgb4_iscsi_init(struct net_device *dev, unsigned 
int tag_mask,
 }

[PATCH V2 02/10] cxgb4: Common platform specific changes for DB Drop Recovery

2011-10-20 Thread Vipul Pandya

- Add platform-specific callback functions for interrupts.  This is
needed to do a single read-clear of the CAUSE register and then call
out to platform specific functions for DB threshold interrupts and DB
drop interrupts.

- Add t4_mem_win_read_len() - mem-window reads for arbitrary lengths.
This is used to read the CIDX/PIDX values from EC contexts during DB
drop recovery.

- Add t4_fwaddrspace_write() - sends addrspace write cmds to the fw.
Needed to flush the sge eq context cache.

Signed-off-by: Vipul Pandya 
Signed-off-by: Steve Wise 
---
Changes:
V2: Corrected the subject for patch.

 drivers/net/ethernet/chelsio/cxgb4/t4_hw.c|   69 +
 drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h |5 ++
 2 files changed, 63 insertions(+), 11 deletions(-)

diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c 
b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
index 13609bf..32e1dd5 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
@@ -868,11 +868,14 @@ int t4_restart_aneg(struct adapter *adap, unsigned int 
mbox, unsigned int port)
return t4_wr_mbox(adap, mbox, &c, sizeof(c), NULL);
 }
 
+typedef void (*int_handler_t)(struct adapter *adap);
+
 struct intr_info {
unsigned int mask;   /* bits to check in interrupt status */
const char *msg; /* message to print or NULL */
short stat_idx;  /* stat counter to increment or -1 */
unsigned short fatal;/* whether the condition reported is fatal */
+   int_handler_t int_handler; /* platform-specific int handler */
 };
 
 /**
@@ -905,6 +908,8 @@ static int t4_handle_intr_status(struct adapter *adapter, 
unsigned int reg,
} else if (acts->msg && printk_ratelimit())
dev_warn(adapter->pdev_dev, "%s (0x%x)\n", acts->msg,
 status & acts->mask);
+   if (acts->int_handler)
+   acts->int_handler(adapter);
mask |= acts->mask;
}
status &= mask;
@@ -1013,9 +1018,9 @@ static void sge_intr_handler(struct adapter *adapter)
{ ERR_INVALID_CIDX_INC,
  "SGE GTS CIDX increment too large", -1, 0 },
{ ERR_CPL_OPCODE_0, "SGE received 0-length CPL", -1, 0 },
-   { F_DBFIFO_LP_INT, NULL, -1, 0 },
-   { F_DBFIFO_HP_INT, NULL, -1, 0 },
-   { ERR_DROPPED_DB, "SGE doorbell dropped", -1, 0 },
+   { F_DBFIFO_LP_INT, NULL, -1, 0, t4_db_full },
+   { F_DBFIFO_HP_INT, NULL, -1, 0, t4_db_full },
+   { F_ERR_DROPPED_DB, NULL, -1, 0, t4_db_dropped },
{ ERR_DATA_CPL_ON_HIGH_QID1 | ERR_DATA_CPL_ON_HIGH_QID0,
  "SGE IQID > 1023 received CPL for FL", -1, 0 },
{ ERR_BAD_DB_PIDX3, "SGE DBP 3 pidx increment too large", -1,
@@ -1036,20 +1041,14 @@ static void sge_intr_handler(struct adapter *adapter)
};
 
v = (u64)t4_read_reg(adapter, SGE_INT_CAUSE1) |
-   ((u64)t4_read_reg(adapter, SGE_INT_CAUSE2) << 32);
+   ((u64)t4_read_reg(adapter, SGE_INT_CAUSE2) << 32);
if (v) {
dev_alert(adapter->pdev_dev, "SGE parity error (%#llx)\n",
-(unsigned long long)v);
+   (unsigned long long)v);
t4_write_reg(adapter, SGE_INT_CAUSE1, v);
t4_write_reg(adapter, SGE_INT_CAUSE2, v >> 32);
}
 
-   err = t4_read_reg(adapter, A_SGE_INT_CAUSE3);
-   if (err & (F_DBFIFO_HP_INT|F_DBFIFO_LP_INT))
-   t4_db_full(adapter);
-   if (err & F_ERR_DROPPED_DB)
-   t4_db_dropped(adapter);
-
if (t4_handle_intr_status(adapter, SGE_INT_CAUSE3, sge_intr_info) ||
v != 0)
t4_fatal_err(adapter);
@@ -1995,6 +1994,54 @@ int t4_wol_pat_enable(struct adapter *adap, unsigned int 
port, unsigned int map,
(var).retval_len16 = htonl(FW_LEN16(var)); \
 } while (0)
 
+int t4_fwaddrspace_write(struct adapter *adap, unsigned int mbox,
+ u32 addr, u32 val)
+{
+   struct fw_ldst_cmd c;
+
+   memset(&c, 0, sizeof(c));
+   c.op_to_addrspace = htonl(V_FW_CMD_OP(FW_LDST_CMD) | F_FW_CMD_REQUEST |
+   F_FW_CMD_WRITE |
+   V_FW_LDST_CMD_ADDRSPACE(FW_LDST_ADDRSPC_FIRMWARE));
+   c.cycles_to_len16 = htonl(FW_LEN16(c));
+   c.u.addrval.addr = htonl(addr);
+   c.u.addrval.val = htonl(val);
+
+   return t4_wr_mbox(adap, mbox, &c, sizeof(c), NULL);
+}
+
+/*
+ * t4_mem_win_read_len - read memory through PCIE memory window
+ * @adap: the adapter
+ * @addr: address of first byte requested aligned on 32b.
+ * @data: len bytes to hold the data read
+ * @len: amount of data to read from window.  Must be <=
+ *MEMWIN0_APERATURE after adjusting for 16B alignm

Re: ib_srpt status

2011-10-20 Thread Roland Dreier

On Wed, Oct 19, 2011 at 11:42 AM, Nicholas A. Bellinger
 wrote:
>> > *) srpt_service_guid: Should this really have global scope..?
>>
>> I'm afraid that having different values of this parameter for
>> different HCAs would break failing over between HCAs.

If I have the possibility to set this per-whatever, I can of course
still choose to set it to the same value everywhere, if I care about
failover (vs. maybe providing SRP service to two different contexts).

How do I choose what value to set this to anyway?

Can we at least put this in configfs with the rest of the target
config options?  It seems like a hassle fo rme as a user to have
to set a module parameter to control this type of config.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib_srpt status

2011-10-20 Thread Roland Dreier

On Wed, Oct 19, 2011 at 11:09 AM, Bart Van Assche  wrote:
>> *) use_port_guid_in_session_name: This appears to be a legacy compat
>>   item, can this be safetly removed for mainline..?
>
> As far as I know some people are still using this parameter.

So what exactly does this parameter do?  How do I as a user
know what to set this to?

For mainline can we just decide what the right value of this is
and get rid of the parameter?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] ib_srpt: Initial SRP Target merge for v3.2-rc1

2011-10-20 Thread Roland Dreier

On Wed, Oct 19, 2011 at 11:47 AM, Bart Van Assche  wrote:
> I'm referring to this code:
>
>                sprintf(sport->port_guid, "0x%04x%04x%04x%04x",
>                                be16_to_cpu(((__be16 *)&device->node_guid)[0]),
>                                be16_to_cpu(((__be16 *)&device->node_guid)[1]),
>                                be16_to_cpu(((__be16 *)&device->node_guid)[2]),
>                                be16_to_cpu(((__be16
> *)&device->node_guid)[3]) + i);
>
> There is no guarantee that the above formula will match the numbers
> assigned by the HCA manufacturer. Also, what will happen if for the
> last two digits the sum exceeds 2**16 ?
>
> I would like to see the above code to be replaced by something like
> the following:
>
>               snprintf(sport->port_guid, sizeof(sport->port_guid), 
> "0x%016llx",
>                        be64_to_cpu(sport->gid.global.interface_id));
>
> Please note that in the InfiniBand Architecture Manual the words "port
> GUID" refer to a manufacturer-assigned EUI-64. So if you are going to
> use the name "port GUID" for an 128-bit quantity I'm afraid you are
> going to really confuse some people.

I agree that this looks broken.  I think the most sensible thing to do
would be to have a port_gid (GID, not GUID) and use GID table
entry 0, rather than doing this math on node GUID (that I agree is
totally bogus).

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] ib_srpt: Initial SRP Target merge for v3.2-rc1

2011-10-20 Thread Roland Dreier

On Wed, Oct 19, 2011 at 11:24 PM, Bart Van Assche  wrote:
> If you have a look at the output of the command ibv_devices -v
> (maintained by Roland), you will see that in the output of that
> command ports are identified by a 64-bit GUID.

Where would I see that?  Seriously, I don't know where ibv_devices
prints anything about ports.  It lists device by "node GUID" which is
a globally unique ID, but that is per-adapter (and of course 2-port
adapters are very common).

It is true that every port does have a port GUID, but those are not
used in most places -- typically we use a port GID.  Again, it is
true that GID table entry 0 must be formed by taking the subnet
prefix (assigned by the subnet manager) and appending the
port GUID.

To be honest I've kind of lost track of where this leads: what
exactly are we trying to figure out how to identify?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

2011-10-20 Thread Steve Wise


On 10/20/2011 11:06 AM, Roland Dreier wrote:

On Thu, Oct 20, 2011 at 2:11 AM,  wrote:

http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt

Line no 66 to 97 states that - at a given point of time, there should be only 
one callback per CQ should be active.
Is this ensured?

compl_handler() is invoked from multiple places flush_qp() and post_qp_event() 
to my mind.

Yes, does look like an issue with this patch :(



And I think this bug exists in the T3 driver too.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

2011-10-20 Thread Roland Dreier

On Thu, Oct 20, 2011 at 2:11 AM,   wrote:
> http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt
>
> Line no 66 to 97 states that - at a given point of time, there should be only 
> one callback per CQ should be active.
> Is this ensured?
>
> compl_handler() is invoked from multiple places flush_qp() and 
> post_qp_event() to my mind.

Yes, does look like an issue with this patch :(

  - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

2011-10-20 Thread Steve Wise


On 10/20/2011 04:11 AM, parav.pan...@emulex.com wrote:

http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt

Line no 66 to 97 states that - at a given point of time, there should be only 
one callback per CQ should be active.
Is this ensured?

compl_handler() is invoked from multiple places flush_qp() and post_qp_event() 
to my mind.



I think you're correct.  post_qp_event() can be called indirectly on the ingress side of things concurrently with the 
application moving the QP to a state that causes a flush.  My first thought is to serialize the call to the 
compl_handler with the CQ spinlock, but I'm pretty certain that we can't hold that around an upcall because the app most 
likely turns around and calls ib_poll_cq() which would attempt to acquire this same lock.





Parav


-Original Message-
From: linux-rdma-ow...@vger.kernel.org 
[mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Steve Wise
Sent: Thursday, October 13, 2011 9:43 PM
To: Roland Dreier
Cc: Kumar Sanghvi; linux-rdma@vger.kernel.org; d...@chelsio.com
Subject: Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

On 10/13/2011 11:01 AM, Roland Dreier wrote:

Would this generate a completion event even if no completion entries are queued?


I guess it can if the QP has no WRs posted at all.


Maybe I'm misunderstanding, but this sounds like a bandaid for broken
applications,
and a bandaid that other hardware drivers won't implement.


I'm not sure other drivers have this issue.  For example, when a mlx or mthca 
QP moves out of RTS, does the HW flush the
pending recv WRs?  For Chelsio devices, the provider driver/library must handle 
this.

This logic is needed to adhere to the iwarp verbs spec which states that when 
the QP moves out of RTS, all WRs that are
pending get completed with FLUSH status.  For T3/T4 devices, this is all done 
in software.  For user mode, the provider
library has to flush the QP (IE the kernel doesn't own the queue state).

The idea is that if an application is expecting a CQ event notification when 
the QP moves out of RTS, and there are only
recv wrs posted, then the T4 (and T3 does this too) driver must post this CQ 
notification, in addition to marking the CQ
as "in error" which means some QP bound to this CQ needs flushing.  Then when 
the app wakes up and polls the CQ, the
libcxgb4 code will flush the QPs in error and thus CQEs will be inserted into 
the CQ.  We have seen certain applications
that rely on this event to discover that a QP has moved out of RTS.  IE they 
don't look at async QP events nor RDMACM
events.



Steve.




On Thu, Oct 13, 2011 at 1:21 AM, Kumar Sanghvi   wrote:

At the time when peer closes connection, iw_cxgb4 will not
send a cq event if ibqp.uobject exists. In that case, its possible
for user application to get blocked in ibv_get_cq_event.

To resolve this, call the cq's comp_handler to unblock any read
from ibv_get_cq_event.

Signed-off-by: Kumar Sanghvi
---
   drivers/infiniband/hw/cxgb4/qp.c |6 +-
   1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/hw/cxgb4/qp.c b/drivers/infiniband/hw/cxgb4/qp.c
index ec3ce67..b59b56c 100644
--- a/drivers/infiniband/hw/cxgb4/qp.c
+++ b/drivers/infiniband/hw/cxgb4/qp.c
@@ -970,8 +970,12 @@ static void flush_qp(struct c4iw_qp *qhp)
 if (qhp->ibqp.uobject) {
 t4_set_wq_in_error(&qhp->wq);
 t4_set_cq_in_error(&rchp->cq);
-   if (schp != rchp)
+   (*rchp->ibcq.comp_handler)(&rchp->ibcq, rchp->ibcq.cq_context);
+   if (schp != rchp) {
 t4_set_cq_in_error(&schp->cq);
+   (*schp->ibcq.comp_handler)(&schp->ibcq,
+   schp->ibcq.cq_context);
+   }
 return;
 }
 __flush_qp(qhp, rchp, schp);
--
1.7.1



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

2011-10-20 Thread Steve Wise


On 10/19/2011 05:12 PM, Roland Dreier wrote:

This looks like the only drivers/infiniband patch that depends on the
drivers/net changes?

  - R.


I believe 5 and 7 have build dependencies.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

ghost cq event in handler

2011-10-20 Thread Benoit Hudzia

Hi,

I am developing in kernel features that use the RDMA or exchanging data.

I am currently testing the RDMA communication engine i coded and i am
experiencing a strange behaviour.

Setup :
kernel : linux 3.0
RDMA : soft Iwarp ( for dev / debug purpose)

Testing setup :
Establish communication between 2 machine A and B

A send a request (IB_POST_SEND / IB_SEND_SIGNALLED)  to B with a
message containing a variable status  set 1
B receive the message , process it and reply back with an and
(IB_WR_SEND + IBSEND_SIGNALED)  . The RDMA write , write a page, the
send , just to notify the reception of the page the message of
notification contain a variable status set to 2.

Note the IB_POST_SEND are only performed if the status change has been
confirmed. so i cannot try to send a message that doesn't contain a
status of 1 or 2 .

I have a flow control that prevent me from overflowing the CQ.
Also i protect my IB_POST_SEND by spinlock with irqsave/restore


Now here is the strange behaviour:

When i send 100 msg, 1000 msg , No problem .

When i start moving toward 100 000 msg and more i start seeing some
strange message in my "CQ send handler ".
I start to get notification of message send , however when i check the
status of the message , it is 0 ;

Note so far this doesn't create any disturbance and the exchange of
the 100k+ message finish ok.

However sometimes , it looks like the system is repeatedly trying to
send the message and  then the value of the status variable change and
is suddenly pushed over to B. Which naturally receive it and discarded
it because of the non valid status value.

It would be ok if it didn't also drop the message it was supposed to
send in the first place at the same time.

So from time to time i  get a random message being dropped ( only when
i push a high number of message as fast as possible).


My question is : did any of you experience similar issue, if yes , how
did you solve it?

Regards
Benoit

--
" The production of too many useful things results in too many useless people"
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

2011-10-20 Thread Parav.Pandit

http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt

Line no 66 to 97 states that - at a given point of time, there should be only 
one callback per CQ should be active.
Is this ensured?

compl_handler() is invoked from multiple places flush_qp() and post_qp_event() 
to my mind.

Parav

-Original Message-
From: linux-rdma-ow...@vger.kernel.org 
[mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Steve Wise
Sent: Thursday, October 13, 2011 9:43 PM
To: Roland Dreier
Cc: Kumar Sanghvi; linux-rdma@vger.kernel.org; d...@chelsio.com
Subject: Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

On 10/13/2011 11:01 AM, Roland Dreier wrote:
> Would this generate a completion event even if no completion entries are 
> queued?
>

I guess it can if the QP has no WRs posted at all.

> Maybe I'm misunderstanding, but this sounds like a bandaid for broken
> applications,
> and a bandaid that other hardware drivers won't implement.
>

I'm not sure other drivers have this issue.  For example, when a mlx or mthca 
QP moves out of RTS, does the HW flush the 
pending recv WRs?  For Chelsio devices, the provider driver/library must handle 
this.

This logic is needed to adhere to the iwarp verbs spec which states that when 
the QP moves out of RTS, all WRs that are 
pending get completed with FLUSH status.  For T3/T4 devices, this is all done 
in software.  For user mode, the provider 
library has to flush the QP (IE the kernel doesn't own the queue state).

The idea is that if an application is expecting a CQ event notification when 
the QP moves out of RTS, and there are only 
recv wrs posted, then the T4 (and T3 does this too) driver must post this CQ 
notification, in addition to marking the CQ 
as "in error" which means some QP bound to this CQ needs flushing.  Then when 
the app wakes up and polls the CQ, the 
libcxgb4 code will flush the QPs in error and thus CQEs will be inserted into 
the CQ.  We have seen certain applications 
that rely on this event to discover that a QP has moved out of RTS.  IE they 
don't look at async QP events nor RDMACM 
events.

Steve.

> On Thu, Oct 13, 2011 at 1:21 AM, Kumar Sanghvi  wrote:
>> At the time when peer closes connection, iw_cxgb4 will not
>> send a cq event if ibqp.uobject exists. In that case, its possible
>> for user application to get blocked in ibv_get_cq_event.
>>
>> To resolve this, call the cq's comp_handler to unblock any read
>> from ibv_get_cq_event.
>>
>> Signed-off-by: Kumar Sanghvi
>> ---
>>   drivers/infiniband/hw/cxgb4/qp.c |6 +-
>>   1 files changed, 5 insertions(+), 1 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/cxgb4/qp.c 
>> b/drivers/infiniband/hw/cxgb4/qp.c
>> index ec3ce67..b59b56c 100644
>> --- a/drivers/infiniband/hw/cxgb4/qp.c
>> +++ b/drivers/infiniband/hw/cxgb4/qp.c
>> @@ -970,8 +970,12 @@ static void flush_qp(struct c4iw_qp *qhp)
>> if (qhp->ibqp.uobject) {
>> t4_set_wq_in_error(&qhp->wq);
>> t4_set_cq_in_error(&rchp->cq);
>> -   if (schp != rchp)
>> +   (*rchp->ibcq.comp_handler)(&rchp->ibcq, 
>> rchp->ibcq.cq_context);
>> +   if (schp != rchp) {
>> t4_set_cq_in_error(&schp->cq);
>> +   (*schp->ibcq.comp_handler)(&schp->ibcq,
>> +   schp->ibcq.cq_context);
>> +   }
>> return;
>> }
>> __flush_qp(qhp, rchp, schp);
>> --
>> 1.7.1
>>
>>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA_CM over RoCE

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

[PATCH] IW_CXGB4: Fix iw_cxgb4 count_rcqes logic

Re: ib_srpt status

Re: RDMA_CM over RoCE

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

RE: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

RE: [PATCH V2 02/10] cxgb4: Common platform specific changes for DB Drop Recovery

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

[PATCH RESEND 03/10] cxgb4: DB Drop Recovery for RDMA and LLD queues.

[PATCH V2 02/10] cxgb4: Common platform specific changes for DB Drop Recovery

Re: ib_srpt status

Re: ib_srpt status

Re: [PATCH] ib_srpt: Initial SRP Target merge for v3.2-rc1

Re: [PATCH] ib_srpt: Initial SRP Target merge for v3.2-rc1

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.

ghost cq event in handler

RE: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel

22 matches

Site Navigation

Mail list logo

Footer information