Re: [openib-general] krping test utility

[Devesh Sharma]

On 5/24/06, Roland Dreier <[EMAIL PROTECTED]> wrote:
Devesh> Hello all, In the krping test utility get_dma_mr is calledDevesh> with access premissionsDevesh> IB_ACCESS_LOCAL_WRITE|IB_ACCESS_REMOTE_WRITE|IB_ACCESS_REMOTE_READ,Devesh> But the lkey we get from get_dma_mr is similar to reserved
Devesh> lkey with which only Local operations are allowed, butDevesh> here it seems violating that statement.No, ib_get_dma_mr() returns an L_Key/R_Key with exactly thepermissions requested.

According to previous disscussions on openIb fourm about ib_get_dma_mr
I have an impression that lkey returned from ib_get_dma_mr is similar
to Reserved L_Key specified in IBTA spec. But now usage in Krping is
requesting remote Read/Write premissions also, So this is confusing me,
please clarify this.
 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] RE: [PATCH]OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix NULL ptrissue

[Eitan Zahavi]

Sorry for my abbreviated late night style.
By "yes it is" I meant that the CL_ASSERT is really mapping to no op in
non debug version.
By "I do not think it is required" I mean that if we have the new "if"
we do not need the CL_ASSERT.

Eitan

> RE:[PATCH]OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix
NULL
> ptrissue
> 
> On Wed, 2006-05-24 at 16:41, Eitan Zahavi wrote:
> > > But isn't CL_ASSERT a debug compile time thing so it's needed when
> > it's
> > > built without debug ?
> > Yes it is - but I do not think it is required.
> 
> I don't understand what you wrote.
> 
> Does "yes it is" mean that you agree it is a debug compile time thing
?
> 
> Why is it not required ? What if the memory allocation fails ?
> 
> -- Hal
> 
> >
> > > >
> > > > You missed the line that asserts on null p_next_step just before
the
> > > > code you changed .
> > >
> > >
> > > -- Hal
> > >
> > > > .
> > > >
> > > > EZ
> > > >
> > > > Eitan Zahavi
> > > > Senior Engineering Director, Software Architect
> > > > Mellanox Technologies LTD
> > > > Tel:+972-4-9097208
> > > > Fax:+972-4-9593245
> > > > P.O. Box 586 Yokneam 20692 ISRAEL
> > > >
> > > >
> > > > > -Original Message-
> > > > > From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> > > > > Sent: Wednesday, May 24, 2006 4:34 PM
> > > > > To: openib-general@openib.org
> > > > > Cc: Eitan Zahavi
> > > > > Subject: [PATCH]
> > > > OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix
> > > > > NULL ptr issue
> > > > >
> > > > > OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_t Fix
NULL
> > ptr
> > > > > issue
> > > > >
> > > > > Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>
> > > > >
> > > > > Index: opensm/osm_ucast_updn.c
> > > > >
> > ===
> > > > > --- opensm/osm_ucast_updn.c   (revision 7435)
> > > > > +++ opensm/osm_ucast_updn.c   (working copy)
> > > > > @@ -121,10 +121,12 @@ __updn_create_updn_next_step_t(IN updn_s
> > > > >p_next_step = (updn_next_step_t*)
> > cl_zalloc(sizeof(*p_next_step));
> > > > >CL_ASSERT (p_next_step != NULL);
> > > > >
> > > > > -  p_next_step->state = state;
> > > > > -  p_next_step->p_sw = p_sw;
> > > > > +  if (p_next_step)
> > > > > +  {
> > > > > +p_next_step->state = state;
> > > > > +p_next_step->p_sw = p_sw;
> > > > > +  }
> > > > >return p_next_step;
> > > > > -
> > > > >  }
> > > > >
> > > > >
> > > >
> >
/**
> > > > >
> > > >
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Compilation issues on rhel4 u3 ppc64 sysfs

[Tziporet Koren]

Paul wrote:
I dont see an rc5 @ 
https://openfabrics.org/svn/gen2/branches/1.0/ofed/releases/


Has anybody gotten the build.sh or install.sh to produce 64bit stuff 
on ppc64 ? (show of hands ? any voodoo required ?)

We succeeded to compile everything on PPC64 on the RC5 tag we opened.
It is not yet published (should be today).

Instruction how to compile PPC64 with lib64 will be in the release mail.

Tziporet
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][3/7]ipoib performance patches -- remove tx_ring

[Shirley Ma]

Here is the new patch.


diff -urpN infiniband-ah/ulp/ipoib/ipoib.h
infiniband-tx/ulp/ipoib/ipoib.h
--- infiniband-ah/ulp/ipoib/ipoib.h        2006-05-23
10:09:05.0 -0700
+++ infiniband-tx/ulp/ipoib/ipoib.h        2006-05-24
11:45:52.0 -0700
@@ -114,11 +114,19 @@ struct ipoib_rx_buf {
         dma_addr_t        mapping;
 };
 
-struct ipoib_tx_buf {
-        struct sk_buff *skb;
-        DECLARE_PCI_UNMAP_ADDR(mapping)
+struct ipoib_skb_prv {
+        dma_addr_t        
       addr;
+        struct ipoib_ah      
         *ah;
+        struct sk_buff      
         *skb;
+        struct list_head      
 list;
 };
 
+#define IPOIB_SKB_PRV_ADDR(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->addr)
+#define IPOIB_SKB_PRV_AH(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->ah)
+#define IPOIB_SKB_PRV_SKB(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->skb)
+#define IPOIB_SKB_PRV_LIST(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->list)
+
+
 /*
  * Device private locking: tx_lock protects members used in TX fast
  * path (and we use LLTX so upper layers don't do extra locking).
@@ -166,12 +174,11 @@ struct ipoib_dev_priv {
 
         struct ipoib_rx_buf *rx_ring;
 
-        spinlock_t        
  tx_lock        cacheline_aligned_in_smp;
-        struct
ipoib_tx_buf *tx_ring;
-        unsigned        
    tx_head;
-        unsigned        
    tx_tail;
+        spinlock_t        
  tx_lock;
         struct ib_sge      
 tx_sge;
         struct ib_send_wr    tx_wr;
+        spinlock_t        
    slist_lock;
+        struct list_head     send_list;
 
         struct list_head dead_ahs;
 
diff -urpN infiniband-ah/ulp/ipoib/ipoib_ib.c infiniband-tx/ulp/ipoib/ipoib_ib.c
--- infiniband-ah/ulp/ipoib/ipoib_ib.c        2006-05-23
10:14:08.0 -0700
+++ infiniband-tx/ulp/ipoib/ipoib_ib.c        2006-05-24
14:57:33.0 -0700
@@ -243,45 +243,39 @@ static void ipoib_ib_handle_send_wc(stru
                  
               
  struct ib_wc *wc)
 {
         struct ipoib_dev_priv *priv = netdev_priv(dev);
-        unsigned int wr_id = wc->wr_id;
-        struct ipoib_tx_buf *tx_req;
+        struct sk_buff *skb;
         unsigned long flags;
+        unsigned long wr_id = wc->wr_id;
 
-        ipoib_dbg_data(priv, "called:
id %d, op %d, status: %d\n",
-                
      wr_id, wc->opcode, wc->status);
-
-        if (wr_id >= ipoib_sendq_size)
{
-                ipoib_warn(priv,
"completion event with wrid %d (> %d)\n",
-                
          wr_id, ipoib_sendq_size);
-        
       return;
-        }
-
-        ipoib_dbg_data(priv, "send complete,
wrid %d\n", wr_id);
-
-        tx_req = &priv->tx_ring[wr_id];
-
-        dma_unmap_single(priv->ca->dma_device,
-                
        pci_unmap_addr(tx_req, mapping),
-                
        tx_req->skb->len,
-                
        DMA_TO_DEVICE);
+        skb = (struct sk_buff *)wr_id;
+        kref_put(&IPOIB_SKB_PRV_AH(skb)->ref,
ipoib_free_ah);
 
+         if (IS_ERR(skb) || skb != IPOIB_SKB_PRV_SKB(skb))
{
+                 ipoib_warn(priv,
"send completion event with corrupted wrid\n");
+                  return;
+          }
+        spin_lock_irqsave(&priv->slist_lock,
flags);
+        list_del(&IPOIB_SKB_PRV_LIST(skb));
+        spin_unlock_irqrestore(&priv->slist_lock,
flags);
+          
+        ipoib_dbg_data(priv, "send complete,
wrid %lu\n", wr_id);
+  
+          dma_unmap_single(priv->ca->dma_device,
+                  
       IPOIB_SKB_PRV_ADDR(skb),
+                
        skb->len,
+                  
        DMA_TO_DEVICE);
+  
         ++priv->stats.tx_packets;
-        priv->stats.tx_bytes += tx_req->skb->len;
-
-        dev_kfree_skb_any(tx_req->skb);
-
-        spin_lock_irqsave(&priv->tx_lock,
flags);
-        ++priv->tx_tail;
-        if (netif_queue_stopped(dev) &&
-            priv->tx_head - priv->tx_tail
<= ipoib_sendq_size >> 1)
-        
       netif_wake_queue(dev);
-        spin_unlock_irqrestore(&priv->tx_lock,
flags);
-
-        if (wc->status != IB_WC_SUCCESS
&&
-            wc->status != IB_WC_WR_FLUSH_ERR)
-                ipoib_warn(priv,
"failed send event "
-                
          "(status=%d, wrid=%d vend_err
%x)\n",
-                
          wc->status, wr_id, wc->vendor_err);
+        priv->stats.tx_bytes += skb->len;
+        dev_kfree_skb_any(skb);
+ 
+        if (netif_queue_stopped(dev))
+                  netif_wake_queue(dev);
+          if (wc->status != IB_WC_SUCCESS
&&
+              wc->status
!= IB_WC_WR_FLUSH_ERR)
+                  ipoib_warn(priv,
"failed send event "
+                  
         "(status=%d, wrid=%lu vend_err
%x)\n",
+                  
         wc->status, wr_id, wc->vendor_err);
 }
 
 void ipoib_ib_send_completion(struct ib_cq *cq, void *dev_ptr)
@@ -313,7 +307,7 @@ void ipoib_ib_recv_completion(struct ib_
 }
 
 static inline int post_send(struct ipoib_dev_priv *priv,
-                
           unsigned int wr_id,
+                
      

Re: [openib-general] [PATCH][3/7]ipoib performance patches -- remove tx_ring

[Shirley Ma]

Oops, I missed one pair of spin_lock_irqsave()/spin_lock_irqrestore()
to protect send_list in ipoib_ib_handle_send_wc().

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [Question]Fast Registration Work Request and Fast Memory Regions

[Ian Jiang]

Hi all!In IBA spec. 1.2 defines a Fast Registration Work Request and it should be used together with the Allocate L Key verb to get a fast registration in my opinion. Fast Memory Regions is used in the ULPs of IB Stack, such as kDAPL, iSER.
The two comfuse me very much. I don't think they are the same thing but I cann't tell the difference between them.Would anybody give an explaination on how they work and where they are usually used?Thanks very much and best wish!
-- Ian Jiang
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH][3/7]ipoib performance patches -- remove tx_ring

[Shirley Ma]

Roland, 

Here is the tx_ring removal patch for
you to review.


diff -urpN infiniband-ah/ulp/ipoib/ipoib.h
infiniband-tx/ulp/ipoib/ipoib.h
--- infiniband-ah/ulp/ipoib/ipoib.h        2006-05-23
10:09:05.0 -0700
+++ infiniband-tx/ulp/ipoib/ipoib.h        2006-05-24
11:45:52.0 -0700
@@ -114,11 +114,19 @@ struct ipoib_rx_buf {
         dma_addr_t        mapping;
 };
 
-struct ipoib_tx_buf {
-        struct sk_buff *skb;
-        DECLARE_PCI_UNMAP_ADDR(mapping)
+struct ipoib_skb_prv {
+        dma_addr_t        
       addr;
+        struct ipoib_ah      
         *ah;
+        struct sk_buff      
         *skb;
+        struct list_head      
 list;
 };
 
+#define IPOIB_SKB_PRV_ADDR(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->addr)
+#define IPOIB_SKB_PRV_AH(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->ah)
+#define IPOIB_SKB_PRV_SKB(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->skb)
+#define IPOIB_SKB_PRV_LIST(skb)        (((struct
ipoib_skb_prv *)(skb)->cb)->list)
+
+
 /*
  * Device private locking: tx_lock protects members used in TX fast
  * path (and we use LLTX so upper layers don't do extra locking).
@@ -166,12 +174,11 @@ struct ipoib_dev_priv {
 
         struct ipoib_rx_buf *rx_ring;
 
-        spinlock_t        
  tx_lock        cacheline_aligned_in_smp;
-        struct
ipoib_tx_buf *tx_ring;
-        unsigned        
    tx_head;
-        unsigned        
    tx_tail;
+        spinlock_t        
  tx_lock;
         struct ib_sge      
 tx_sge;
         struct ib_send_wr    tx_wr;
+        spinlock_t        
    slist_lock;
+        struct list_head     send_list;
 
         struct list_head dead_ahs;
 
diff -urpN infiniband-ah/ulp/ipoib/ipoib_ib.c infiniband-tx/ulp/ipoib/ipoib_ib.c
--- infiniband-ah/ulp/ipoib/ipoib_ib.c        2006-05-23
10:14:08.0 -0700
+++ infiniband-tx/ulp/ipoib/ipoib_ib.c        2006-05-24
11:58:46.0 -0700
@@ -243,45 +243,36 @@ static void ipoib_ib_handle_send_wc(stru
                  
               
  struct ib_wc *wc)
 {
         struct ipoib_dev_priv *priv = netdev_priv(dev);
-        unsigned int wr_id = wc->wr_id;
-        struct ipoib_tx_buf *tx_req;
-        unsigned long flags;
-
-        ipoib_dbg_data(priv, "called:
id %d, op %d, status: %d\n",
-                
      wr_id, wc->opcode, wc->status);
-
-        if (wr_id >= ipoib_sendq_size)
{
-                ipoib_warn(priv,
"completion event with wrid %d (> %d)\n",
-                
          wr_id, ipoib_sendq_size);
-                return;
-        }
-
-        ipoib_dbg_data(priv, "send complete,
wrid %d\n", wr_id);
+        struct
sk_buff *skb;
+        unsigned long wr_id = wc->wr_id;
 
-        tx_req = &priv->tx_ring[wr_id];
-
-        dma_unmap_single(priv->ca->dma_device,
-                
        pci_unmap_addr(tx_req, mapping),
-                
        tx_req->skb->len,
-                
        DMA_TO_DEVICE);
+        skb = (struct sk_buff *)wr_id;
+        kref_put(&IPOIB_SKB_PRV_AH(skb)->ref,
ipoib_free_ah);
 
+         if (IS_ERR(skb) || skb != IPOIB_SKB_PRV_SKB(skb))
{
+                 ipoib_warn(priv,
"send completion event with corrupted wrid\n");
+                  return;
+          }
+        list_del(&IPOIB_SKB_PRV_LIST(skb));
+  
+        ipoib_dbg_data(priv, "send complete,
wrid %lu\n", wr_id);
+  
+          dma_unmap_single(priv->ca->dma_device,
+                  
       IPOIB_SKB_PRV_ADDR(skb),
+                
        skb->len,
+                  
        DMA_TO_DEVICE);
+  
         ++priv->stats.tx_packets;
-        priv->stats.tx_bytes += tx_req->skb->len;
-
-        dev_kfree_skb_any(tx_req->skb);
-
-        spin_lock_irqsave(&priv->tx_lock,
flags);
-        ++priv->tx_tail;
-        if (netif_queue_stopped(dev) &&
-            priv->tx_head - priv->tx_tail
<= ipoib_sendq_size >> 1)
-                netif_wake_queue(dev);
-        spin_unlock_irqrestore(&priv->tx_lock,
flags);
-
-        if (wc->status != IB_WC_SUCCESS
&&
-        
   wc->status != IB_WC_WR_FLUSH_ERR)
-                ipoib_warn(priv,
"failed send event "
-                
          "(status=%d, wrid=%d vend_err
%x)\n",
-                
          wc->status, wr_id, wc->vendor_err);
+        priv->stats.tx_bytes += skb->len;
+        dev_kfree_skb_any(skb);
+ 
+        if (netif_queue_stopped(dev))
+                  netif_wake_queue(dev);
+          if (wc->status != IB_WC_SUCCESS
&&
+              wc->status
!= IB_WC_WR_FLUSH_ERR)
+                  ipoib_warn(priv,
"failed send event "
+                  
         "(status=%d, wrid=%lu vend_err
%x)\n",
+                  
         wc->status, wr_id, wc->vendor_err);
 }
 
 void ipoib_ib_send_completion(struct ib_cq *cq, void *dev_ptr)
@@ -313,7 +304,7 @@ void ipoib_ib_recv_completion(struct ib_
 }
 
 static inline int post_send(struct ipoib_dev_priv *priv,
-                
           unsigned int wr_id,
+                
           unsigned long wr_id,
                  
          struct ib_ah *address, u32

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

Shirley> And in the reap AH code, tx_tail/tx_head isn't
Shirley> consistently protected by tx_lock. It uses priv->lock.

Roland> Hmm, that may be a bug.  I'll take a look.

Something like this (untested still) should fix things up:

--- infiniband/ulp/ipoib/ipoib_ib.c (revision 7485)
+++ infiniband/ulp/ipoib/ipoib_ib.c (working copy)
@@ -84,15 +84,9 @@ void ipoib_free_ah(struct kref *kref)
 
unsigned long flags;
 
-   if ((int) priv->tx_tail - (int) ah->last_send >= 0) {
-   ipoib_dbg(priv, "Freeing ah %p\n", ah->ah);
-   ib_destroy_ah(ah->ah);
-   kfree(ah);
-   } else {
-   spin_lock_irqsave(&priv->lock, flags);
-   list_add_tail(&ah->list, &priv->dead_ahs);
-   spin_unlock_irqrestore(&priv->lock, flags);
-   }
+   spin_lock_irqsave(&priv->lock, flags);
+   list_add_tail(&ah->list, &priv->dead_ahs);
+   spin_unlock_irqrestore(&priv->lock, flags);
 }
 
 static int ipoib_ib_post_receive(struct net_device *dev, int id)
@@ -376,19 +370,16 @@ static void __ipoib_reap_ah(struct net_d
struct ipoib_ah *ah, *tah;
LIST_HEAD(remove_list);
 
-   spin_lock_irq(&priv->lock);
+   spin_lock_irq(&priv->tx_lock);
+   spin_lock(&priv->lock);
list_for_each_entry_safe(ah, tah, &priv->dead_ahs, list)
if ((int) priv->tx_tail - (int) ah->last_send >= 0) {
list_del(&ah->list);
-   list_add_tail(&ah->list, &remove_list);
+   ib_destroy_ah(ah->ah);
+   kfree(ah);
}
-   spin_unlock_irq(&priv->lock);
-
-   list_for_each_entry_safe(ah, tah, &remove_list, list) {
-   ipoib_dbg(priv, "Reaping ah %p\n", ah->ah);
-   ib_destroy_ah(ah->ah);
-   kfree(ah);
-   }
+   spin_unlock(&priv->lock);
+   spin_unlock_irq(&priv->tx_lock);
 }
 
 void ipoib_reap_ah(void *dev_ptr)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

Roland Dreier <[EMAIL PROTECTED]> wrote on 05/24/2006
05:52:31 PM:
>     Shirley> With tx_ring removal patch, RCU can be done.
> 
> OK, I guess I'll wait and see.  But to be honest I don't see
how RCU
> helps anything.
> 
>  - R.

I am continuing to sumit the tx_ring
patch with atomic operation for you
to review, let's discuss the AH reap
solution later.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

Shirley> With tx_ring removal patch, RCU can be done.

OK, I guess I'll wait and see.  But to be honest I don't see how RCU
helps anything.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

Roland Dreier <[EMAIL PROTECTED]> wrote on 05/24/2006
05:11:12 PM:

>     Shirley> Roland, My idea is to remove this AH reap
thread. We can
>     Shirley> use RCU to do the same work without lots
of coding. Do
>     Shirley> you agree?
> 
> No, I don't see how that will help.  How does RCU know when it's
safe
> to free an AH?

With tx_ring removal patch, RCU can
be done.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

Shirley> Roland, My idea is to remove this AH reap thread. We can
Shirley> use RCU to do the same work without lots of coding. Do
Shirley> you agree?

No, I don't see how that will help.  How does RCU know when it's safe
to free an AH?

Shirley> And in the reap AH code, tx_tail/tx_head isn't
Shirley> consistently protected by tx_lock. It uses priv->lock.

Hmm, that may be a bug.  I'll take a look.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

Roland,

My idea is to remove this AH reap thread.
We can use RCU to do the same work without lots of coding. Do you agree?

And in the reap AH code, tx_tail/tx_head
isn't consistently protected by tx_lock. It uses priv->lock.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

Shirley> I don't think two atomic operation is that expensive
Shirley> compare to reaping AHs in process context according to
Shirley> the test results and profiling data. Or we can use RCU
Shirley> instead.

What is the cost of reaping AHs?  Under most workloads I would expect AHs
essentially never get freed.

Are you saying that you see __ipoib_reap_ah() show up in your profiles?
If so then we should fix the underlying cause of that rather than
slowing down the ipoib fast path.

BTW, since it is allowed to call ib_destroy_ah() with a lock held (cf
Documentation/infiniband/core_locking.txt) then I think the patch
below is a reasonable cleanup:

--- infiniband/ulp/ipoib/ipoib_ib.c (revision 7485)
+++ infiniband/ulp/ipoib/ipoib_ib.c (working copy)
@@ -380,15 +380,10 @@ static void __ipoib_reap_ah(struct net_d
list_for_each_entry_safe(ah, tah, &priv->dead_ahs, list)
if ((int) priv->tx_tail - (int) ah->last_send >= 0) {
list_del(&ah->list);
-   list_add_tail(&ah->list, &remove_list);
+   ib_destroy_ah(ah->ah);
+   kfree(ah);
}
spin_unlock_irq(&priv->lock);
-
-   list_for_each_entry_safe(ah, tah, &remove_list, list) {
-   ipoib_dbg(priv, "Reaping ah %p\n", ah->ah);
-   ib_destroy_ah(ah->ah);
-   kfree(ah);
-   }
 }
 
 void ipoib_reap_ah(void *dev_ptr)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Sean Hefty]

Sean Hefty wrote:
Reviewing the code, the multicast module should cancel all SA queries 
and wait for them to complete before unloading.  (Even if it didn't 
perform the cancel, it should still wait for any outstanding SA query to 
complete.)


As a thought, is there any chance this crash is related to the ib_sa (and ib_cm, 
 rdma_cm, ib_addr) interface, module unload issue that was reported?


- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

Roland Dreier <[EMAIL PROTECTED]> wrote on 05/24/2006
04:07:58 PM:
> To reiterate: freeing AHs is a rare,
> "slow path" operation that can be done asynchronously.  It
is not a
> good tradeoff to do two atomic_t operations for every sent packet,
> just to avoid occasionally reaping AHs in process context.

I don't think two atomic operation is that expensive
compare to 
reaping AHs in process context according to the test
results and 
profiling data. Or we can use RCU instead.

>  > > But can you guarantee that the AH stays around until
after the send
>  > > completes (which could be an arbitrarily long delay)?
> 
>  > I checked negih_add_path(), for unicast it is true always.
See code below.
>  > 
>  > static void neigh_add_path(..)
>  > {
>  > ...
>  >         if (path->ah) {
>  >                
kref_get(&path->ah->ref);
>  >                
neigh->ah = path->ah;
>  >             ipoib_send(dev,
skb, path->ah... 
>  > } 
> 
> Again, I don't understand how this is a response at all.  The
AH
> cannot be freed until after the send operation is actually fully
> completed, which could be a long time after ib_post_send() returns.
> 
> If an AH is freed after ipoib_send() returns but before the send is
> executed, then the HCA may use stale data, which could lead to a send
> error.
> 
> To summarize: the patch is broken (leads to incorrect lifetimes for
> AHs), and in any case makes the send fast path slower.
> 
>  - R.

That's a value point. This problem will be addressed
in next tx_ring removal 
patch. The kref_put was called in ipoib_ib_handle_send_wc().

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Sean Hefty]

Michael S. Tsirkin wrote:

The last trunk build causes kernel oops on 2.6.16 while restarting the driver.

(the previous build -rev 7422- works fine)


Note that ipoib moved to ib_multicast in rev 7401.


May 24 16:00:40 sw037 kernel: Modules linked in: ib_sa ib_uverbs ib_umad


ib_mthca ib_mad ib_core


Can you please verify that ib_multicast was loaded prior to the crash?


{:ib_sa:ib_sa_mcmember_rec_callback+76}


I've been unable to reproduce this after 100 or so tests of doing an ifconfig 
while unloading the modules.  I've tried with the SM both up and down, and with 
ping running at the same time.


Reviewing the code, the multicast module should cancel all SA queries and wait 
for them to complete before unloading.  (Even if it didn't perform the cancel, 
it should still wait for any outstanding SA query to complete.)



Right, reverting to 7401 solved the problem.


Are you able to duplicate the problem with the latest svn rev?

- Sean


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

 > > But freeing AHs is something that happens infrequently and can be done
 > > asynchronously.  You're replacing that cost with two atomic operations
 > > per send packet!

 > No, actually it didn't free during sending during my test.

Sorry, I don't understand that answer at all.  It doesn't seem to be
responding to my point at all.  To reiterate: freeing AHs is a rare,
"slow path" operation that can be done asynchronously.  It is not a
good tradeoff to do two atomic_t operations for every sent packet,
just to avoid occasionally reaping AHs in process context.

 > > But can you guarantee that the AH stays around until after the send
 > > completes (which could be an arbitrarily long delay)?

 > I checked negih_add_path(), for unicast it is true always. See code below.
 > 
 > static void neigh_add_path(..)
 > {
 > ...
 > if (path->ah) {
 > kref_get(&path->ah->ref);
 > neigh->ah = path->ah;
 > ipoib_send(dev, skb, path->ah... 
 > } 

Again, I don't understand how this is a response at all.  The AH
cannot be freed until after the send operation is actually fully
completed, which could be a long time after ib_post_send() returns.

If an AH is freed after ipoib_send() returns but before the send is
executed, then the HCA may use stale data, which could lead to a send
error.

To summarize: the patch is broken (leads to incorrect lifetimes for
AHs), and in any case makes the send fast path slower.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

Roland,

Roland Dreier <[EMAIL PROTECTED]> wrote on 05/24/2006
03:01:01 PM:
>     Shirley> Compared to have a single thread handling
AH, I don't
>     Shirley> think this atomic operation is expensive.
> 
> But freeing AHs is something that happens infrequently and can be
done
> asynchronously.  You're replacing that cost with two atomic operations
> per send packet!

No, actually it didn't free during sending during
my test.

>     Shirley> It is true for unicast, it has a reference
count before
>     Shirley> ipoib_send(). I need to look at multicast.
> 
> But can you guarantee that the AH stays around until after the send
> completes (which could be an arbitrarily long delay)?
> 
>  - R.

I checked negih_add_path(), for unicast
it is true always. See code below.

static void neigh_add_path(..)
{
...
        if (path->ah)
{
           
    kref_get(&path->ah->ref);
           
    neigh->ah = path->ah;
         
  ipoib_send(dev, skb, path->ah...        
}        

Please correct me if I am wrong.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: which dapl/udapl changes in trunk should be imported into OFED branch? (patch enclosed)

[James Lentini]

On Wed, 24 May 2006, Jack Morgenstein wrote:

> Hi,
> 
> Below is a patch file of differences between the OFED dapl library 
> and the openib main trunk dapl library.
> 
> Please indicate which of the dapl library changes are necessary for 
> the Intel MPI to work correctly in OFED.

How recent is the ucm code in OFED? 
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

Shirley> Compared to have a single thread handling AH, I don't
Shirley> think this atomic operation is expensive.

But freeing AHs is something that happens infrequently and can be done
asynchronously.  You're replacing that cost with two atomic operations
per send packet!

Shirley> It is true for unicast, it has a reference count before
Shirley> ipoib_send(). I need to look at multicast.

But can you guarantee that the AH stays around until after the send
completes (which could be an arbitrarily long delay)?

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

Roland,

Roland Dreier <[EMAIL PROTECTED]> wrote on 05/24/2006
02:03:34 PM:

>     Shirley> My observation is the atomic operation is
not that
>     Shirley> expensive.
> 
> It's just about the worst thing to do.  For example, on x86/x86-64
an
> instruction with the lock; prefix is quite slow.  If you look
at an
> instruction level profile you can see that quite clearly.

Compared to have a single thread handling AH, I don't
think this atomic operation is expensive.

>     Shirley> I thought the path holding another AH reference
to
>     Shirley> prevent it to be freed?
> 
> If that were true then why would we want to reference count sends
at
> all?  The whole point is that a path might be destroyed before
the
> send is executed.
> 
>  - R.

It is true for unicast, it has a reference count before ipoib_send(). I
need to look at multicast.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH] ib_multicast: set member state correctly

[Sean Hefty]

Once a join request completes, we need to set the state to MCAST_MEMBER.
Otherwise, we won't try to leave the group when ib_free_multicast() is called,
which will leave the port in the multicast group.

Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
---
This fix shouldn't have any effect on stability.

Index: multicast.c
===
--- multicast.c (revision 7442)
+++ multicast.c (working copy)
@@ -332,6 +332,7 @@ static int send_leave(struct mcast_group
 static void join_group(struct mcast_group *group, struct mcast_member *member,
   u8 join_state)
 {
+   member->state = MCAST_MEMBER;
adjust_membership(group, join_state, 1);
group->rec.join_state |= join_state;
member->multicast.rec = group->rec;

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] RE: [PATCH]OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix NULL ptr issue

[Hal Rosenstock]

On Wed, 2006-05-24 at 16:41, Eitan Zahavi wrote:
> > But isn't CL_ASSERT a debug compile time thing so it's needed when
> it's
> > built without debug ?
> Yes it is - but I do not think it is required.

I don't understand what you wrote. 

Does "yes it is" mean that you agree it is a debug compile time thing ?

Why is it not required ? What if the memory allocation fails ?

-- Hal

> 
> > >
> > > You missed the line that asserts on null p_next_step just before the
> > > code you changed .
> > 
> > 
> > -- Hal
> > 
> > > .
> > >
> > > EZ
> > >
> > > Eitan Zahavi
> > > Senior Engineering Director, Software Architect
> > > Mellanox Technologies LTD
> > > Tel:+972-4-9097208
> > > Fax:+972-4-9593245
> > > P.O. Box 586 Yokneam 20692 ISRAEL
> > >
> > >
> > > > -Original Message-
> > > > From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> > > > Sent: Wednesday, May 24, 2006 4:34 PM
> > > > To: openib-general@openib.org
> > > > Cc: Eitan Zahavi
> > > > Subject: [PATCH]
> > > OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix
> > > > NULL ptr issue
> > > >
> > > > OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_t Fix NULL
> ptr
> > > > issue
> > > >
> > > > Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>
> > > >
> > > > Index: opensm/osm_ucast_updn.c
> > > >
> ===
> > > > --- opensm/osm_ucast_updn.c (revision 7435)
> > > > +++ opensm/osm_ucast_updn.c (working copy)
> > > > @@ -121,10 +121,12 @@ __updn_create_updn_next_step_t(IN updn_s
> > > >p_next_step = (updn_next_step_t*)
> cl_zalloc(sizeof(*p_next_step));
> > > >CL_ASSERT (p_next_step != NULL);
> > > >
> > > > -  p_next_step->state = state;
> > > > -  p_next_step->p_sw = p_sw;
> > > > +  if (p_next_step)
> > > > +  {
> > > > +p_next_step->state = state;
> > > > +p_next_step->p_sw = p_sw;
> > > > +  }
> > > >return p_next_step;
> > > > -
> > > >  }
> > > >
> > > >
> > >
> /**
> > > >
> > >

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: Compilation issues on rhel4 u3 ppc64 sysfs.o

[Paul]

Yeah. I got it (had to build it myself from the src rpm). I now have a functioning 64bit openIB install. Currently running into some issues with open-mpi  but I am taking that up with the open-mpi mailling list.
Thanks for the help guys.On 5/24/06, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote:
Quoting r. Paul <[EMAIL PROTECTED]>:> Subject: Re: Compilation issues on rhel4 u3 ppc64 sysfs.o>> Scott,>   Upon further inspection the 
build.sh and install.sh scripts built 32bit libraries and binaries. If I export CFLAGS (and the like) to include -m64 then the build dies while looking for a 64bit libsysfs. rhel4 u3 does not include a ppc64 sysfsutils, nor have I been able to find an actual 64bit version of it. Is there a workaround for getting things to build actual ppc64 binaries/libraries ?
>> The actual error is:> checking for dlsym in -ldl... yes> checking for pthread_mutex_init in -lpthread... yes> checking for sysfs_open_class in -lsysfs... no> configure: error: sysfs_open_class() not found. libibverbs requires libsysfs.
You need the 64 bit version of libsysfs - verbs require this.--MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

Shirley> My observation is the atomic operation is not that
Shirley> expensive.

It's just about the worst thing to do.  For example, on x86/x86-64 an
instruction with the lock; prefix is quite slow.  If you look at an
instruction level profile you can see that quite clearly.

Shirley> I thought the path holding another AH reference to
Shirley> prevent it to be freed?

If that were true then why would we want to reference count sends at
all?  The whole point is that a path might be destroyed before the
send is executed.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

Roland Dreier <[EMAIL PROTECTED]> wrote on 05/24/2006
01:50:37 PM:

> NAK to this patch.  Not only is is a step backwards in performance
--
> you've essentially added two (expensive) atomic operations for every
> packet sent 

My observation is the atomic operation is not that
expensive.

> -- but the patch is actually wrong:
> 
>  > +       err = post_send(priv, priv->tx_head
& (ipoib_sendq_size - 1), 
>  > +                
      address->ah, qpn, addr, skb->len); 
>  > +       kref_put(&address->ref, ipoib_free_ah);
> 
> The whole point of the complexity in AH handling in IPoIB is that
AHs
> cannot be freed until the driver knows that all sends referring to
> them have _completed_.  As you've written your patch, an AH can
easily
> be freed before the HCA has a chance to execute the corresponding
send
> request.
> 
>  - R.

I thought the path holding another AH
reference to prevent it to be freed?

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: Compilation issues on rhel4 u3 ppc64 sysfs.o

[Michael S. Tsirkin]

Quoting r. Paul <[EMAIL PROTECTED]>:
> Subject: Re: Compilation issues on rhel4 u3 ppc64 sysfs.o
> 
> Scott,
>   Upon further inspection the build.sh and install.sh scripts built 32bit 
> libraries and binaries. If I export CFLAGS (and the like) to include -m64 
> then the build dies while looking for a 64bit libsysfs. rhel4 u3 does not 
> include a ppc64 sysfsutils, nor have I been able to find an actual 64bit 
> version of it. Is there a workaround for getting things to build actual ppc64 
> binaries/libraries ?
> 
> The actual error is:
> checking for dlsym in -ldl... yes
> checking for pthread_mutex_init in -lpthread... yes
> checking for sysfs_open_class in -lsysfs... no
> configure: error: sysfs_open_class() not found. libibverbs requires libsysfs.

You need the 64 bit version of libsysfs - verbs require this.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Roland Dreier]

NAK to this patch.  Not only is is a step backwards in performance --
you've essentially added two (expensive) atomic operations for every
packet sent -- but the patch is actually wrong:

 > +   err = post_send(priv, priv->tx_head & (ipoib_sendq_size - 1), 
 > +   address->ah, qpn, addr, skb->len); 
 > +   kref_put(&address->ref, ipoib_free_ah);

The whole point of the complexity in AH handling in IPoIB is that AHs
cannot be freed until the driver knows that all sends referring to
them have _completed_.  As you've written your patch, an AH can easily
be freed before the HCA has a chance to execute the corresponding send
request.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH] mthca: fix posting list of 256 to SRQ

[Roland Dreier]

Thanks, applied.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH][1/7]ipoib performance patches -- remove ah_reap

[Shirley Ma]

I have changed the order of these patches
in order to make the splitting easy.


diff -urpN infiniband-split-cq/ulp/ipoib/ipoib.h
infiniband-ah/ulp/ipoib/ipoib.h
--- infiniband-split-cq/ulp/ipoib/ipoib.h        2006-05-23
10:07:15.0 -0700
+++ infiniband-ah/ulp/ipoib/ipoib.h        2006-05-23
10:09:05.0 -0700
@@ -86,7 +86,6 @@ enum {
         IPOIB_PKEY_STOP      
    = 4,
         IPOIB_FLAG_SUBINTERFACE   = 5,
         IPOIB_MCAST_RUN      
    = 6,
-        IPOIB_STOP_REAPER      
  = 7,
         IPOIB_MCAST_STARTED    
  = 8,
 
         IPOIB_MAX_BACKOFF_SECONDS = 16,
@@ -147,7 +146,6 @@ struct ipoib_dev_priv {
         struct work_struct mcast_task;
         struct work_struct flush_task;
         struct work_struct restart_task;
-        struct work_struct ah_reap_task;
 
         struct ib_device *ca;
         u8          
           port;
@@ -197,7 +195,6 @@ struct ipoib_ah {
         struct ib_ah      *ah;
         struct list_head   list;
         struct kref        ref;
-        unsigned        
  last_send;
 };
 
 struct ipoib_path {
@@ -263,7 +260,6 @@ int ipoib_add_pkey_attr(struct net_devic
 
 void ipoib_send(struct net_device *dev, struct sk_buff *skb,
                 struct
ipoib_ah *address, u32 qpn);
-void ipoib_reap_ah(void *dev_ptr);
 
 void ipoib_flush_paths(struct net_device *dev);
 struct ipoib_dev_priv *ipoib_intf_alloc(const char *format);
diff -urpN infiniband-split-cq/ulp/ipoib/ipoib_ib.c infiniband-ah/ulp/ipoib/ipoib_ib.c
--- infiniband-split-cq/ulp/ipoib/ipoib_ib.c        2006-05-23
10:07:51.0 -0700
+++ infiniband-ah/ulp/ipoib/ipoib_ib.c        2006-05-23
10:14:08.0 -0700
@@ -65,7 +65,6 @@ struct ipoib_ah *ipoib_create_ah(struct 
                 return
NULL;
 
         ah->dev       =
dev;
-        ah->last_send = 0;
         kref_init(&ah->ref);
 
         ah->ah = ib_create_ah(pd, attr);
@@ -83,17 +82,9 @@ void ipoib_free_ah(struct kref *kref)
         struct ipoib_ah *ah = container_of(kref,
struct ipoib_ah, ref);
         struct ipoib_dev_priv *priv = netdev_priv(ah->dev);
 
-        unsigned long flags;
-
-        if ((int) priv->tx_tail - (int)
ah->last_send >= 0) {
-                ipoib_dbg(priv,
"Freeing ah %p\n", ah->ah);
-                ib_destroy_ah(ah->ah);
-                kfree(ah);
-        } else {
-                spin_lock_irqsave(&priv->lock,
flags);
-                list_add_tail(&ah->list,
&priv->dead_ahs);
-                spin_unlock_irqrestore(&priv->lock,
flags);
-        }
+        ipoib_dbg(priv, "Freeing ah %p\n",
ah->ah);
+        ib_destroy_ah(ah->ah);
+        kfree(ah);
 }
 
 static int ipoib_ib_post_receive(struct net_device *dev, int id)
@@ -344,13 +335,16 @@ void ipoib_send(struct net_device *dev, 
         struct ipoib_dev_priv *priv = netdev_priv(dev);
         struct ipoib_tx_buf *tx_req;
         dma_addr_t addr;
+        int err;
 
+        kref_get(&address->ref);
         if (skb->len > dev->mtu +
INFINIBAND_ALEN) {
                 ipoib_warn(priv,
"packet len %d (> %d) too long to send, dropping\n",
                  
         skb->len, dev->mtu + INFINIBAND_ALEN);
                 ++priv->stats.tx_dropped;
                 ++priv->stats.tx_errors;
                 dev_kfree_skb_any(skb);
+                kref_put(&address->ref,
ipoib_free_ah);
                 return;
         }
 
@@ -370,8 +364,10 @@ void ipoib_send(struct net_device *dev, 
                  
            DMA_TO_DEVICE);
         pci_unmap_addr_set(tx_req, mapping,
addr);
 
-        if (unlikely(post_send(priv, priv->tx_head
& (ipoib_sendq_size - 1),
-                
              address->ah,
qpn, addr, skb->len))) {
+        err = post_send(priv, priv->tx_head
& (ipoib_sendq_size - 1),        
+                
       address->ah, qpn, addr, skb->len);

+        kref_put(&address->ref,
ipoib_free_ah);
+        if (unlikely(err)) {
                 ipoib_warn(priv,
"post_send failed\n");
                 ++priv->stats.tx_errors;
                 dma_unmap_single(priv->ca->dma_device,
addr, skb->len,
@@ -380,7 +376,6 @@ void ipoib_send(struct net_device *dev, 
         } else {
                 dev->trans_start
= jiffies;
 
-                address->last_send
= priv->tx_head;
                 ++priv->tx_head;
 
                 if
(priv->tx_head - priv->tx_tail == ipoib_sendq_size) {
@@ -390,38 +385,6 @@ void ipoib_send(struct net_device *dev, 
         }
 }
 
-static void __ipoib_reap_ah(struct net_device *dev)
-{
-        struct ipoib_dev_priv *priv = netdev_priv(dev);
-        struct ipoib_ah *ah, *tah;
-        LIST_HEAD(remove_list);
-
-        spin_lock_irq(&priv->lock);
-        list_for_each_entry_safe(ah, tah,
&priv->dead_ahs, list)
-                if
((int) priv->tx_tail - (int) ah->last_send >= 0) {
-                
       list_del(&ah->list);
-                
       list_add_tail(&ah->list, &remove_list);
-                }
-        spin_unlock_irq(&priv->lock);
-
-        list_for_each_entry_safe(ah, tah,
&remove_

[openib-general] RE: [PATCH]OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix NULL ptr issue

[Eitan Zahavi]

> But isn't CL_ASSERT a debug compile time thing so it's needed when
it's
> built without debug ?
Yes it is - but I do not think it is required.

> >
> > You missed the line that asserts on null p_next_step just before the
> > code you changed .
> 
> 
> -- Hal
> 
> > .
> >
> > EZ
> >
> > Eitan Zahavi
> > Senior Engineering Director, Software Architect
> > Mellanox Technologies LTD
> > Tel:+972-4-9097208
> > Fax:+972-4-9593245
> > P.O. Box 586 Yokneam 20692 ISRAEL
> >
> >
> > > -Original Message-
> > > From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> > > Sent: Wednesday, May 24, 2006 4:34 PM
> > > To: openib-general@openib.org
> > > Cc: Eitan Zahavi
> > > Subject: [PATCH]
> > OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix
> > > NULL ptr issue
> > >
> > > OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_t Fix NULL
ptr
> > > issue
> > >
> > > Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>
> > >
> > > Index: opensm/osm_ucast_updn.c
> > >
===
> > > --- opensm/osm_ucast_updn.c   (revision 7435)
> > > +++ opensm/osm_ucast_updn.c   (working copy)
> > > @@ -121,10 +121,12 @@ __updn_create_updn_next_step_t(IN updn_s
> > >p_next_step = (updn_next_step_t*)
cl_zalloc(sizeof(*p_next_step));
> > >CL_ASSERT (p_next_step != NULL);
> > >
> > > -  p_next_step->state = state;
> > > -  p_next_step->p_sw = p_sw;
> > > +  if (p_next_step)
> > > +  {
> > > +p_next_step->state = state;
> > > +p_next_step->p_sw = p_sw;
> > > +  }
> > >return p_next_step;
> > > -
> > >  }
> > >
> > >
> >
/**
> > >
> >
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Compilation issues on rhel4 u3 ppc64 sysfs

[Paul]

I dont see an rc5 @ https://openfabrics.org/svn/gen2/branches/1.0/ofed/releases/ Has anybody gotten the build.sh or install.sh to produce 64bit stuff on ppc64 ? (show of hands ? any voodoo required ?)
On 5/24/06, Scott Weitzenkamp (sweitzen) <[EMAIL PROTECTED]> wrote:





I know Vlad made some changes for rc5 in this area, at 
least for libsdp, not sure if other libs got changed as 
well.
 
Scott 
Weitzenkamp
SQA and Release 
Manager
Server Virtualization 
Business Unit
Cisco Systems
 

  
  
  From: Paul [mailto:
[EMAIL PROTECTED]] 
  Sent: Wednesday, May 24, 2006 11:00 AMTo: Scott 
  Weitzenkamp (sweitzen)Cc: 
  openib-general@openib.orgSubject: Re: [openib-general] Compilation 
  issues on rhel4 u3 ppc64 sysfs.o
  Scott,  Upon further inspection 
  the build.sh and install.sh scripts built 32bit libraries and binaries. If I 
  export CFLAGS (and the like) to include -m64 then the build dies while looking 
  for a 64bit libsysfs. rhel4 u3 does not include a ppc64 sysfsutils, nor have I 
  been able to find an actual 64bit version of it. Is there a workaround for 
  getting things to build actual ppc64 binaries/libraries ? The actual 
  error is:checking for dlsym in -ldl... yeschecking for 
  pthread_mutex_init in -lpthread... yeschecking for sysfs_open_class in 
  -lsysfs... noconfigure: error: sysfs_open_class() not found. libibverbs 
  requires libsysfs. 


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] Compilation issues on rhel4 u3 ppc64 sysfs.o

[Scott Weitzenkamp (sweitzen)]

I know Vlad made some changes for rc5 in this area, at 
least for libsdp, not sure if other libs got changed as 
well.
 
Scott 
Weitzenkamp
SQA and Release 
Manager
Server Virtualization 
Business Unit
Cisco Systems
 

  
  
  From: Paul [mailto:[EMAIL PROTECTED] 
  Sent: Wednesday, May 24, 2006 11:00 AMTo: Scott 
  Weitzenkamp (sweitzen)Cc: 
  openib-general@openib.orgSubject: Re: [openib-general] Compilation 
  issues on rhel4 u3 ppc64 sysfs.o
  Scott,  Upon further inspection 
  the build.sh and install.sh scripts built 32bit libraries and binaries. If I 
  export CFLAGS (and the like) to include -m64 then the build dies while looking 
  for a 64bit libsysfs. rhel4 u3 does not include a ppc64 sysfsutils, nor have I 
  been able to find an actual 64bit version of it. Is there a workaround for 
  getting things to build actual ppc64 binaries/libraries ? The actual 
  error is:checking for dlsym in -ldl... yeschecking for 
  pthread_mutex_init in -lpthread... yeschecking for sysfs_open_class in 
  -lsysfs... noconfigure: error: sysfs_open_class() not found. libibverbs 
  requires libsysfs. 
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Compilation issues on rhel4 u3 ppc64 sysfs.o

[Paul]

Scott,  Upon further inspection the build.sh and install.sh scripts built 32bit libraries and binaries. If I export CFLAGS (and the like) to include -m64 then the build dies while looking for a 64bit libsysfs. rhel4 u3 does not include a ppc64 sysfsutils, nor have I been able to find an actual 64bit version of it. Is there a workaround for getting things to build actual ppc64 binaries/libraries ? 
The actual error is:checking for dlsym in -ldl... yeschecking for pthread_mutex_init in -lpthread... yeschecking for sysfs_open_class in -lsysfs... noconfigure: error: sysfs_open_class() not found.  libibverbs requires libsysfs.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Sean Hefty]

Michael S. Tsirkin wrote:

It's not completely trivial to reproduce.  I tried loading and
unloading ib_ipoib a few times, then I tried to load ib_ipoib and
unload ib_mthca, I tried pinging in between loading and unloading, and
I didn't see any crashes.



Maybe SM was down.


I've tested loading, ifconfig, unloading a substantial number of times with the 
SM up and down, and I can't reproduce this.  I'll continue to look into this, 
but if you can provide any more information about the test setup, it would be 
helpful.


- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: Re: ipoib use of multicast module on trunk causes kernel oops on 
> 2.6.16
> 
> It's not completely trivial to reproduce.  I tried loading and
> unloading ib_ipoib a few times, then I tried to load ib_ipoib and
> unload ib_mthca, I tried pinging in between loading and unloading, and
> I didn't see any crashes.

Maybe SM was down.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Michael S. Tsirkin]

Quoting r. Sean Hefty <[EMAIL PROTECTED]>:
> Subject: Re: [openib-general] Re: ipoib use of multicast module on trunk 
> causes kernel oops on 2.6.16
> 
> Michael S. Tsirkin wrote:
> >Ali here says he's just bringing the ib0 up, and then unloads the module.
> >Linux is sending broadcasts out all the time, so ...
> 
> Were ipoib and ib_multicast already loaded, and he was just doing an 
> ifconfig? Did he unload both ipoib and ib_multicast, or just ipoib?

modprobe -r on ipoib which I think unloads ipoib and then multicast.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Sean Hefty]

Michael S. Tsirkin wrote:

Ali here says he's just bringing the ib0 up, and then unloads the module.
Linux is sending broadcasts out all the time, so ...


Were ipoib and ib_multicast already loaded, and he was just doing an ifconfig? 
Did he unload both ipoib and ib_multicast, or just ipoib?


- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Sean Hefty]

Michael S. Tsirkin wrote:

What modules were unloaded and being unloaded when this occurred?


I think this can be seen in oops.


I'm guessing that ipoib and ib_multicast were unloaded, and the crash occurred 
unloading ib_sa.  Is this correct?


- Sean


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [IPoIB] executing iperf over IPoIB causes to multicast (IP) packets to be recieved out-of-order

[Roland Dreier]

Michael> This is on back to back. I don't think hardware does this
Michael> - this is ipoib software thing.

It seems to be in the stack above IPoIB.  IPoIB posts sends and
collects receive completions exactly in the order they happen.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Roland Dreier]

Michael> I think this can be seen in oops.

Not which modules were being unloaded...
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Roland Dreier]

It's not completely trivial to reproduce.  I tried loading and
unloading ib_ipoib a few times, then I tried to load ib_ipoib and
unload ib_mthca, I tried pinging in between loading and unloading, and
I didn't see any crashes.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [IPoIB] executing iperf over IPoIB causes to multicast (IP) packets to be recieved out-of-order

[Michael S. Tsirkin]

Quoting r. Hal Rosenstock <[EMAIL PROTECTED]>:
> Subject: Re: [IPoIB] executing iperf over IPoIB causes to multicast (IP) 
> packets to be recieved out-of-order
> 
> On Wed, 2006-05-24 at 11:51, Dotan Barak wrote:
> > Hi.
> > 
> > when executing iperf over IPoIB with multicast IP packets, there are some 
> > packets that being received out of order.
> 
> With IPmc, there is no ordering guarantee. I'm not sure what your
> network configuration is exactly and whether this could be related or
> whether a dropped packet is being reported as out of order.

This is on back to back. I don't think hardware does this - this is
ipoib software thing.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Michael S. Tsirkin]

Quoting r. Sean Hefty <[EMAIL PROTECTED]>:
> Subject: Re: [openib-general] Re: ipoib use of multicast module on trunk 
> causes kernel oops on 2.6.16
> 
> Michael S. Tsirkin wrote:
> >Looks like this. Note its not enough to cancel query, you must wait for it
> >to complete as well.
> >Cancelling outstanding queries properly is one of the harder things in 
> >ipoib
> >multicast code.
> 
> I will look at this.  The multicast module should have handled this.
> 
> What modules were unloaded and being unloaded when this occurred?

I think this can be seen in oops.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Sean Hefty]

Michael S. Tsirkin wrote:

Looks like this. Note its not enough to cancel query, you must wait for it
to complete as well.
Cancelling outstanding queries properly is one of the harder things in ipoib
multicast code.


I will look at this.  The multicast module should have handled this.

What modules were unloaded and being unloaded when this occurred?

- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: ipoib use of multicast module on trunk causes kernel oops on 
> 2.6.16
> 
> What is this test doing exactly?

Ali here says he's just bringing the ib0 up, and then unloads the module.
Linux is sending broadcasts out all the time, so ...

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> It looks like ib_multicast may be unloaded without canceling all of
> its SA queries, and ib_sa ends up calling back into an unloaded
> module.

Looks like this. Note its not enough to cancel query, you must wait for it
to complete as well.
Cancelling outstanding queries properly is one of the harder things in ipoib
multicast code.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: ipoib use of multicast module on trunk causes kernel oops on 
> 2.6.16
> 
> What is this test doing exactly?
> 
> One thing that seems strange is that I don't see ib_multicast here:
> 
>  > Modules linked in: ib_sa ib_uverbs ib_umad ib_mthca ib_mad ib_core
> 
> It looks like ib_multicast may be unloaded without canceling all of
> its SA queries, and ib_sa ends up calling back into an unloaded
> module.

I think its not a specific test - Ali was simply doing things and
unloaded the driver in process.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [IPoIB] executing iperf over IPoIB causes to multicast (IP) packets to be recieved out-of-order

[Hal Rosenstock]

On Wed, 2006-05-24 at 11:51, Dotan Barak wrote:
> Hi.
> 
> when executing iperf over IPoIB with multicast IP packets, there are some 
> packets that being received out of order.

With IPmc, there is no ordering guarantee. I'm not sure what your
network configuration is exactly and whether this could be related or
whether a dropped packet is being reported as out of order.

-- Hal

> Here are my machine attributes:
> *
> Host Architecture : x86_64
> Linux Distribution: Fedora Core release 4 (Stentz)
> Kernel Version: 2.6.11-1.1369_FC4smp
> Memory size   : 4071672 kB
> Driver Version: openib_gen2-20060524-1700 (REV=7460)
> HCA ID(s) : mthca0
> HCA model(s)  : 25208
> FW version(s) : 4.7.600
> Board(s)  : MT_00A0010001
> *
> 
> here are the iperf command line:
> sender command:./iperf -c 10.4.3.86 -u -T 3 -t 400 -i 2 -b 1000M -l 100
> receiver command: ./iperf -s -u -B 10.4.3.86 -i 2
> 
> here is the output of the receiver execution:
> 
> Execute the receiver over IPoIB
> 
> Server listening on UDP port 5001
> Binding to local address 11.4.3.86
> Receiving 1470 byte datagrams
> UDP buffer size:  132 KByte (default)
> 
> [  3] local 11.4.3.86 port 5001 connected with 11.4.3.87 port 4
> [ ID] Interval   Transfer Bandwidth   Jitter   Lost/Total 
> Datagrams
> [  3]  0.0- 2.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms  232/359959 
> (0.064%)
> [  3]  0.0- 2.0 sec  45 datagrams received out-of-order
> [  3]  2.0- 4.0 sec  34.3 MBytes   144 Mbits/sec  0.004 ms0/359391 (0%)
> [  3]  2.0- 4.0 sec  43 datagrams received out-of-order
> [  3]  4.0- 6.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms0/359470 (0%)
> [  3]  4.0- 6.0 sec  31 datagrams received out-of-order
> [  3]  6.0- 8.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms0/359195 (0%)
> [  3]  6.0- 8.0 sec  38 datagrams received out-of-order
> [  3]  8.0-10.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms0/359379 (0%)
> [  3]  8.0-10.0 sec  48 datagrams received out-of-order
> Waiting for server threads to complete. Interrupt again to force quit.
> [  3]  0.0-21.2 sec   364 MBytes   144 Mbits/sec  0.005 ms  232/3813508 
> (0.0061%)
> [  3]  0.0-21.2 sec  595 datagrams received out-of-order
> 
> Execute the receiver over IP
> 
> Server listening on UDP port 5001
> Binding to local address 10.4.3.86
> Receiving 1470 byte datagrams
> UDP buffer size:  132 KByte (default)
> 
> [  3] local 10.4.3.86 port 5001 connected with 10.4.3.87 port 4
> [ ID] Interval   Transfer Bandwidth   Jitter   Lost/Total 
> Datagrams
> [  3]  0.0- 2.0 sec  14.4 MBytes  60.2 Mbits/sec  0.034 ms  355/150976 (0.24%)
> [  3]  2.0- 4.0 sec  14.4 MBytes  60.2 Mbits/sec  0.010 ms0/150599 (0%)
> [  3]  4.0- 6.0 sec  14.4 MBytes  60.2 Mbits/sec  0.012 ms0/150597 (0%)
> [  3]  6.0- 8.0 sec  14.4 MBytes  60.2 Mbits/sec  0.011 ms0/150599 (0%)
> [  3]  8.0-10.0 sec  14.4 MBytes  60.2 Mbits/sec  0.009 ms0/150598 (0%)
> [  3] 10.0-12.0 sec  14.4 MBytes  60.2 Mbits/sec  0.010 ms0/150584 (0%)
> [  3] 12.0-14.0 sec  14.4 MBytes  60.2 Mbits/sec  0.009 ms0/150583 (0%)
> [  3] 14.0-16.0 sec  14.4 MBytes  60.2 Mbits/sec  0.019 ms0/150601 (0%)
> [  3] 16.0-18.0 sec  14.4 MBytes  60.2 Mbits/sec  0.009 ms0/150576 (0%)
> [  3] 18.0-20.0 sec  14.4 MBytes  60.2 Mbits/sec  0.021 ms0/150583 (0%)
> [  3] 20.0-22.0 sec  14.4 MBytes  60.2 Mbits/sec  0.010 ms0/150591 (0%)
> Waiting for server threads to complete. Interrupt again to force quit.
> [  3] 22.0-24.0 sec  14.4 MBytes  60.2 Mbits/sec  0.028 ms0/150582 (0%)
> 
> 
> did anyone notice this issue before?
> 
> thanks
> Dotan
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Roland Dreier]

What is this test doing exactly?

One thing that seems strange is that I don't see ib_multicast here:

 > Modules linked in: ib_sa ib_uverbs ib_umad ib_mthca ib_mad ib_core

It looks like ib_multicast may be unloaded without canceling all of
its SA queries, and ib_sa ends up calling back into an unloaded
module.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [Bug 95] New: Stack seems to reorder multicast entries

[bugzilla-daemon]

http://openib.org/bugzilla/show_bug.cgi?id=95

   Summary: Stack seems to reorder multicast entries
   Product: OpenFabrics Linux
   Version: 1.0rc3
  Platform: All
OS/Version: Other
Status: NEW
  Severity: normal
  Priority: P2
 Component: IPoIB
AssignedTo: [EMAIL PROTECTED]
ReportedBy: [EMAIL PROTECTED]


When running iperf to test multicast performance, we seen many out-of-order 
errors when the receive node is running OFED RC3 or RC4.

Case 1:  IBGD (or gen2) sending to gen2 receive node
  Receive node: 
- iperf -s -u -B 224.0.6.66 -i 2

  Send node:
- Add multicast route to interface
- iperf -c 224.0.6.66 -u -T 3 -t 400 -i 2 -b 100M -l 100

The receive node should report no lost or out of order frames.

Case 2: same sender, using OFED RC3 or RC4 receive node
  Run the same test as above.  You will see many multicast frames report out 
of order.

We have reproduced this problem in two different sites.  In all cases we are 
using SUSE10 as the OS hosting the receive stack (gen2 or OFED).



--- You are receiving this mail because: ---
You are the assignee for the bug, or are watching the assignee.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [IPoIB] executing iperf over IPoIB causes to multicast (IP) packets to be recieved out-of-order

[Dotan Barak]

Hi.

when executing iperf over IPoIB with multicast IP packets, there are some 
packets that being received out of order.


Here are my machine attributes:
*
Host Architecture : x86_64
Linux Distribution: Fedora Core release 4 (Stentz)
Kernel Version: 2.6.11-1.1369_FC4smp
Memory size   : 4071672 kB
Driver Version: openib_gen2-20060524-1700 (REV=7460)
HCA ID(s) : mthca0
HCA model(s)  : 25208
FW version(s) : 4.7.600
Board(s)  : MT_00A0010001
*

here are the iperf command line:
sender command:./iperf -c 10.4.3.86 -u -T 3 -t 400 -i 2 -b 1000M -l 100
receiver command: ./iperf -s -u -B 10.4.3.86 -i 2

here is the output of the receiver execution:

Execute the receiver over IPoIB

Server listening on UDP port 5001
Binding to local address 11.4.3.86
Receiving 1470 byte datagrams
UDP buffer size:  132 KByte (default)

[  3] local 11.4.3.86 port 5001 connected with 11.4.3.87 port 4
[ ID] Interval   Transfer Bandwidth   Jitter   Lost/Total Datagrams
[  3]  0.0- 2.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms  232/359959 (0.064%)
[  3]  0.0- 2.0 sec  45 datagrams received out-of-order
[  3]  2.0- 4.0 sec  34.3 MBytes   144 Mbits/sec  0.004 ms0/359391 (0%)
[  3]  2.0- 4.0 sec  43 datagrams received out-of-order
[  3]  4.0- 6.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms0/359470 (0%)
[  3]  4.0- 6.0 sec  31 datagrams received out-of-order
[  3]  6.0- 8.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms0/359195 (0%)
[  3]  6.0- 8.0 sec  38 datagrams received out-of-order
[  3]  8.0-10.0 sec  34.3 MBytes   144 Mbits/sec  0.003 ms0/359379 (0%)
[  3]  8.0-10.0 sec  48 datagrams received out-of-order
Waiting for server threads to complete. Interrupt again to force quit.
[  3]  0.0-21.2 sec   364 MBytes   144 Mbits/sec  0.005 ms  232/3813508 
(0.0061%)
[  3]  0.0-21.2 sec  595 datagrams received out-of-order

Execute the receiver over IP

Server listening on UDP port 5001
Binding to local address 10.4.3.86
Receiving 1470 byte datagrams
UDP buffer size:  132 KByte (default)

[  3] local 10.4.3.86 port 5001 connected with 10.4.3.87 port 4
[ ID] Interval   Transfer Bandwidth   Jitter   Lost/Total Datagrams
[  3]  0.0- 2.0 sec  14.4 MBytes  60.2 Mbits/sec  0.034 ms  355/150976 (0.24%)
[  3]  2.0- 4.0 sec  14.4 MBytes  60.2 Mbits/sec  0.010 ms0/150599 (0%)
[  3]  4.0- 6.0 sec  14.4 MBytes  60.2 Mbits/sec  0.012 ms0/150597 (0%)
[  3]  6.0- 8.0 sec  14.4 MBytes  60.2 Mbits/sec  0.011 ms0/150599 (0%)
[  3]  8.0-10.0 sec  14.4 MBytes  60.2 Mbits/sec  0.009 ms0/150598 (0%)
[  3] 10.0-12.0 sec  14.4 MBytes  60.2 Mbits/sec  0.010 ms0/150584 (0%)
[  3] 12.0-14.0 sec  14.4 MBytes  60.2 Mbits/sec  0.009 ms0/150583 (0%)
[  3] 14.0-16.0 sec  14.4 MBytes  60.2 Mbits/sec  0.019 ms0/150601 (0%)
[  3] 16.0-18.0 sec  14.4 MBytes  60.2 Mbits/sec  0.009 ms0/150576 (0%)
[  3] 18.0-20.0 sec  14.4 MBytes  60.2 Mbits/sec  0.021 ms0/150583 (0%)
[  3] 20.0-22.0 sec  14.4 MBytes  60.2 Mbits/sec  0.010 ms0/150591 (0%)
Waiting for server threads to complete. Interrupt again to force quit.
[  3] 22.0-24.0 sec  14.4 MBytes  60.2 Mbits/sec  0.028 ms0/150582 (0%)


did anyone notice this issue before?

thanks
Dotan
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] RE: [PATCH] OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix NULL ptr issue

[Hal Rosenstock]

On Wed, 2006-05-24 at 11:33, Eitan Zahavi wrote:
> Hi Hal
> 
> You missed the line that asserts on null p_next_step just before the 
> code you changed .

But isn't CL_ASSERT a debug compile time thing so it's needed when it's
built without debug ?

-- Hal

> .
> 
> EZ
> 
> Eitan Zahavi
> Senior Engineering Director, Software Architect
> Mellanox Technologies LTD
> Tel:+972-4-9097208
> Fax:+972-4-9593245
> P.O. Box 586 Yokneam 20692 ISRAEL
> 
> 
> > -Original Message-
> > From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> > Sent: Wednesday, May 24, 2006 4:34 PM
> > To: openib-general@openib.org
> > Cc: Eitan Zahavi
> > Subject: [PATCH]
> OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix
> > NULL ptr issue
> > 
> > OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_t Fix NULL ptr
> > issue
> > 
> > Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>
> > 
> > Index: opensm/osm_ucast_updn.c
> > ===
> > --- opensm/osm_ucast_updn.c (revision 7435)
> > +++ opensm/osm_ucast_updn.c (working copy)
> > @@ -121,10 +121,12 @@ __updn_create_updn_next_step_t(IN updn_s
> >p_next_step = (updn_next_step_t*) cl_zalloc(sizeof(*p_next_step));
> >CL_ASSERT (p_next_step != NULL);
> > 
> > -  p_next_step->state = state;
> > -  p_next_step->p_sw = p_sw;
> > +  if (p_next_step)
> > +  {
> > +p_next_step->state = state;
> > +p_next_step->p_sw = p_sw;
> > +  }
> >return p_next_step;
> > -
> >  }
> > 
> >
> /**
> > 
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] krping test utility

[Roland Dreier]

Devesh> Hello all, In the krping test utility get_dma_mr is called
Devesh> with access premissions
Devesh> IB_ACCESS_LOCAL_WRITE|IB_ACCESS_REMOTE_WRITE|IB_ACCESS_REMOTE_READ,
Devesh> But the lkey we get from get_dma_mr is similar to reserved
Devesh> lkey with which only Local operations are allowed, but
Devesh> here it seems violating that statement.

No, ib_get_dma_mr() returns an L_Key/R_Key with exactly the
permissions requested.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] RE: [PATCH]OpenSM/osm_sa_mcmember_record.c::OpenSM/osm_sa_mcmember_record.c Return 0when no table can be allocated

[Eitan Zahavi]

Looks right

EZ

> -Original Message-
> From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, May 24, 2006 4:59 PM
> To: openib-general@openib.org
> Cc: Eitan Zahavi
> Subject:
> [PATCH]OpenSM/osm_sa_mcmember_record.c::OpenSM/osm_sa_mcmember_record
> .c Return 0when no table can be allocated
> 
> OpenSM/osm_sa_mcmember_record.c Return 0 when no used MLID table can
be
> allocated
> 
> Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>
> 
> Index: opensm/osm_sa_mcmember_record.c
> ===
> --- opensm/osm_sa_mcmember_record.c   (revision 7441)
> +++ opensm/osm_sa_mcmember_record.c   (working copy)
> @@ -326,6 +326,8 @@ __get_new_mlid(
>/* track all used mlids in the array (by mlid index) */
>used_mlids_array =
>  (uint8_t *)cl_zalloc(sizeof(uint8_t)*max_num_mlids);
> +  if (!used_mlids_array)
> +return 0;
> 
>/* scan all available multicast groups in the DB and fill in the
table */
>while( p_mgrp != (osm_mgrp_t*)cl_qmap_end( &p_subn->mgrp_mlid_tbl )
)
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] RE: [PATCH] OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix NULL ptr issue

[Eitan Zahavi]

Hi Hal

You missed the line that asserts on null p_next_step just before the 
code you changed ...

EZ

Eitan Zahavi
Senior Engineering Director, Software Architect
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL


> -Original Message-
> From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, May 24, 2006 4:34 PM
> To: openib-general@openib.org
> Cc: Eitan Zahavi
> Subject: [PATCH]
OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_tFix
> NULL ptr issue
> 
> OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_t Fix NULL ptr
> issue
> 
> Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>
> 
> Index: opensm/osm_ucast_updn.c
> ===
> --- opensm/osm_ucast_updn.c   (revision 7435)
> +++ opensm/osm_ucast_updn.c   (working copy)
> @@ -121,10 +121,12 @@ __updn_create_updn_next_step_t(IN updn_s
>p_next_step = (updn_next_step_t*) cl_zalloc(sizeof(*p_next_step));
>CL_ASSERT (p_next_step != NULL);
> 
> -  p_next_step->state = state;
> -  p_next_step->p_sw = p_sw;
> +  if (p_next_step)
> +  {
> +p_next_step->state = state;
> +p_next_step->p_sw = p_sw;
> +  }
>return p_next_step;
> -
>  }
> 
>
/**
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Talpey, Thomas]

At 10:52 AM 5/24/2006, Roland Dreier wrote:
>Michael> No idea - the site seems to be down :)
>
>It's working from here -- must be an issue in your network.
>

I saw the same error, but adding "www." to the "openib.org" url fixes it.

Tom.


>Anyway the report is:
>
>*
>Host Architecture : x86_64
>Linux Distribution: Fedora Core release 4 (Stentz)
>Kernel Version: 2.6.11-1.1369_FC4smp
>Memory size   : 4071672 kB
>Driver Version: OFED-1.0-rc5-pre5
>HCA ID(s) : mthca0
>HCA model(s)  : 25208
>FW version(s) : 4.7.600
>Board(s)  : MT_00A0010001
>*
>
>posting a list of multiples of 256 WR to SRQ or QP may be corrupted.
>The WR list that is being posted may be posted to a different QP than
>the QP
>number of the QP handle.
>
>test to reproduce it: qp_test
>daemon:
>qp_test --daemon
>client:
>qp_test --thread=15 --oust=256 --srq CLIENT SR 1 1
> or
>qp_test --thread=15 --oust=256 CLIENT SR 1 1
>___
>openib-general mailing list
>openib-general@openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Michael S. Tsirkin]

Quoting r. Michael S. Tsirkin <[EMAIL PROTECTED]>:
> Subject: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> 
> Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> > Subject: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> > 
> > Michael> BTW, srq will have the same problem in tavor, won't it?
> > Michael> Both kernel and userspace code look quite similiar.
> > 
> > I'll check.
> 
> Yes, just got a report that posting list of 256 entries on SRQ on tavor fails.
> Its the same problem.

I've just tested and posted a fix. Please queue for 2.6.17 and libibverbs 1.0.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH] mthca: fix posting list of 256 to SRQ

[Michael S. Tsirkin]

Here's the same patch for kernel.

---

Fix posting WQE lists of length 246 to SRQ for tavor.
This is really the same bug we fixed for non-shared RQ.

Signed-off-by: Michael S. Tsirkin <[EMAIL PROTECTED]>

Index: linux-2.6.16/drivers/infiniband/hw/mthca/mthca_srq.c
===
--- linux-2.6.16.orig/drivers/infiniband/hw/mthca/mthca_srq.c   2006-05-24 
18:07:24.0 +0300
+++ linux-2.6.16/drivers/infiniband/hw/mthca/mthca_srq.c2006-05-24 
18:07:19.0 +0300
@@ -490,26 +490,7 @@ int mthca_tavor_post_srq_recv(struct ib_
 
first_ind = srq->first_free;
 
-   for (nreq = 0; wr; ++nreq, wr = wr->next) {
-   if (unlikely(nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB)) {
-   nreq = 0;
-
-   doorbell[0] = cpu_to_be32(first_ind << srq->wqe_shift);
-   doorbell[1] = cpu_to_be32(srq->srqn << 8);
-
-   /*
-* Make sure that descriptors are written
-* before doorbell is rung.
-*/
-   wmb();
-
-   mthca_write64(doorbell,
- dev->kar + MTHCA_RECEIVE_DOORBELL,
- 
MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
-
-   first_ind = srq->first_free;
-   }
-
+   for (nreq = 0; wr; wr = wr->next) {
ind = srq->first_free;
 
if (ind < 0) {
@@ -569,6 +550,25 @@ int mthca_tavor_post_srq_recv(struct ib_
 
srq->wrid[ind]  = wr->wr_id;
srq->first_free = next_ind;
+
+   if (unlikely(++nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB)) {
+   nreq = 0;
+
+   doorbell[0] = cpu_to_be32(first_ind << srq->wqe_shift);
+   doorbell[1] = cpu_to_be32(srq->srqn << 8);
+
+   /*
+* Make sure that descriptors are written
+* before doorbell is rung.
+*/
+   wmb();
+
+   mthca_write64(doorbell,
+ dev->kar + MTHCA_RECEIVE_DOORBELL,
+ 
MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock));
+
+   first_ind = srq->first_free;
+   }
}
 
if (likely(nreq)) {

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH] libmthca: fix posting 256 WRs to SRQ in tavor

[Michael S. Tsirkin]

libmthca: fix posting 256 WRs to SRQ in tavor.
Same bug as for regular RQ really, so same fix.

Signed-off-by: Michael S. Tsirkin <[EMAIL PROTECTED]>

Index: openib/src/userspace/libmthca/src/srq.c
===
--- openib/src/userspace/libmthca/src/srq.c (revision 7317)
+++ openib/src/userspace/libmthca/src/srq.c (working copy)
@@ -97,24 +97,7 @@ int mthca_tavor_post_srq_recv(struct ibv
 
first_ind = srq->first_free;
 
-   for (nreq = 0; wr; ++nreq, wr = wr->next) {
-   if (nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB) {
-   nreq = 0;
-
-   doorbell[0] = htonl(first_ind << srq->wqe_shift);
-   doorbell[1] = htonl(srq->srqn << 8);
-
-   /*
-* Make sure that descriptors are written
-* before doorbell is rung.
-*/
-   mb();
-
-   mthca_write64(doorbell, to_mctx(ibsrq->context), 
MTHCA_RECV_DOORBELL);
-
-   first_ind = srq->first_free;
-   }
-
+   for (nreq = 0; wr; wr = wr->next) {
ind = srq->first_free;
 
if (ind < 0) {
@@ -172,6 +155,23 @@ int mthca_tavor_post_srq_recv(struct ibv
 
srq->wrid[ind]  = wr->wr_id;
srq->first_free = next_ind;
+
+   if (++nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB) {
+   nreq = 0;
+
+   doorbell[0] = htonl(first_ind << srq->wqe_shift);
+   doorbell[1] = htonl(srq->srqn << 8);
+
+   /*
+* Make sure that descriptors are written
+* before doorbell is rung.
+*/
+   mb();
+
+   mthca_write64(doorbell, to_mctx(ibsrq->context), 
MTHCA_RECV_DOORBELL);
+
+   first_ind = srq->first_free;
+   }
}
 
if (nreq) {

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Dotan Barak]

On Wednesday 24 May 2006 17:59, Michael S. Tsirkin wrote:
> Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> > Subject: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> > 
> > Michael> No idea - the site seems to be down :)
> > 
> > It's working from here -- must be an issue in your network.
> > 

yes. i opened a bug on this issue (posting 256 WR to SRQ) and the previous 
issue (posting 256 WR to QP).

Dotan
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] which dapl/udapl changes in trunk should be imported into OFED branch? (patch enclosed)

[Jack Morgenstein]

Hi,

Below is a patch file of differences between the OFED dapl library and the 
openib main trunk dapl library.

Please indicate which of the dapl library changes are necessary for the Intel 
MPI to work correctly in OFED.

Thanks!
- Jack



Index: test/dapltest/test/dapl_server.c
===
--- test/dapltest/test/dapl_server.c(.../branches/1.0/src/userspace/dapl)   
(revision 7464)
+++ test/dapltest/test/dapl_server.c(.../trunk/src/userspace/dapl)  
(revision 
7464)
@@ -50,7 +50,7 @@ DT_cs_Server (Params_t * params_ptr)
 Started_server_t*temp_list  = NULL;
 Started_server_t*pre_list   = NULL;
 unsigned char   *buffp  = NULL;
-unsigned char   *module = "DT_cs_Server";
+char*module = "DT_cs_Server";
 
 DAT_DTO_COOKIE dto_cookie;
 DAT_DTO_COMPLETION_EVENT_DATA dto_stat;
@@ -842,7 +842,7 @@ send_control_data (
Per_Server_Data_t   *ps_ptr,
Per_Test_Data_t *pt_ptr)
 {
-unsigned char   *module = "send_control_data";
+char*module = "send_control_data";
 DAT_DTO_COOKIE dto_cookie;
 DAT_DTO_COMPLETION_EVENT_DATA dto_stat;
 
Index: test/dapltest/test/dapl_bpool.c
===
--- test/dapltest/test/dapl_bpool.c (.../branches/1.0/src/userspace/dapl)   
(revision 7464)
+++ test/dapltest/test/dapl_bpool.c (.../trunk/src/userspace/dapl)  
(revision 
7464)
@@ -56,7 +56,7 @@ DT_BpoolAlloc (
   DAT_BOOLEANenable_rdma_write,
   DAT_BOOLEANenable_rdma_read)
 {
-unsigned char   *module = "DT_BpoolAlloc";
+char*module = "DT_BpoolAlloc";
 unsigned char   *alloc_ptr = 0;
 Bpool   *bpool_ptr = 0;
 DAT_COUNTalloc_size;
@@ -254,7 +254,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p
  DT_Tdep_Print_Head *phead,
  Bpool * bpool_ptr)
 {
-unsigned char *module = "DT_Bpool_Destroy";
+char   *module = "DT_Bpool_Destroy";
 bool  rval = true;
 
 if (bpool_ptr)
Index: test/dapltest/test/dapl_test_util.c
===
--- test/dapltest/test/dapl_test_util.c (.../branches/1.0/src/userspace/dapl)   
(revision 7464)
+++ test/dapltest/test/dapl_test_util.c (.../trunk/src/userspace/dapl)  
(revision 7464)
@@ -38,7 +38,7 @@ DT_query (   Per_Test_Data_t *pt_ptr,
DAT_IA_HANDLE   ia_handle,
DAT_EP_HANDLE   ep_handle)
 {
-unsigned char   *module = "DT_query";
+char*module = "DT_query";
 DAT_EVD_HANDLE  async_evd_hdl;  /* not used */
 DAT_EP_PARAMep_params;
 DAT_RETURN  ret;
Index: test/dapltest/test/dapl_client.c
===
--- test/dapltest/test/dapl_client.c(.../branches/1.0/src/userspace/dapl)   
(revision 7464)
+++ test/dapltest/test/dapl_client.c(.../trunk/src/userspace/dapl)  
(revision 
7464)
@@ -57,7 +57,7 @@ DT_cs_Client (Params_t * params_ptr,
 Performance_Cmd_t   *Performance_Cmd= NULL;
 Bpool   *bpool  = NULL;
 DAT_IA_ADDRESS_PTR  server_netaddr  = NULL;
-unsigned char   *module = "DT_cs_Client";
+char*module = "DT_cs_Client";
 unsigned int   did_connect = 0;
 unsigned int   retry_cnt   = 0;
 DAT_DTO_COOKIE dto_cookie;
Index: test/dapltest/test/dapl_transaction_util.c
===
--- test/dapltest/test/dapl_transaction_util.c  
(.../branches/1.0/src/userspace/dapl)   (revision 7464)
+++ test/dapltest/test/dapl_transaction_util.c  (.../trunk/src/userspace/dapl)  
(revision 7464)
@@ -641,7 +641,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p
  */
 bool
 DT_check_params (Per_Test_Data_t  *pt_ptr,
-   unsigned char*module)
+char *module)
 {
 Transaction_Cmd_t * cmd= &pt_ptr->Params.u.Transaction_Cmd;
 unsigned long  num_recvs   = 0U;
Index: test/dapltest/test/dapl_transaction_test.c
===
--- test/dapltest/test/dapl_transaction_test.c  
(.../branches/1.0/src/userspace/dapl)   (revision 7464)
+++ test/dapltest/test/dapl_transaction_test.c  (.../trunk/src/userspace/dapl)  
(revision 7464)
@@ -972,38 +972,42 @@ retry:
}   /* end foreach op */
 
/*
-* Send our memory info (synchronously)
+* Send our memory info. The client performs the first send to comply
+* with the iWARP MPA protocol's "Connection Startup Rules".
 */
DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Sending %s Memory Info\n",

[openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> 
> Michael> No idea - the site seems to be down :)
> 
> It's working from here -- must be an issue in your network.
> 
> Anyway the report is:
> 
> *
> Host Architecture : x86_64
> Linux Distribution: Fedora Core release 4 (Stentz)
> Kernel Version: 2.6.11-1.1369_FC4smp
> Memory size   : 4071672 kB
> Driver Version: OFED-1.0-rc5-pre5
> HCA ID(s) : mthca0
> HCA model(s)  : 25208
> FW version(s) : 4.7.600
> Board(s)  : MT_00A0010001
> *
> 
> posting a list of multiples of 256 WR to SRQ or QP may be corrupted.
> The WR list that is being posted may be posted to a different QP than
> the QP
> number of the QP handle.
> 
> test to reproduce it: qp_test
> daemon:
> qp_test --daemon
> client:
> qp_test --thread=15 --oust=256 --srq CLIENT SR 1 1
>  or
> qp_test --thread=15 --oust=256 CLIENT SR 1 1
> 

Yep, looks like it.
Testing the following patch now, will let you know.

Index: openib/src/userspace/libmthca/src/srq.c
===
--- openib/src/userspace/libmthca/src/srq.c (revision 7317)
+++ openib/src/userspace/libmthca/src/srq.c (working copy)
@@ -97,24 +97,7 @@
 
first_ind = srq->first_free;
 
-   for (nreq = 0; wr; ++nreq, wr = wr->next) {
-   if (nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB) {
-   nreq = 0;
-
-   doorbell[0] = htonl(first_ind << srq->wqe_shift);
-   doorbell[1] = htonl(srq->srqn << 8);
-
-   /*
-* Make sure that descriptors are written
-* before doorbell is rung.
-*/
-   mb();
-
-   mthca_write64(doorbell, to_mctx(ibsrq->context), 
MTHCA_RECV_DOORBELL);
-
-   first_ind = srq->first_free;
-   }
-
+   for (nreq = 0; wr; wr = wr->next) {
ind = srq->first_free;
 
if (ind < 0) {
@@ -172,6 +155,23 @@
 
srq->wrid[ind]  = wr->wr_id;
srq->first_free = next_ind;
+
+   if (++nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB) {
+   nreq = 0;
+
+   doorbell[0] = htonl(first_ind << srq->wqe_shift);
+   doorbell[1] = htonl(srq->srqn << 8);
+
+   /*
+* Make sure that descriptors are written
+* before doorbell is rung.
+*/
+   mb();
+
+   mthca_write64(doorbell, to_mctx(ibsrq->context), 
MTHCA_RECV_DOORBELL);
+
+   first_ind = srq->first_free;
+   }
}
 
if (nreq) {

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] krping test utility

[Steve Wise]

On Wed, 2006-05-24 at 09:59 +0530, Devesh Sharma wrote:
> Hello all,
> 
> In the krping test utility get_dma_mr is called with access
> premissions IB_ACCESS_LOCAL_WRITE|IB_ACCESS_REMOTE_WRITE|
> IB_ACCESS_REMOTE_READ, But the lkey we get from get_dma_mr is similar
> to reserved lkey with which only Local operations are allowed, but
> here it seems violating that statement. 

What exactly do you think is incorrect? I don't understand your
question.

Steve.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Roland Dreier]

Michael> No idea - the site seems to be down :)

It's working from here -- must be an issue in your network.

Anyway the report is:

*
Host Architecture : x86_64
Linux Distribution: Fedora Core release 4 (Stentz)
Kernel Version: 2.6.11-1.1369_FC4smp
Memory size   : 4071672 kB
Driver Version: OFED-1.0-rc5-pre5
HCA ID(s) : mthca0
HCA model(s)  : 25208
FW version(s) : 4.7.600
Board(s)  : MT_00A0010001
*

posting a list of multiples of 256 WR to SRQ or QP may be corrupted.
The WR list that is being posted may be posted to a different QP than
the QP
number of the QP handle.

test to reproduce it: qp_test
daemon:
qp_test --daemon
client:
qp_test --thread=15 --oust=256 --srq CLIENT SR 1 1
 or
qp_test --thread=15 --oust=256 CLIENT SR 1 1
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> 
> BTW, any comment on this:
> 
> http://openib.org/bugzilla/show_bug.cgi?id=94
> 
> Is this the same bug?

No idea - the site seems to be down :)

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH 1 of 10] ipath - fix spinlock recursion bug

[Bryan O'Sullivan]

On Tue, 2006-05-23 at 14:31 -0700, Roland Dreier wrote:

> It's probably OK as long as it's pure code motion.

I'll recheck and make sure that it is before I send you anything.
Thanks.

> What I want to
> avoid is the giant combo patch that does several different things,
> because if someone later bisects a regression back to that patch,
> we're kind of screwed...

Yeah, I've been doing some educating lately about that :-)

http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> 
> Michael> BTW, srq will have the same problem in tavor, won't it?
> Michael> Both kernel and userspace code look quite similiar.
> 
> I'll check.

Yes, just got a report that posting list of 256 entries on SRQ on tavor fails.
Its the same problem.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Compilation issues on rhel4 u3 ppc64 sysfs.o

[Paul]

Scott,  That worked. Thanks for the info.Regards.On 5/23/06, Scott Weitzenkamp (sweitzen) <[EMAIL PROTECTED]
> wrote:




No clue, I know if you grab OFED 1.0 rc4 tarball and run 
install.sh, it should work.
 
Scott 
Weitzenkamp
SQA and Release 
Manager
Server Virtualization 
Business Unit
Cisco Systems
 

  
  
  From: Paul [mailto:[EMAIL PROTECTED]] 
  Sent: Tuesday, May 23, 2006 12:42 PMTo: Scott 
  Weitzenkamp (sweitzen)Cc: 
  openib-general@openib.orgSubject: Re: [openib-general] Compilation 
  issues on rhel4 u3 ppc64 sysfs.o
  Scott,   Thanks for the confirmation and the quick 
  reply. Any ideas as to what might be causing the error in question 
  ?Regards.
  On 5/23/06, Scott 
  Weitzenkamp (sweitzen) <[EMAIL PROTECTED]> wrote:
  


OFED 1.0 
rc4 does compile and run on RHEL4 U3 ppc64.
 
Scott 
Weitzenkamp
SQA and Release 
Manager
Server Virtualization 
Business Unit
Cisco 
Systems
 

  
  
  From: [EMAIL PROTECTED] [mailto:
[EMAIL PROTECTED]] On Behalf Of 
  Paul LundinSent: Tuesday, May 23, 2006 12:34 
  PMTo: openib-general@openib.orgSubject: 
  [openib-general] Compilation issues on rhel4 u3 ppc64 
  sysfs.o

Hi All, I just started working with 
openIB in the past week. I am having an issue getting the kernel modules to 
compile with the stock rhel4 u3 kernel. I have applied the patches found at 
https://openib.org/svn/gen2/branches/backport/2.6.9_U3/ 
and followed the instructions from https://openib.org/tiki/tiki-index.php?page=Installation+Cheat+Sheet 
but I have been getting the following error:LD 
/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/built-in.oLD 
/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/built-in.oCC 
[M] /usr/src/kernels/2.6.9- 
34.EL-ppc64/drivers/infiniband/core/index.oCC [M] 
/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/addr.oCC [M] 
/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/cm.o/usr/src/kernels/2.6.9- 
34.EL-ppc64/drivers/infiniband/core/cm.c: In function 
`ib_cm_cleanup':/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/cm.c:3367: 
warning: implicit declaration of function `idr_destroy'CC [M] 
/usr/src/kernels/2.6.9- 34.EL-ppc64/drivers/infiniband/core/packer.oCC 
[M] 
/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/ud_header.oCC 
[M] /usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/verbs.oCC 
[M] /usr/src/kernels/2.6.9- 
34.EL-ppc64/drivers/infiniband/core/sysfs.o/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/sysfs.c:693: 
error: unknown field `uevent' specified in 
initializer/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/sysfs.c:693: 
warning: initialization from incompatible pointer type make[2]: *** 
[/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core/sysfs.o] Error 
1make[1]: *** 
[/usr/src/kernels/2.6.9-34.EL-ppc64/drivers/infiniband/core] Error 
2make: *** [_module_/usr/src/kernels/2.6.9- 
34.EL-ppc64/drivers/infiniband] Error 2make: Leaving directory 
`/usr/src/kernels/2.6.9-34.EL-ppc64'Any help would be appreciated. 
As noted this is on a ppc64 machine. The rhel4 u3 install does *NOT* 
configure openIB by default like it does on intel architectures. I was 
wondering if openIB has been tested at all on ppc64 and if this was even 
possible at this point. Regards.Paul



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Roland Dreier]

BTW, any comment on this:

http://openib.org/bugzilla/show_bug.cgi?id=94

Is this the same bug?

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH] IB/iser remove usage of ISCSI_VERSION_STR

[Roland Dreier]

Thanks, applied and pushed out.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH 1/2] mthca support for max_map_per_fmr device attribute

[Roland Dreier]

Or> OK, fair enough, i will need at least some kickoff
Or> helping... can you comment if the above calculation is indeed
Or> broken under memfree? if yes, is it broken under both
Or> Arbel/Sinai? where should i look into the driver or i should
Or> look in the PRM?

Sorry, I don't have time to look back at all the details now.  You
will have to read the relevant PRMs and code to figure it out.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Roland Dreier]

Michael> BTW, srq will have the same problem in tavor, won't it?
Michael> Both kernel and userspace code look quite similiar.

I'll check.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: Fwd: [Bug 91] sizeof(srp_indirect_buf) wrongon 64-bit platforms

[Roland Dreier]

Arne> Do you (or anyone else?) want (me) to forward the fix to the
Arne> stable team anyway?

I don't see any reason, given that it has no practical impact right
now.

  - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Plans for libibverbs 1.1

[Roland Dreier]

Gleb> What about madvice patch? Is it scheduled to go into 1.1?

Yes, in some form.

 - R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH] IB/iser remove usage of ISCSI_VERSION_STR

[Or Gerlitz]

Concatenate iSER's DRV_VER instead of ISCSI_VERSION_STR to the name field of
the scsi host template.

Signed-off-by: Or Gerlitz <[EMAIL PROTECTED]>

--- b/drivers/infiniband/ulp/iser/iscsi_iser.c  2006-05-24 16:41:56.936748675 
+0300
+++ linux-2.6.16/drivers/infiniband/ulp/iser/iscsi_iser.c   2006-05-24 
16:38:07.011802445 +0300
@@ -679,8 +679,7 @@ iscsi_iser_ep_disconnect(__u64 ep_handle
 }
 
 static struct scsi_host_template iscsi_iser_sht = {
-   .name   = "iSCSI Initiator over iSER, v."
- ISCSI_VERSION_STR,
+   .name   = "iSCSI Initiator over iSER, v." DRV_VER,
.queuecommand   = iscsi_queuecommand,
.can_queue  = ISCSI_XMIT_CMDS_MAX - 1,
.sg_tablesize   = ISCSI_ISER_SG_TABLESIZE,
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH] OpenSM/osm_sa_mcmember_record.c::OpenSM/osm_sa_mcmember_record.c Return 0 when no table can be allocated

[Hal Rosenstock]

OpenSM/osm_sa_mcmember_record.c Return 0 when no used MLID table can be
allocated

Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>

Index: opensm/osm_sa_mcmember_record.c
===
--- opensm/osm_sa_mcmember_record.c (revision 7441)
+++ opensm/osm_sa_mcmember_record.c (working copy)
@@ -326,6 +326,8 @@ __get_new_mlid(
   /* track all used mlids in the array (by mlid index) */
   used_mlids_array =
 (uint8_t *)cl_zalloc(sizeof(uint8_t)*max_num_mlids);
+  if (!used_mlids_array)
+return 0;
  
   /* scan all available multicast groups in the DB and fill in the table */
   while( p_mgrp != (osm_mgrp_t*)cl_qmap_end( &p_subn->mgrp_mlid_tbl ) )



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [PATCH] OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_t Fix NULL ptr issue

[Hal Rosenstock]

OpenSM/osm_ucast_updn.c::__updn_create_updn_next_step_t Fix NULL ptr
issue

Signed-off-by: Hal Rosenstock <[EMAIL PROTECTED]>

Index: opensm/osm_ucast_updn.c
===
--- opensm/osm_ucast_updn.c (revision 7435)
+++ opensm/osm_ucast_updn.c (working copy)
@@ -121,10 +121,12 @@ __updn_create_updn_next_step_t(IN updn_s
   p_next_step = (updn_next_step_t*) cl_zalloc(sizeof(*p_next_step));
   CL_ASSERT (p_next_step != NULL);
 
-  p_next_step->state = state;
-  p_next_step->p_sw = p_sw;
+  if (p_next_step)
+  {
+p_next_step->state = state;
+p_next_step->p_sw = p_sw;
+  }
   return p_next_step;
-
 }
 
 /**



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] ipoib use of multicast module on trunk causes kernel oops on 2.6.16

[Michael S. Tsirkin]

Hi!
Looks like moving ipoib to the new multicast module caused some instability.
See below.

- Forwarded message from Ali Ayoub <[EMAIL PROTECTED]> -

> -Original Message-
> From: Michael S. Tsirkin
> To: Ali Ayoub
> 
> Quoting r. Ali Ayoub <[EMAIL PROTECTED]>:
> > Subject: [gen2 trunk] kernel oops on 2.6.16
> >
> > The last trunk build causes kernel oops on 2.6.16 while restarting the 
> > driver.
> >
> > (the previous build -rev 7422- works fine)
> >
> >
> >
> > May 24 16:00:40 sw037 kernel: Unable to handle kernel paging request at
> 8804bb17 RIP:
> > May 24 16:00:40 sw037 kernel: []
> > May 24 16:00:40 sw037 kernel: PGD 103027 PUD 105027 PMD 17f69d067 PTE 0
> > May 24 16:00:40 sw037 kernel: Oops:  [1] SMP
> > May 24 16:00:40 sw037 kernel: CPU 1
> > May 24 16:00:40 sw037 kernel: Modules linked in: ib_sa ib_uverbs ib_umad
> ib_mthca ib_mad ib_core
> > May 24 16:00:40 sw037 kernel: Pid: 4355, comm: modprobe Not tainted
> 2.6.16 #9
> > May 24 16:00:40 sw037 kernel: RIP: 0010:[]
> []
> > May 24 16:00:40 sw037 kernel: RSP: :810179ca7d40  EFLAGS:
> 00010246
> > May 24 16:00:40 sw037 kernel: RAX: 0005 RBX:
> 810179ca7df0 RCX: 88045b49
> > May 24 16:00:40 sw037 kernel: RDX: 81017c0f1760 RSI:
>  RDI: fffc
> > May 24 16:00:40 sw037 kernel: RBP: 810179ca7da8 R08:
> 81017a83fb68 R09: 81017c54cc40
> > May 24 16:00:40 sw037 kernel: R10: 81017c8c4848 R11:
> 0020 R12: 81017e21a3b8
> > May 24 16:00:40 sw037 kernel: R13: fffc R14:
>  R15: 0080
> > May 24 16:00:40 sw037 kernel: FS:  ()
> GS:81017fc772a8() knlGS:
> > May 24 16:00:40 sw037 ifdown: Interface not available and no
> configuration found.
> > May 24 16:00:40 sw037 kernel: CS:  0010 DS:  ES:  CR0:
> 8005003b
> > May 24 16:00:40 sw037 kernel: CR2: 8804bb17 CR3:
> 00017af3f000 CR4: 06e0
> > May 24 16:00:40 sw037 kernel: Process modprobe (pid: 4355, threadinfo
> 810179ca6000, task 81017f9209a0)
> > May 24 16:00:40 sw037 kernel: Stack: 88045b95 81017c54cc38
> 81017c54c000 810179ca7d98
> > May 24 16:00:40 sw037 kernel:80173b26 810179ca7d98
> 801736ad 
> > May 24 16:00:40 sw037 kernel:81017fc00040 81017c8c4840
> > May 24 16:00:40 sw037 kernel: Call Trace:
> {:ib_sa:ib_sa_mcmember_rec_callback+76}
> > May 24 16:00:40 sw037 kernel:
> {cache_free_debugcheck+568}
> {poison_obj+58}
> > May 24 16:00:40 sw037 kernel:
> {:ib_sa:send_handler+80}
> {:ib_mad:ib_unregister_mad_agent+359}
> > May 24 16:00:40 sw037 kernel:
> {:ib_sa:free_sm_ah+0}
> {:ib_sa:ib_sa_remove_one+80}
> > May 24 16:00:40 sw037 kernel:
> {:ib_core:ib_unregister_client+72}
> > May 24 16:00:40 sw037 kernel:
> {:ib_sa:ib_sa_cleanup+16}
> {sys_delete_module+513}
> > May 24 16:00:40 sw037 kernel:{__up_write+293}
> {system_call+126}
> > May 24 16:00:40 sw037 kernel:
> > May 24 16:00:40 sw037 kernel: Code:  Bad RIP value.
> > May 24 16:00:40 sw037 kernel: RIP [] RSP
> 
> > May 24 16:00:40 sw037 kernel: CR2: 8804bb17
> > May 24 16:00:40 sw037 kernel:  BUG: modprobe/4355, lock held at task
> exit time!
> > May 24 16:00:40 sw037 kernel:  [8800c9e0] {device_mutex}
> > May 24 16:00:40 sw037 kernel: .. held by:  modprobe: 4355
> [81017f9209a0, 118]
> > May 24 16:00:40 sw037 kernel: ... acquired at:
> ib_unregister_client+0x1a/0x108 [ib_core]
> > May 24 16:00:42 sw037 ifdown: Interface not available and no
> configuration found.
> > May 24 16:00:42 sw037 ifdown: Interface not available and no
> >
> >
> 
> Must be ipoib multicast change by Sean. Please try reverting r7401.

Right, reverting to 7401 solved the problem.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: Fwd: [Bug 91] sizeof(srp_indirect_buf) wrongon 64-bit platforms

[Arne Redlich]

Roland Dreier <[EMAIL PROTECTED]> writes:

> >>> I'm afraid it *does* have an effect, unfortunately.
>
> Hmm, go ahead and forward the fix from 2.6.17 to the stable team for
> kernel 2.6.16 if this bug affects your target.
>
> Thanks,
>   Roland

It doesn't affect our target during regular operation, as the indirect 
descriptor table is always included entirely in the IU, so RDMA reading it 
isn't necessary. I just stumbled accross this issue while enforcing an RDMA 
read of the descriptor table for testing purposes - so having the fix queued 
for 2.6.17 is sufficient for us.

Do you (or anyone else?) want (me) to forward the fix to the stable team anyway?

Arne

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH 1/2] mthca support for max_map_per_fmr device attribute

[Michael S. Tsirkin]

Quoting r. Or Gerlitz <[EMAIL PROTECTED]>:
> Subject: Re: [PATCH 1/2] mthca support for max_map_per_fmr device attribute
> 
> Roland Dreier wrote:
> >Or> When "requests" come fast enough, there's a window in time
> >Or> when there's an unmapping of N FMRs running at batch, but out
> >Or> of the remaining N FMRs some are already dirty and can't be
> >Or> used to serve a credit. So the app fails temporally... So,
> >Or> setting the watermark to 0.5N might solve this, but since
> >Or> enlarging the number of remaps is trivial, i'd like to do it
> >Or> first.

You will still be limited wrt the number of remaps in memfree architecture.
And the memory registration code is hard enough to stress test.
So I have to say setting the watermark at 0.5N makes much more sense to me.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] NFS/RDMA for Linux: client and server update release 5

[Talpey, Thomas]

OBTW, I just noticed that your server printed the message:

nfsd: request from insecure port 140.221.134.220, port=32768!

This means the /mnt/rdma export isn't configured with "insecure",
and causes the server to close the connection. Because the IB CM
does not allow the client to use so-called secure ports (< 1024), you
need to set this flag on any RDMA exports, this is mentioned in our
README.

The jury is out on whether it's worth implementing the source port
emulation in the IB CM. The problem is that to do so requires the
CM to interface with the local IP port space, or manage one of its
own. So for now, NFS/RDMA just recommends using the exports
flag. Frankly, it provides no additional security, and is misnamed...

Tom.

At 07:25 AM 5/24/2006, Talpey, Thomas wrote:
>[Cutting down the reply list to more relevant parties...]
>
>It's hard to say what is crashing, but I suspect the CM code, due
>to the process context being ib_cm. Is there some reason you're
>not getting symbols in the stack trace? If you could feed this oops
>text to ksymoops it will give us more information.
>
>In any case, it appears the connection is succeeding at the server,
>but the client RPC code isn't being signalled that it has done so.
>Perhaps this is due to a lost reply, but the NFS code hasn't actually
>started to do anything. So, I would look for IB-level issues. Is the
>client running the current OpenFabrics svn top-of-tree?
>
>Let's take this offline to diagnose, unless someone has an idea why
>the CM would be failing. The ksymoops analysis would help.
>
>Tom.
>
>
>
>At 07:19 PM 5/23/2006, helen chen wrote:
>>Hi Tom,
>>
>>I have downloaded your release 5 of the NFS/RDMA and am having trouble
>>mounting the rdma nfs, the 
>>"./nfsrdmamount -o rdma on16-ib:/mnt/rdma /mnt/rdma" command never
>>returned. and the dmesg for client and server are:
>>
>>-- demsg from client -
>>RPCRDMA Module Init, register RPC RDMA transport
>>Defaults:
>>MaxRequests 50
>>MaxInlineRead 1024
>>MaxInlineWrite 1024
>>Padding 0
>>Memreg 5
>>RPC: Registered rdma transport module.
>>RPC: Registered rdma transport module.
>>RPC:   xprt_setup_rdma: 140.221.134.221:2049
>>nfs: server on16-ib not responding, timed out
>>Unable to handle kernel NULL pointer dereference at 
>>RIP:
>>[<>]
>>PGD a9f2b067 PUD a8ca2067 PMD 0
>>Oops: 0010 [1] PREEMPT SMP
>>CPU 1
>>Modules linked in: xprtrdma ib_srp iscsi_tcp scsi_transport_iscsi
>>scsi_mod
>>Pid: 346, comm: ib_cm/1 Not tainted 2.6.16.16 #4
>>RIP: 0010:[<>] [<>]
>>RSP: 0018:8100af5a1c30  EFLAGS: 00010246
>>RAX: 8100aeff2400 RBX: 8100aeff2400 RCX: 8100afc9e458
>>RDX:  RSI: 8100af5a1d48 RDI: 8100aeff2440
>>RBP: 8100aeff2440 R08:  R09: 
>>R10: 0003 R11:  R12: 8100aeff2500
>>R13: ff99 R14: 8100af5a1d48 R15: 8036c72c
>>FS:  00505ae0() GS:810003ce25c0()
>>knlGS:
>>CS:  0010 DS: 0018 ES: 0018 CR0: 8005003b
>>CR2:  CR3: ad587000 CR4: 06a0
>>Process ib_cm/1 (pid: 346, threadinfo 8100af5a, task
>>8100afea8100)
>>Stack: 8802a331 8100aeff2500 0001
>>8100aeff2440
>>   804011fd  8802a343
>>8100afdd6100
>>   80364ee4 0100
>>Call Trace: [] []
>>   [] [] []
>>   [] [] []
>>   [] [] []
>>   [] [] []
>>   [] [] []
>>   [] [] []
>>   [] [] []
>>   [] [] []
>>   []
>>
>>Code:  Bad RIP value.
>>RIP [<>] RSP 
>>CR2: 
>>
>>--dmesg from server --
>>nfsd: request from insecure port 140.221.134.220, port=32768!
>>svc_rdma_recvfrom: transport 81007e8f2800 is closing
>>svc_rdma_put: Destroying transport 81007e8f2800,
>>cm_id=81007e945200, sk_flags=154, sk_inuse=0
>>
>>Did I forget to configure necessary components into my kernel?
>>
>>Thanks,
>>Helen
>>
>>On Mon, 2006-05-22 at 13:25, Talpey, Thomas wrote:
>>> Network Appliance is pleased to announce release 5 of the NFS/RDMA
>>> client and server for Linux 2.6.16.16. This update to the April 19 release
>>> adds improved server parallel performance and fixes various issues. This
>>> code supports both Infiniband and iWARP transports.
>>> 
>>> 
>>> 
>>> 
>>id=191427>
>>> 
>>> Comments and feedback welcome. We're especially interested in
>>> successful test reports! Thanks.
>>> 
>>> Tom Talpey, for the various NFS/RDMA projects.
>>> 
>>> ___
>>> openib-general mailing list
>>> openib-general@openib.org
>>> http://openib.org/mailman/listinfo/openib-general
>>> 
>>> To unsubscribe, please visit 
>>http://openib.org/mailman

Re: [openib-general] Plans for libibverbs 1.1

[Gleb Natapov]

On Tue, May 23, 2006 at 09:57:03AM -0700, Roland Dreier wrote:
> I'm planning on branching the libibverbs tree so that I can open a 1.1
> development branch where ABI/API stability is not a requirement.  My
> current plan is to copy the current src/userspace/libibverbs tree in
> svn to src/userspace/libibverbs-1.0.  The libibverbs-1.0 tree would be
> used for stable maintainence (only changes that preserve ABI and API
> stability will be accepted), and the libibverbs tree would be used for
> new development.
> 
> I would expect a libibverbs 1.1-pre1 snapshot release shortly, with
> the goal of a full stable libibverbs 1.1 release in 3 or 4 months.
> 
> So far I have the changes below queued up for the new libibverbs 1.1
> tree.  The main changes are getting rid of libsysfs use, and removing
> the deprecated ib_XXX symbols.
> 
What about madvice patch? Is it scheduled to go into 1.1?

--
Gleb.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] NFS/RDMA for Linux: client and server update release 5

[Talpey, Thomas]

[Cutting down the reply list to more relevant parties...]

It's hard to say what is crashing, but I suspect the CM code, due
to the process context being ib_cm. Is there some reason you're
not getting symbols in the stack trace? If you could feed this oops
text to ksymoops it will give us more information.

In any case, it appears the connection is succeeding at the server,
but the client RPC code isn't being signalled that it has done so.
Perhaps this is due to a lost reply, but the NFS code hasn't actually
started to do anything. So, I would look for IB-level issues. Is the
client running the current OpenFabrics svn top-of-tree?

Let's take this offline to diagnose, unless someone has an idea why
the CM would be failing. The ksymoops analysis would help.

Tom.



At 07:19 PM 5/23/2006, helen chen wrote:
>Hi Tom,
>
>I have downloaded your release 5 of the NFS/RDMA and am having trouble
>mounting the rdma nfs, the 
>"./nfsrdmamount -o rdma on16-ib:/mnt/rdma /mnt/rdma" command never
>returned. and the dmesg for client and server are:
>
>-- demsg from client -
>RPCRDMA Module Init, register RPC RDMA transport
>Defaults:
>MaxRequests 50
>MaxInlineRead 1024
>MaxInlineWrite 1024
>Padding 0
>Memreg 5
>RPC: Registered rdma transport module.
>RPC: Registered rdma transport module.
>RPC:   xprt_setup_rdma: 140.221.134.221:2049
>nfs: server on16-ib not responding, timed out
>Unable to handle kernel NULL pointer dereference at 
>RIP:
>[<>]
>PGD a9f2b067 PUD a8ca2067 PMD 0
>Oops: 0010 [1] PREEMPT SMP
>CPU 1
>Modules linked in: xprtrdma ib_srp iscsi_tcp scsi_transport_iscsi
>scsi_mod
>Pid: 346, comm: ib_cm/1 Not tainted 2.6.16.16 #4
>RIP: 0010:[<>] [<>]
>RSP: 0018:8100af5a1c30  EFLAGS: 00010246
>RAX: 8100aeff2400 RBX: 8100aeff2400 RCX: 8100afc9e458
>RDX:  RSI: 8100af5a1d48 RDI: 8100aeff2440
>RBP: 8100aeff2440 R08:  R09: 
>R10: 0003 R11:  R12: 8100aeff2500
>R13: ff99 R14: 8100af5a1d48 R15: 8036c72c
>FS:  00505ae0() GS:810003ce25c0()
>knlGS:
>CS:  0010 DS: 0018 ES: 0018 CR0: 8005003b
>CR2:  CR3: ad587000 CR4: 06a0
>Process ib_cm/1 (pid: 346, threadinfo 8100af5a, task
>8100afea8100)
>Stack: 8802a331 8100aeff2500 0001
>8100aeff2440
>   804011fd  8802a343
>8100afdd6100
>   80364ee4 0100
>Call Trace: [] []
>   [] [] []
>   [] [] []
>   [] [] []
>   [] [] []
>   [] [] []
>   [] [] []
>   [] [] []
>   [] [] []
>   []
>
>Code:  Bad RIP value.
>RIP [<>] RSP 
>CR2: 
>
>--dmesg from server --
>nfsd: request from insecure port 140.221.134.220, port=32768!
>svc_rdma_recvfrom: transport 81007e8f2800 is closing
>svc_rdma_put: Destroying transport 81007e8f2800,
>cm_id=81007e945200, sk_flags=154, sk_inuse=0
>
>Did I forget to configure necessary components into my kernel?
>
>Thanks,
>Helen
>
>On Mon, 2006-05-22 at 13:25, Talpey, Thomas wrote:
>> Network Appliance is pleased to announce release 5 of the NFS/RDMA
>> client and server for Linux 2.6.16.16. This update to the April 19 release
>> adds improved server parallel performance and fixes various issues. This
>> code supports both Infiniband and iWARP transports.
>> 
>> 
>> 
>> 
>
>> 
>> Comments and feedback welcome. We're especially interested in
>> successful test reports! Thanks.
>> 
>> Tom Talpey, for the various NFS/RDMA projects.
>> 
>> ___
>> openib-general mailing list
>> openib-general@openib.org
>> http://openib.org/mailman/listinfo/openib-general
>> 
>> To unsubscribe, please visit 
>http://openib.org/mailman/listinfo/openib-general
>> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH] opensm: fix byte ordering in ib_member_get/set_sl_flow_hop()

[Hal Rosenstock]

On Mon, 2006-05-22 at 11:10, Sasha Khapyorsky wrote:
> This fixes net/host byte ordering in ib_member_get/set_sl_flow_hop()
> functions.
> 
> Signed-off-by: Sasha Khapyorsky <[EMAIL PROTECTED]>

Thanks. Applied to trunk only.

-- Hal

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Michael S. Tsirkin]

Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> 
> Thanks, applied.
> 

BTW, srq will have the same problem in tavor, won't it?
Both kernel and userspace code look quite similiar.

-- 
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] different send and receive CQs

[Or Gerlitz]

Eric Barton wrote:


BTW, Or Gerlitz reckons there is a performance penalty for using multiple
CQs.  The reason I'm interested in separate CQs is to avoid CQ overflow as I
add connections.  On other stacks (Voltaire, Cisco, Silverstorm) I size a
single CQ large enough for 'n' connections (i.e. cluster size - 1), but that
means I have to refuse connections when 'n' have been established.  


Talking about CQ wrt adding connections here's my take: the max CQ size 
(reported by struct ib_device_attr->max_cqe of ib_query_device) is 128K

(this is on memfull HCA, you would need to check the memfree HCA). So
when the number of RX credits per connection is low it allows for many-K 
connections to use the same CQ (eg eight credits allow for 120K 
connections which is much more then the ~48K limit on LMC0 IB clusters 
size...). If you need more connections (QPs) than a single CQ can carry, 
 create another one and attach it to new QPs. The CQ callback gets the 
CQ pointer as its first element, so you need not change you 
polling/arming logic.


Also note that a 128K entries CQ consumes about 4MB (Roland can you 
confirm?) of the HCA attached memory (or host memory for memfree),

so per my taste, coding apps for the cq_resize is kind of over doing.

> In one stack it also stressed vmalloc() and prevented me from using a 
> single whole-memory mapping.


Is there a chance that you are confusing CQs with QPs? Before 
implementing FMR scheme for the voltaire NAL, you were creating a giant 
QP for which the gen1 driver was allocating the host side memory using 
vmalloc, so it could not allocate more then ~300 QPs.


With the mthca driver you should be able to allocate a CQ with the 
maximum allowed size (and if not it will be fixed...)


Or.




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH 1/2] mthca support for max_map_per_fmr device attribute

[Or Gerlitz]

Roland Dreier wrote:

Or> Also, if the patch makes sense and the memfree issue is
Or> resolved, i'd like to change the name of the device attribute
Or> from max_map_per_fmr to max_remaps_per_fmr, i can resend this
Or> patch series with this fix.

The patch makes sense, although of course you need to make sure you
understand and handle the mem-free case as well if you want it
applied.  


+ /* FMR can be remapped 2^B - 1 times where B < 32 is the number of
+ *  bits which are not used for MPT addressing */
+ max_map_per_fmr = (1 << (32 - long_log2(mdev->limits.num_mpts))) - 1;

OK, fair enough, i will need at least some kickoff helping... can you 
comment if the above calculation is indeed broken under memfree? if yes, 
is it broken under both Arbel/Sinai? where should i look into the driver 
or i should look in the PRM?


> I'm not sure changing to max_remaps_per_fmr is really
> clearer, since the value counts the first mapping of the FMR (which is
> not a remapping).  But I guess I could be convinced if more people
> think it's clearer.

remaps means all the maps except the first one, for this stands the "-1" 
in the calculation above. Let me know what makes sense more to you (or 
others if they choose to respond) calling it max_maps_per_fmr (which 
counts also the first map) or calling it max_remaps_per_fmr (so it does 
not count the first map).


Or.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH 1/2] mthca support for max_map_per_fmr device attribute

[Or Gerlitz]

Roland Dreier wrote:

Or> When "requests" come fast enough, there's a window in time
Or> when there's an unmapping of N FMRs running at batch, but out
Or> of the remaining N FMRs some are already dirty and can't be
Or> used to serve a credit. So the app fails temporally... So,
Or> setting the watermark to 0.5N might solve this, but since
Or> enlarging the number of remaps is trivial, i'd like to do it
Or> first.

I don't quite understand how increasing the max remap count really
helps you that much.  Increasing it would just make this failure less
frequent, but it would still occur, right?


Increasing the max remap count --really-- helps me b/c it takes >> time 
for free FMRs to become dirty, and this window is enough for the batch 
unmap to complete, so practically there are always free N FMRs with the 
scheme suggested above (allocate 2N with watermark at N, publish N to 
upper layers)


Indeed, the code can not --count-- on that, so when iSER get -EAGAIN 
return code from ib_fmr_pool_map_phys() it would try later. The current 
retry scheme is just trying over and over (you can not see it easily 
from the iser code, its related to the interaction with libiscsi), i 
have on my TODO an item to register a flush callback with the pool, 
suspend the iser TX flow when getting EAGAIN from fmr_map and resume TX 
from the flush callback.


Or.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH 2/2] port the fmr pool to use the max_map_per_fmr device attribute

[Or Gerlitz]

Roland Dreier wrote:

 > + struct ib_device_attr device_attr;

How big is struct ib_device_attr?  I've usually been reluctant to put
this type of thing on the stack to avoid bloating stack usage too
much.


Oh, its 168 bytes on x86_64, I will fix it to be allocated dynamically.

Or.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] different send and receive CQs

[Eric Barton]

> Are you not seeing any completions when you poll the CQ, or are you
> not getting completion events?  Some things to check would be that you
> are requesting notification on all the CQs you want events on,

Doh! I must have been having a bit of a Homer moment...

BTW, Or Gerlitz reckons there is a performance penalty for using multiple
CQs.  The reason I'm interested in separate CQs is to avoid CQ overflow as I
add connections.  On other stacks (Voltaire, Cisco, Silverstorm) I size a
single CQ large enough for 'n' connections (i.e. cluster size - 1), but that
means I have to refuse connections when 'n' have been established.  In one
stack it also stressed vmalloc() and prevented me from using a single
whole-memory mapping.

Is there a consensus?  

Cheers,
Eric


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general