[openib-general] local QP error notification
0. after disconnecting a cable (we just used a switch interface to disable a port) for a connected qpair, we get errors when try to send a packet on that port (this is expected). but 1. don't get errors when try to post receive a packet. is this expected? 2. don't get any of the IB_EVENT_CQ_ERR, IB_EVENT_QP_FATAL, (ib_event_type) errors.. a. when is a good time for ULPs to issue disconnect and destroy qpairs in such cases? should it do it in step 0? should it expect step 1 and 2 to happen? b. what is expected when we connect the cable back? I assume for RC QP we must destroy the connection/QP in case of such error, it will never come back to ok state. is that right? thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] remote node/port going down notification
is there a way to get remote node/port down notification (other part of
a connected qpair)?
I can't associate any of these async events to remote node/port going down..
I assume these are for local port/node...
thanks, som.
enum ib_event_type {
IB_EVENT_CQ_ERR,
IB_EVENT_QP_FATAL,
IB_EVENT_QP_REQ_ERR,
IB_EVENT_QP_ACCESS_ERR,
IB_EVENT_COMM_EST,
IB_EVENT_SQ_DRAINED,
IB_EVENT_PATH_MIG,
IB_EVENT_PATH_MIG_ERR,
IB_EVENT_DEVICE_FATAL,
IB_EVENT_PORT_ACTIVE,
IB_EVENT_PORT_ERR,
IB_EVENT_LID_CHANGE,
IB_EVENT_PKEY_CHANGE,
IB_EVENT_SM_CHANGE,
IB_EVENT_SRQ_ERR,
IB_EVENT_SRQ_LIMIT_REACHED,
IB_EVENT_QP_LAST_WQE_REACHED
};
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote:
> Tang, Changqing wrote:
>
>> Is there a way (not limited to cm) to know that path 1 is back again and
>> reload it as the new alternate path ? If path 1 is down, we can not set
>> it as alternate path, right ?
>>
>> If we can not bring a path back on fly, the usage of APM is limited.
>
>
> Yes - the usage is currently limited, but that is being worked on.
> Support is needed in both the ib_cm and ib_sa. The ib_sa needs SA
> informinfo/notice support, so users can receive fabric event
> notifications. And the ib_cm needs to know when to switch to the
> alternate path. Venkatesh has a couple of patches that assist with
> both of these.
>
> - Sean
Sean,
will there be a new API for SA event notification?
today we already get this IB_EVENT_PATH_MIG (as defined below), will
"path1 is back again" event
be delivered the same way?
thanks, som.
enum ib_event_type {
IB_EVENT_CQ_ERR,
IB_EVENT_QP_FATAL,
IB_EVENT_QP_REQ_ERR,
IB_EVENT_QP_ACCESS_ERR,
IB_EVENT_COMM_EST,
IB_EVENT_SQ_DRAINED,
IB_EVENT_PATH_MIG,
IB_EVENT_PATH_MIG_ERR,
IB_EVENT_DEVICE_FATAL,
IB_EVENT_PORT_ACTIVE,
IB_EVENT_PORT_ERR,
IB_EVENT_LID_CHANGE,
IB_EVENT_PKEY_CHANGE,
IB_EVENT_SM_CHANGE,
IB_EVENT_SRQ_ERR,
IB_EVENT_SRQ_LIMIT_REACHED,
IB_EVENT_QP_LAST_WQE_REACHED
};
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
I don't see that API( ib_cm_init_rearm_attr() the build I am using.
so I I use a function like this:
modifyqp_rearm(con_t *con) {
struct ib_qp_attr qp_attr;
int qp_attr_mask =0;
int ib_stat =0;
memset(&qp_attr, 0, sizeof(struct ib_qp_attr));
qp_attr.qp_state = IB_QPS_RTS;
ib_stat = ib_cm_init_qp_attr(
con ->cm_id,
&qp_attr,
&qp_attr_mask);
if (ib_stat) {
return ib_stat;
}
qp_attr.path_mig_state = IB_MIG_REARM;
qp_attr_mask |= IB_QP_PATH_MIG_STATE;
ib_stat = ib_modify_qp(con->qp, &qp_attr, qp_attr_mask);
if (ib_stat) {
return ib_stat;
}
return 0;
}
Active side sends the LAP. Both side is called with APR event.
I call this function at the APR handler and it returns me the error -22
when tried to ib_modify_qp().
see anything wrong? do I have to set anything else?
or is it broken in the build I am using?
thanks, som.
Venkatesh Babu wrote:
> 1. No my application has to make the state trasition from RTS to RTS.
> In case of failover set path_mig_state to IB_MIG_MIGRATED.
>
> In case of rearming call the function ib_cm_init_rearm_attr(). This
> function is defined in the bug#172.
>
> 2. No, I have my own ULP module which sits alongside to the SDP, SRP,
> IPoIB modules.
>
> VBabu
>
> somenath wrote:
>
>> Venkatesh Babu wrote:
>>
>>> 1. Yes, I can rearm the alternate path by sending LAP and APR messages.
>>
>>
>>
>>
>> does the qpair go to rearm state just by sending LAP and APR messages?
>> I mean, you don't have to change the QP state to REARM explicitely?
>>
>>>
>>> 2. I was sending some network traffic (netperf) while doing these
>>> failovers.
>>>
>> so, I assume its SDP's APM feature gets tested? is that true?
>>
>> thanks, som.
>>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote:
> somenath wrote:
>
>> any idea when this stuff will get done?
>
>
> As a rough estimate, Q1, though I will try to get to it after adding
> support for SA events, which likely would be near late November, early
> December. The framework in the ib_cm is there; I just need a way to
> signal that failover has occurred.
>
>> another question: If one brings back the old path (after the first
>> failure) and use the old path record
>> to do lap/apr then reenabling migration using LAP/APR should work,
>> right?
>
>
> Yes.
>
>> next question is: which component is suppossed to change QP state to
>> REARM? spec just says:
>> "based on a command from a management entity, the QP state is set to
>> REARM" .
>
>
> The ib_cm does not perform any QP state transitions. That is left up
> to the user, since the CM is unaware of other actions related to the
> QP that the user may be performing.
but this function
cm_init_qp_rts_attr(struct cm_id_private *cm_id_priv,
struct ib_qp_attr *qp_attr,
int *qp_attr_mask)
does perform the state transition using
if (cm_id_priv->alt_av.ah_attr.dlid) {
*qp_attr_mask |= IB_QP_PATH_MIG_STATE;
qp_attr->path_mig_state = IB_MIG_REARM;
}
which works for me the first time I load path (using send REQ) and
change state to RTS.
so, I am bit confused here. do you mean that CM will do the state
transition first time and
not the next time?
thanks, som.
>
> - Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote: >> 1. does re-enabling Migration (as defined in vol1 of ib spec in >> 17.2.8.1.4) work for you? >> (I mean after the 1st path failure, you do lap/apr packet transfer) > > > I believe that there's other issues that need to be fixed for this to > fully work. The ib_cm uses the primary_path specified during > connection establishment to send CM MADs related to that connection. > If the primary_path fails and stays unavailable, future MADs (e.g. > LAP) sent by the CM would also fail. The CM records the alternate > path, but currently doesn't do anything with it. It needs to know > when to switch to the alternate path for its MADs. > > - Sean any idea when this stuff will get done? another question: If one brings back the old path (after the first failure) and use the old path record to do lap/apr then reenabling migration using LAP/APR should work, right? next question is: which component is suppossed to change QP state to REARM? spec just says: "based on a command from a management entity, the QP state is set to REARM" . does the current CM code do it automatically? In my case, its not happening... thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Venkatesh Babu wrote: > 1. Yes, I can rearm the alternate path by sending LAP and APR messages. does the qpair go to rearm state just by sending LAP and APR messages? I mean, you don't have to change the QP state to REARM explicitely? > > 2. I was sending some network traffic (netperf) while doing these > failovers. > so, I assume its SDP's APM feature gets tested? is that true? thanks, som. > VBabu > > somenath wrote: > >> hi Venkatesh: >> >> Two questions: >> >> 1. does re-enabling Migration (as defined in vol1 of ib spec in >> 17.2.8.1.4) work for you? >> (I mean after the 1st path failure, you do lap/apr packet transfer) >> >> 2. What applications you are testing with? >> >> thanks, som. >> >> Venkatesh Babu wrote: >> >>> >>> I have added couple of patches to the OFED stack as described in >>> bug#160, bug#172, and bug#159 and with this successfully tested the >>> APM functionality, except one issue. >>> >>> Configuration: >>> 2 Nodes >>> CPU: AMD Opteron(tm) Processor 252 Dual processor >>> CA type: MT25208 >>> Firmware version: 5.1.4 >>> OS: CentOS release 4.2 >>> IB: OFED 1.0 >>> >>> 2 Flextronics 24 port switchs >>> >>> Node1 Port1 connected to Switch1 >>> Node1 Port2 connected to Switch2 >>> Node2 Port1 connected to switch1 >>> Node 2 Port 2 connected to Switch2 >>> >>> Node1 : Active side of the RC QP >>> Node 2 : Passive side of the RC QP >>> >>> Test1: >>> Failover simulation on Node1 >>> 1. Simulate the port1 failure, RC QP migrates the path to port2 >>> 2. Simulate the port1 UP to rearm the alternate path from port1 >>> 3. Simulate the port2 failure, RC QP migrate the path to port1 >>> 4. Simulate the port2 IP to rearm the alternate path from port2 >>> >>> Test2: >>> Real failover my manually pulling the cable >>> 1. Simulate the failover/failback by pulling cable of Node1 port1 >>> 2. Simulate the failover/failback by pulling cable of Node1 port2 >>> 3. Simulate the failover/failback by pulling cable of Node2 port1 >>> 4. Simulate the failover/failback by pulling cable of Node2 port2 >>> >>> >>> ISSUE: >>> If I pull the both the cables then there are no paths to the >>> destination, so RC QP connection is supposed to tear down. But it >>> is not working. >>> >>> 1. Create a RC QP and load both primary and alternate path >>>(I was setting rnr_retry_count = 6, retry_count = 6, >>> packet_life_time field of struct ib_sa_path_rec to 15 and also tried >>> with 12) >>> 2. Send some traffic over RC QP >>> 3. Disconnect the cable belonging to the primary path >>> 4. It smoothly fails over to alternate path and it becomes primary >>> path. >>> >>> No affect to the traffic on that RC QP >>> 5. Remove the second cable belonging to the new primary path. >>> 6. Obviously traffic stops since there are no paths to the >>> destination. But for the outstanding WRs in the RC QP I don't get >>> any callback from the verbs layer describing whether it succeeded or >>> failed due to some error like IB_WC_RETRY_EXC_ERR. >>> When I query the RC QP properties it still shows that it is in >>> IB_QPS_RTS state. >>> >>> >>> Without APM functionality it behaves correctly - >>> 1. Create a RC QP and load only primary path >>>(I was setting rnr_retry_count = 6, retry_count = 6, >>> packet_life_time field of struct ib_sa_path_rec to 15 and also tried >>> with 12) >>> 2. Send some traffic over RC QP >>> 3. Disconnect the cable belonging to the primary path >>> 4. Obviously traffic stops since there are no paths to the >>> destination. For the outstanding WRs in the RC QP I do get a >>> callback from the verbs layer describing the first WR that it failed >>> due to error IB_WC_RETRY_EXC_ERR and for all other WRs I get >>> IB_WC_WR_FLUSH_ERR. >>> I will close this RC QP. >>> >>> VBabu >>> >>> Date: Mon, 16 Oct 2006 14:03:50 -0700 >>> From: "Sean Hefty" <[EMAIL PROTECTED]> >>> Subject: Re: [openib-general] APM support in openib stack >>> To: [EMAIL PROTECTED] >>> Cc: openib-general@openib.org >>> Message-ID: <[EMAIL PROTECTED]> >>> Content-Type: text/plain; charset=iso-8859-1; format=flowed >>> >>> somenath wrote: >>> >>>>>>> Doesn't ib_cm_init_qp_attr() set this for you? >>>>>> >>>>>> >>>>>> >>>>> >>>>> No, it doesn't. it returns me >>>>> attr_mask=0x12d181 >>>>> port=0x0 alt_port=0x0 >>>> >>>> >>>> >>>> >>>> >>> >>> Okay - there was a fix to the cm.c file (svn rev 8267) that added >>> setting the alternate port number when initializing the QP >>> attributes. Apparently that fix did not make it into the release >>> that you're using. >>> >>> - Sean >>> >>> >>> >>> >>> >> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
hi Venkatesh: Two questions: 1. does re-enabling Migration (as defined in vol1 of ib spec in 17.2.8.1.4) work for you? (I mean after the 1st path failure, you do lap/apr packet transfer) 2. What applications you are testing with? thanks, som. Venkatesh Babu wrote: > > I have added couple of patches to the OFED stack as described in > bug#160, bug#172, and bug#159 and with this successfully tested the > APM functionality, except one issue. > > Configuration: > 2 Nodes > CPU: AMD Opteron(tm) Processor 252 Dual processor > CA type: MT25208 > Firmware version: 5.1.4 > OS: CentOS release 4.2 > IB: OFED 1.0 > > 2 Flextronics 24 port switchs > > Node1 Port1 connected to Switch1 > Node1 Port2 connected to Switch2 > Node2 Port1 connected to switch1 > Node 2 Port 2 connected to Switch2 > > Node1 : Active side of the RC QP > Node 2 : Passive side of the RC QP > > Test1: > Failover simulation on Node1 > 1. Simulate the port1 failure, RC QP migrates the path to port2 > 2. Simulate the port1 UP to rearm the alternate path from port1 > 3. Simulate the port2 failure, RC QP migrate the path to port1 > 4. Simulate the port2 IP to rearm the alternate path from port2 > > Test2: > Real failover my manually pulling the cable > 1. Simulate the failover/failback by pulling cable of Node1 port1 > 2. Simulate the failover/failback by pulling cable of Node1 port2 > 3. Simulate the failover/failback by pulling cable of Node2 port1 > 4. Simulate the failover/failback by pulling cable of Node2 port2 > > > ISSUE: > If I pull the both the cables then there are no paths to the > destination, so RC QP connection is supposed to tear down. But it is > not working. > > 1. Create a RC QP and load both primary and alternate path >(I was setting rnr_retry_count = 6, retry_count = 6, > packet_life_time field of struct ib_sa_path_rec to 15 and also tried > with 12) > 2. Send some traffic over RC QP > 3. Disconnect the cable belonging to the primary path > 4. It smoothly fails over to alternate path and it becomes primary path. > > No affect to the traffic on that RC QP > 5. Remove the second cable belonging to the new primary path. > 6. Obviously traffic stops since there are no paths to the > destination. But for the outstanding WRs in the RC QP I don't get any > callback from the verbs layer describing whether it succeeded or > failed due to some error like IB_WC_RETRY_EXC_ERR. > When I query the RC QP properties it still shows that it is in > IB_QPS_RTS state. > > > Without APM functionality it behaves correctly - > 1. Create a RC QP and load only primary path >(I was setting rnr_retry_count = 6, retry_count = 6, > packet_life_time field of struct ib_sa_path_rec to 15 and also tried > with 12) > 2. Send some traffic over RC QP > 3. Disconnect the cable belonging to the primary path > 4. Obviously traffic stops since there are no paths to the > destination. For the outstanding WRs in the RC QP I do get a callback > from the verbs layer describing the first WR that it failed due to > error IB_WC_RETRY_EXC_ERR and for all other WRs I get IB_WC_WR_FLUSH_ERR. > I will close this RC QP. > > VBabu > > Date: Mon, 16 Oct 2006 14:03:50 -0700 > From: "Sean Hefty" <[EMAIL PROTECTED]> > Subject: Re: [openib-general] APM support in openib stack > To: [EMAIL PROTECTED] > Cc: openib-general@openib.org > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=iso-8859-1; format=flowed > > somenath wrote: > >>>>> Doesn't ib_cm_init_qp_attr() set this for you? >>>> >>> >>> >>> No, it doesn't. it returns me >>> attr_mask=0x12d181 >>> port=0x0 alt_port=0x0 >> >> >> > > Okay - there was a fix to the cm.c file (svn rev 8267) that added > setting the alternate port number when initializing the QP > attributes. Apparently that fix did not make it into the release that > you're using. > > - Sean > > > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote: >>now I am able to create connections with pri_path = path1, alt_path = path2 >>with the following change in the code. >> >>I specify the port_num and al_port_num before calling ib_modify_qp() to >>change >>state to RTR (earlier I was changing this when modifying state to >>IB_QPS_INIT). >> >> > >Doesn't ib_cm_init_qp_attr() set this for you? > > No, it doesn't. it returns me attr_mask=0x12d181 port=0x0 alt_port=0x0 other attribs are: qp_state=0x2 cur_qp_state=0x0 path_mtu=0x4 path_mig_state=0x0 qkey=0x0 rq_psn=0x0 sq_psn=0x0 dest_qp_num=0x407 qp_access_flags=0x0 pkey_index=0x0 min_rnr_timer=0x0 port_num=0x0 timeout=0x0 retry_cnt=0x0 rnr_retry=0x0 alt_port_num=0x0 cap: max_send_wr=0x0 max_recv_wr=0x0 max_send_sge=0x0 max_recv_sge=0x0 max_inline_date=0x0 thanks, som., > > >>( but it fails if I do this qp_attr_mask |= IB_QP_PORT, should it fail?) >> >> > >This sets the primary physical port, which is only valid transitioning to INIT. > >- Sean > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote: >>>pri_path = alt_path = path 1 >>>works >>> >>> >>> >>no, I haven't tested that. I can try that too, if u think that can >>provide useful info.. >> >> > >I misunderstood one of your earlier e-mails then. I threw together a test case >to try this, and it worked for me. Can you see if the same works for you? If >not, then my guess is that the release you're using is missing some needed >patches. (You may be able to work around the issue in your code, however, so >we'll see what can be done.) > >My systems only have one path between them, so I until I can physically add >another path, I won't be able to test the case where pri_path != alt_path. > >- Sean > > Sean: now I am able to create connections with pri_path = path1, alt_path = path2 with the following change in the code. I specify the port_num and al_port_num before calling ib_modify_qp() to change state to RTR (earlier I was changing this when modifying state to IB_QPS_INIT). ( but it fails if I do this qp_attr_mask |= IB_QP_PORT, should it fail?) but the connection migration to alt_path etc. is not yet tested, so not sure how much is working...will check what happens next. thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
somenath wrote: > Sean Hefty wrote: > >> Can you confirm that I have your test failure correctly? >> >> pri_path = path 1 >> alt_path = NULL >> works > > > > yes. > >> >> pri_path = path 2 >> alt_path = NULL >> works >> > yes. > >> pri_path = alt_path = path 1 >> works >> > > no, I haven't tested that. I can try that too, if u think that can > provide useful info.. > >> pri_path = alt_path = path 2 >> works > > > > no, not tried that either. > >> >> pri_path = path 1 >> alt_path = path 2 >> fails >> > > yes, that fails to change the QP state to RTR in req_recv_handler().. > (if I don't change the state to RTR, REP sending will be successful) > >> Are the only difference between path 1 and 2 the S/DGIDs and S/DLIDs? >> > that's how its coded, but I haven't verified that in req_recv_handler(). > I will verify that and let you know (by dumping both the paths in > rec_recv_handler()). for node1: primary_path: dlid=0x1f06 slid=0x1d06 raw=0x0 flow=0x0 hop=0x0 tra=0x0 rev=0x1 numb=0x0 pkey=0x sl=0x0 mtus=0x2 mtu=0x4 rate=0x2 plts=0x3 plt=0x2 pref=0x2 alt_path: dlid=0x2006 slid=0x1e06 raw=0x0 flow=0x0 hop=0x0 tra=0x0 rev=0x1 numb=0x0 pkey=0x sl=0x0 mtus=0x2 mtu=0x4 rate=0x2 plts=0x3 plt=0x2 pref=0x2 they seem to be same, thanks, som. > thanks, som. > > > > >> - Sean > > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote: > Can you confirm that I have your test failure correctly? > > pri_path = path 1 > alt_path = NULL > works yes. > > pri_path = path 2 > alt_path = NULL > works > yes. > pri_path = alt_path = path 1 > works > no, I haven't tested that. I can try that too, if u think that can provide useful info.. > pri_path = alt_path = path 2 > works no, not tried that either. > > pri_path = path 1 > alt_path = path 2 > fails > yes, that fails to change the QP state to RTR in req_recv_handler().. (if I don't change the state to RTR, REP sending will be successful) > Are the only difference between path 1 and 2 the S/DGIDs and S/DLIDs? > that's how its coded, but I haven't verified that in req_recv_handler(). I will verify that and let you know (by dumping both the paths in rec_recv_handler()). thanks, som. > - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote: > somenath wrote: > >> I must be missing something here: do I have to do anything with: >> >> 1.path_mig_state field of ib_qp_attr? > > > This should be set when transitioning to RTS. The field is set by the > ib_cm in ib_cm_init_qp_attr. (Actually, are you using this call to > set your QP attributes?) yes, I use ib_cm_init_qp_attr() > >> 2. when do I set alt_port_num? I do that using ib_modify_qp() using >> IB_QP_PORT attrib mask. > > > This is set when transitioning to RTR. > yes, i do that. >> 3. how do I set alt_timeout and alt_pkey_index? with what attrib >> mask? 4. is there any sample working code using APM? > > > Not that I'm aware of. thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote:
>>ib_sa_path_rec_get(
>> device,
>> HCA_PRM_PORT, /* first port =1, second
>>port=2 */
>>
>>
>
>Note that this tells the ib_sa module which port to send the request out on.
>It's separate from the actual path information being requested, which is based
>on the GIDs.
>
>
>
that's right: so I just verified that by dumping dgid, sgid in primary
and alternate path
obtained in req_recv_handler(); they look ok to me, but just dumping
here, in case u find
something here:
for node1:
destination gid===
gid fe80:0:0:0:5:ad00:3:a801
source gid===
gid fe80:0:0:0:5:ad00:3:9abd
APM destination gid===
gid fe80:0:0:0:5:ad00:3:a802
APM source gid===
gid fe80:0:0:0:5:ad00:3:9abe
===
for node2:
destination gid===
gid fe80:0:0:0:5:ad00:3:9abd
source gid===
gid fe80:0:0:0:5:ad00:3:a801
APM destination gid===
gid fe80:0:0:0:5:ad00:3:9abe
APM source gid===
gid fe80:0:0:0:5:ad00:3:a802
==
req_handler() {
// create qpairs, connection handle, ...
ib_stat = modifyqp_rtr(connection);;
if (ib_stat) {
goto exit; > i get error here
}
ib_stat = ib_send_cm_rep(
connection->cm_id,
&accept_param);
if (ib_stat) {
goto exit;
}
}
if I do, modifyqp_rtr() after sending ib_send_cm_rep() then REP sending
(just to try out)
is successful (and other side recvs the REP), however changing state to
RTR still fails after that.
I must be missing something here: do I have to do anything with:
1.path_mig_state field of ib_qp_attr?
2. when do I set alt_port_num? I do that using ib_modify_qp() using
IB_QP_PORT attrib mask.
3. how do I set alt_timeout and alt_pkey_index? with what attrib mask?
4. is there any sample working code using APM?
thanks, som.
>> &path_rec,
>> IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID |
>> IB_SA_PATH_REC_NUMB_PATH
>> | IB_SA_PATH_REC_PKEY,
>>
>>
>...
>
>
>>do you remember when you checked in the working code?
>>I am wondering if the RHEL4 U4 binary distrition of redhat has your changes.
>>
>>
>
>That, I'm not sure about. The fact that it works when alt_path = primary_path
>is what's confusing me at the moment.
>
>- Sean
>
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] APM support in openib stack
Sean Hefty wrote: >>3. in req_handler() we follow the same steps as we have done without APM.. >> i.e. create qpairs, change qp state to RTR and then send REP. >> >>however, when trying to change state to RTR usinb ib_modify_qp() I get >>an error (-22). >> >>two info: same code will work if I pass alt_path as NULL or change the >>alt_path as primary path. >> >>I must be missing something here, I assume this basic APM feature works >>in RHEL4 update 4 distribtion >>of openib stack. >> >> > >I added code to the ib_cm to handle APM, but haven't ever tested it myself. I >believe others have used it successfully though. > >What differences are there between the primary and alternate paths? I.e. are >just the LIDs different, or are other values also different? > >- Sean > > Sean: thanks for ur reply. I use ib_sa_path_rec_get( device, HCA_PRM_PORT, /* first port =1, second port=2 */ &path_rec, IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID | IB_SA_PATH_REC_NUMB_PATH | IB_SA_PATH_REC_PKEY, 5000, GFP_KERNEL, func_completion, context, &query) to get the primary path, and make the same call with HCA_ALT_PORT (=2) to get the alternate path. primary path has the source and destination gid for the HCA port 1, alternate path the source and destination gid for the HCA port 2. using these two paths, I send the REQ, otherwise gets the REQ... (I can dump the primary and alternate path received in req handler to check everything is ok, will try that next..) do you remember when you checked in the working code? I am wondering if the RHEL4 U4 binary distrition of redhat has your changes. thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] APM support in openib stack
hi, I am trying to use the APM support in openib kernel stack and facing some problems. here are the steps I follow: 1. first resolve both the path, primary and alternate path. 2. send REQ using: active_param.primary_path = path; active_param.alternate_path = alt_path; ib_send_cm_req( cm_id, &active_param); this call doesn't return any error 3. in req_handler() we follow the same steps as we have done without APM.. i.e. create qpairs, change qp state to RTR and then send REP. however, when trying to change state to RTR usinb ib_modify_qp() I get an error (-22). two info: same code will work if I pass alt_path as NULL or change the alt_path as primary path. I must be missing something here, I assume this basic APM feature works in RHEL4 update 4 distribtion of openib stack. thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] HCAs with and without memory
is there any performance difference observed between memFree and non-memFree HCAs? thanks, som. Dotan Barak wrote: >Hi john. > >john t wrote: > > >>Hi OpenIB group, >> >>What is the difference between HCAs with memory and without memory. >>How is the on-board memory used by HCAs? Is it that data is first >>copied into this memory and then into physical memory? >> >>Regards, >>John T. >> >> > >If you are asking about Mellanox HCAs i can answer you: > >The difference is the technology which those HCAs are using: >The HCAs without the attached memory are using the memfree technology. > >The main difference between the 2 HCAs is where the context of the >various resources is located: in the host memory or in the attached memory. > >The data itself (during data movement) is not stored in this memory at >any point in the attached memory. > >Dotan > > >___ >openib-general mailing list >openib-general@openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] openib tools to update mellnox firmware
Michael S. Tsirkin wrote: >Quoting r. somenath <[EMAIL PROTECTED]>: > > >>thanks Tziporet. >>from ur link , I am looking here >>http://www.mellanox.com/support/firmware_table.php >> >>for the exact HCA I have from your list. I don't find the exact match of >>OPN and Card Rev (PSID) and I don't know how to find those. >>all I found about that card is this: >>[EMAIL PROTECTED] bin]# ./ibstat >>CA 'mthca0' >>CA type: MT25208 (MT23108 compat mode) >>Number of ports: 2 >>Firmware version: 4.6.0 >>Hardware version: a0 >>Node GUID: 0x0005ad039abc >>System image GUID: 0x0005ad000100d050 >> >>Is there another tool which will give me OPN/Card Rev (PSID) so that I >>can match exactly from the table above? >> >> >> > >Try this: > >cat /sys/class/infiniband/mthca0/board_id > > in my setup that doesn't produce any output...if I do this: [EMAIL PROTECTED] mthca0]# cat /sys/class/infiniband/mthca0/board_id that's an empty output. however cat'ing other files give some stuff e.g.: [EMAIL PROTECTED] mthca0]# cat /sys/class/infiniband/mthca0/fw_ver 4.6.0 == [EMAIL PROTECTED] mthca0]# cat /sys/class/infiniband/mthca0/hca_type MT25208 (MT23108 compat mode) == [EMAIL PROTECTED] mthca0]# cat /sys/class/infiniband/mthca0/hw_rev a0 == thanks, som. >>also, another question: is the latest firmware backward compatible with >>the openib stack I am using? (from redhat binary distribtution RH4 >>update 3)? >> >>thanks, som. >> >> > >Should be. > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] openib tools to update mellnox firmware
Tziporet Koren wrote: > somenath wrote: > >> I see this messages while booting my system: >> >> Jul 17 10:35:37 alekhine kernel: ib_mthca :03:00.0: HCA FW >> version 4.6.0 is old (4.7.0 is current). >> Jul 17 10:35:37 alekhine kernel: ib_mthca :03:00.0: If you have >> problems, try updating your HCA FW. >> >> where can I find the latest firmware for ib_mthca (pci express card) >> and tools to update this? >> >> thanks, som. >> >> >> > > Most updated FW is located at: > http://www.mellanox.com/support/firmware_download.php > > You can download a binary image and burn it using mstflint (which is > part of OFED), or download also MFT tools (from the same location) to > create a binary image. > > Tziporet > thanks Tziporet. from ur link , I am looking here http://www.mellanox.com/support/firmware_table.php for the exact HCA I have from your list. I don't find the exact match of OPN and Card Rev (PSID) and I don't know how to find those. all I found about that card is this: [EMAIL PROTECTED] bin]# ./ibstat CA 'mthca0' CA type: MT25208 (MT23108 compat mode) Number of ports: 2 Firmware version: 4.6.0 Hardware version: a0 Node GUID: 0x0005ad039abc System image GUID: 0x0005ad000100d050 Is there another tool which will give me OPN/Card Rev (PSID) so that I can match exactly from the table above? also, another question: is the latest firmware backward compatible with the openib stack I am using? (from redhat binary distribtution RH4 update 3)? thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] API clarification: ib_get_dma_mr()
only documentation I find on this API is: /** * ib_get_dma_mr - Returns a memory region for system memory that is * usable for DMA. * @pd: The protection domain associated with the memory region. * @mr_access_flags: Specifies the memory access rights. */ I see there is no address associated with it, so I assume this is some kind of reserved key stuff. now my questions are: 1. it basically returns l_key and r_key: can those keys be used with any phys/bus addresseses? (address returned by dma_map_single(), dma_map_xxx(), virt_to_phys() etc) 2. can the same l_key and r_key be used for different addresses at the same time? I mean can one just use the API once to get a pair of l_key and r_key and use for all post_recv and send ops (including rdma's)? and my next question is on (valid for another similar API on fast memory reg too): struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd, struct ib_phys_buf *phys_buf_array, int num_phys_buf, int mr_access_flags, u64 *iova_start); q1:is there any use of iova_start returned by this API? phys_buf_arrary, l_key and r_key are only used in subsequent APIs.. q2: is there any merit of using this API (or the one with fast memory reg) vs ib_get_dma_mr()? I see the advatage that its little bit more secure that it gives seperate l_key/r_key for a given phys address. Is there any other benefit? thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] openib tools to update mellnox firmware
I see this messages while booting my system: Jul 17 10:35:37 alekhine kernel: ib_mthca :03:00.0: HCA FW version 4.6.0 is old (4.7.0 is current). Jul 17 10:35:37 alekhine kernel: ib_mthca :03:00.0: If you have problems, try updating your HCA FW. where can I find the latest firmware for ib_mthca (pci express card) and tools to update this? thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
Sean Hefty wrote:
> somenath wrote:
>
>>> I think this evaluates ib_poll_cq(..) < 0 before doing the
>>> assignment. Since the expression evaluates to false, count is
>>> assigned 0. Can you try modifying this to:
>>>
>>> if ((count = ib_poll_cq(..)) < 0)
>>>
>>> - Sean
>>
>>
>>
>> I added that stuff, but it didn't make a difference...it still
>> returned 0
>
>
> If ib_poll_cq() is truly returning 0, but with a valid wc, then this
> is a bug. (I'm pretty sure that the additional parens are needed in
> any case, and you'll also want to swap your ib_poll_cq and
> ib_req_notify calls back around.) I have never seen this issue, and
> I'm sure that we would have heard about it if others had. Nothing
> obvious jumped out at me when looking at the mthca completion code.
>
> Maybe you've tried this, but can you break the count = ib_poll_cq()
> out from the if statement, print count and the wc structure
> immediately before and after the call, and post the results?
I tried this code too...(an retrying just now) and seeing it still
returns 0...
count = ib_poll_cq(cq, 1, &wc);
if (count < 0) {
stat = xxx_st_error;
goto error; //it has never gone to error
from this location.
}
I am ready to try any other suggestions and will change this portion of
the code too,
but currently I am just trying to get a valid wc handle with
ib_poll_cq() returning 1!!!
that never happens in anyway I try.
>
>> looks like, I may be getting a valid entry even if ib_poll_cq() is
>> returning zero...
>> is that possible?
>
>
> You should not be getting a valid wc if the call returns 0. For most
> applications this would result in a lost completion.
>
>> dump of wc entry from recv side, looks like many are valid entries
>> (like wr_id, length, op etc..)
>
>
> This could just be left over data from the stack, but...
>
>> wr_id=0x10118163c00 status=0x0 op=0x80
>
>
> Does the wr_id match the value that you set on the work request? (I'm
> assuming that it does based on your previous comment.) The opcode
> does match IB_WC_RECV.
>
'
yes, wr_id, lenght, opcode, status match...
next, i will check the data
> - Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
Sean Hefty wrote:
> somenath wrote:
>
>> int
>> io_complete(
>>struct ib_cq *cq,
>>void *passed_arg)
>> {
>>xxx_connection_t*arg = passed_arg;
>>xxx_status_tstat = xxx_st_ok;
>>struct ib_wc wc;
>>int count = 0;
>>
>>if (count = ib_poll_cq(cq, 1, &wc) < 0) {
>
>
> I think this evaluates ib_poll_cq(..) < 0 before doing the
> assignment. Since the expression evaluates to false, count is
> assigned 0. Can you try modifying this to:
>
> if ((count = ib_poll_cq(..)) < 0)
>
> - Sean
I added that stuff, but it didn't make a difference...it still returned
0
thanks anyway for suggesting
looks like, I may be getting a valid entry even if ib_poll_cq() is
returning zero...
is that possible?
I am going to try it anyway, even if the ib_poll_cq() returns zero, go
ahead and try to use the wc..
dump of wc entry from recv side, looks like many are valid entries (like
wr_id, length, op etc..)
wr_id=0x10118163c00 status=0x0 op=0x80
vendor_err=0x246 byte_len=0x2028 imm_data=0xa0367eb8
qp_num=0x404 src_qp=0x68 wc_flags=0x0
pkey_index=0x0 slid=0x0 sl=0x0
dlid_path_bits=0x0 port_num=0xff
som.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
Sean Hefty wrote:
> Can you also post your code, including the completion handler routines
> and QP creation / initialization sections?
>
> - Sean
curretnly this is what I am trying to test, just to get one notification
successfully with a proper wc element.
thanks, som.
int
recvio_complete(
struct ib_cq *cq,
void *passed_arg) {
io_complete(cq, passed_arg);
}
int
sendio_complete(
struct ib_cq *cq,
void *passed_arg) {
io_complete(cq, passed_arg);
}
int
io_complete(
struct ib_cq *cq,
void *passed_arg)
{
xxx_connection_t*arg = passed_arg;
xxx_status_tstat = xxx_st_ok;
struct ib_wc wc;
int count = 0;
if (count = ib_poll_cq(cq, 1, &wc) < 0) {
stat = xxx_st_error;
arg = NULL;
goto error;
}
if(stat = ib_req_notify_cq(cq, IB_CQ_NEXT_COMP)) {
stat = xxx_st_error;
goto error;
}
if (count == 0) {
stat = xxx_st_ok;
goto error;
}
if ((arg != NULL) && (cq != NULL)) {
io_process(conn_handle);
} else {
stat = xxx_st_error;
goto error;
}
error:
return stat;
}
xxx_status_t
xxx_create_qpairs(
xxx_connection_t *conn_handle)
{
xxx_status_txxx_stat = xxx_st_ok;
int ib_stat = 0;
int entries_send, entries_recv;
struct ib_qp_init_attr qp_attr = { 0 };
unsignedlongflags;
conn_handle->llp_send_cq =
ib_create_cq(
xxx_openib.xxx_device,
sendio_complete,
xxx_async_cb,
conn_handle,
XXX_SEND_CQ_DEPTH);
if (IS_ERR(conn_handle->llp_send_cq)) {
ib_stat = PTR_ERR(conn_handle->llp_send_cq);
xxx_stat = map_ib_to_xxx_stat(ib_stat);
goto free_out;
}
conn_handle->llp_recv_cq =
ib_create_cq(
xxx_openib.xxx_device,
recvio_complete,
xxx_async_cb,
conn_handle,
XXX_RECV_CQ_DEPTH);
if (IS_ERR(conn_handle->llp_recv_cq)) {
ib_stat = PTR_ERR(conn_handle->llp_recv_cq);
xxx_stat = map_ib_to_xxx_stat(ib_stat);
ib_destroy_cq(conn_handle->llp_send_cq);
goto free_out;
}
if(ib_stat = ib_req_notify_cq(conn_handle->llp_send_cq,
IB_CQ_NEXT_COMP)) {
xxx_stat = map_ib_to_xxx_stat(ib_stat);
ib_destroy_cq(conn_handle->llp_send_cq);
ib_destroy_cq(conn_handle->llp_recv_cq);
goto free_out;
}
if(ib_stat = ib_req_notify_cq(conn_handle->llp_recv_cq,
IB_CQ_NEXT_COMP)) {
xxx_stat = map_ib_to_xxx_stat(ib_stat);
ib_destroy_cq(conn_handle->llp_recv_cq);
goto free_out;
}
memset(&qp_attr, 0, sizeof qp_attr);
qp_attr.event_handler = xxx_qevent_cb;
qp_attr.qp_context = conn_handle;
qp_attr.send_cq = conn_handle->llp_send_cq;
qp_attr.recv_cq = conn_handle->llp_recv_cq;
qp_attr.cap.max_send_wr = XXX_SENDQ_DEPTH;
qp_attr.cap.max_recv_wr = XXX_RECVQ_DEPTH;
qp_attr.cap.max_send_sge = 1;
qp_attr.cap.max_recv_sge = 1;
qp_attr.sq_sig_type = IB_SIGNAL_ALL_WR;
qp_attr.qp_type = IB_QPT_RC;
qp_attr.port_num = HCA_PRM_PORT; /* special QP types only */
conn_handle->llp_qp =
ib_create_qp(
xxx_openib.xxx_global_pd,
&qp_attr);
if (IS_ERR(conn_handle->llp_qp)) {
ib_destroy_cq(conn_handle->llp_send_cq);
ib_destroy_cq(conn_handle->llp_recv_cq);
ib_stat = PTR_ERR(conn_handle->llp_qp);
xxx_stat = map_ib_to_xxx_stat(ib_stat);
goto free_out;
}
xxx_modifyqp_init(conn_handle);
free_out:
return (xxx_stat);
} 102,1 68%
1,1 Top
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
Sean Hefty wrote:
> somenath wrote:
>
>> just to make sure I conveyed the exact thing I meant, if I change
>> the above code as follows:
>>
>> while (ib_poll_cq(cq, 1, &wc) > 0) {
>> process completion();
>> }
>> rearm CQ;
>>
>> then I just get notification once, and don't get any futher
>> notifications...so I assume rearm CQ should be done even if
>> ib_poll_cq() returns zero.
>
>
> Hmm... what HCA cards and svn version are you using? Mellanox cards
> generate new events if any completions remain on the CQ. So, if
> ib_poll_cq() isn't finding any completions, then another event should
> be generated. If ib_poll_cq() is returning a completion, then it
> sounds like there's a bug with ib_poll_cq() returning the wrong number
> of completions.
>
> - Sean
hardware config: PCI express machine with PCI-Express cards of Mellanox
HCA.
Exact same configuration was working with gen1 stack (no change in the
hardware setup).
you can find rest of the details here:
===
[EMAIL PROTECTED] bin]# ./ibstat
CA 'mthca0'
CA type: MT25208 (MT23108 compat mode)
Number of ports: 2
Firmware version: 4.6.0
Hardware version: a0
Node GUID: 0x0005ad039abc
System image GUID: 0x0005ad000100d050
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 114
LMC: 0
SM lid: 1
Capability mask: 0x00510a68
Port GUID: 0x0005ad039abd
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00510a68
Port GUID: 0x0005ad039abe
=
I am using the version of gen2 code as distributed in RH version 4 update 3
since I am trying to get a port on this version of the distribution.
(so don't know the exact svn version they pulled from, I just use the RPM's
in their binary distribution).
thanks, som.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
somenath wrote:
>Rimmer, Todd wrote:
>
>
>
>>>From: Sean Hefty
>>>somenath wrote:
>>>
>>>
>>>
>>>
>>>>2. why is the io completion routine called when ib_poll_cq() returns
>>>>zero? does this kind of notification contain any information?
>>>>is there some error happening here? what are some possible problem
>>>>
>>>>
>>>>
>>>>
>>>areas?
>>>
>>>
>>>
>>>
>>>>any wild guess...?
>>>>
>>>>
>>>>
>>>>
>>>Can you clarify what's happening? Are you calling ib_poll_cq() from
>>>
>>>
>>>
>>>
>>your
>>
>>
>>
>>
>>>ib_comp_handler() and not finding a completion?
>>>
>>>
>>>
>>>
>>Be aware that an IB completion handler should be coded as:
>>
>>while poll_cq returns a completion
>> process completion
>>rearm CQ
>>
>>
>>
>>
>
>that's what I am doing, to be more specific:
>
>rearm CQ;
>while (ib_poll_cq(cq, 1, &wc) > 0) {
> process completion();
>}
>
>is that what you meant?
>
>
just to make sure I conveyed the exact thing I meant, if I change
the above code as follows:
while (ib_poll_cq(cq, 1, &wc) > 0) {
process completion();
}
rearm CQ;
then I just get notification once, and don't get any futher
notifications...so I assume rearm CQ should be done even if
ib_poll_cq() returns zero.
thanks, som.
>
>
>>while poll_cq returns a completion
>> process completion
>>
>>Per the IBTA spec, rearm CQ is not required to generate an event for
>>completions already on the CQ, just for newly arriving ones (Mellanox
>>HCAs will generate a completion event if any CQEs remain on the CQ).
>>
>>Due to various race situations between the HCA and the software/ULP,
>>there are some valid cases where a completion event could occur after
>>poll_cq has already processed the completion.
>>
>>
>>
>>
>
>in my case, poll_cq() always returns 0, so I never get a valid wc entry...
>
>
>
>>Todd Rimmer
>>
>>
>>
>>
>
>
>___
>openib-general mailing list
>openib-general@openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
Rimmer, Todd wrote:
>>From: Sean Hefty
>>somenath wrote:
>>
>>
>>>2. why is the io completion routine called when ib_poll_cq() returns
>>>zero? does this kind of notification contain any information?
>>>is there some error happening here? what are some possible problem
>>>
>>>
>>areas?
>>
>>
>>>any wild guess...?
>>>
>>>
>>Can you clarify what's happening? Are you calling ib_poll_cq() from
>>
>>
>your
>
>
>>ib_comp_handler() and not finding a completion?
>>
>>
>
>Be aware that an IB completion handler should be coded as:
>
>while poll_cq returns a completion
> process completion
>rearm CQ
>
>
that's what I am doing, to be more specific:
rearm CQ;
while (ib_poll_cq(cq, 1, &wc) > 0) {
process completion();
}
is that what you meant?
>while poll_cq returns a completion
> process completion
>
>Per the IBTA spec, rearm CQ is not required to generate an event for
>completions already on the CQ, just for newly arriving ones (Mellanox
>HCAs will generate a completion event if any CQEs remain on the CQ).
>
>Due to various race situations between the HCA and the software/ULP,
>there are some valid cases where a completion event could occur after
>poll_cq has already processed the completion.
>
>
in my case, poll_cq() always returns 0, so I never get a valid wc entry...
>Todd Rimmer
>
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
Sean Hefty wrote:
> somenath wrote:
>
>> 2. why is the io completion routine called when ib_poll_cq() returns
>> zero? does this kind of notification contain any information?
>> is there some error happening here? what are some possible problem
>> areas?
>> any wild guess...?
>
>
> Can you clarify what's happening? Are you calling ib_poll_cq() from
> your ib_comp_handler() and not finding a completion?
>
> - Sean
That's right Sean. I establish a RC connection and pre-post a buffer of
size 4K in one side,
and try to send 4K packet from the other side. each side gets a
completion event when ib_poll_cq()
rets zero.
ib_recv_comp_hanlder(cq, arg) {
struct ib_wc wc;
ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); //i check for errors, there is
no error;
count = ib_poll_cq(cq, 1, &wc);
// here I find count = 0
}
same happens to send_comp_handler too..
so, everytime I get a cq notification, I get a zero entry (send and recv
completion
event occurs on respective nodes).
thanks, som.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] is wc valid if ib_poll_cq() returns zero
Sean Hefty wrote: > somenath wrote: > >> 1. if ib_poll_cq(cq, 1, &wc) returns zero, does wc contain a valid >> entry? > > > no > > * Poll a CQ for (possibly multiple) completions. If the return value > * is < 0, an error occurred. If the return value is >= 0, it is the > * number of completions returned. If the return value is > * non-negative and strictly less than num_entries, then the CQ was > * emptied. > > - Sean Thanks Sean! As per the above definiton "no error" occurred since return value is 0.. And as per your clarification, wc doesn't contain a valid entry. So my next question: 2. why is the io completion routine called when ib_poll_cq() returns zero? does this kind of notification contain any information? is there some error happening here? what are some possible problem areas? any wild guess...? anyone else seen this kind of stuff? ( I get send completion and recv completion both your 0 return value). thanks, som. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] is wc valid if ib_poll_cq() returns zero
two Qs refers to openib kernel stack found in Redhat distribution (RHEL4 update
3)
(redhat kernel version: 2.4.18-14 #1):
1. if ib_poll_cq(cq, 1, &wc) returns zero, does wc contain a valid entry?
[just for reference what happend in gen1 stack
following code is from gen1 code where if similar routine returned zero
send_entry was still valid...
if ( target->state == TARGET_ACTIVE_CONNECTION ) {
status = ib_cq_poll( target->active_conn->cqs_hndl, &send_entry
);
if ( status == 0 ) {
cq_send_handler( NULL, &send_entry, target );
}
}
]
2. why is the io completion routine called when ib_poll_cq() returns zero? is
there some error happening here?
thanks, som.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Mellanox HCAs: outstanding RDMAs
Talpey, Thomas wrote: At 10:03 AM 6/3/2006, Rimmer, Todd wrote: Yes, the limit of outstanding RDMAs is not related to the send queue depth. Of course you can post many more than 4 RDMAs to a send queue -- the HCA just won't have more than 4 requests outstanding at a time. To further clarity, this parameter only affects the number of concurrent outstanding RDMA Reads which the HCA will process. Once it hits this limit, the send Q will stall waiting for issued reads to complete prior to initiating new reads. It's worse than that - the send queue must stall for *all* operations. Otherwise the hardware has to track in-progress operations which are queued after stalled ones. It really breaks the initiation model. possibility of stalling is scary! is there any way one can figure out: 1. number of outstanding sends at a given point of time in Send Q? 2. maximum number of outstanding sends ever posted (during the lifetime of Q)? its possible to measure those in ULPs, but then that may not match exactly what is seen in the real Q...so, is there any low level tool to measure this? thanks, som. Semantically, the provider is not required to provide any such flow control behavior by the way. The Mellanox one apparently does, but it is not a requirement of the verbs, it's a requirement on the upper layer. If more RDMA Reads are posted than the remote peer supports, the connection may break. The number of outstanding RDMA Reads is negotiated by the CM during connection establishment and the QP which is sending the RDMA Read must have a value configured for this parameter which is <= the remote ends capability. In other words, we're probably stuck at 4. :-) I don't think there is any Mellanox-based implementation that has ever supported > 4. In previous testing by Mellanox on SDR HCAs they indicated values beyond 2-4 did not improve performance (and in fact required more RDMA resources be allocated for the corresponding QP or HCA). Hence I suspect a very large value like 128 would offer no improvement over values in the 2-8 range. I am not so sure of that. For one thing, it's dependent on VERY small latencies. The presence of a switch, or link extenders will make a huge difference. Second, heavy multi-QP firmware loads will increase the latencies. Third, constants are pretty much never a good idea in networking. The NFS/RDMA client tries to set the maximum IRD value it can obtain. RDMA Read is used quite heavily by the server to fetch client data segments for NFS writes. Tom. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Mellanox HCAs: outstanding RDMAs
What happens if one tries to do RDMA (say write for example) higher than 4 (or 128 in changed case)? does it just wait till previos operation is completed? I don't remember seeing any error it was only limited by the send Q-depth which can go much larger value. thanks, som. Roland Dreier wrote: Manpreet> Mellanox HCA can handle has been configured at 4 Manpreet> (mthca_main.c: default_profile: rdb_per_qp). And the Manpreet> HCAs can support a much higher value (128 I think). Manpreet> Could we move this value higher or atleast make it Manpreet> configurable? Leonid Arsh has a patch that I will integrate soon that makes this configurable. However, I'm curious. Do you have a workload where this actually makes a measurable difference? It seems that having 4 RDMA requests outstanding on the wire should be enough to get things to pipeline pretty well. If you haven't tested this, right now you can of course edit mthca_main.c to change the default value and recompile. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
