Re: [openib-general] port num in port priv
On Fri, 2006-02-10 at 13:56, Hal Rosenstock wrote: > On Thu, 2006-02-09 at 10:55, Roland Dreier wrote: > > The current code probably needs to be extended to give > > the upper layers the port number where a MAD was received, and also to > > make it possible to specify the port number that a directed route MAD > > will be sent on. > > On the receive side, the port_num is part of the WC for DR SMPs. > On the send side, the port_num is part of the WR (for DR SMPs). > So at least that part is covered as far as I can tell. While this is supported, one issue is that in order to obtain the port_num on the send side, the optional ib_query_ah must be implemented as the ib_mad_send_buf only contains an ah and not a port_num. Is there a better way to handle this ? -- Hal > There are some changes I can see and of course, the SMI has not been > exercised for switch forwarding so there could be some issues there. > > -- Hal > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] port num in port priv
On Thu, 2006-02-09 at 10:55, Roland Dreier wrote: > The current code probably needs to be extended to give > the upper layers the port number where a MAD was received, and also to > make it possible to specify the port number that a directed route MAD > will be sent on. On the receive side, the port_num is part of the WC for DR SMPs. On the send side, the port_num is part of the WR (for DR SMPs). So at least that part is covered as far as I can tell. There are some changes I can see and of course, the SMI has not been exercised for switch forwarding so there could be some issues there. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hi Suri, On Fri, 2006-02-10 at 09:09, Suresh Shelvapille wrote: > Hal: > The way I gather port stats locally on my device is by doing a "cat" on the > "/sys/class/Infiniband/mysw/port/0/counters" file. As Roland mentioned this > is limited to port0 counters on a switch. Base switch port 0 does not support PortCounters; only enhanced switch port 0. > If I need stats for port#N I need to expand this by providing more class > attributes and getting to them that way Switches are a little different than HCAs in terms of this so you may want to change the model. > I am not using user space verbs library, is there any other way to get at > the different port#N stats? Yes, by sending the appropriate PerfMgt Get for PortCounters to the local LID (of port 0) with the the PortSelect component set to the switch external port you want and decoding the GetResp. That's what the perfquery tool does. Your switch needs to support PMA for this to work. -- Hal > Thanks, > Suri > > > > The PMA counters are gathered by a tool named perfquery. Port number is > > in the PortSelect component of the PMA attribute. You can't get PMA > > counters until the port is active. > > > > -- Hal > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hi Suri, You can send a PMA PortCounters MAD either using "process local" or to the device using its LID. Eitan > I am not using user space verbs library, is there any other way to get at > the different port#N stats? > > Thanks, > Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hal: The way I gather port stats locally on my device is by doing a "cat" on the "/sys/class/Infiniband/mysw/port/0/counters" file. As Roland mentioned this is limited to port0 counters on a switch. If I need stats for port#N I need to expand this by providing more class attributes and getting to them that way I am not using user space verbs library, is there any other way to get at the different port#N stats? Thanks, Suri > The PMA counters are gathered by a tool named perfquery. Port number is > in the PortSelect component of the PMA attribute. You can't get PMA > counters until the port is active. > > -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hi Suri, On Thu, 2006-02-09 at 18:06, Suresh Shelvapille wrote: > Hal and folks: > > > > > For switches, the one from the WC needs to be filled in and passed so > > that sounds wrong and needs fixing. Do you want to take a crack at this > > or should I ? > > > [SS] I am not as IB core stack savvy as you guys are, so I would appreciate > it if you can provide the changes...I will be happy to review and test it. I'll supply a patch but you will need to test it and provide feedback. OK ? -- Hal > Right now, I have kludged my sw driver to handle DR SMPs to make the subnet > manager happy. > > Thanks a lot, > Suri > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
On Thu, 2006-02-09 at 17:56, Suresh Shelvapille wrote: > > > > Unfortunately the sysfs PMA counters support will not be that useful, > > since it will only show counters for port 0; but that's a different issue. > > > > - R. > [SS] That's a bummer, so don't we need to create as many file descriptors as > the number of physical ports on a switch, so that we can gather different > port related stats from the device locally (not talking about subnet manager > queries)! The PMA counters are gathered by a tool named perfquery. Port number is in the PortSelect component of the PMA attribute. You can't get PMA counters until the port is active. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hal and folks: > > For switches, the one from the WC needs to be filled in and passed so > that sounds wrong and needs fixing. Do you want to take a crack at this > or should I ? > [SS] I am not as IB core stack savvy as you guys are, so I would appreciate it if you can provide the changes...I will be happy to review and test it. Right now, I have kludged my sw driver to handle DR SMPs to make the subnet manager happy. Thanks a lot, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
> > Unfortunately the sysfs PMA counters support will not be that useful, > since it will only show counters for port 0; but that's a different issue. > > - R. [SS] That's a bummer, so don't we need to create as many file descriptors as the number of physical ports on a switch, so that we can gather different port related stats from the device locally (not talking about subnet manager queries)! Thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hi Suresh,
On Thu, 2006-02-09 at 17:22, Suresh Shelvapille wrote:
> Hal:
>
> I am only addressing the process_mad issue here:
>
>
>
> > > As a result, it appears like the process_mad function is always called
> > with a port number of
> > > zero.
> >
> > That should be OK. The switch external port it was received on should be
> > in the ib_wc as follows:
> >
> > struct ib_wc {
> > u64 wr_id;
> > enum ib_wc_status status;
> > enum ib_wc_opcode opcode;
> > u32 vendor_err;
> > u32 byte_len;
> > __be32 imm_data;
> > u32 qp_num;
> > u32 src_qp;
> > int wc_flags;
> > u16 pkey_index;
> > u16 slid;
> > u8 sl;
> > u8 dlid_path_bits;
> > u8 port_num; /* valid only for DR SMPs
> > on switches */
> > };
> >
> > and that is the one that needs to be used in the DR return path.
> >
> [SS] if we are supposed to get the port num from the ib_wc, in
> process_mad(0) I can do that for all SMP methods such as getportinfo()
> etc...
> But, the ib_wc parameter is null when process_mad is called from
> show_pma_counter() function in sysfs.c?
PMA uses LR (LID routed) rather than DR packets. LR packets do not fill
in port number. Note that there are both LR and DR SMPs. port_num is
only valid for DR SMPs.
> So should I be switching between the port_num parameter vs. sib_wc->port_num
> depending on whether ib_wc is NULL or not?
I think the driver needs to fill in the port_num field in the ib_wc and
this is only needed for DR SMPs (as the comment indicates).
-- Hal
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] port num in port priv
Suresh> [SS] if we are supposed to get the port num from the Suresh> ib_wc, in process_mad(0) I can do that for all SMP methods Suresh> such as getportinfo() etc... But, the ib_wc parameter is Suresh> null when process_mad is called from show_pma_counter() Suresh> function in sysfs.c? For performance management queries, you should not need to look at the physical port where the MAD was received. The port being queried is in the PortSelect field of the PortCounters query. Unfortunately the sysfs PMA counters support will not be that useful, since it will only show counters for port 0; but that's a different issue. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hal:
I am only addressing the process_mad issue here:
> > As a result, it appears like the process_mad function is always called
> with a port number of
> > zero.
>
> That should be OK. The switch external port it was received on should be
> in the ib_wc as follows:
>
> struct ib_wc {
> u64 wr_id;
> enum ib_wc_status status;
> enum ib_wc_opcode opcode;
> u32 vendor_err;
> u32 byte_len;
> __be32 imm_data;
> u32 qp_num;
> u32 src_qp;
> int wc_flags;
> u16 pkey_index;
> u16 slid;
> u8 sl;
> u8 dlid_path_bits;
> u8 port_num; /* valid only for DR SMPs
> on switches */
> };
>
> and that is the one that needs to be used in the DR return path.
>
[SS] if we are supposed to get the port num from the ib_wc, in
process_mad(0) I can do that for all SMP methods such as getportinfo()
etc...
But, the ib_wc parameter is null when process_mad is called from
show_pma_counter() function in sysfs.c?
So should I be switching between the port_num parameter vs. sib_wc->port_num
depending on whether ib_wc is NULL or not?
Thanks,
Suri
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] port num in port priv
On Thu, 2006-02-09 at 10:55, Roland Dreier wrote: > Suresh> For a switch only one port_priv object is created in > Suresh> mad.c. As a result, it appears like the process_mad > Suresh> function is always called with a port number of zero. And > Suresh> the return path is always filled with zero as well (in > Suresh> smi.c). Should not this be the physical port number from > Suresh> which the mad packet came? > > You are breaking new ground by running the Linux IB stack on top of a > switch device. Yes, this is new ground as far as I know. > The current code probably needs to be extended to give > the upper layers the port number where a MAD was received, This is supported at the WC level. I didn't trace it all the way back to see what changes if any changes are needed at the MAD layer to pass this port number to SMI, but I think that at least is the first layer of this onion. > and also to > make it possible to specify the port number that a directed route MAD > will be sent on. > Also I'm not sure whether the directed route handling will work for > MADs whose final destination is not the local switch and which > therefore need to be forwarded instead of processed. This clearly has not been tested. -- Hal > So there are quite a few things in the core that you will need to add > for your device. > > Suresh> Do I have to do something with the return path attributes > Suresh> when I send the packet out in my switch driver? I am using > Suresh> linux 2.6.12. > > I would suggest using a newer kernel. Otherwise you will be > struggling to merge your change back and forth. > > - R. > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] port num in port priv
Hi Suresh,
On Thu, 2006-02-09 at 10:40, Suresh Shelvapille wrote:
> Hi:
>
> For a switch
Note that this is some new ground for this code.
> only one port_priv object is created in mad.c.
A switch only has one port through which all MADs are processed (switch
port 0 (either base or enhanced)) so I think this is correct.
> As a result, it appears like the process_mad function is always called with
> a port number of
> zero.
That should be OK. The switch external port it was received on should be
in the ib_wc as follows:
struct ib_wc {
u64 wr_id;
enum ib_wc_status status;
enum ib_wc_opcode opcode;
u32 vendor_err;
u32 byte_len;
__be32 imm_data;
u32 qp_num;
u32 src_qp;
int wc_flags;
u16 pkey_index;
u16 slid;
u8 sl;
u8 dlid_path_bits;
u8 port_num; /* valid only for DR SMPs on
switches */
};
and that is the one that needs to be used in the DR return path.
> And the return path is always filled with zero as well (in smi.c).
For switches, the one from the WC needs to be filled in and passed so
that sounds wrong and needs fixing. Do you want to take a crack at this
or should I ?
> Should
> not this be the physical port number from which the mad packet came?
Yes.
> Do I have to do something with the return path attributes when I send the
> packet out in my switch driver? I am using linux 2.6.12.
There haven't been any changes here since then that would affect this.
-- Hal
> Am I not reading something right?
>
> Thanks,
> Suri
>
>
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] port num in port priv
Suresh> For a switch only one port_priv object is created in Suresh> mad.c. As a result, it appears like the process_mad Suresh> function is always called with a port number of zero. And Suresh> the return path is always filled with zero as well (in Suresh> smi.c). Should not this be the physical port number from Suresh> which the mad packet came? You are breaking new ground by running the Linux IB stack on top of a switch device. The current code probably needs to be extended to give the upper layers the port number where a MAD was received, and also to make it possible to specify the port number that a directed route MAD will be sent on. Also I'm not sure whether the directed route handling will work for MADs whose final destination is not the local switch and which therefore need to be forwarded instead of processed. So there are quite a few things in the core that you will need to add for your device. Suresh> Do I have to do something with the return path attributes Suresh> when I send the packet out in my switch driver? I am using Suresh> linux 2.6.12. I would suggest using a newer kernel. Otherwise you will be struggling to merge your change back and forth. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
