Re: [openib-general] patches to 2.6.19.1 kernel for switch Operation
Hal: Ref: comment on mad.c (ib_mad_recv_done_handler(). Even if I make the relevant changes to smi.c functions how do I get the packet to get forwarded, without making additional changes in this function? Meaning, when smi_handle_dr_smp_send(),smi_check_forward_dr_smp() are called and you determine that the packet has to be forwarded instead of consuming where do you actually do the send? I think this chain is missing! My initial thought was what I wrote but in looking at this further, as you point out, the SMI routines are only updating the packet and indicating its disposition. The actual sending needs to be elsewhere. I'm not sure what the code ends up looking like with the changes suggested and would just like this to look as clean as possible and use the SMI routines where appropriate here. Does this make sense ? I am not sure I follow this last statement. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] patches to 2.6.19.1 kernel for switch Operation
Hal: We are upgrading to 2.6.19.1 kernel and I finally ported the changes required for Switch operation from my current kernel (2.6.12) version. I have tested these changes for a switch with different SM(s). But I need the community's help to test the changes on different HCAs to make sure I have not broken anything. Please see if the changes look OK. Thanks, Suri smi.c.ptch Description: Binary data agent.c.ptch Description: Binary data mad.c.ptch Description: Binary data ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Utilities for sending traffic with different SL
Folks: Is there a utility within the OFED1.0 package which can be used for generating traffic on different SL (akin to the Voltaire perf_main utility)? Many thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Perf_main for RDMA
Folks: Has any one run perf_main(RDMA) tests simulating high latency networks? I am trying to simulate RDMA over WAN and looking for some benchmarks to compare with. Any pointers to links will be great. Many thanks in advance, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCH 2/3] SA: add call to initializeib_ah_attr from a path record
Jason: This might work in this case. But, if you look at RFC4391(section 4.1 Broadcast GID params, item 4. other parameters) : The broadcast-GID's scope bits need to be set based on whether the IPoIB link is confined within an IB subnet or the IPoIB link spans multiple IB subnets. A default of local-subnet scope (i.e., 0x2) is RECOMMENDED. A node might determine the scope bits to use by interactively searching for a broadcast- GID of ever greater scope by first starting with the local- scope. Or, an implementation might include the scope bits as a configuration parameter How can a node determine the scope param in Roland's crazy setup(when the scope bits are not a config param)? Thanks, Suri -Original Message- From: [EMAIL PROTECTED] [mailto:openib-general- [EMAIL PROTECTED] On Behalf Of Jason Gunthorpe Sent: Thursday, May 18, 2006 11:10 PM To: Sean Hefty Cc: 'Roland Dreier'; openib-general@openib.org Subject: Re: [openib-general] [PATCH 2/3] SA: add call to initializeib_ah_attr from a path record On Thu, May 18, 2006 at 04:24:16PM -0700, Sean Hefty wrote: I should note that I compared the subnet prefixes to determine if the GRH should be used. Reading back over the 'GRH flag in ib_ah_attr' thread, it looks like there's consensus that hop_limit 1 is the check that we want. I will update the code accordingly. I think it is also prudent to not use a GRH if the DGID's prefix is fe80::/64 (link local scope). Jason ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: [RFC] [PATCH 1/2] multicast support formultiple users
Sean/Roland:
I may be the only one using the openib stack for a switch. I don't need the
module in question at this point, but I request the group to consider
putting all changes for a switch (as well) while adding new functionality.
This way, if and when I get to a point of needing the functionality, I would
have something to start with! And I promise to test and send in changes (if
required).
BTW, as far as the ib_mad/smi stuff goes, I have sent my changes to Hal,
since I am running 2.6.12 and it is best if he does the merge (I think he
said so already in his email).
Thanks a lot,
Suri
-Original Message-
From: [EMAIL PROTECTED] [mailto:openib-general-
[EMAIL PROTECTED] On Behalf Of Sean Hefty
Sent: Thursday, April 06, 2006 4:17 PM
To: Roland Dreier
Cc: openib-general@openib.org
Subject: Re: [openib-general] Re: [RFC] [PATCH 1/2] multicast support
formultiple users
Roland Dreier wrote:
+dev = kmalloc(sizeof *dev + device-phys_port_cnt * sizeof
*port,
+ GFP_KERNEL);
+if (!dev)
+return;
+
+for (i = 1; i = device-phys_port_cnt; i++) {
Seems like this is implicitly assuming that the IB device is a CA.
Maybe we should give up the ghost and stop trying to support IB
switches?
I can go either way here. Would someone want to use this module on a
switch?
If so, I can fix up the code so that it can work on one, or at least
change the
checks that the device is a CA.
I thought that someone was running at least ib_mad on a switch. I'm just
not
sure which other modules are likely to run on them.
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-
general
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Open IB stack for Switch
Folks: I am happy to announce that the Switch implementation of OpenIB stack is working very well. I have tested this with Voltaire and Silver Storm SM. Of course, this is a very limited implementation in that I am only using the core-stack and nothing else. Also, I have changed mad.c and agent.c to make this work, and I will send it to Hal as he made the initial changes. Unfortunately, this is all on 2.6.12, and I am not sure when we will move to a newer kernel. Many thanks to this group! Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ping delay
Folks: Two commercial switches are connected to each other via two of our devices (which act like switches). One HCA is connected to each of the commercial switches. Upon power up, everything is normal and when I ping one HCA from the other everything is fine. When everything is operational, if I move one of the HCAs from one switch port to another on the same switch, then the pings start to take an enormous amount of time When a port move happens, is there some SMP sent from the SM(the SM is on the commercial switch) which we may not be handling in our device, which could be causing this delay? Thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH] mad.c: Add support for switch SMI
Hal: From a switch perspective, and from the little testing that I have done with different SM, I don't know how I can get out port num in the query AH. Because, from what I have observed, the out port num is different depending on whether the switch is in the middle or at the end of the query and hence one may have to compute this info from either return_path, or init_path or from the input port number(physical). Pardon my ignorance, but will I have access to the MAD packet in query_ah? Thanks a lot, Suri 2. On the send side, the driver must support the optional query_ah verb in order to obtain the send side port number (actual switch external port on which to send the DR SMP). ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hal: The way I gather port stats locally on my device is by doing a cat on the /sys/class/Infiniband/mysw/port/0/counters file. As Roland mentioned this is limited to port0 counters on a switch. If I need stats for port#N I need to expand this by providing more class attributes and getting to them that way I am not using user space verbs library, is there any other way to get at the different port#N stats? Thanks, Suri The PMA counters are gathered by a tool named perfquery. Port number is in the PortSelect component of the PMA attribute. You can't get PMA counters until the port is active. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] port num in port priv
Hi: For a switch only one port_priv object is created in mad.c. As a result, it appears like the process_mad function is always called with a port number of zero. And the return path is always filled with zero as well (in smi.c). Should not this be the physical port number from which the mad packet came? Do I have to do something with the return path attributes when I send the packet out in my switch driver? I am using linux 2.6.12. Am I not reading something right? Thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hal:
I am only addressing the process_mad issue here:
As a result, it appears like the process_mad function is always called
with a port number of
zero.
That should be OK. The switch external port it was received on should be
in the ib_wc as follows:
struct ib_wc {
u64 wr_id;
enum ib_wc_status status;
enum ib_wc_opcode opcode;
u32 vendor_err;
u32 byte_len;
__be32 imm_data;
u32 qp_num;
u32 src_qp;
int wc_flags;
u16 pkey_index;
u16 slid;
u8 sl;
u8 dlid_path_bits;
u8 port_num; /* valid only for DR SMPs
on switches */
};
and that is the one that needs to be used in the DR return path.
[SS] if we are supposed to get the port num from the ib_wc, in
process_mad(0) I can do that for all SMP methods such as getportinfo()
etc...
But, the ib_wc parameter is null when process_mad is called from
show_pma_counter() function in sysfs.c?
So should I be switching between the port_num parameter vs. sib_wc-port_num
depending on whether ib_wc is NULL or not?
Thanks,
Suri
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Unfortunately the sysfs PMA counters support will not be that useful, since it will only show counters for port 0; but that's a different issue. - R. [SS] That's a bummer, so don't we need to create as many file descriptors as the number of physical ports on a switch, so that we can gather different port related stats from the device locally (not talking about subnet manager queries)! Thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] port num in port priv
Hal and folks: For switches, the one from the WC needs to be filled in and passed so that sounds wrong and needs fixing. Do you want to take a crack at this or should I ? [SS] I am not as IB core stack savvy as you guys are, so I would appreciate it if you can provide the changes...I will be happy to review and test it. Right now, I have kludged my sw driver to handle DR SMPs to make the subnet manager happy. Thanks a lot, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] IB traffic generators
Folks: can you please point me to some traffic generators out there. Thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] initialization -udev
Folks: Another basic question, I just built my 2.6.10 kernel with core Infiniband stack. I only have CONFIG_INFINIBAND=y and nothing else (I don't need anything else for now!). Couple of questions: 1. Do I need to load any of the kernel modules by hand? I don't have etc/udev directory (none of the modules listed in MST's installation cheat sheet are applicable to me I think!) 2. What should I see to confirm that the core Infiniband stack is loaded correctly? Thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] initialization -udev
1. I did not add the CONFIG_INFINIBAND=y by hand. I used make menuconfig and the GUI let me choose y to select the Infiniband core. And the result was, in .config which I presume is the one generated by the GUI has CONIFG_INFINIBAND=y. From what you say, the ib_core and ib_mad modules are not loaded if CONFIG_INFINIBAND=y and not 'm'? 2. When I do /sbin/lsmod --I see nothing though 3. BTW I have not loaded my device driver yet...I will try it once the core modules get loaded correctly! Thanks a lot in advance, Suri -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Monday, October 17, 2005 4:42 PM To: Suresh Shelvapille Cc: openib-general@openib.org Subject: Re: [openib-general] initialization -udev Hi Suri, On Mon, 2005-10-17 at 16:15, Suresh Shelvapille wrote: Folks: Another basic question, I just built my 2.6.10 kernel with core Infiniband stack. I only have CONFIG_INFINIBAND=y and nothing else (I don't need anything else for now!). If you want modules, set it to m not y. Couple of questions: 1. Do I need to load any of the kernel modules by hand? I don't have etc/udev directory (none of the modules listed in MST's installation cheat sheet are applicable to me I think!) 2. What should I see to confirm that the core Infiniband stack is loaded correctly? You should do an /sbin/lsmod | grep ib_ and see that ib_mthca (or whatever your driver is), ib_mad, and ib_core are loaded. I think that's allz you need right now. -- Hal Thanks, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] rdma/ib_verbs.h
Folks: While writing a switch driver, I noticed that the alloc_pd and create_cq function signatures are different depending on rdma/ib_verbs.h vs. ib_verbs.h. I don't need RDMA for now, so going with the func signature as in ib_verbs.h is OK or is there a necessity to use rdma/ib_verbs.h? Thanks in advance. Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] rdma/ib_verbs.h
I am not sure when I SVNed but, I have the same files under both include/ and include/rdma in my copy of the Infiniband tree!! So, there should not be any header files under 'include', but they should all be under include/rdma? Please clarify...thanks, Suri -Original Message- From: Roland Dreier [mailto:[EMAIL PROTECTED] Sent: Friday, October 14, 2005 5:30 PM To: Suresh Shelvapille Cc: openib-general@openib.org Subject: Re: [openib-general] rdma/ib_verbs.h Suresh Folks: While writing a switch driver, I noticed that the Suresh alloc_pd and create_cq function signatures are different Suresh depending on rdma/ib_verbs.h vs. ib_verbs.h. I don't need Suresh RDMA for now, so going with the func signature as in Suresh ib_verbs.h is OK or is there a necessity to use Suresh rdma/ib_verbs.h? Which two files are you comparing? There should only be one ib_verbs.h file in your kernel tree. Otherwise you're just asking for trouble. You may be getting confused because the include files were moved from drivers/infiniband/ to include/rdma/ between kernel version 2.6.13 and 2.6.14-rc1. However the files were just moved. You should just work against the latest version of ib_verbs.h when writing your driver. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
FW: [openib-general] drivers.diff patch
Just for the benefit of the community, I am forwarding a mail in which Hal already answered my question (I forgot to hit reply all...). -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 27, 2005 12:27 PM To: Suresh Shelvapille Subject: RE: [openib-general] drivers.diff patch Hi Suresh, On Tue, 2005-09-27 at 12:14, Suresh Shelvapille wrote: Thanks for your help. It is really painful to get this going. If I applied any of the patches from branches/backport-to-2.6.9 the kernel would not built. So I managed to look at the changes needed for drivers/kconfig and drivers/Makefile in the kernel-patch diff file and applied it, so that the menu showed the infiniband. And then applied a few patches from the branches/backport/2.6.9. Now the Kernel builds with the CONFIG_INFINIBAND=y and CONFIG_INFININBAND_USER_MAD=y. If I have anything else set in the CONFIG the build fails. As I said before I am afraid to take any of the patches from the branches/backport-to-2.6.9 as even the core modules won't build. Not sure what is broke. Question is: Our system (at least in the initial release) will act like a bear bones ib switch, That configuration (run as a switch; most everyone has run as HCA) is untested so this may need a tweak or two. What I'm referring to is not build related though. and I am hoping that only MAD packet support would be needed within the kernel. Are the above two CONFIG_INFINIBAND and CONFIG_INFININBAND_USER_MAD sufficient? If you want to run OpenSM and/or the management diagnostics, yes. Otherwise, you won't need user MAD configured. You will need some switch driver which I presume you would be adding locally. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Generic device driver
Another newbee question. I only see a Mellanox HCA driver in the gen2/trunk, which is a PCI device. Where as, we have a proprietary implementation of the HCA, where the PHY is accessed by the host via an FPGA (Host-bridge). 1. Is there a Generic Device Driver (I see it being mentioned in many places). 2. If none, can I replace some files/functions within the Mellanox driver with the low level addresses etc, to achieve the same. Thanks in advance for the advice, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Support for Linux 2.6.10
Hi: I am very new to Infiniband. I see from the FAQ that the IB stack has been integrated into 2.6.11 Linux kernel. I also see some patches for 2.6.9. Unfortunately we are using a 2.6.10 kernel. Is there a patch for 2.6.10? Thanks a lot, Suri ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
