Re: [openib-general] IPoIB Question
Is this with a combination of TCP and UDP or just TCP? Sean Scott Weitzenkamp (sweitzen) wrote: > We see 3.6 Gb/sec with IPoIB using RHEL4U4 2.6.9-42 x86_64 kernel on > Dell PE1950 Woodcrest systems. > > In my testing, faster hardware is more important than newer kernels, but > I don't try newer kernels much. > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
We see 3.6 Gb/sec with IPoIB using RHEL4U4 2.6.9-42 x86_64 kernel on Dell PE1950 Woodcrest systems. In my testing, faster hardware is more important than newer kernels, but I don't try newer kernels much. Scott Weitzenkamp SQA and Release Manager Server Virtualization Business Unit Cisco Systems > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Greg Lindahl > Sent: Tuesday, October 24, 2006 1:16 PM > To: Sean Hubbell > Cc: openib-general@openib.org > Subject: Re: [openib-general] IPoIB Question > > On Tue, Oct 24, 2006 at 08:35:18AM -0500, Sean Hubbell wrote: > > > We are currently looking at the new tickless kernel. Do you > have one > > that you recommend? > > The main one to less-recommend is 2.6.9-based kernels, those are the > slowest at TCP. Modern kernels, like the ones you see in Fedora 4 and > up and SLES 10, seem to all be good and about equal in this area. > > I don't think we've tried a tickless kernel. We do most of our testing > on the various kernels that ship with distros, plus the tip-of-tree > kernel.org kernel. > > -- greg > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
On Tue, Oct 24, 2006 at 08:35:18AM -0500, Sean Hubbell wrote: > We are currently looking at the new tickless kernel. Do you have one > that you recommend? The main one to less-recommend is 2.6.9-based kernels, those are the slowest at TCP. Modern kernels, like the ones you see in Fedora 4 and up and SLES 10, seem to all be good and about equal in this area. I don't think we've tried a tickless kernel. We do most of our testing on the various kernels that ship with distros, plus the tip-of-tree kernel.org kernel. -- greg ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
At 10:00 PM 10/23/2006, Greg Lindahl wrote: >On Mon, Oct 23, 2006 at 07:53:06AM -0500, Hubbell, Sean C >Contractor/Decibel wrote: > > > I currently have several applications that uses a legacy IPv4 protocol > > and I use IPoIB to utilize my infiniband network which works great. I > > have completed some timing and throughput analysis and noticed that I do > > not get very much more if I use an infiniband network interface than > > using my GigE network interface. > >You might want to note that different InfinBand implementations have >quite different performance of IPoIB, especially for UDP. > >Another issue is that IPoIB has quite different performance with >different Linux kernels. This is especially evident for TCP, although >you can use SDP to accelerate TCP sockets and avoid this issue. > > > My question is, am I using IPoIB correctly or are these the typical > > numbers that everyone is seeing? > >It is certainly the case that there are some message patterns and >situations for which InfiniBand is not much of an improvement over >gigE. Unfortunately, the comparison of IB to GbE are often apple-to-orange comparisons even for IP over IB. Until a HCA supplies the same level of functional off-load enabled by the IP network stack that is used with Ethernet, it really isn't a fair comparison. The same is also true for many of the marketroids and their comparisons of IB to Ethernet based solutions. Fortunately, most customers are getting a bit smarter and not falling for the marketing drivel these days - certainly the OEM don't fall for it thought the marketroids continue to come in and try to convince people it isn't an apple-to-orange comparison.The fact is both technologies have their pros / cons and it is really the workload or production environment that determines which is the best fit instead of the force fit. In any case, not really a development issue so will drop further discussion. Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Greg Lindahl wrote: > On Mon, Oct 23, 2006 at 07:53:06AM -0500, Hubbell, Sean C Contractor/Decibel > wrote: > > >> I currently have several applications that uses a legacy IPv4 protocol >> and I use IPoIB to utilize my infiniband network which works great. I >> have completed some timing and throughput analysis and noticed that I do >> not get very much more if I use an infiniband network interface than >> using my GigE network interface. >> > > You might want to note that different InfinBand implementations have > quite different performance of IPoIB, especially for UDP. > > Another issue is that IPoIB has quite different performance with > different Linux kernels. This is especially evident for TCP, although > you can use SDP to accelerate TCP sockets and avoid this issue. > > We are currently looking at the new tickless kernel. Do you have one that you recommend? Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
On Mon, Oct 23, 2006 at 07:53:06AM -0500, Hubbell, Sean C Contractor/Decibel wrote: > I currently have several applications that uses a legacy IPv4 protocol > and I use IPoIB to utilize my infiniband network which works great. I > have completed some timing and throughput analysis and noticed that I do > not get very much more if I use an infiniband network interface than > using my GigE network interface. You might want to note that different InfinBand implementations have quite different performance of IPoIB, especially for UDP. Another issue is that IPoIB has quite different performance with different Linux kernels. This is especially evident for TCP, although you can use SDP to accelerate TCP sockets and avoid this issue. > My question is, am I using IPoIB correctly or are these the typical > numbers that everyone is seeing? It is certainly the case that there are some message patterns and situations for which InfiniBand is not much of an improvement over gigE. -- greg ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
At 10:59 AM 10/23/2006, Sean Hubbell wrote: >Thanks Michael I looked at iperf and that looks like a very nice tool. Something else about Iperf is, that it supports multiple streams. Which maybe closer to the way some apps operate. * Correspondence * This email contains no programmatic content that requires independent ADC review ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Perfect, I'll check with my vendor to see if this is possible. If so, this rocks! Thanks! Sean -- Original message -- Date: Mon, 23 Oct 2006 11:04:40 -0700 From: Michael Krause <[EMAIL PROTECTED]> Reply-To: Michael Krause <[EMAIL PROTECTED]> To: Michael S. Tsirkin <[EMAIL PROTECTED]>, Sean Hubbell <[EMAIL PROTECTED]> CC: openib-general@openib.org Subject: Re: [openib-general] IPoIB Question At 10:19 AM 10/23/2006, Michael S. Tsirkin wrote: >Quoting r. Sean Hubbell <[EMAIL PROTECTED]>: > > I am looking at libsdp for the TCP funcationality and wanted to know if > > libsdp supports UDP as well > >AFAIK, SDP can only emulate TCP sockets. SDP is defined to work with AF_INET applications. If using a shared library approach / pre-load, one can transparently enable any AF_INET application to utilize SDP without a recompile, etc. The SDP Port Mapper specification for iWARP / service id for IB enable the connection management or whatever service it is implemented within to application-transparent discover the real target listen port and establish a SDP session nominally during connection establishment.Implementations may vary in the robustness or policies used to determine what to off-load, number of off-load sessions, etc. - in other words, a lot of opportunity and flexibility is provided to use SDP. Note: WinSocks Direct on Windows provides an equivalent service though uses a proprietary protocol. Vista will have SDP as defined in the specifications. There are currently no plans to develop an equivalent for datagram applications. Any datagram application (user or kernel) can already access the hardware directly and given RDMA is not defined for datagram, it was felt such a specification would provide minimal value. Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
At 10:19 AM 10/23/2006, Michael S. Tsirkin wrote: >Quoting r. Sean Hubbell <[EMAIL PROTECTED]>: > > I am looking at libsdp for the TCP funcationality and wanted to know if > > libsdp supports UDP as well > >AFAIK, SDP can only emulate TCP sockets. SDP is defined to work with AF_INET applications. If using a shared library approach / pre-load, one can transparently enable any AF_INET application to utilize SDP without a recompile, etc. The SDP Port Mapper specification for iWARP / service id for IB enable the connection management or whatever service it is implemented within to application-transparent discover the real target listen port and establish a SDP session nominally during connection establishment.Implementations may vary in the robustness or policies used to determine what to off-load, number of off-load sessions, etc. - in other words, a lot of opportunity and flexibility is provided to use SDP. Note: WinSocks Direct on Windows provides an equivalent service though uses a proprietary protocol. Vista will have SDP as defined in the specifications. There are currently no plans to develop an equivalent for datagram applications. Any datagram application (user or kernel) can already access the hardware directly and given RDMA is not defined for datagram, it was felt such a specification would provide minimal value. Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Quoting r. Sean Hubbell <[EMAIL PROTECTED]>: > I am looking at libsdp for the TCP funcationality and wanted to know if > libsdp supports UDP as well AFAIK, SDP can only emulate TCP sockets. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Thanks Michael I looked at iperf and that looks like a very nice tool. I will be using that when I evaluate and check performance of my applications. I am also interested in getting more bandwidth out of my applications leveraging a current or planned capability for IPoIB. This way, I will not have to modify my source code and I can just actually change out the interfaces that my applications send and receive on. So, I am looking at libsdp for the TCP funcationality and wanted to know if libsdp supports UDP as well or is there another library that I can use to maximize the bandwidth when transmitting and sending over infiniband? Sean Michael S. Tsirkin wrote: > Quoting r. Sean Hubbell <[EMAIL PROTECTED]>: > >> Subject: Re: IPoIB Question >> >> Scott, >> >> Thanks for the reply again. The third party api that we use leverages a >> combination of UDP and TCP socket conntections for speed. Is there >> something for UCP as well? >> > > iperf supports UDP as well. Again, check out the -P flag. > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Nothing today in OF to accelerate UDP sockets. Scott > Thanks for the reply again. The third party api that we use > leverages a > combination of UDP and TCP socket conntections for speed. Is there > something for UCP as well? > > Sean > > Scott Weitzenkamp (sweitzen) wrote: > > If you are using TCP, you can use SDP transparently via > libsdp to get > > improved latency and throughput. > > > > Scott ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Quoting r. Sean Hubbell <[EMAIL PROTECTED]>: > Subject: Re: IPoIB Question > > Scott, > > Thanks for the reply again. The third party api that we use leverages a > combination of UDP and TCP socket conntections for speed. Is there > something for UCP as well? iperf supports UDP as well. Again, check out the -P flag. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Scott, Thanks for the reply again. The third party api that we use leverages a combination of UDP and TCP socket conntections for speed. Is there something for UCP as well? Sean Scott Weitzenkamp (sweitzen) wrote: > If you are using TCP, you can use SDP transparently via libsdp to get > improved latency and throughput. > > Scott > > >> -Original Message- >> From: Sean Hubbell [mailto:[EMAIL PROTECTED] >> Sent: Monday, October 23, 2006 8:56 AM >> To: Scott Weitzenkamp (sweitzen) >> Cc: openib-general@openib.org >> Subject: Re: [openib-general] IPoIB Question >> >> We currently have a non-homogeneous cluster so that seems that would >> possible explain a few of the differences that I have seen on >> some of my >> tests. I will look at netperf.org and see what they have to offer. >> >> On another note, is there plans to have IPoIB support the full >> throughput that infiniband 4x or 12x has? Specifically, can I keep my >> legacy apps and just upgrade the network to take advantage of >> the bandwidth? >> >> Sean >> >> Scott Weitzenkamp (sweitzen) wrote: >> >>> IPoIB performance will vary quite a bit depending on what >>> >> motherboard, >> >>> CPU speed, and HCA type you have. What are the specs on >>> >> the systems >> >>> you are using? >>> >>> Netperf (www.netperf.org <http://www.netperf.org>) is a >>> >> good tool to >> >>> measure IPoIB performance. >>> >>> Scott Weitzenkamp >>> SQA and Release Manager >>> Server Virtualization Business Unit >>> Cisco Systems >>> >>> >>> >>> >> ---------- >> -- >> >>> *From:* [EMAIL PROTECTED] >>> [mailto:[EMAIL PROTECTED] *On Behalf >>> >> Of *Hubbell, >> >>> Sean C Contractor/Decibel >>> *Sent:* Monday, October 23, 2006 5:53 AM >>> *To:* openib-general@openib.org >>> *Cc:* Sean Hubbell >>> *Subject:* [openib-general] IPoIB Question >>> >>> Hello, >>> >>> I currently have several applications that uses a legacy IPv4 >>> protocol and I use IPoIB to utilize my infiniband network which >>> works great. I have completed some timing and >>> >> throughput analysis >> >>> and noticed that I do not get very much more if I use an >>> infiniband network interface than using my GigE network >>> >> interface. >> >>> My question is, am I using IPoIB correctly or are these the >>> typical numbers that everyone is seeing? Is there a standard >>> application that I may use to test my current configuration? >>> >>> Thanks in advance, >>> >>> Sean >>> >>> > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
If you are using TCP, you can use SDP transparently via libsdp to get improved latency and throughput. Scott > -Original Message- > From: Sean Hubbell [mailto:[EMAIL PROTECTED] > Sent: Monday, October 23, 2006 8:56 AM > To: Scott Weitzenkamp (sweitzen) > Cc: openib-general@openib.org > Subject: Re: [openib-general] IPoIB Question > > We currently have a non-homogeneous cluster so that seems that would > possible explain a few of the differences that I have seen on > some of my > tests. I will look at netperf.org and see what they have to offer. > > On another note, is there plans to have IPoIB support the full > throughput that infiniband 4x or 12x has? Specifically, can I keep my > legacy apps and just upgrade the network to take advantage of > the bandwidth? > > Sean > > Scott Weitzenkamp (sweitzen) wrote: > > IPoIB performance will vary quite a bit depending on what > motherboard, > > CPU speed, and HCA type you have. What are the specs on > the systems > > you are using? > > > > Netperf (www.netperf.org <http://www.netperf.org>) is a > good tool to > > measure IPoIB performance. > > > > Scott Weitzenkamp > > SQA and Release Manager > > Server Virtualization Business Unit > > Cisco Systems > > > > > > > -- > -- > > *From:* [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] *On Behalf > Of *Hubbell, > > Sean C Contractor/Decibel > > *Sent:* Monday, October 23, 2006 5:53 AM > > *To:* openib-general@openib.org > > *Cc:* Sean Hubbell > > *Subject:* [openib-general] IPoIB Question > > > > Hello, > > > > I currently have several applications that uses a legacy IPv4 > > protocol and I use IPoIB to utilize my infiniband network which > > works great. I have completed some timing and > throughput analysis > > and noticed that I do not get very much more if I use an > > infiniband network interface than using my GigE network > interface. > > My question is, am I using IPoIB correctly or are these the > > typical numbers that everyone is seeing? Is there a standard > > application that I may use to test my current configuration? > > > > Thanks in advance, > > > > Sean > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
We currently have a non-homogeneous cluster so that seems that would possible explain a few of the differences that I have seen on some of my tests. I will look at netperf.org and see what they have to offer. On another note, is there plans to have IPoIB support the full throughput that infiniband 4x or 12x has? Specifically, can I keep my legacy apps and just upgrade the network to take advantage of the bandwidth? Sean Scott Weitzenkamp (sweitzen) wrote: > IPoIB performance will vary quite a bit depending on what motherboard, > CPU speed, and HCA type you have. What are the specs on the systems > you are using? > > Netperf (www.netperf.org <http://www.netperf.org>) is a good tool to > measure IPoIB performance. > > Scott Weitzenkamp > SQA and Release Manager > Server Virtualization Business Unit > Cisco Systems > > > > *From:* [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] *On Behalf Of *Hubbell, > Sean C Contractor/Decibel > *Sent:* Monday, October 23, 2006 5:53 AM > *To:* openib-general@openib.org > *Cc:* Sean Hubbell > *Subject:* [openib-general] IPoIB Question > > Hello, > > I currently have several applications that uses a legacy IPv4 > protocol and I use IPoIB to utilize my infiniband network which > works great. I have completed some timing and throughput analysis > and noticed that I do not get very much more if I use an > infiniband network interface than using my GigE network interface. > My question is, am I using IPoIB correctly or are these the > typical numbers that everyone is seeing? Is there a standard > application that I may use to test my current configuration? > > Thanks in advance, > > Sean > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Quoting r. Scott Weitzenkamp (sweitzen) <[EMAIL PROTECTED]>: > Netperf (www.netperf.org) is a good tool to measure IPoIB performance. Of special note is the -T flag which often lets you get more consistent results by pinning the test to a single CPU. Another useful tool is iperf, which has a -P option for running multiple socket tests in parallel. In TCP, multi-socket performance often exceeds that of a single socket. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB Question
Title: IPoIB Question IPoIB performance will vary quite a bit depending on what motherboard, CPU speed, and HCA type you have. What are the specs on the systems you are using? Netperf (www.netperf.org) is a good tool to measure IPoIB performance. Scott Weitzenkamp SQA and Release Manager Server Virtualization Business Unit Cisco Systems From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hubbell, Sean C Contractor/DecibelSent: Monday, October 23, 2006 5:53 AMTo: openib-general@openib.orgCc: Sean HubbellSubject: [openib-general] IPoIB Question Hello, I currently have several applications that uses a legacy IPv4 protocol and I use IPoIB to utilize my infiniband network which works great. I have completed some timing and throughput analysis and noticed that I do not get very much more if I use an infiniband network interface than using my GigE network interface. My question is, am I using IPoIB correctly or are these the typical numbers that everyone is seeing? Is there a standard application that I may use to test my current configuration? Thanks in advance, Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] IPoIB Question
Title: IPoIB Question Hello, I currently have several applications that uses a legacy IPv4 protocol and I use IPoIB to utilize my infiniband network which works great. I have completed some timing and throughput analysis and noticed that I do not get very much more if I use an infiniband network interface than using my GigE network interface. My question is, am I using IPoIB correctly or are these the typical numbers that everyone is seeing? Is there a standard application that I may use to test my current configuration? Thanks in advance, Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] ipoib question when running on the same node as opensm
Quoting r. Ira Weiny <[EMAIL PROTECTED]>: > Do I need to ensure that opensm is up before all > ipoib requests in the future? Shouldn't be required, thing work well for me, anyway. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ipoib question when running on the same node as opensm
We just brought another cluster up and had an issue with our management node (node running opensm) not coming up on ipoib. Here is what happened and how I got it working and I had some questions. 1) We had both opensm running and a switch based Voltaire SM running. This caused problems. 2) We stopped the Voltaire SM and restarted all the nodes. This got all of the nodes except the one with opensm running to work. 3) I had to unload all the modules, load only those needed by opensm, start opensm, and then bring up the ipoib interface. At this point the node seemed to be in the multicast group and ipoib worked fine. Does this seem like proper behavior? I would think that on boot if ipoib does not find a SM running it will delay setting up a connection until the SM comes on-line? (ie when the opensm init script gets run.) It seems like the card saves some information (from the Voltaire SM) across a soft reboot? I know that it was not coming up in the multicast group with the opensm. Is this by design? At this point ipoib seems to work fine after a reboot even though the interface is brought up before opensm. Do I need to ensure that opensm is up before all ipoib requests in the future? Thanks, Ira Weiny [EMAIL PROTECTED] ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ipoib: question
Roland, where exactly does the following math come from?
static inline struct ipoib_neigh **to_ipoib_neigh(struct neighbour *neigh)
{
return (struct ipoib_neigh **) (neigh->ha + 24 -
(offsetof(struct neighbour, ha) & 4));
}
1. What does & 4 do here?
2. Why are we subsrctucting a function of ha offset?
4. What is 24? Is it related to INFINIBAND_ALEN?
Thanks,
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] IPoIB question/problem
Hello, Roland!
While debugging a (gen1) problem with IPoIB,
I have noticed the following code in function neigh_update:
net/core/neighbour.c:1015
if (lladdr != neigh->ha) {
memcpy(&neigh->ha, lladdr, dev->addr_len);
neigh_update_hhs(neigh);
if (!(new & NUD_CONNECTED))
neigh->confirmed = jiffies -
(neigh->parms->base_reachable_time << 1);
#ifdef CONFIG_ARPD
notify = 1;
#endif
}
It appears, therefore, that the neighbour ha field may get updated
without destroying the neighbour.
Assuming that a remote node is replaced and its address changes
(e.g. gid change), it seems that the ha field will gets out of sync with
the address handle stored in ipoib_neigh->ah, with the result that the
ah field would point to an incorrect path, resulting in all packets being lost.
Does this analysis make sense? If yes, what would be the best way to fix this?
Thanks,
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IPoIB question
On Tue, 2005-09-27 at 04:11, Abhijit Gadgil wrote: > Hi All, > > I am new to IPoIB. I have a query, as per the IPoIB Architecture > document, whenever an IPoIB interface is brought up, it needs to do a > Full Member Join to the "broadcast" Multicast group. Where exactly in > the code, is this taking place? I have been able to trace a little bit - > eg. in ipoib_add_port() there is a call to ipoib_intf_alloc() which in > turn creates a Work Queue for the ipoib_mcast_restart_task(). In this > task, subsequently there is _ipoib_mcast_join() and so on where it > finally reaches ipoib_mcast_attach() (in ipoib_verbs.c). What is not > clear at this point is, Why is it looking for cached PKey? It needs the PKey as this is part of the MGID to join (for the broadcasy group and other multicast groups). > Is it not > something that needs to be sent by the SM? Yes. > Further, I am putting SM in testability 'debug' mode (DEBUG=10 in > /etc/opensm.conf), however I am still not seeing any dump of messages > about FullMember join whenever I try restarting the IB interfaces. What > should be log-level to put SM to dump those messages? Already answered by Eitan. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] IPoIB question
Title: RE: [openib-general] IPoIB question > Further, I am putting SM in testability 'debug' mode (DEBUG=10 in > /etc/opensm.conf), however I am still not seeing any dump of messages > about FullMember join whenever I try restarting the IB interfaces. What > should be log-level to put SM to dump those messages? [EZ] Seems you are using a Gen1 Mellanox distribution (IBGD). You can use OpenIB OpenSM too. If still want to use IBGD OpenSM you need to use DEBUG="-V" instead of DEBUG=10 . If you want to use OpenIB OpenSM you will need to run : opensm -V EZ ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] IPoIB question
Hi All, I am new to IPoIB. I have a query, as per the IPoIB Architecture document, whenever an IPoIB interface is brought up, it needs to do a Full Member Join to the "broadcast" Multicast group. Where exactly in the code, is this taking place? I have been able to trace a little bit - eg. in ipoib_add_port() there is a call to ipoib_intf_alloc() which in turn creates a Work Queue for the ipoib_mcast_restart_task(). In this task, subsequently there is _ipoib_mcast_join() and so on where it finally reaches ipoib_mcast_attach() (in ipoib_verbs.c). What is not clear at this point is, Why is it looking for cached PKey? Is it not something that needs to be sent by the SM? Further, I am putting SM in testability 'debug' mode (DEBUG=10 in /etc/opensm.conf), however I am still not seeing any dump of messages about FullMember join whenever I try restarting the IB interfaces. What should be log-level to put SM to dump those messages? Thanks in advance. Regards. -abhijit ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
