Re: [openib-general] InfiniBand incompatible with the Linux kernel?

[Troy Benjegerdes]

Well, fortunately this has turned out to be a non-issue.
I just went to www.infinibandta.org and the 1.2 spec is available for 
download.

http://www.infinibandta.org/specs/register/publicspec/vol1r1_2.zip
http://www.infinibandta.org/specs/register/publicspec/vol2r1_2.zip
Roland Dreier wrote:
   Roland> it's orthogonal to any IP issues.  Since the Linux kernel
   Roland> contains a lot of code written to specs available only
   Roland> under NDA (and even reverse-engineered code where specs
   Roland> are completely unavailable), I don't think the expense
   Roland> should be an issue.
   Francois> One can say good bye to peer review.
Yes and no.  Certainly people without specs can't review spec
compliance, but review for coding style, locking bugs, etc. is if
anything more valuable.
Thanks,
 Roland
___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
 

___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] kernel 2.6.11 and userland packages?

[Troy Benjegerdes]

I have in my office a shiny new kernel.org 2.6.11 64 bit kernel running
on my Mac G5, with the drivers/infiniband modules loaded.

What do I need to do to verify this all works?

Also, I'd really like to make debian packages of the userland utilities
and libraries, and get a debian/ subdirectory into the subversion
release, so the packages can be rebuilt easily.

Where should I start on this?

-- 
--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] http://openib.org/downloads/

[Troy Benjegerdes]

On Tue, Mar 08, 2005 at 06:56:47PM -0800, Grant Grundler wrote:
> On Tue, Mar 08, 2005 at 03:31:48PM -0800, Matt Leininger wrote:
> >   You can grab the openib source code from the subversion repository.
> > See  http://www.openib.org/tools.html.   If you want everything run 'svn
> > co https://openib.org/svn'  
> 
> Matt,
> probably best to just add a short blurb to tools.html
> that includes an example using gen2 branch. That's what
> we want people to focus on I think.

having just waded into this stuff, I'd really like a
"just_build_it_all.sh" script.

Well, actually, what I'd really like is to do:

svn co https://openib.org/some/path
cd some/path
fakeroot dpkg-buildpackage

and get me some debian packages ;)

FYI, I'm hopeing that at least the PPC debian 2.6.11 kernel packages
with have IB modules enabled in the .config
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Getting rid of pinned memory requirement

[Troy Benjegerdes]

The current InfiniBand model of using 'mlock()' to maintain a constant virtual
to physical mapping for registered memory pages is not going to work with NUMA
page migration and memory hotplug.

I want to get some discussion started on this list, and once we have an idea
what's feasable from the infiniband side, to bring up the discussion on
linux-kernel, and get the memory hotplug and numa page migration people
involved as well.

I think the following list covers the major points. Are there any big
"gotcha's" involved? 

* Add "registered" flag to linux/mm.h (VM_REGISTERED 0x0100)

* Need to define a 'registered memory' api. Maybe using 'madvise()' ?

* Kernel needs to be able to unpin registered memory and shoot down cached
  mappings in network cards (treat IB/Iwarp cards like a TLB)

* Requires IB/Iwarp card to dispatch an interrupt on a mapping 'miss'

* This model allows applications to register more memory than physically
  exists, and the kernel manages what is actually pinned.

* Requires adding hooks in MM code to dispatch driver mapping shootdowns. (A
  per-VM area list of adapters to be notified for the mapping?) 


I know that having the card dispatch an interrupt on an incoming packet that's
not mapped is outside the spec. The alternative is that if the kernel wants to
move some memory around that's registered, it's got to have some way to either
kill the application, or tear down and re-establish all the QP's. I suppose
an alternative would be a "SIG_I_KILLED_YOUR_MAPPINGS" type signal to tell
the application (or library) that it needs to re-establish all it's pinned
memory might work.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Getting rid of pinned memory requirement

[Troy Benjegerdes]

On Mon, Mar 14, 2005 at 03:29:06PM -0800, Caitlin Bestler wrote:
>  
> 
> > -Original Message-
> > From: [EMAIL PROTECTED] 
> > [mailto:[EMAIL PROTECTED] On Behalf Of Troy 
> > Benjegerdes
> > Sent: Monday, March 14, 2005 3:01 PM
> > To: openib-general@openib.org
> > Subject: [openib-general] Getting rid of pinned memory requirement
> > 
> > The current InfiniBand model of using 'mlock()' to maintain a 
> > constant virtual to physical mapping for registered memory 
> > pages is not going to work with NUMA page migration and 
> > memory hotplug.
> > 
> > I want to get some discussion started on this list, and once 
> > we have an idea what's feasable from the infiniband side, to 
> > bring up the discussion on linux-kernel, and get the memory 
> > hotplug and numa page migration people involved as well.
> > 
> > I think the following list covers the major points. Are there 
> > any big "gotcha's" involved? 
> > 
> > * Add "registered" flag to linux/mm.h (VM_REGISTERED 0x0100)
> > 
> > * Need to define a 'registered memory' api. Maybe using 'madvise()' ?
> > 
> > * Kernel needs to be able to unpin registered memory and 
> > shoot down cached
> >   mappings in network cards (treat IB/Iwarp cards like a TLB)
> > 
> > * Requires IB/Iwarp card to dispatch an interrupt on a mapping 'miss'
> > 
> 
> The point of requiring that the memory be pinned is so that
> the IB/iWARP card does not have to deal with the kernel on
> a per-placement basis.
> 
> That includes having to double-check any host memory resources
> to see if there is anything to 'miss' in the mapping.

I guess I wasn't implying any 'double-checking'.. What I want is for the
kernel to be able to unpin memory and tell the card it did so, instead
of being locked into never being able to move that memory around. This
requires no host memory interaction.

By doing this, I can register a whole lot *more* memory, and the kernel
can still keep buggy applications from trashing the whole system.

[snip]

> Fundamental to any definition of RDMA is that the application
> controls the avialability of target memory -- not the kernel.
> That is why traditional buffer flow controls do not apply.

While hardware designers may like this idea, I would like to make the
point that if you want the application to *absolutely* control the
availability of physical memory, you shouldn't be writing userspace
applications that run on Linux.

There's always going to be a limit on how much memory you can mlock. And
right now the only option the kernel has for unlocking that memory is to
kill the application. I think there's got to be a reasonable way to deal
with this that doesn't make the application responsible for everything
in the world. We don't want to have to rewrite every RDMA application to
be able to support memory hotplug. This is an obvious layer that can and
should be abstracted by the kernel.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Getting rid of pinned memory requirement

[Troy Benjegerdes]

On Mon, Mar 14, 2005 at 04:33:19PM -0800, Caitlin Bestler wrote:
> > 
> > While hardware designers may like this idea, I would like to 
> > make the point that if you want the application to 
> > *absolutely* control the availability of physical memory, you 
> > shouldn't be writing userspace applications that run on Linux.
> > 
> 
> This is not just a hardware design issue. It is fundamental to
> why RDMA is able to optimize end-to-end traffic flow. The application
> is directly advertising the availability of buffers (through RKeys)
> to the other side. It is bad network engineering for the kernel
> to revoke that good faith advertisement and count on the HCA/RNIC
> to say "oops" when the data does arrive but the targeted buffer
> is not in memory.
> 
> But that does not mean that you cannot design mechanisms below
> the application to allow the kernel to re-organize physical
> memory -- it just means that the kernel had best not be playing
> overcommit tricks behind the applications back.
> 
> To use a banking analogy, an adverised RKey is like a certified
> check. The application has sent this RKey to its peer, and it
> expects the HCA/RNIC to honor that check when RDMA Writes are
> made to that memory.  But just as a bank does not have to 
> guarantee in advance which specific bills will be used to
> cash a guaranteed check, there is nothing to say that the
> virtual to physical mappings are permanent and immutable.
> 
> It would be possible to design an interface that allowed
> the kernel to:
> 
> a)suspend the use of a memory region.
>   1) outputs referencing the suspend LKey would be
>   temporarily held by the HCA/RNIC.
>   2) inputs referencing the suspend memory region
>   would be delayed (RNR NAK, internal buffers,
>   etc.)
>   3) possibly ask the peer to similarly suspend
>   sending. This is trickier though.
> b)Update the virtual to physical mappings, or at least
>   provide the RDMA layer with "physical page X replaced
>   by physical page Y".
> c)unsuspend the memory region.
> 
> The key is that the entire operation either has to be fast
> enough so that no connection or application session layer
> time-outs occur, or an end-to-end agreement to suspend the
> connetion is a requirement. The first option seems more
> plausible to me, the second essentially reuqires extending
> the CM protocol. That's a tall order even for InfiniBand,
> and it's even worse for iWARP where the CM functionality
> typically ends when the connection is established.
 
I'll buy the good network design argument.

I suppose if the kernel wants to revoke a card's pinned memory, we
should be able to guarantee that it gets new pinned memory within a
bounded time. What sort of timing do we need? Milliseconds?
Microseconds?

In the case of iWarp, isn't this just TCP underneath? If so, can't we
just drop any packets in the pipe on the floor and let them get
retransmitted? (I suppose the same argument goes for infiniband..
what sort of a time window do we have for retransmission?)

What are the limits on end-to-end flow control in IB and iWarp?

> 
> 
> 
> > There's always going to be a limit on how much memory you can 
> > mlock. And right now the only option the kernel has for 
> > unlocking that memory is to kill the application. I think 
> > there's got to be a reasonable way to deal with this that 
> > doesn't make the application responsible for everything in 
> > the world. We don't want to have to rewrite every RDMA 
> > application to be able to support memory hotplug. This is an 
> > obvious layer that can and should be abstracted by the kernel.
> >
> 
> Yes, there are limits on how much memory you can mlock, or
> even allocate. Applications are required to reqister memory
> precisely because the required guarantess are not there by
> default. Eliminating those guarantees *is* effectively
> rewriting every RDMA application without even letting
> them know.

Some of this argument is a policy issue, which I would argue shouldn't
be hard-coded in the code or in the network hardware.

At least in my view, the guarantees are only there to make applications
go fast. We are getting low latency and high performance with infiniband
by making memory registration go really really slow. If, to make big HPC
simulation applications work, we wind up doing memcpy() to put the data
into a registered buffer because we can't register half of physical
memory, the application isn't going very fast.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] uverbs security

[Troy Benjegerdes]

On Tue, Mar 15, 2005 at 08:23:52AM +0200, Michael S. Tsirkin wrote:
> Hi, Roland!
> Looking at uverbs kernel module, I notice that in some instances
> it passes some parameters from userspace directly to ib core, without
> verifying their sanity.
> 
> One example of this is qp attributes in create and modify qp.
> 
> For example, modify qp and alloc qp will simply copy the attributes.
> This might create issues since the core may assume it works against a
> trusted kernel client, so it may get confused if passed illegal
> parameter values.
> 
> For example, qp type could be IB_QPT_SMI or IB_QPT_GSI. Will this create
> a problem? Hard for me to tell ...
> 
> I think the best approach is to validate *all* user-given parameters
> before passing them on to core. What do you think?

Yes. We should be validating all user parameters, and be thinking about
malicious userspace apps. This is another reason I think we ought to
have the linux MM support a 'VM_REGISTERED' flag, and things like
selinux can have different security policies for registered memory vs
not-registered.

I think we should probably also have (possibly compile-time) options for
IB core to sanity check everything, regardless of whether it came from
userspace or kernelspace. (Kind of like CONFIG_DEBUG_KERNEL and the
like)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of NetPIPE-3.6.2 to OpenIB userspace verbs

[Troy Benjegerdes]

On Tue, Mar 15, 2005 at 02:25:50PM -0800, Woodruff, Robert J wrote:
> 
> Roland> I just spent a little time creating a new "ibv" module for
> NetPIPE
> >that runs on top of the userspace verbs I've been developing on the
> >roland-uverbs branch.  
> 
> Cool. this will be very useful. Any idea if/when the netpipe folks will
> release a version of netpipe that has this patch included ?

I'll ask Dave Turner what he wants to do about this..

Once I get it built and tested locally, I'll probably stick some results
and a link up at http://scl.ameslab.gov/Projects/InfiniBand/

Sooo... what's the easiest way for me to test this if I have opterons
with 2.6.11.4 kernels?

(aka, just replace drivers/infiniband from the roland-uverbs branch? And
does anyone have a clean way of building all the userspace stuff? What
I've seen so far is pretty tedious)

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of NetPIPE-3.6.2 to OpenIB userspace verbs

[Troy Benjegerdes]

On Tue, Mar 15, 2005 at 02:06:29PM -0800, Roland Dreier wrote:
> I just spent a little time creating a new "ibv" module for NetPIPE
> that runs on top of the userspace verbs I've been developing on the
> roland-uverbs branch.  This is pretty much a straight port of the
> current Mellanox VAPI "ib" module, with the main changes coming from
> the fact that OpenIB doesn't support the non-standard "unsignaled
> receive" extension, and the fact that a completion event thread is no
> longer created automatically.
> 
> I found several bugs in the verbs support while making this work, but
> it seems quite stable now, although I haven't tried all option
> combinations.  I also have not had a chance to compare Mellanox VAPI
> and OpenIB verbs performance on identical hardware -- it would be very
> useful to see this comparison on a variety of systems.
> 

I'm having trouble building opensm et all from roland-uverb... (and I
can't really test NetPIPE without an SM )

.o -o opensm  -L/afs/scl/project/infiniband/openib/roland-uverbs/src/lib
-lpthread
/afs/scl/project/infiniband/openib/roland-uverbs/src/lib/libumad.so
/afs/scl/project/infiniband/openib/roland-uverbs/src/lib/libcomplib.so
/afs/scl/project/infiniband/openib/roland-uverbs/src/lib/libcommon.so
-Wl,--rpath -Wl,/afs/scl/project/infiniband/openib/roland-uverbs/src/lib
-Wl,--rpath -Wl,/afs/scl/project/infiniband/openib/roland-uverbs/src/lib
osm_switch.o(.text+0x25): In function `osm_switch_init':
/afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:111:
multiple definition of `no symbol'
osm_switch.o(.text+0x0):/afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:98:
first defined here
/usr/bin/ld: Warning: size of symbol `' changed from 37 in osm_switch.o
to 242 in osm_switch.o
osm_switch.o(.text+0x117): In function `osm_switch_destroy':
/afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:163:
multiple definition of `no symbol'
osm_switch.o(.text+0x0):/afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:98:
first defined here

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of NetPIPE-3.6.2 to OpenIB userspace verbs

[Troy Benjegerdes]

Note to self: check build twice in case your filesystem has bogons.

it builds fine. I must have had a bogus object file. But, does this mean
the port is up, or only that the physical link is active?

[EMAIL PROTECTED]:/afs/scl/project/infiniband/openib/roland-uverbs/src/bin$ 
./ibstat
CA 'mthca0'
CA type: MT23108
Number of ports: 2
Firmware version: 2.0.0
Hardware version: a1
Node GUID: 0x0002c90108cd8ba0
System image GUID: 0x0002c90108cd8ba3
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x00100a6a
Port GUID: 0x0002c90108cd8ba1


On Wed, Mar 16, 2005 at 10:58:17PM -0600, Troy Benjegerdes wrote:
> On Tue, Mar 15, 2005 at 02:06:29PM -0800, Roland Dreier wrote:
> > I just spent a little time creating a new "ibv" module for NetPIPE
> > that runs on top of the userspace verbs I've been developing on the
> > roland-uverbs branch.  This is pretty much a straight port of the
> > current Mellanox VAPI "ib" module, with the main changes coming from
> > the fact that OpenIB doesn't support the non-standard "unsignaled
> > receive" extension, and the fact that a completion event thread is no
> > longer created automatically.
> > 
> > I found several bugs in the verbs support while making this work, but
> > it seems quite stable now, although I haven't tried all option
> > combinations.  I also have not had a chance to compare Mellanox VAPI
> > and OpenIB verbs performance on identical hardware -- it would be very
> > useful to see this comparison on a variety of systems.
> > 
> 
> I'm having trouble building opensm et all from roland-uverb... (and I
> can't really test NetPIPE without an SM )
> 
> .o -o opensm  -L/afs/scl/project/infiniband/openib/roland-uverbs/src/lib
> -lpthread
> /afs/scl/project/infiniband/openib/roland-uverbs/src/lib/libumad.so
> /afs/scl/project/infiniband/openib/roland-uverbs/src/lib/libcomplib.so
> /afs/scl/project/infiniband/openib/roland-uverbs/src/lib/libcommon.so
> -Wl,--rpath -Wl,/afs/scl/project/infiniband/openib/roland-uverbs/src/lib
> -Wl,--rpath -Wl,/afs/scl/project/infiniband/openib/roland-uverbs/src/lib
> osm_switch.o(.text+0x25): In function `osm_switch_init':
> /afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:111:
> multiple definition of `no symbol'
> osm_switch.o(.text+0x0):/afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:98:
> first defined here
> /usr/bin/ld: Warning: size of symbol `' changed from 37 in osm_switch.o
> to 242 in osm_switch.o
> osm_switch.o(.text+0x117): In function `osm_switch_destroy':
> /afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:163:
> multiple definition of `no symbol'
> osm_switch.o(.text+0x0):/afs/scl/project/infiniband/openib/roland-uverbs/src/userspace/management/osm/opensm/osm_switch.c:98:
> first defined here
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

-- 
--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of NetPIPE-3.6.2 to OpenIB userspace verbs

[Troy Benjegerdes]

Okay, it's late. 

I have NetPIPE TCP running on ipoib, peak is around 1.9 gigabits/sec or
so with no socket buffer tuning. (this means I have an SM and such). 

Tommorow I will try the netpipe on the uverbs.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of NetPIPE-3.6.2 to OpenIB userspace verbs

[Troy Benjegerdes]

On Wed, Mar 16, 2005 at 08:58:54AM -, Paul Baxter wrote:
> From: "Troy Benjegerdes" <[EMAIL PROTECTED]>
> >Once I get it built and tested locally, I'll probably stick some results
> >and a link up at http://scl.ameslab.gov/Projects/InfiniBand/
> >
> >Sooo... what's the easiest way for me to test this if I have opterons
> >with 2.6.11.4 kernels?
> >
> >(aka, just replace drivers/infiniband from the roland-uverbs branch? And
> >does anyone have a clean way of building all the userspace stuff? What
> >I've seen so far is pretty tedious)
> 
> Troy,
> 
> While I appreciate your keenness , I think its a little unfair to criticise 
> the build status and organisation of code that is still being written and 
> is subject to change. I'd far rather everyone gets a working core before 
> worrying so much about how it might be packaged. That does need to be 
> addressed, of course.
> 
> Your comments at your URL regarding complexity and size of the software 
> stack making progress slow are IMHO unfair to openib.
> They've worked hard on getting a streamlined set of functionality into the 
> kernel and now need to finish off key parts of userspace support and only 
> then 'package' it so that you will find it easier to compile and test.

Actually, my comments about InfiniBand software refferred to all the
previous vendor "released" software stacks.

It's been about two days since I started looking at this stuff, and I'm
now able to run Netpipe. I haven't crashed the kernel once from the
infiniband driver. I'm quite impressed with openib progress.

OpenIB is relatively modular, and the individual modules are nicely
named, and much smaller than anything else I've seen for IB drivers. In
fact, the code size of the IB support looks to be smaller than the ipv6
module.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of NetPIPE-3.6.2 to OpenIB userspace verbs

[Troy Benjegerdes]

On Wed, Mar 16, 2005 at 08:52:56AM -0800, Roland Dreier wrote:
> Troy> Sooo... what's the easiest way for me to test this if I have
> Troy> opterons with 2.6.11.4 kernels?
> 
> Troy> (aka, just replace drivers/infiniband from the roland-uverbs
> Troy> branch? And does anyone have a clean way of building all the
> Troy> userspace stuff? What I've seen so far is pretty tedious)
> 
> Yes, the roland-uverbs src/linux-kernel/infiniband directory should
> just drop in and replace the existing drivers/infiniband.  You'll want
> to turn on CONFIG_INFINIBAND_USER_VERBS in your config (a new option)
> to enable userspace verbs, load the ib_uverbs module (if you don't
> build support into your kernel), and create /dev/infiniband/uverbs
> device nodes (easiest way is to add
> 
> KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
> 
> to your udev rules).
> 
> To build the userspace verbs support, you just need to build
> libibverbs and libmthca libraries (using the usual "./autogen.sh &&
> ./configure && make && make install" recipe).  I agree that the
> management subdirectory has a few too many little pieces right now,
> but it's not needed if you already have a subnet manager running
> somewhere.

The management/Makefile and management/Readme in roland-uverbs were very 
usefull. 

The only real gotcha I had was I didn't install in /usr/local/lib, and
set LD_LIBRARY_PATH to point to the libraries.. thus libibverbs couldn't
find libmthca.so, and the error messages weren't particularly
informative.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Causes of interrupt problems?

[Troy Benjegerdes]

What would cause the following?

ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004)
ib_mthca: Initializing Mellanox Technology MT23108 InfiniHost (:04:00.0)
ib_mthca :04:00.0: NOP command failed to generate interrupt, aborting.
ib_mthca :04:00.0: BIOS or ACPI interrupt routing problem?


I've seen this on two Opteron systems, one Tyan board, one Rioworks
HDAMA. Is there some bios setting I should look for? Things are working
fine on another Rioworks HDAMA board.


-- 
--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Causes of interrupt problems?

[Troy Benjegerdes]

On Fri, Mar 18, 2005 at 08:23:12PM -0800, Roland Dreier wrote:
>  > What would cause the following?
> 
>  > ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004)
>  > ib_mthca: Initializing Mellanox Technology MT23108 InfiniHost 
> (:04:00.0)
>  > ib_mthca :04:00.0: NOP command failed to generate interrupt, aborting.
>  > ib_mthca :04:00.0: BIOS or ACPI interrupt routing problem?
> 
>  > I've seen this on two Opteron systems, one Tyan board, one Rioworks
>  > HDAMA. Is there some bios setting I should look for? Things are working
>  > fine on another Rioworks HDAMA board.
> 
> It seems that the fact that the HCA appears as a PCI device with a
> huge BAR behind a PCI bridge confuses some BIOS/ACPI implementations.
> 
> Looking at that error message I realize it might be nice to be able to
> see what IRQ the driver is trying.  If you change the line in
> mthca_main.c that prints the error to something like
> 
>   mthca_err(dev, "NOP command failed to generate interrupt (IRQ 
> %d), aborting.\n",
> dev->mthca_flags & MTHCA_FLAG_MSI_X ?
> dev->eq_table.eq[MTHCA_EQ_CMD].msi_x_vector :
> dev->pdev->irq);

Can you add this, as well as a check for recent firmware and/or card
revision?

I have some cards with ancient firmware revisions, which seem like they
don't implement NOP. The bios was actually fine on this machine, and
everything was happy once I put a card with a newer firmware in.

FYI, I've now got nfs over ipoib running, and I'm getting about 110-120
MB/sec read throughput from nfs using 'dd if=nfsfile of=/dev/null'.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] NFS over IPoIB performance..

[Troy Benjegerdes]

Well, I'm quite impressed.. I've been running NFS over IPoIB to a server
with a 3ware SATA raid card in it, and nothing's crashed ;)

This is with a 2.4.11.4 kernel and roland-uverbs branch. (although I'm
not using uverbs at the moment)


da0:64bit:~$ ifconfig ib0
ib0   Link encap:UNSPEC  HWaddr 
00-00-00-14-FE-80-00-00-00-00-00-00-00-00-00-00
  inet addr:10.0.0.10  Bcast:10.255.255.255  Mask:255.255.0.0
  inet6 addr: fe80::206:6a00:a000:43c/64 Scope:Link
  UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
  RX packets:209004632 errors:0 dropped:0 overruns:0 frame:0
  TX packets:959650153 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:128
  RX bytes:309422700203 (288.1 GiB)  TX bytes:1863296583515 (1.6 TiB)

My only real problem is there seems to be no good way to set readahead
for the NFS client. I'm averaging 75mb/sec or so, and I've seen peaks of
100MB/sec, with clients from two different machines running the GAMESS
computational chemistry code.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] NFS over IPoIB performance..

[Troy Benjegerdes]

On Thu, Mar 24, 2005 at 05:31:36PM -0800, Libor Michalek wrote:
> On Tue, Mar 22, 2005 at 08:57:53PM -0600, Troy Benjegerdes wrote:
> > Well, I'm quite impressed.. I've been running NFS over IPoIB to a server
> > with a 3ware SATA raid card in it, and nothing's crashed ;)
> > 
> > My only real problem is there seems to be no good way to set readahead
> > for the NFS client. I'm averaging 75mb/sec or so, and I've seen peaks of
> > 100MB/sec, with clients from two different machines running the GAMESS
> > computational chemistry code.
> 
>   I'm curious, where you running NFS over UDP or TCP? If the former
> could you try the later?
> 
> -Libor

With TCP:
10.0.0.10:/scratch/md1 on /nfs/da0/md1 type nfs
(rw,v3,rsize=32768,wsize=32768,hard,tcp,lock,addr=10.0.0.10)

opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 113.188032 seconds (94863547 bytes/sec)


With UDP:

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 96.758483 seconds (110971337 bytes/sec)

And one more time with tcp, in case there are buffer-cache effects.

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3,tcp
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 129.839903 seconds (82697368 bytes/sec)

One last time with UDP:

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd troy/
opteron1:64bit:/nfs/da0/troy$ cd ../md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 96.915829 seconds (110791172 bytes/sec)

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of ISC DHCP-3.0.2 to OpenIB IPoIB

[Troy Benjegerdes]

On Thu, Mar 31, 2005 at 06:06:30PM -0500, Hal Rosenstock wrote:
> On Wed, 2005-03-30 at 20:04, Josh England wrote: 
> > Are there any plans to modify the linux DHCP client so it would be
> > possible to do kernel-level DHCP and NFSroot over IB?
> 
> I took a quick look at this and it looks pretty straightforward. Stay
> tuned...

I'd say don't.

Using initrd/initramfs is a much better solution. At some point the
in-kernel dhcp is going to get so buggy and old it's going to get
removed.

I boot all my cluster systems with NFS root servers, and I'm trying to
get everything moved to using Debian packaged kernels with initrd's.
With an initrd, you at least have a chance to get a shell and figure out
why you couldn't find your nfs server, instead of "kernel panic, I'm
going to die now" you get with in-kernel dhcp/nfs.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Port of ISC DHCP-3.0.2 to OpenIB IPoIB

[Troy Benjegerdes]

On Mon, Apr 04, 2005 at 08:17:43PM -0700, Josh England wrote:
> Troy Benjegerdes wrote:
> > On Thu, Mar 31, 2005 at 06:06:30PM -0500, Hal Rosenstock wrote:
> > 
> >>On Wed, 2005-03-30 at 20:04, Josh England wrote: 
> >>
> >>>Are there any plans to modify the linux DHCP client so it would be
> >>>possible to do kernel-level DHCP and NFSroot over IB?
> >>
> >>I took a quick look at this and it looks pretty straightforward. Stay
> >>tuned...
> > 
> > 
> > I'd say don't.
> > 
> > Using initrd/initramfs is a much better solution. At some point the
> > in-kernel dhcp is going to get so buggy and old it's going to get
> > removed.
> 
> I know...it's just crummy to have ship another 1.3 Megs out to every node.
> 
> > I boot all my cluster systems with NFS root servers, and I'm trying to
> > get everything moved to using Debian packaged kernels with initrd's.
> > With an initrd, you at least have a chance to get a shell and figure out
> > why you couldn't find your nfs server, instead of "kernel panic, I'm
> > going to die now" you get with in-kernel dhcp/nfs.
> 
> Check out oneSIS (http://onesis.org).  It can build initrds for you that
> do NFSroot (and drop to a shell when things go sour).  I'd love to hear
> some feedback from people familiar with running NFSroot.

Debian has a package called "lessdisks" that does some similiar stuff..
If I do "apt-get install initrd-netboot-tools", and then install a
debian kernel image, it builds an initrd that can netboot. I suppose my
next trick after I get the debian kernel maintainers to make sure
infiniband is enabled is to try booting over IPoIB.

What dhcp client does onesis use? The debian lessdisks stuff uses
'udhcpc', advertised as a "very small DHCP client".
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

[Troy Benjegerdes]

> In particular, the memory pinning code in in uverbs_mem.c could stand
> a looking over.  In addition, a sanity check of the write()-based
> scheme for passing commands into the kernel in uverbs_main.c and
> uverbs_cmd.c is probably worthwhile.

How is memory pinning handled? (I haven't had time to read all the code,
so please excuse my ignorance of something obvious).

The old mellanox drivers used to have a hack to call 'sys_mlock', and
promiscuously lock memory any old userspace application asked for. What
is the API for the new uverbs memory registration, and how will things
like memory hotplug and NUMA page migration be able to unpin pages
locked by a user program?

I have applications that would benefit from being able to register 15GB
of memory on a machine with 16GB. Right now, MPI and other possible
users of infiniband in userspace have to play cacheing games and limit
what they can register. But locking all that memory without providing
the kernel a way to unlock it under memory pressure or for page
migration seems like a bad idea.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

[Troy Benjegerdes]

On Mon, Apr 11, 2005 at 08:34:19AM -0700, Roland Dreier wrote:
> Troy> How is memory pinning handled? (I haven't had time to read
> Troy> all the code, so please excuse my ignorance of something
> Troy> obvious).
> 
> The userspace library calls mlock() and then the kernel does
> get_user_pages().

Is there a check in the kernel that the memory is actually mlock()ed?

What if a malicious (or broken) application does ibv_reg_mr() but
doesn't lock the memory? Does the IB card get a physical address for a
page that might get swapped out?

> Troy> The old mellanox drivers used to have a hack to call
> Troy> 'sys_mlock', and promiscuously lock memory any old userspace
> Troy> application asked for. What is the API for the new uverbs
> Troy> memory registration, and how will things like memory hotplug
> Troy> and NUMA page migration be able to unpin pages locked by a
> Troy> user program?
> 
> The API for uverbs memory registration is ibv_reg_mr(), and right now
> the memory is pinned and that's it.
> 
>  - R.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

[Troy Benjegerdes]

On Mon, Apr 11, 2005 at 09:56:53AM -0700, Roland Dreier wrote:
> Troy> Is there a check in the kernel that the memory is actually
> Troy> mlock()ed?
> 
> No.
> 
> Troy> What if a malicious (or broken) application does
> Troy> ibv_reg_mr() but doesn't lock the memory? Does the IB card
> Troy> get a physical address for a page that might get swapped
> Troy> out?
> 
> No, the kernel does get_user_pages().  So the pages that the HCA gets
> will not be swapped or used for anything else.  The only thing a
> malicious userspace app can do is screw itself up.
> 
>  - R.

Do we even need the mlock in userspace then?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

[Troy Benjegerdes]

On Mon, Apr 18, 2005 at 10:07:12PM +0200, Bernhard Fischer wrote:
> On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote:
> >On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote:
> >> Arjan van de Ven wrote:
> >> 
> >> > this is a myth; linux is free to move the page about in physical memory
> >> > even if it's mlock()ed!!
> darn, yes, this is true.
> I know people who introduced
> #define VM_RESERVED 0x0008  /* Don't unmap it from swap_out
> */

Someone (aka Tospin, infinicon, and Amasso) should probably post a patch
adding '#define VM_REGISTERD 0x0100', and some extensions to
something like 'madvise' to set pages to be registered.

My preference is said patch will also allow a way for the kernel to
reclaim registered memory from an application under memory pressure.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] IBM eHCA Device Driver for gen1 IB stack

[Troy Benjegerdes]

On Wed, Apr 20, 2005 at 09:44:45AM -0700, Roland Dreier wrote:
> > Hi, we've just released the first linux device driver for
> > the IBM eServer HCA for Power5.  It's gen1 based and runs
> > on SLES9 SP1.  Main testvehicle for this code was IPoIB.
> 
> > gen2 and full userspace support will be next.
> 
> Excellent, I'm glad to see this released.  I'm looking forward to
> seeing the gen2 support.
> 
> If I may make a small suggestion for future releases: please have the
> tar file contain a top-level directory like ehca-0021, with everything
> contained in that directory.  It's a little annoying to unpack a tar
> file and have it spread 5 files in your working directory, especially
> when some have generic names like "INSTALL" or "patches."

What will it take to get the Gen2 support into the openib.org subversion
tree?

How much is the low-level driver likely to change once it's written and
working?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] umad abi 2 v 3 and multicast join failed

[Troy Benjegerdes]

I was running a crufty version of opensm (compiled from the
roland-uverbs branch), and I started getting these kinds of errors for
no apparent reason:

ib0: multicast join failed for ff12:401b::0:0:0::, status
-22
ib0: multicast join failed for ff12:401b::0:0:0::, status
-22

I'm running 2.6.11 kernels, and 'stock' modules.. I just tried
rebuilding opensm from the latest SVN, but it apparently needs a new
umad driver..

warn: [24878] umad_init: wrong ABI version:
/sys/class/infiniband_mad/abi_version is 2 but library ABI is 3


I suppose I need to rebuild the kernel ib_umad (and maybe everything
else for good measure).. And if I do that, should I expect OpenSM to
work better regarding the multicast issue?

Also, what will happen if I run opensm on two different nodes? Will they
fight, or will one of them figure out how to be a backup slave SM if the
first goes down?

-- 
--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

[Troy Benjegerdes]

> Then someone would have to start implementing a low-level driver for a
> specific RNIC, and find which modifications to the existing verbs are


I'll believe that RNICs are actually going to work and it's worth
talking about OpenRDMA when I can see code that runs. Initially, I'd say
that extending OpenIB is going to be the best way forward, once there is
a working RNIC driver.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: umad abi 2 v 3 and multicast join failed

[Troy Benjegerdes]

On Thu, May 26, 2005 at 10:29:09AM +0300, Michael S. Tsirkin wrote:
> Quoting r. Grant Grundler <[EMAIL PROTECTED]>:
> > Subject: Re: umad abi 2 v 3 and multicast join failed
> > 
> > On Wed, May 25, 2005 at 09:07:25PM -0700, Roland Dreier wrote:
> > > In general, given that kernels 2.6.11 and 2.6.12 are shipping with ABI
> > > version 2, does it make sense to avoid problems like this by keeping
> > > the old userspace code around and having the library decide at runtime
> > > which ABI to use?
> > 
> > Only if you want to continue providing fixes for 2.6.11 and 2.6.12.
> > It would be nice if someone put together a source snapshot that
> > would work with those releases. But I'm not sure how to
> > make that available to some random person who is using it.
> 
> Making a subversion branch could be one way to start.
> 
> Actually, it would be quite useful if whatever is submitted to mainline
> is kept on some subversion branch, too.
> Roland - would that disrupt your development process a lot?

What's the status on getting ABI ver 3 into 2.6.1X ?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

I'm having intermittent problems with opensm.. It seems after a while
IPoIB stops working and if I restart opensm, it starts spitting out
errors. Do I have a misbehaving switch somewhere?

ibnetdiscover seems to work fine.


(this is from running 'opensm -v -o -r')


-
OpenSM Rev:openib-1.0.0
Command Line Arguments:
 Verbose option -v (log flags = 0x7)
 Run Once
 Reassign LIDs
-
Jun 02 18:14:35 [B7E7F600] -> OpenSM Rev:openib-1.0.0
Jun 02 18:14:35 [B7E7F600] -> osm_opensm_init: Forcing single threaded 
dispatcher.
Jun 02 18:14:35 [B7E7F600] -> osm_report_notice: Reporting Generic Notice 
type:3 num:66 from LID:0x GID:0xfe80,0x
Jun 02 18:14:35 [B7E7F600] -> osm_report_notice: Reporting Generic Notice 
type:3 num:66 from LID:0x GID:0xfe80,0x
Jun 02 18:14:35 [B7E7F600] -> osm_vendor_get_all_port_attr: assign CA mthca0 
port 1 guid (0x66a00a44d) as the default port.
using default guid 0x66a00a44d
Jun 02 18:14:35 [B7E7F600] -> osm_sm_mad_ctrl_bind: Binding to port 
0x66a00a44d.
Jun 02 18:14:35 [B7E7F600] -> osm_vendor_bind: Binding to port 0x66a00a44d.
Jun 02 18:14:35 [B7E7F600] -> osm_sa_mad_ctrl_bind: Binding to port GUID = 
0x66a00a44d.
Jun 02 18:14:35 [B7E7F600] -> osm_vendor_bind: Binding to port 0x66a00a44d.
Jun 02 18:14:35 [B7E7F600] -> __osm_sm_state_mgr_discovering_msg: 


**
 ENTERING SM DISCOVERING STATE ***
**


Jun 02 18:14:35 [B7E7F600] -> __osm_state_mgr_sweep_heavy_msg: 


**
 INITIATING HEAVY SWEEP **
**


Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 0 hop path:
Initial path = [0]
Return path  = [0]
Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Channel 
Adapter node,
GUID = 0x66a009800044d, TID = 0x1234.
Jun 02 18:14:35 [B567DBB0] -> __osm_nd_rcv_process_nd: Node 0x66a009800044d
Description = MT23108 InfiniHost Mellanox 
Technologies.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 1 with 
GUID = 0x66a00a44d for parent node GUID = 0x66a009800044d, TID = 0x1236.
Jun 02 18:14:35 [B567DBB0] -> osm_vendor_set_sm: setting IS_SM capability mask 
failed; errno 2
Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 
port_num 1 with GUID = 0x66a00a44d for parent node GUID = 0x66a009800044d, 
TID = 0x1237.
Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 
port_num 1 with GUID = 0x66a00a44d for parent node GUID = 0x66a009800044d, 
TID = 0x1238.
Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 1 hop path:
Initial path = [0][1]
Return path  = [0][4]
Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Switch 
node,
GUID = 0x84e00014a, TID = 0x1239.
Jun 02 18:14:35 [B567DBB0] -> __osm_nd_rcv_process_nd: Node 0x84e00014a
Description = DivergeNet Meshlink 4X Switch.
Jun 02 18:14:35 [B567DBB0] -> __osm_si_rcv_process_new:Subnet max multicast lid 
is 0xC400
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 0 with 
GUID = 0x84e00014a for parent node GUID = 0x84e00014a, TID = 0x123c.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x0.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 1 with 
GUID = 0x84e00014a for parent node GUID = 0x84e00014a, TID = 0x123d.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x1.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 2 with 
GUID = 0x84e00014a for parent node GUID = 0x84e00014a, TID = 0x123e.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x2.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 3 with 
GUID = 0x84e00014a for parent node GUID = 0x84e00014a, TID = 0x123f.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x3.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 4 with 
GUID = 0x84e00014a for parent node GUID = 0x84e00014a, TID = 0x1240.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 5 with 
GUID = 0x84e00014a for parent node GUID = 0x84e00014a, TID = 0x1241.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x5.
Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discov

Re: [openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

On Thu, Jun 02, 2005 at 06:23:31PM -0500, Troy Benjegerdes wrote:
> I'm having intermittent problems with opensm.. It seems after a while
> IPoIB stops working and if I restart opensm, it starts spitting out
> errors. Do I have a misbehaving switch somewhere?
> 
> ibnetdiscover seems to work fine.
> 
> 
> (this is from running 'opensm -v -o -r')
> 

Some more info.. I rebooted the switches, and tried to re-run it.

I found that ibnetdiscover showed everything with a LID of 0 except 1
HCA card.. when I found that machine and did 'rmmod ib_mthca', opensm
seemed to get unstuck and mapped all the other lids.

And just now, as a sanity check, I was going to reload all the IB
modules, but got the following panic:

gozer.scl.ameslab.gov login: ib_mad: Invalid directed route
ib_mthca :03:00.0: dma_pool_destroy mthca_av, f71c2000 busy
Unable to handle kernel NULL pointer dereference at virtual address 
 printing eip:
c012d348
*pde = 
Oops: 0002 [#1]
SMP
Modules linked in: ib_umad nfsd lockd sunrpc ipv6 evdev floppy pcspkr ib_mad 
ib_core shpchp pci_hotplug ohci_hcd usbcore serverworks i2c_piix4 i2c_core 
sworks_agp agpgart aic7xxx tg3 xfs exportfs capability commoncap ide_cd 
ide_core cdrom genrtc isofs ext2 ext3 jbd mbcache sd_mod aacraid scsi_mod unix 
fbcon font bitblit vesafb cfbcopyarea cfbimgblt cfbfillrect
CPU:0
EIP:0060:[]Tainted: GF VLI
EFLAGS: 00010812   (2.6.11-1-686-smp)
EIP is at __queue_work+0x38/0x70
eax: c2157414   ebx: c2157400   ecx:    edx: f77ea2d4
esi: f77ea2d0   edi: 0286   ebp: c012d3f0   esp: f7277f44
ds: 007b   es: 007b   ss: 0068

Process default.hotplug (pid: 3102, threadinfo=f7276000 task=f70d7a60)
Stack: f7841580 f77ea200 f77ea2d0 c200c9a0 c012d432 c2157400 f77ea2d0 f77ea2d0
   0100 c01262e6 f77ea2d0 f7277fa0 c0118093  f7276000 f7277f80
   f7277f80 01f0 0011 c035ff68 c0397aa0  c0121cca c035ff68

Call Trace:
 [] delayed_work_timer_fn+0x42/0x50
 [] run_timer_softirq+0xd6/0x1c0
 [] scheduler_tick+0x63/0x320
 [] __do_softirq+0xba/0xd0
 [] do_softirq+0x2d/0x30
 [] apic_timer_interrupt+0x1c/0x24

Code: 24 08 8b 74 24 18 89 d8 89 7c 24 0c e8 22 fb 17 00 89 5e 14 89 c7 8d 56 
04 8d 43 0c 8b 48 04 89 46 04 89 50 04 8d 43 14 89 4a 04 <89> 11 ba 03 00 00 00 
b9 01 00 00 00 ff 43 08 c7 04 24 00 00 00
 <0>Kernel panic - not syncing: Fatal exception in interrupt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] cable test/error count utilities?

[Troy Benjegerdes]

Some of my problems seem to be from intermittent cables.. 

Is there anything for OpenIB that can read error counters?

What I'd really like to see is something that I can integrate with
nagios ( http://www.nagios.org/about ) 

-- 
--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] cable test/error count utilities?

[Troy Benjegerdes]

On Fri, Jun 03, 2005 at 06:14:34AM -0400, Hal Rosenstock wrote:
> On Thu, 2005-06-02 at 20:25, Troy Benjegerdes wrote:
> > Some of my problems seem to be from intermittent cables.. 
> > 
> > Is there anything for OpenIB that can read error counters?
> 
> Aside from pulling these from the driver via
> /sys/class/infiniband/mthca0/ports/1/counters/, there is also perfquery
> which displays the portcounters (which contains the error counters):
> 
> Usage: perfquery [-d(ebug) -G(uid_addr) -a(ll_ports) -r(reset_after_read) -C 
> ca_name -P hca_port -R(eset_only) -t timeout_ms -V(ersion) -h(elp)] 
> [ [[port] [reset_mask]]]

How can I query a port between two switches?

[1] {00066a009800044d}
DR path [0][1] -> new remote switch {00084e00014a} portnum 0 lid 0x3-0x3 "Di
vergeNet Meshlink 4X Switch"
[4] {00084e00014a}
DR path [0][1] -> processing switch {00084e00014a} portnum 0 lid 0x2-0x2 "Di
vergeNet Meshlink 4X Switch"
[5] {00084e00014a}
DR path [0][1][5] -> new remote switch {00084e000152} portnum 0 lid 0x1-0x1
"DivergeNet Meshlink 4X Switch"
[6] {00084e000152}
DR path [0][1][5] -> processing switch {00084e000152} portnum 0 lid 0x0-0x0
"DivergeNet Meshlink 4X Switch"
[2] {00084e000152}
DR path [0][1][5][2] -> new remote hca {00066a009800043c} portnum 1 lid 0x7-0x7
"MT23108 InfiniHost Mellanox Technologies"
[1] {00066a009800043c}
warn: [17598] _do_madrpc: send failed; Invalid argument
warn: [17598] handle_port: Nodeinfo on DR path [0][1][5][3] port 3 failed, skipp
ing port
[4] {00084e000152}
DR path [0][1][5][4] -> new remote hca {00066a0098000458} portnum 1 lid 0x9-0x9
"MT23108 InfiniHost Mellanox Technologies"
[1] {00066a0098000458}
[8] {00084e000152}
DR path [0][1][5][8] -> new remote hca {00066a0098000444} portnum 1 lid 0x8-0x8
"MT23108 InfiniHost Mellanox Technologies"
[1] {00066a0098000444}
#
# Topology file: generated on Fri Jun  3 12:13:51 2005
#

switchguids=0x84e00014a
Switch  8 "S-00084e00014a"  # DivergeNet Meshlink 4X Switch port 0 l
id 3
[5] "S-00084e000152"[6]
[4] "H-00066a009800044d"[1]

switchguids=0x84e000152
Switch  8 "S-00084e000152"  # DivergeNet Meshlink 4X Switch port 0 l
id 1
[8] "H-00066a0098000444"[1]
[4] "H-00066a0098000458"[1]
[2] "H-00066a009800043c"[1]
[6] "S-00084e00014a"[5]

hcaguids=0x66a0098000444
Hca 2 "H-00066a0098000444"  # MT23108 InfiniHost Mellanox Technologi
es
[1] "S-00084e000152"[8] # lid 8 lmc 0

hcaguids=0x66a0098000458
Hca 2 "H-00066a0098000458"  # MT23108 InfiniHost Mellanox Technologi
es
[1] "S-00084e000152"[4] # lid 9 lmc 0

hcaguids=0x66a009800043c
Hca 2 "H-00066a009800043c"  # MT23108 InfiniHost Mellanox Technologi
es
[1] "S-00084e000152"[2] # lid 7 lmc 0

hcaguids=0x66a009800044d
Hca 2 "H-00066a009800044d"  # MT23108 InfiniHost Mellanox Technologi
es
[1] "S-00084e00014a"[4] # lid 2 lmc 0


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

On Fri, Jun 03, 2005 at 08:37:04AM -0400, Hal Rosenstock wrote:
> On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: 
> > I'm having intermittent problems with opensm.. It seems after a while
> > IPoIB stops working
> 
> Wonder if there is some relation to the two: intermittent IPoIB and lack
> of response to SM query.
> 
> >  and if I restart opensm,
> 
> How did you get around the ABI version mismatch issue ?
> 
> >  it starts spitting out
> > errors. Do I have a misbehaving switch somewhere?
> 
> It appears that a node is not responding to a discovery packet (SM Get
> NodeInfo (attrID 0x11)). It's direct route initial path (an array of
> port numbers at the start of the next hop) is:
> Initial path = [1][81][1] which means that starting at the node running
> OpenSM, port 1 then port 129 then port 1. Is there a large switch in the
> middle ? Can you send the output of ibnetdiscover ? If that is valid,
> which HCA (port) is not responding (what is the GUID) ? 
> 
> Unfortunately on such an error osm does not appear to give up  (it
> retries forever and is locked on such a node). This is obviously not
> good.
> 
> > ibnetdiscover seems to work fine.
> 
> Are you sure it displays all HCA and switches and their ports ? I
> wouldn't think it would respond to ibnetdiscover if it didn't respond to
> osm. 

I'm running a subversion checkout as of yesterday, so that's how I
got around the ABI version stuff.

the [81] port indicator is definitely bogus. All I have are 8 port
switches. I've also seen [0][0][0] path indicators.. are those allowed
as well?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

On Fri, Jun 03, 2005 at 01:59:42PM -0400, Hal Rosenstock wrote:
> On Fri, 2005-06-03 at 13:17, Troy Benjegerdes wrote:
> > I've also seen [0][0][0] path indicators.. are those allowed
> > as well?
> 
> That is bogus too. Are all the bogus initial paths after a umad_send
> error (message like "umad_receiver: send completed with error" in the
> osm.log) ? Thanks.

I can't say for sure, but I think so.

One of the problems turned out to be a crashed Xserve G5, which seemed
to have the link up, but the driver was obviously not responding to MAD
packets.


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

On Fri, Jun 03, 2005 at 01:52:31PM -0400, Hal Rosenstock wrote:
> Hi Troy,
> 
> On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote:
> > I'm having intermittent problems with opensm.. It seems after a while
> > IPoIB stops working and if I restart opensm, it starts spitting out
> > errors. 
> 
> Please try the following workaround and let me know if this makes things
> better.
> 
> -- Hal
> 
> Index: libvendor/osm_vendor_ibumad.c
> ===
> --- libvendor/osm_vendor_ibumad.c   (revision 2520)
> +++ libvendor/osm_vendor_ibumad.c   (working copy)
> @@ -402,7 +402,7 @@
>  
> p_vend->p_log = p_log;
> p_vend->timeout = timeout;
> -   p_vend->max_retries = OSM_DEFAULT_RETRY_COUNT;
> +   p_vend->max_retries = 1;
>  
> p_vend->umad_port_id = -1;
> p_vend->issmfd = -1;

No, it doesn't seem to help. To get anything to work at all, I seem to
need to reload all the IB modules on every maching I want to use ipoib
on.

There have been two times now I've been able to see about 4 ping
packets, and then one of the arp entries seems to go away.

(On the sm machine, also the machine I am trying to ping)
10.40.5.213  (incomplete)   ib0

(on another machine, trying to ping from..)
10.40.137.12ether   00:00:04:04:FE:80:00C   ib0
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

> > No, it doesn't seem to help. To get anything to work at all, I seem to
> > need to reload all the IB modules on every maching I want to use ipoib
> > on.
> > 
> > There have been two times now I've been able to see about 4 ping
> > packets, and then one of the arp entries seems to go away.
> > 
> > (On the sm machine, also the machine I am trying to ping)
> > 10.40.5.213  (incomplete)   ib0
> > 
> > (on another machine, trying to ping from..)
> > 10.40.137.12ether   00:00:04:04:FE:80:00C   ib0
> 
> That may be another issue. Are all your links active and the OpenSM
> appears to be behaving better now ?
> 

The links all seem better.. opensm definitely started working better
when the hung machine was rebooted.

I can't tell anything conclusive from the patch though.

How do I go about debugging the multicast arp/ipoib stuff?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

On Fri, Jun 03, 2005 at 03:33:48PM -0400, Hal Rosenstock wrote:
> On Fri, 2005-06-03 at 15:24, Troy Benjegerdes wrote:
> > I can't tell anything conclusive from the patch though.
> 
> OK. So you can't recreate the original problem ?
> 
> > How do I go about debugging the multicast arp/ipoib stuff?
> 
> Do you have any additional known good cables to try ?

I have several cables I *could* try, but I have no idea which ones are
good, or what ports are getting errors.

Some lids seem to respond to 'perfquery', but I haven't been able to get
any of my 8 port switches to respond. Is this something only in the 24
port switch chips?

Also, I have two machines in a state right now where they are printing
out:

kernel: unregister_netdevice: waiting for ib0 to become free. Usage
count = 1

Finally.. perfquery has two 'RcvBytes' fields.. should the second one be
'RcvPkts' ?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] opensm fails to bring up subnet..

[Troy Benjegerdes]

On Fri, Jun 03, 2005 at 04:08:05PM -0400, Hal Rosenstock wrote:
> On Fri, 2005-06-03 at 15:58, Troy Benjegerdes wrote:
> > > Do you have any additional known good cables to try ?
> > 
> > I have several cables I *could* try, but I have no idea which ones are
> > good, or what ports are getting errors.
> > 
> > Some lids seem to respond to 'perfquery', but I haven't been able to get
> > any of my 8 port switches to respond. Is this something only in the 24
> > port switch chips?
> 
> No. It may be the following: Although PMA is a required IB subagent per
> the IB spec, it might not be available in all vendor products. That's
> one possibility. Do all HCAs respond but not your switches ?

That's my suspicion too.


> > Also, I have two machines in a state right now where they are printing
> > out:
> > 
> > kernel: unregister_netdevice: waiting for ib0 to become free. Usage
> > count = 1
> 
> That's an IPoIB issue... What was the sequence of events which caused
> this ?

IPoIB quit working, so I wrote a script to ifconfig down ib0, unload all the
modules, then reload them. The 'waiting for ib0 to become free' happened
when the script tried to unload the modules.

Also, the following fixes ibchecknet for me.. (I seem to have a bad
cable or two)

gozer:~/gen2/trunk/src/userspace/management/diags/scripts# svn diff
Index: ibcheckerrors
===
--- ibcheckerrors   (revision 2532)
+++ ibcheckerrors   (working copy)
@@ -122,7 +122,7 @@
 function blue(s)
 {
if (mono)
-   normal(s)
+   printf s
else if (!quiet) {
printf "\033[1;034m" s
printf "\033[0;39m"
Index: ibcheckport
===
--- ibcheckport (revision 2532)
+++ ibcheckport (working copy)
@@ -65,7 +65,7 @@
 function blue(s)
 {
if (mono)
-   normal(s)
+   printf s
else if (!quiet) {
printf "\033[1;034m" s
printf "\033[0;39m"

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] 2.6.11.11 NFS over IPoIB crash

[Troy Benjegerdes]

We are running NFS over IPOIB, and are getting kernel panics under heavy 
NFS I/O. This is on a PowerMac G5, and the server is a dual opteron 
running 2.6.11 with the OpenIB code from subversion. It looks like a bug 
in nfs.. but we've only seen it using IPoIB... Is it worth trying to 
reproduce this over gigabit ethernet?


Brett Bode wrote:

Well I finally got 2.6.11.11 running this morning, but crashed it too. 
I guess I will try updating to the latest OpenIB code in subversion? 
Here is the kernel panic:


Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 POWERMAC
Modules linked in: ib_ipoib ib_sa eth1394 ib_mthca ib_mad ib_core 
ohci1394 ieee1394
NIP: C0038B58 XER:  LR: C00390C8 CTR: 
C003B340

REGS: c00254c845f0 TRAP: 0300   Not tainted  (2.6.11.11-G5)
MSR: 90001032 EE: 0 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 42028488
DAR: c0060045b218 DSISR: 4001
TASK: c19047d0[2004] 'IOTest.ppc.x' THREAD: c00254c84000 
CPU: 1
GPR00: 00060010 C00254C84870 C045F280 
C19047D0
GPR04: C04B4480  3B9A1A60 
C04B4488
GPR08: C045B208 C00254C84000 C0328C40 
C0328C40
GPR12: 28028428 C037E000 C043B170 
C0470340
GPR16:  0001  
C044EE20
GPR20:   0003 

GPR24: CFCC4000  CFD0B030 
0001
GPR28: C04B3B40 0001 C0391A90 
C00254C84870

NIP [c0038b58] .resched_task+0x38/0xcc
LR [c00390c8] .try_to_wake_up+0x304/0x324
Call Trace:
[c00254c848f0] [c00390c8] .try_to_wake_up+0x304/0x324
[c00254c849d0] [c003b3e0] .__wake_up_common+0x84/0xe8
[c00254c84a90] [c003b4a0] .__wake_up+0x5c/0x94
[c00254c84b40] [c0057840] .__queue_work+0x80/0xb4
[c00254c84be0] [c0057908] .queue_work+0x94/0xb0
[c00254c84c60] [c0280df8] .rpc_make_runnable+0xe0/0x154
[c00254c84cf0] [c0281324] .__rpc_do_wake_up_task+0xc8/0x24c
[c00254c84d80] [c0281608] .rpc_wake_up_task+0xb0/0xd8
[c00254c84e20] [c027e61c] .xprt_complete_rqst+0x90/0x188
[c00254c84ed0] [c027ec3c] .udp_data_ready+0x230/0x290
[c00254c84fa0] [c0236714] .udp_queue_rcv_skb+0x268/0x51c
[c00254c85040] [c0236fc8] .udp_rcv+0x600/0x7f0
[c00254c85160] [c0209e8c] .ip_local_deliver_finish+0xd4/0x35c
[c00254c851f0] [c01fa928] .nf_hook_slow+0x184/0x1bc
[c00254c852d0] [c020a360] .ip_local_deliver+0x24c/0x3bc
[c00254c85360] [c020a6f4] .ip_rcv_finish+0x224/0x378
[c00254c85410] [c01fa928] .nf_hook_slow+0x184/0x1bc
[c00254c854f0] [c020ad30] .ip_rcv+0x4e8/0x610
[c00254c855a0] [c01eac68] .netif_receive_skb+0x2ec/0x3a0
[c00254c85650] [c01eae34] .process_backlog+0x118/0x278
[c00254c85730] [c01eb07c] .net_rx_action+0xe8/0x24c
[c00254c857f0] [c0047ea8] .__do_softirq+0xdc/0x1b8
[c00254c858b0] [c004800c] .do_softirq+0x88/0x90
[c00254c85940] [c00480b4] .local_bh_enable+0xa0/0xa4
[c00254c859c0] [c01ea274] .dev_queue_xmit+0x2f4/0x3a0
[c00254c85a60] [c01f3308] .neigh_connected_output+0x104/0x1a0
[c00254c85b00] [c020e824] .ip_finish_output2+0x100/0x2dc
[c00254c85bc0] [c020fa18] .ip_fragment+0x4d8/0x800
[c00254c85cd0] [c024f768] .ip_refrag+0xa4/0xb0
[c00254c85d70] [c01fa2a4] .nf_iterate+0x128/0x190
[c00254c85e30] [c01fa86c] .nf_hook_slow+0xc8/0x1bc
[c00254c85f10] [c020ebd8] .ip_finish_output+0x1d8/0x36c
[c00254c85fd0] [c020fa18] .ip_fragment+0x4d8/0x800
[c00254c860e0] [c020e1e4] .dst_output+0x4c/0x80
[c00254c86170] [c01fa928] .nf_hook_slow+0x184/0x1bc
[c00254c86250] [c021162c] .ip_push_pending_frames+0x4ec/0x550
[c00254c86300] [c02353b8] 
.udp_push_pending_frames+0x16c/0x260

[c00254c863c0] [c0235af0] .udp_sendmsg+0x644/0x758
[c00254c86570] [c023fdd0] .inet_sendmsg+0x88/0xb8
[c00254c86610] [c01dba14] .sock_sendmsg+0xdc/0x13c
[c00254c86810] [c01dbab0] .kernel_sendmsg+0x3c/0x64
[c00254c868a0] [c01dff94] .sock_no_sendpage+0x9c/0xb8
[c00254c86970] [c0235d88] .udp_sendpage+0x184/0x200
[c00254c86a60] [c023fe9c] .inet_sendpage+0x9c/0x108
[c00254c86b10] [c028cd24] .xdr_sendpages+0x280/0x34c
[c00254c86c80] [c027fad0] .xprt_transmit+0x170/0x550
[c00254c86d50] [c027cfd4] .call_transmit+0x1cc/0x2bc
[c00254c86e00] [c02822fc] .__rpc_execute+0xf4/0x524
[c00254c86f60] [c00ff690] .nfs_execute_write+0x50/0x88
[c00254c87000] [c00ffbc0] .nfs_flush_list+0x4f8/0x644
[c00254

[openib-general] SDP: device mthca0 does not support fast memory regions

[Troy Benjegerdes]

I'm getting a 'SDP: device mthca0 does nto support fast memory regions'
error on PPC64 systems.. Is there something that needs to be done for
PPC, or could I have an older version of the mthca module hanging
around?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: SDP: device mthca0 does not support fast memory regions

[Troy Benjegerdes]

On Thu, Jun 16, 2005 at 10:03:55PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <[EMAIL PROTECTED]>:
> > Subject: Re: SDP: device mthca0 does not support fast memory regions
> > 
> > On Wed, Jun 15, 2005 at 04:30:47PM -0500, Troy Benjegerdes wrote:
> > > I'm getting a 'SDP: device mthca0 does nto support fast memory regions'
> > > error on PPC64 systems.. Is there something that needs to be done for
> > > PPC, or could I have an older version of the mthca module hanging
> > > around?
> > 
> >   I'm not sure about the state or plans for FMR support on PPC64, Roland
> > will have a better idea. 
> 
> Should just work AFAIK.
> But maybe your card hides the onboard memory?
> Try lspci -vv to see if the card has
> the huge bar (100Mbyte or more) corresponding to that memory.

Yes, the DDR is hidden on a couple of the machines, since they are IBM
Power4 systems, which have some stupid issue with the firmware.

However, I'm getting the following problem on an Xserve G5:

xl0:~# cat /proc/version
Linux version 2.6.11.11-G5 ([EMAIL PROTECTED]) (gcc version 3.4.2) #1 SMP
Wed Jun 812:11:29 CDT 2005

ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004)
ib_mthca: Initializing Mellanox Technology MT23108 InfiniHost
(0001:07:00.0)
WARN: : Failed to find lock pointer.
WARN: : Error <-22> initializing memory locking.
WARN: : Error <-22> initializing SDP IOCB table.
WARN: : INIT: Error <-22> initializing connection table.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation

[Troy Benjegerdes]

On Tue, Jun 28, 2005 at 05:27:09PM -0700, Greg KH wrote:
> On Tue, Jun 28, 2005 at 04:03:43PM -0700, Roland Dreier wrote:
> > +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 
> > 15:20:04.363963991 -0700
> > @@ -0,0 +1,708 @@
> > +/*
> > + * Copyright (c) 2005 Topspin Communications.  All rights reserved.
> > + * Copyright (c) 2005 Cisco Systems.  All rights reserved.
> > + *
> > + * This software is available to you under a choice of one of two
> > + * licenses.  You may choose to be licensed under the terms of the GNU
> > + * General Public License (GPL) Version 2, available from the file
> > + * COPYING in the main directory of this source tree, or the
> > + * OpenIB.org BSD license below:
> 
> Ok, I've complained about this before, but due to the fact that you are
> calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for
> it for someone to use the BSD license on it in the future, is pretty
> much impossible, right?

Only if someone tries to use it under a BSD license, strips off the GPL
notices, and then builds it against *Linux*. If linux-kernel is going to
be that fascist about licensing, let's please clean up all the binary
firmware blobs in header files first.

It seems reasonable to me that distribution and modification of the
*source code* can be under either license. But as soon as you build a
binary agaist the linux kernel, the binary is irrevocably GPL licensed.

--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation

[Troy Benjegerdes]

On Wed, Jun 29, 2005 at 09:12:09AM -0700, Greg KH wrote:
> On Tue, Jun 28, 2005 at 11:13:22PM -0500, Troy Benjegerdes wrote:
> > On Tue, Jun 28, 2005 at 05:27:09PM -0700, Greg KH wrote:
> > > On Tue, Jun 28, 2005 at 04:03:43PM -0700, Roland Dreier wrote:
> > > > +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 
> > > > 15:20:04.363963991 -0700
> > > > @@ -0,0 +1,708 @@
> > > > +/*
> > > > + * Copyright (c) 2005 Topspin Communications.  All rights reserved.
> > > > + * Copyright (c) 2005 Cisco Systems.  All rights reserved.
> > > > + *
> > > > + * This software is available to you under a choice of one of two
> > > > + * licenses.  You may choose to be licensed under the terms of the GNU
> > > > + * General Public License (GPL) Version 2, available from the file
> > > > + * COPYING in the main directory of this source tree, or the
> > > > + * OpenIB.org BSD license below:
> > > 
> > > Ok, I've complained about this before, but due to the fact that you are
> > > calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for
> > > it for someone to use the BSD license on it in the future, is pretty
> > > much impossible, right?
> > 
> > Only if someone tries to use it under a BSD license, strips off the GPL
> > notices, and then builds it against *Linux*.
> 
> Exactly, that's my point.  It's pretty useless, and if you are going to
> build this code for another OS, well, that's going to be a tough job :)
> 
> > If linux-kernel is going to be that fascist about licensing, let's
> > please clean up all the binary firmware blobs in header files first.
> 
> I'm not being "fascist", I'm just saying it's pretty pointless to try to
> dual license this code, that's all.

Ahh.. I think the point of the dual-license is that there is a lot of
non linux-specific Infiniband code that will (hopefully) be usefull on
other platforms where a BSD license might be more usefull. If for some
reason I decided I wanted to run MacOSX, I would at least want to be
running the OpenIB infiniband stack, and not some proprietary module.

Does anyone have some nice scripts to audit for useage of
EXPORT_SYMBOL_GPL only functions? Maybe it's worth trying to clean up
the code to clearly deliniate what depends on GPL functions and what
doesn't.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: TSO and IPoIB performance degradation

[Troy Benjegerdes]

On Thu, Apr 27, 2006 at 05:16:29PM -0700, Greg Lindahl wrote:
> On Thu, Apr 27, 2006 at 04:22:40PM -0700, Grant Grundler wrote:
> 
> > Anything preventnig such a gateway from routing SDP to ethernet?
> > Those gateways obviously will grok IB protocols.
> > I'm asking becuase I don't understand/know if there is a real
> > barrier to an IB -> ethernet gateway _without_ IPoIB.
> 
> I don't know if a SDP to ethernet gateway even exists, but I do know
> that it's a lot more work than just an IPoIB to ethernet gateway --
> the gateway is going to have to pass all its data through a TCP stack.
> So I would expect SDP to ethernet to not run very fast, especially on
> a gateway with lots of streams going.

And this is exactly the reason that we should not be playing games with
"infiniband specific" TCP optimzations. If you stay on the IB network,
use SDP or verbs. If you are going to cross networks, you want to be
running the full host TCP stack that has been well tested and is robust
to all the kinds of failures you see crossing networks. This does not
mean that it won't be fast, but you *will* have more overhead than on a
single network fabric.

If someone has a configureation where full TCP processing on
the host is a bottleneck and not the IPoIB to ethernet gateway, then
let's have this discussion again. But I don't believe such a
configuration actually exists anywhere. If you think you have some
problem like this, I would love to be able to run some benchmarks on the
system.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] opensm segfault?

[Troy Benjegerdes]

I got this after an indeterminate amount of time running opensm..


(gdb) bt
#0  0x2b90b0dbebf3 in cl_memcpy (p_dest=0x2ac88850, p_src=0x0,
count=64) at cl_memory_osd.c:87
#1  0x00415053 in osm_pkey_tbl_sync_new_blocks (
p_pkey_tbl=0x2ad99228) at osm_pkey.c:127
#2  0x00416687 in osm_pkey_mgr_process (p_osm=0x580e40)
at osm_pkey_mgr.c:407
#3  0x0043bb22 in osm_state_mgr_process (p_mgr=0x581ad8,
signal=3)
at osm_state_mgr.c:2243
#4  0x0043c88f in __osm_state_mgr_ctrl_disp_callback (
context=0x5819e8, p_data=0x3) at osm_state_mgr_ctrl.c:70
#5  0x2b90b0db9437 in __cl_disp_worker (context=0x5831f0)
at cl_dispatcher.c:108
#6  0x2b90b0dc1ca3 in __cl_thread_pool_routine (context=0x583268)
at cl_threadpool.c:78
#7  0x2b90b0dc1ae2 in __cl_thread_wrapper (arg=0x584750) at
cl_thread.c:61
#8  0x2b90b0fe3b1c in start_thread () from /lib/libpthread.so.0
#9  0x2b90b12c8273 in clone () from /lib/libc.so.6



And why the heck is "cl_memcpy" just a call to 'memcpy' anyway? This
just seems like excessive uneeded abstraction.

I'm running opensm from subversion rev 7091.. 

May 10 16:27:53 145969 [] -> OpenSM Rev:openib-1.2.0 OpenIB svn
6251:7091M

the only local changes are as follows:

[EMAIL PROTECTED]:/usr/src/openib-src/userspace/management$ svn diff
Index: osm/opensm/osm_port_info_rcv.c
===
--- osm/opensm/osm_port_info_rcv.c  (revision 7091)
+++ osm/opensm/osm_port_info_rcv.c  (working copy)
@@ -469,9 +469,14 @@
   goto Exit;
 }

+#if 0
 /* Check for IBM eHCA firmware defect in reporting partition
 * enforcement cap */
 if (cl_ntoh32(ib_node_info_get_vendor_id(&p_node->node_info)) ==
IBM_VENDOR_ID)
   p_switch->switch_info.enforce_cap = 0;
+#endif
+/* Check for busted divergenet switch on ameslab network */
+if (cl_ntoh64(p_node->node_info.node_guid) == 0x00084e000152)
+   p_switch->switch_info.enforce_cap = 0;

 /* Bail out if this is a switch with no partition enforcement
 * capability */
 if (cl_ntoh16(p_switch->switch_info.enforce_cap) == 0)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] EHCA broken for 2.6.16?

[Troy Benjegerdes]

Okay guys, what's up this time?

Kernel 2.6.16.. 

  CC [M]  drivers/infiniband/hw/ehca/ehca_main.o
In file included from drivers/infiniband/hw/ehca/ehca_qes.h:47,
 from drivers/infiniband/hw/ehca/ipz_pt_fn.h:46,
 from drivers/infiniband/hw/ehca/ehca_classes.h:46,
 from drivers/infiniband/hw/ehca/ehca_main.c:43:
drivers/infiniband/hw/ehca/ehca_tools.h: In function 'ehca2ib_return_code':
drivers/infiniband/hw/ehca/ehca_tools.h:404: error: 'H_SUCCESS'
undeclared (first use in this function)
drivers/infiniband/hw/ehca/ehca_tools.h:404: error: (Each undeclared
identifier is reported only once
drivers/infiniband/hw/ehca/ehca_tools.h:404: error: for each function it 
appears in.)
drivers/infiniband/hw/ehca/ehca_tools.h:406: error: 'H_BUSY' undeclared
(first use in this function)
drivers/infiniband/hw/ehca/ehca_tools.h:408: error: 'H_NO_MEM'
undeclared (first use in this function)
drivers/infiniband/hw/ehca/ehca_main.c: In function
'ehca_sense_attributes':

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] libehca.conf required, but no documentation..

[Troy Benjegerdes]

Please provide some documentation for libecha.conf.

There is not even a README in userspace/libehca.

Also, what firmware are the people doing echa development using? Most
versions I have tried seem to have some sort of nasty issue if the SM
doesn't bring the port active fast enough, which seems like some kind of
fundamental design flaw to me. Could someone please explain this so I
understand what is supposed to be happening?


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] [ANNOUNCE] NetPIPE 3.7 release candidate 1

[Troy Benjegerdes]

I am preparing to release an update to the NetPIPE benchmark
( http://scl.ameslab.gov/Projects/NetPIPE/NetPIPE.html ), and I would
very much like to hear some feedback on the OpenIB verbs implementation
(NPibv), and take any patches to make it build on Windows as well.

I would also like hear from anyone that has run NetPIPE on RDMA ethernet
hardware, since I currently don't have any to test with.

There is a tarball available at
http://scl.ameslab.gov/~troy/NetPIPE-3.7-rc1.tar.gz

and a mercurial repository at:

http://source.scl.ameslab.gov/hg/netpipe3-dev

If you have any interest in NetPIPE, please take a look and send any
comments and/or patches to [EMAIL PROTECTED] 

Thanks.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [ANNOUNCE] NetPIPE 3.7 release candidate 1

[Troy Benjegerdes]

Pradipta Kumar Banerjee wrote:

> Troy Benjegerdes wrote:
>
>> I am preparing to release an update to the NetPIPE benchmark
>> ( http://scl.ameslab.gov/Projects/NetPIPE/NetPIPE.html ), and I would
>> very much like to hear some feedback on the OpenIB verbs implementation
>> (NPibv), and take any patches to make it build on Windows as well.
>>
>> I would also like hear from anyone that has run NetPIPE on RDMA ethernet
>> hardware, since I currently don't have any to test with.
>
> Troy,
>  This unfortunately doesn't work with the iWARP devices. I tested it 
> on Ammasso iWARP cards. Ammasso iWARP cards create two interfaces - 
> one the usual ethernet interface (ethX) and another iWARP interface 
> (iwX).
> The RDMA protocol is run over the iWARP interface which doesn't allow 
> for normal socket based communication. Hence bind/listen etc is on the 
> ethX interface whereas for RDMA these should be on the iwX interface.
> Using RDMA CM solves the problem. Its a generic solution for both IB 
> and iWARP.
> ( For more information look at the following thread 
> http://thread.gmane.org/gmane.linux.drivers.openib/27629/focus=27629 )
>
> If you are interested I can try making the relevant changes to NetPIPE 
> and send it across to you.
>
> Thanks,
> Pradipta Kumar.

Is the connection manager api considered 'stable' yet? Last I knew it 
was still undergoing a lot of development.

Wouldn't it be possible to adapt the existing tcp connection setup code 
to work for Iwarp without using the connection manager? Most of the time 
when I test IB networks, I just bind to ethernet eth0 interfaces anyway.



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [ANNOUNCE] NetPIPE 3.7 release candidate 1

[Troy Benjegerdes]

Pradipta Kumar Banerjee wrote:
> Sean Hefty wrote:
>> Troy Benjegerdes wrote:
>>> Is the connection manager api considered 'stable' yet? Last I knew 
>>> it was still undergoing a lot of development.
>>
>> The connection manager API is relatively stable, and has been for a 
>> while. Changes to the API have come in the form of new options and 
>> calls, rather than changes to the existing calls.
> True. We have been using it for quite some time for all our work on 
> iWarp.
>
>>
>>> Wouldn't it be possible to adapt the existing tcp connection setup 
>>> code to work for Iwarp without using the connection manager? Most of 
>>> the time when I test IB networks, I just bind to ethernet eth0 
>>> interfaces anyway.
>>
>> My understanding is that it's not possible to connect iWarp QPs 
>> without using the connection manager.  For IB, it's possible to 
>> connect over sockets and exchange QP information over a TCP 
>> connection, but the same is not true for iWarp.
>>
>> - Sean
> Troy, the NetPIPE code needs to use the cm for successful working with 
> iwarp. May be the changes can be in a seperate file altogether (like 
> cm.c as in ibv.c) and have a seperate option in the Makefile to enable 
> the same.
>
>
in the future, I'd kinda like to deprecate the old out of bad IP QP 
connection setup, but it would be nice to separate the code into 
something like 'ib-cm.c'.

I would be happy to take a patch to use the CM instead of the IP 
connection setup we do now. I'm also interested in how to use the CM for 
things like the OpenFabrics verbs port of PVFS we are working on (which 
works on IB right now)


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] ehca issues, again

[Troy Benjegerdes]

This is the latest svn ehca code, 2.6.17 kernel.

Can I also request that the EHCA driver print out what PHYP firmware it 
is known to work with, just like mthca prints out a warning if the 
mellanox card firmware is out of date? And while I'm asking about PHYP, 
what version are the ehca developer running?


p5l6:~# modprobe hcad_mod nr_ports=1
eHCA Infiniband Device Driver (Rel.: SVNEHCA_0009)
xics_enable_irq: irq=36868: ibm_int_on returned -3
p5l6:~#
p5l6:~# ibv_devinfo
libibverbs: Fatal: couldn't read uverbs ABI version.
No IB devices found
p5l6:~# modprobe ib_uverbs
p5l6:~# ibv_devinfo
hca_id: ehca0
node_guid:  0002:5500:0003:5a00
sys_image_guid: :::
vendor_id:  0x5076
vendor_part_id: 0
hw_ver: 0x102
phys_port_cnt:  1
port:   1
state:  PORT_DOWN (1)
max_mtu:2048 (4)
active_mtu: 2048 (4)
sm_lid: 0
port_lid:   0
port_lmc:   0x00

p5l6:~# ibv_rc_pingpong
PU0003 000e0091:ehca_hcall_7arg_7ret HCAD_ERROR  opcode=168 
ret=ffd3 arg1=1304 arg2=2006 
arg3=ac0 arg4=e7a9b000 arg5=0 arg6=0 arg7=0 out1=0 out2=0 
out3=0 out4=0 out5=0 out6=800dad58 out7=0
PU0003 000b04c9:internal_modify_qp HCAD_ERROR  hipz_h_modify_qp() failed 
rc=ffd3 ehca_qp=c000e3626c80 qp_num=6
Failed to modify QP to INIT
p5l6:~# modprobe ib_umad
PU0007 000b0073:ehca_define_sqp HCAD_ERROR  Port 1 is not active.
PU0007 000b0258:ehca_create_qp HCAD_ERROR  ehca_define_sqp() failed 
rc=
PU0007 000b02c0:ehca_create_qp <<< failed ret=ffea
ib_mad: Couldn't create ib_mad QP1
ib_mad: Couldn't open ehca0 port 1




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] debian packages for ehca?

[Troy Benjegerdes]

I see there are debian packages in testing for libibverbs and mthca.. 
is the ehca userspace component ready to be packaged and put in debian
testing? If not, what does it need yet?

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] more ehca errors..

[Troy Benjegerdes]

Does this mean I have a mismatched kernel/firmware/ehca revision?

This is the 2.6.17 kernel with a relatively recent ehca driver from
subversion (SVNEHCA_0009)

p5l3:~#
p5l3:~# strace -ewrite -ewrite=all ibv_rc_pingpong
write(3, "\0\0\0\0\0\4\0\2\0\0\0\0\377\350\10\230", 16) = 16
 | 0  00 00 00 00 00 04 00 02  00 00 00 00 ff e8 08 98    |
write(3, "\0\0\0\3\0\4\0\1\0\0\0\0\377\350\10\270", 16) = 16
 | 0  00 00 00 03 00 04 00 01  00 00 00 00 ff e8 08 b8    |
write(3, "\0\0\0\t\0\f\0\3\0\0\0\0\377\350\10h\0\0\0\0\20\1`\0\0"..., 48) = -1 
EINVAL (Invalid argument)
write(2, "Couldn\'t allocate MR\n", 21Couldn't allocate MR ) = 21
   | 0  43 6f 75 6c 64 6e 27 74  20 61 6c 6c 6f 63 61 74  Couldn't allocat |
   | 00010  65 20 4d 52 0ae MR.|
p5l3:~#
p5l3:~#
p5l3:~# dmesg
PU0007 000e0091:ehca_hcall_7arg_7ret HCAD_ERROR  opcode=160
ret=ffd4 arg1=1304 arg2=5 arg3=10016000
arg4=1000 arg5=80 arg6=451573e0 arg7=0 out1=0 out2=0
out3=0 out4=0 out5=0 out6=0 out7=0
PU0007 00090454:ehca_reg_mr HCAD_ERROR  hipz_alloc_mr failed,
h_ret=ffd4 hca_hndl=1304
PU0007 00090478:ehca_reg_mr <<< ret=ffea shca=c2509800
e_mr=c003452d8e80 iova_start=10016000 size=1000 acl=1
e_pd=c003451573e0 pginfo=c001c4e3ba70 num_pages=1 num_4k=1
PU0007 00090176:ehca_reg_user_mr <<< rc=ffea
pd=c003451573e0 region=c003429d1ed0 mr_access_flags=1
udata=c001c4e3bba0


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] making sense of dapl (and dat.conf)

[Troy Benjegerdes]

So, let's suppose I build ibverbs, libecha/libmthca, and dapl from
subversion trunk.. what should my /etc/dat.conf file look like so things
actually work?

Right now I have:

OpenIB-cma-ip u1.2 nonthreadsafe default /usr/local/lib/libdaplcma.so
mv_dapl.1.2 "10.40.4.56 0" ""
OpenIB-cma-ip u1.2 nonthreadsafe default /usr/local/lib/libdaplcma.so
mv_dapl.1.2 "10.40.4.54 0" ""
OpenIB-cma-name u1.2 nonthreadsafe default /usr/local/lib/libdaplcma.so
mv_dapl.1.2 "p5l6.ib 0" ""
OpenIB-cma-name u1.2 nonthreadsafe default /usr/local/lib/libdaplcma.so
mv_dapl.1.2 "p5l4.ib 0" ""
OpenIB-cma-netdev u1.2 nonthreadsafe default
/usr/local/lib/libdaplcma.so mv_dapl.1.2 "ib0 0" ""

Is there any way I can avoid having to have a different config file for
each machine? Is this documented anywhere what all these fields actually
mean in a way that makes sense to those of us who haven't read the DAT
specification, or are familiar with ibverbs?

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] xt3 troubles (with OFED 1.0.1)

[Troy Benjegerdes]

On Tue, Aug 01, 2006 at 05:39:49PM -0400, Makia Minich wrote:
> So, after flailing about with my IPOIB issue on the XT3, I decided that
> perhaps a firmware upgrade (from 3.3.3 to 3.4.0) might be in order.  Prior
> to the upgrade, I was able to bring the entire stack online and see the
> infiniband network (provided I refrained from ifconfig'ing the ipoib
> interfaces).  Now, on loading ib_mthca I get the following out of dmesg:
> 
> ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February 14, 2006)
> ib_mthca: Initializing :04:00.0
> PCI: Unable to reserve mem region #3:[EMAIL PROTECTED] for device
> :04:00.0
> ib_mthca :04:00.0: Cannot obtain PCI resources, aborting.
> ib_mthca: probe of :04:00.0 failed with error -16
> 
> I saw an early-2005 thread about the P615 that seemed to have the same
> problem, but I couldn't find a resolution.  Does anyone know what might be
> able to be done?  Or perhaps where I should start looking?

You could probably start by encourageing your linux vendor to post their kernel
patches.. I bet they did something hokey in the PCI memory mapping.

Can you post the output of lspci -v ?

On the P615's, this was due to the firmware not allocating a big enough
MMIO region allocated.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] 'Couldn't post send' error?

[Troy Benjegerdes]

What's this mean?

da4:~/NetPIPE_3.6.2# ibv_rc_pingpong 10.1.5.218
  local address:  LID 0x0002, QPN 0x0d0404, PSN 0x599dea
  remote address: LID 0x0001, QPN 0x090404, PSN 0x93b0c8
Couldn't post send

I'm trying to get NetPIPE to work again, and it gets a similiar error.

It looks like 'uc' and 'ud' versions work just fine.

-- 
------
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 'Couldn't post send' error?

[Troy Benjegerdes]

On Fri, Aug 12, 2005 at 09:42:58PM -0500, Troy Benjegerdes wrote:
> What's this mean?
> 
> da4:~/NetPIPE_3.6.2# ibv_rc_pingpong 10.1.5.218
>   local address:  LID 0x0002, QPN 0x0d0404, PSN 0x599dea
>   remote address: LID 0x0001, QPN 0x090404, PSN 0x93b0c8
> Couldn't post send
> 
> I'm trying to get NetPIPE to work again, and it gets a similiar error.
> 
> It looks like 'uc' and 'ud' versions work just fine.

Interesting.. it looks like it's size dependent..

da5:~# ibv_rc_pingpong -s 65536
  local address:  LID 0x0001, QPN 0x250404, PSN 0xabcbf7
  remote address: LID 0x0002, QPN 0x2a0404, PSN 0xa2f603
131072000 bytes in 0.15 seconds = 7225.03 Mbit/sec
1000 iters in 0.15 seconds = 145.13 usec/iter
da5:~# ibv_rc_pingpong -s 512
  local address:  LID 0x0001, QPN 0x260404, PSN 0xa7be00
  remote address: LID 0x0002, QPN 0x2b0404, PSN 0x534952
Couldn't post send


I'm running the kernel modules from 2.6.13-rc6, and userspace libs
checked out from subversion sometime last week.

Do I need to have matching kernel and userspace versions? I can't find
anything in the kernel log messages. This is also with a new single-port
memfree card in a dual opteron iwill DK8NES motherboard, with the cards
connected back-to-back with no switch.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] mpi drop in openib tree

[Troy Benjegerdes]

On Thu, Aug 25, 2005 at 12:50:04PM -0400, Dhabaleswar Panda wrote:
> Hi Roland,
> 
> > As for whether MPI should be in the OpenIB subversion tree or not, my
> > personal opinion is that having MPI there is only appropriate if the
> > svn tree is being used as the primary development source tree.  I
> > don't think it's appropriate to use the OpenIB svn server as a release
> > distribution mechanism.
> 
> Yes, we plan to use the svn tree as the primary development source
> tree for mvapich-gen2 not as a release distribution mechanism.
> 
> In fact, during the last two days we have received a couple of
> patches, bug fixes, and suggestions and have checked-in these patches,
> fixes, and enhancements to the SVN tree with credits to the people who
> supplied us the patches. Thanks to those who sent us the patches!!
> 
> We encourage people in the OpenIB community to test the latest version
> of mvapich-gen2 from the svn tree and provide us feedbacks and
> comments so that we can keep on enhancing it.

What is the plan for merging with future MPICH updates? I've dealt with
keeping full copies of the linux kernel in various version control
systems, and merging with upstream always turns into a huge headache.
Of course, if there aren't going to be any more MPICH-1 updates, then
this is really no big deal.

My first suggestion is to remove the java .jar and configure files.
There is also probably a lot of other stuff that really doesn't need to
be there as well. Do we really need to keep windows specific code in the
(currently linux-only) OpenIB project svn tree?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [ANNOUCEv2] OpenIB OpenSM 1.1.0: trunk now supports 1.8.0 features

[Troy Benjegerdes]

On Tue, Sep 13, 2005 at 07:20:27AM -0400, Hal Rosenstock wrote:
> [This is a minor update to the previous announcement on this.]
> 
> OpenIB OpenSM 1.1.0 now includes the OpenSM 1.8.0 functionality.
> 
> Major thanks go to Yael Kalka and Eitan Zahavi of Mellanox.
> 
> This is a complete merge of the osm-1.8.0-branch up through version
> 3368.
> 
> There are 2 known caveats with this so far:
> 1. Some Anafa ports cannot be brought to active if not most recent
> firmware (5.3.0)
> 2. Solaris interoperability needs work again (I'm working on this)
> 
> New Features
> 
> Semistatic LID assignment 
>   No LID change on SM restart or node reboot 
>   Critical for IPoIB to avoid communication loss 
> Irresponsive port scan during light sweep 
>   No response but Link state not down 

We just had a node crash on our network, and it caused our OpenSM to
stop working.. we were running version openib-1.0.0..

I suppose this means I should start beating up on 1.1.0 now, right?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH v1/RFC] IB: Add SCSI RDMA Protocol (SRP) initiator

[Troy Benjegerdes]

Is there anyplace I can find an SRP target for Linux? What is available?
(Ideally, I'd like one for 2.6.1[3,4] )

On Tue, Sep 13, 2005 at 11:14:55AM -0700, Roland Dreier wrote:
> Sorry to interrupt the SAS arguments, but...
> 
> Here's the latest version of the InfiniBand SRP initiator.  I think
> it's ready for merging; I implemented error handling, which was the
> main thing pointed out from the generally positive reviews of my
> previous posting.  I've done a decent amount of testing without seeing
> any problems, and John Kingman has also tested against his SRP target.
> 
> Since this is a completely new driver and can't break anything,
> assuming the code looks good, does it seem OK to merge for 2.6.14?
> 
> Thanks,
>   Roland
> 
> 
> Add an InfiniBand SCSI RDMA Protocol (SRP) initiator.  This lets us
> talk to InfiniBand SRP targets (storage devices).
> 
> Signed-off-by: Roland Dreier <[EMAIL PROTECTED]>
> 
> ---
> 
>  drivers/infiniband/Kconfig  |2 
>  drivers/infiniband/Makefile |1 
>  drivers/infiniband/ulp/srp/Kbuild   |3 
>  drivers/infiniband/ulp/srp/Kconfig  |   11 
>  drivers/infiniband/ulp/srp/ib_srp.c | 1637 
> +++
>  drivers/infiniband/ulp/srp/ib_srp.h |  324 +++
>  6 files changed, 1978 insertions(+), 0 deletions(-)
> 
> 891de9c3d67dc4afc2e5c941bba96613bca81ae1
> diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
> --- a/drivers/infiniband/Kconfig
> +++ b/drivers/infiniband/Kconfig
> @@ -33,4 +33,6 @@ source "drivers/infiniband/hw/mthca/Kcon
>  
>  source "drivers/infiniband/ulp/ipoib/Kconfig"
>  
> +source "drivers/infiniband/ulp/srp/Kconfig"
> +
>  endmenu
> diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile
> --- a/drivers/infiniband/Makefile
> +++ b/drivers/infiniband/Makefile
> @@ -1,3 +1,4 @@
>  obj-$(CONFIG_INFINIBAND) += core/
>  obj-$(CONFIG_INFINIBAND_MTHCA)   += hw/mthca/
>  obj-$(CONFIG_INFINIBAND_IPOIB)   += ulp/ipoib/
> +obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/
> diff --git a/drivers/infiniband/ulp/srp/Kbuild 
> b/drivers/infiniband/ulp/srp/Kbuild
> new file mode 100644
> --- /dev/null
> +++ b/drivers/infiniband/ulp/srp/Kbuild
> @@ -0,0 +1,3 @@
> +EXTRA_CFLAGS += -Idrivers/infiniband/include
> +
> +obj-$(CONFIG_INFINIBAND_SRP) += ib_srp.o
> diff --git a/drivers/infiniband/ulp/srp/Kconfig 
> b/drivers/infiniband/ulp/srp/Kconfig
> new file mode 100644
> --- /dev/null
> +++ b/drivers/infiniband/ulp/srp/Kconfig
> @@ -0,0 +1,11 @@
> +config INFINIBAND_SRP
> + tristate "InfiniBand SCSI RDMA Protocol"
> + depends on INFINIBAND && SCSI
> + ---help---
> +   Support for the SCSI RDMA Protocol over InfiniBand.  This
> +   allows you to access storage devices that speak SRP over
> +   InfiniBand.
> +
> +   The SRP protocol is defined by the INCITS T10 technical
> +   committee.  See .
> +
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c 
> b/drivers/infiniband/ulp/srp/ib_srp.c
> new file mode 100644
> --- /dev/null
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -0,0 +1,1637 @@
> +/*
> + * Copyright (c) 2005 Cisco Systems.  All rights reserved.
> + *
> + * This software is available to you under a choice of one of two
> + * licenses.  You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + *  - Redistributions of source code must retain the above
> + *copyright notice, this list of conditions and the following
> + *disclaimer.
> + *
> + *  - Redistributions in binary form must reproduce the above
> + *copyright notice, this list of conditions and the following
> + *disclaimer in the documentation and/or other materials
> + *provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + *
> + * $Id: ib_srp.c 3395 2005-09-13 05:10:39Z roland $
> + */
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#include 
> +
> +#include 
> +#include 
> +#include 
> +
> +#include 
> +
> +#include "ib_srp.h"
> +
> +#define DRV_NAME "ib_srp"
> +#define PFX

Re: [openib-general] IBM eHCA Device Driver for gen2 IB stack

[Troy Benjegerdes]

On Fri, Jul 22, 2005 at 01:41:31PM +0200, IBMEHCA DD wrote:
> Hi,
> we've completed the first alpha code drop of the Power5 IBM eHCA Device 
> Driver for the for the gen2 openib.org stack.
> We're running IPoIB and ibv userspace programs successfully with this code 
> in our lab setup.
> 
> The source files can be downloaded from
> https://sourceforge.net/projects/ibmehcad/
> ehca2_0011e

Has this been updated at all? Are there any new drops?

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH v1/RFC] IB: Add SCSI RDMA Protocol (SRP) initiator

[Troy Benjegerdes]

On Tue, Sep 13, 2005 at 02:52:06PM -0700, Roland Dreier wrote:
> >>>>> "Troy" == Troy Benjegerdes <[EMAIL PROTECTED]> writes:
> 
> Troy> Is there anyplace I can find an SRP target for Linux? What
> Troy> is available?  (Ideally, I'd like one for 2.6.1[3,4] )
> 
> I don't believe there are any Linux targets available.  Just FC
> gateways and IB storage devices.
> 
> (And I don't think it takes quoting a 2000-line email and top-posting
> to ask the question either ;)

I coulda swore I deleted the rest. At least I trimmed lkml from the
list, or I'd really get flamed ;)

Does anyone even have a binary SRP target module for $SOME_LINUX_DISTRO?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [ANNOUCEv2] OpenIB OpenSM 1.1.0: trunk now supports 1.8.0 features

[Troy Benjegerdes]

> Were you running any other ULPs other than IPoIB ?

Not at the time. Maybe only MPI.

> > I've got a log now of the latest opensm-1.1.0 attached.
> > 
> > The time (was) off on that machine, FYI. 
> > 
> > At the log entry 'Sep 13 12:06:55', I plugged in the node that is 
> > hung/crashed
> > .. which caused a bunch of opensm errors.. 
> 
> Thanks. That helps orient me. What was the opensm crash ? The log just
> ends abruptly.

OpenSM didn't crash, but the IPoIB multicast seemed broken. The opensm
instance that generated that log is still currently running. Brett can
hopefully tell us tommorow what (if anything) is still broken.

We have another node that crashed, and have seen this behavior before
where one node crashes and gets into some strange state that OpenSM
can't handle well.

> 
> > I have since unplugged that
> > node, and can put it back in tommorow if you want more debug info.
> 
> Great. More later on the log itself...
> 
> -- Hal
> 

-- 
--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH] udapl: PPC64 cpuinfo change

[Troy Benjegerdes]

On Thu, Oct 06, 2005 at 02:14:08PM -0700, Grant Grundler wrote:
> On Thu, Oct 06, 2005 at 11:48:02AM -0600, Todd Bowman wrote:
> > /proc/cpuinfo on PPC64 prints different label for processor speed.
> ...
> 
> ISTR the "clock" value in cpuinfo is NOT the same as the CPU MHz.
> Can you remind me if "clock" value * "mtfb" results in
> "wall clock" time units?
> 
> If not, then use of DT_CPU_MHZ_MHZ needs to be reviewed since
> it typically makes that assumption.
> 
> Also, if someone cares about sparc (hey Tom! :^) ),
> then might leverage the get_clock.c code on:
>   http://svn.gnumonks.org/trunk/mmio_test/

Oh boy is there some reason 'gettimeofday' does not work? Trying to
infer timebase/clock/rtsc frequency is going to be a mess. Think cpus
that dynamically change frequency.. Laptops do now.. how long before
something with infiniband does and breaks this code horribly? (think
embedded systems)

There are a couple of implementations of gettimeofday fully in userspace
that hide the details and still read the high-res hardware counters. Google
for 'vDSO gettimeofday'.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] IBM eHCA testing..

[Troy Benjegerdes]

I have two IBM eHCA cards installed and it appears that OpenSM
is happily talking to the firmware and bringing up the links.

So now I'm looking at the install instructions for the ehca2_EHCA2_0025.tgz
code drop, and wondering what (if any) issues there are with a 2.6.13
kernel, or later OpenIB svn drops. 

Is there a later code drop I can get ahold of? Is the nr_ports issue
something in the driver? I wound up connecting to the lower port in the
Openpower720 machine.. do you know if that's port 1 or 2?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] IBM eHCA testing..

[Troy Benjegerdes]

On Fri, Oct 07, 2005 at 09:33:27AM -0700, Shirley Ma wrote:
> Hi, Troy,
> 
> There is INSTALL file in the EHCA driver package.
> In OpenPower 720 port 1 is at the top, port 2 is at the bottom.
> In P570, port1 is at the bottom, port2 is at the top.

Okay, I guess I should read more carefully ;)

What is the issue with needing to use port 1? Can that be fixed in the
driver, or does that need a firmware update?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] IBM eHCA testing..

[Troy Benjegerdes]

What's the status on getting the ehca driver integrated into subversion?
If there's something holding it up, can we at least get a version that
can be dropped into drivers/infiniband/hw ?

Also, one final note, is it really appropriate to have ehca/ebus in the
infiniband directory? It's really a PPC64 specific driver that works for
more than just the ehca device, correct?

I have the correct port plugged in now, and I can see the logical HCA device
in the output of 'ibnetdiscover' (from another node), but trying to bring up 
ib0 caused this:

[  381.453731] eHCA Infiniband Device Driver (Rel.: EHCA2_0025)
[  381.458602] xics_enable_irq: irq=36868: ibm_int_on returned fffd
[  393.378143] eHCA Infiniband Device Driver (Rel.: EHCA2_0025)
[  452.658083] PU0002 000b0075:ehca_define_sqp HCAD_ERROR  Port 1 is not
active.
[  452.658106] PU0002 000b0383:ehca_create_qp HCAD_ERROR
ehca_define_sqp() failed rc=
[  452.821917] PU0002 000b03aa:ehca_create_qp <<< failed ret=ffea
[  452.821939] ib_mad: Couldn't create ib_mad QP1
[  453.313412] ib_mad: Couldn't open ehca0 port 1
[  475.132318] PU0002 00060100:ehca_parse_ec  EHCA port 1 is available.
[  518.249381] PU0007 000b00b9:plpar_hcall_7arg_7ret HCAD_ERROR
HCALL77_IN r3=168 r4=1304 r5=2008
r6=8a40 r7=1e4e49000 r8=0 r9=0 r10=0
[  518.249411] PU0007 000b00c0:plpar_hcall_7arg_7ret HCAD_ERROR
HCALL77_OUT r3=ffd3 r4=0 r5=0 r6=0 r7=4 r8=0
r9=8005aa18 r10=0
[  518.249438] PU0007 000b0560:internal_modify_qp HCAD_ERROR
hipz_h_modify_qp() failed rc=ffd3 ehca_qp=cf2cd080
qp_num=8
[  518.249460] ib0: failed to modify QP to init, ret = -22
[  518.418976] ib0: ipoib_qp_create returned -22
[  528.813491] Oops: Kernel access of bad area, sig: 11 [#1]
[  528.813505] SMP NR_CPUS=8 NUMA PSERIES LPAR
[  528.813517] Modules linked in: ib_ipoib ib_sa ib_mad hcad_mod ib_core
ebus
[  528.813540] NIP: D0049C6C XER: 2020 LR: D00760A0
CTR: D0049C60
[  528.813554] REGS: cf1eb1d0 TRAP: 0300   Not tainted
(2.6.13.3-power5)
[  528.813568] MSR: 80009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
CR: 22028422
[  528.813580] DAR:  DSISR: 4000
[  528.813592] TASK: c209a9a0[2021] 'ifconfig' THREAD:
cf1e8000 CPU: 0
[  528.813605] GPR00: D00760A0 CF1EB450 D005FFF0

[  528.813625] GPR04: CF1EB548 0071 CF1EB540
0001
[  528.813645] GPR08: 000B 0001 0004
D0049C60
[  528.813664] GPR12: D00774C0 C04B4000 100C
100A
[  528.813685] GPR16:   1002
1002
[  528.813704] GPR20: 1001E71C C001E466C000 8914
C001E46D4810
[  528.813725] GPR24: C001E46D4800 CF43B900 CF1EBD10
0002
[  528.813745] GPR28:  C001E466C380 D0084640
CF1EB548
[  528.813768] NIP [d0049c6c] .ib_modify_qp+0xc/0x40 [ib_core]
[  528.813797] LR [d00760a0] .ipoib_qp_create+0xe0/0x1c0
[ib_ipoib]
[  528.813822] Call Trace:
[  528.813829] [cf1eb450] [434849c5] 0x434849c5
(unreliable)
[  528.813846] [cf1eb4d0] [d00760a0]
.ipoib_qp_create+0xe0/0x1c0 [ib_ipoib]
[  528.813873] [cf1eb5f0] [d007261c]
.ipoib_ib_dev_open+0x2c/0x120 [ib_ipoib]
[  528.813899] [cf1eb680] [d006f38c]
.ipoib_open+0x7c/0x190 [ib_ipoib]
[  528.813923] [cf1eb720] [c032a650]
.dev_open+0xc0/0x120
[  528.813942] [cf1eb7c0] [c0328c70]
.dev_change_flags+0x180/0x1c0
[  528.813961] [cf1eb860] [c037a02c]
.devinet_ioctl+0x81c/0x850
[  528.813980] [cf1eb970] [c037a65c]
.inet_ioctl+0x27c/0x2d0
[  528.813998] [cf1eba00] [c031bc4c]
.sock_ioctl+0x8c/0x440
[  528.814016] [cf1ebaa0] [c00c22f0]
.do_ioctl+0x60/0x120
[  528.814033] [cf1ebb40] [c00c244c]
.vfs_ioctl+0x9c/0x4d0
[  528.814050] [cf1ebbf0] [c00c28cc]
.sys_ioctl+0x4c/0xa0
[  528.814066] [cf1ebca0] [c001bb24]
.dev_ifsioc+0x84/0x390
[  528.814084] [cf1ebd70] [c00e4d88]
.compat_sys_ioctl+0x158/0x500
[  528.814103] [cf1ebe30] [c000d300]
syscall_exit+0x0/0x18
[  528.814119] Instruction dump:
[  528.814126] 7c601b78 38210080 7c030378 e8010010 7c0803a6 4e800020
6000 6000
[  528.814150] 6000 7c0802a6 f8010010 f821ff81  e9490170
e80a f8410028
[  528.814174]  <7>RTAS: event: 3, Type: Platform Error, Severity: 2

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] IBM eHCA testing..

[Troy Benjegerdes]

On Tue, Oct 11, 2005 at 09:13:20AM -0700, Shirley Ma wrote:
> The IB stack doesn't handle errors during client initialization. This 
> problem is easy to reproduce by inducing errors (resouce allocation 
> failure or query failure) in mad_client or sa_client registration. I am 
> working on a patch, but I am in class the whole week, don't have time to 
> verify the patch. I hope the patch will be available early next week to 
> fix the panic. 

I'd be happy to verify the patch, but I need to get the latest version
of the ehca driver, ideally already integrated into the subversion tree.
Otherwise a tar.gz I can extract and drop in drivers/infiniband/hw/ehca
would work just fine.

I'm still not sure I got an answer why the ehca is so senstive to which 
port is plugged in.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] IBM eHCA testing..

[Troy Benjegerdes]

What is the turnaround time on a firmware change? If we can get an
update, I think that would be the best solution. I'll be happy to test
this.

On Wed, Oct 12, 2005 at 11:36:59AM +0200, IBMEHCA DD wrote:
> This is basically the answer why its so "sensitive" which port is plugged.
> We're working on a solution to that problem.
> But currently we only see a chance to change this behaviour by also 
> changing the firmware interface,
> which needs to be coordinated with firmware development.
> 
> Roland Dreier wrote on 10.10.2005 23:44:21:
> 
> > IBMEHCA> So you need some kind of signal from the operating system
> > IBMEHCA> to system firmware, which in the eHCA case is the
> > IBMEHCA> H_DEFINE_AQP1 triggered by ib_create_qp with IB_QPT_GSI
> > IBMEHCA> parameter.  AFTER that call handshaking between system
> > IBMEHCA> firmware and the SM will start, here's a new adapter
> > IBMEHCA> active on a switch port... what's your guid? here's your
> > IBMEHCA> LID, p_key, SM lid  ...and after all that it's
> > IBMEHCA> possible to send and receive packets from the fabric.
> > IBMEHCA> The openib stack expects that a port is fully functional
> > IBMEHCA> after this create_qp returns, and starts to do all sorts
> > IBMEHCA> of modify QP and post send.  So the only choice we have
> > IBMEHCA> there is to delay create_qp until the complete
> > IBMEHCA> handshaking between system firmware and the SM has
> > IBMEHCA> finished (until we see a IB_PORT_ACTIVE in hcad_mod). If
> > IBMEHCA> we don't see that until EHCA_PORT_ACTIVE_TIMEOUT we have
> > IBMEHCA> to return an error code to openib, otherwise we're
> > IBMEHCA> seriously in trouble (tried that).
> > 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: IBM eHCA testing..

[Troy Benjegerdes]

On Wed, Oct 12, 2005 at 01:04:37PM +0200, IBMEHCA DD wrote:
> I just released the ehca2_0028 which uses svn 3615 on 
> https://sourceforge.net/projects/ibmehcad/
> As you might notice the license already has changed to the openib.org 
> license.
> 
> With 2.6.13 we had the non-issue that our maun focus was on 2.6.5-7.191 
> and we're only now moving to the latest kernel.

I just built against svn 3774, and 2.6.13.3, with the timeout set to 120
seconds. There's some bad interaction going on with OpenSM.

p5l2:~# modprobe hcad_mod ehca_nr_ports=1
[ 6186.855237] eBus Device Driver
[ 6186.907578] eHCA Infiniband Device Driver (Rel.: EHCA2_0028)
[ 6186.912203] xics_enable_irq: irq=36868: ibm_int_on returned fffd
p5l2:~# modprobe ib_ipoib
hang for awhile.. entries appear in osm.log ***
[ 6309.683651] PU0003 00060103:ehca_parse_ec  EHCA port 1 is available.
[ 6310.253303] kernel BUG in dma_map_single at arch/ppc64/kernel/dma.c:86!
[ 6310.253320] Oops: Exception in kernel mode, sig: 5 [#1]
[ 6310.253339] SMP NR_CPUS=8 NUMA PSERIES LPAR
[ 6310.253364] Modules linked in: ib_mad hcad_mod ib_core ebus
[ 6310.253383] NIP: C000FA10 XER: 0020 LR: C000F9B0 CTR: 
C000F980
[ 6310.253400] REGS: cf3bb770 TRAP: 0700   Not tainted (2.6.13.3-power5)
[ 6310.253421] MSR: 80029032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 
24002444
[ 6310.253436] DAR:  DSISR: 
[ 6310.253471] TASK: c209f060[1874] 'modprobe' THREAD: 
cf3b8000CPU: 7
[ 6310.253492] GPR00: C04B3660 CF3BB9F0 C05EE948 
C001DBEC5C18
[ 6310.253513] GPR04: C003CB5B1D0C 0128 0002 
0008
[ 6310.253532] GPR08: C003CBD5EEE8  CF67FC00 
C000F980
[ 6310.253553] GPR12: D00621D0 C04B7800 10017078 

[ 6310.253609] GPR16:   0001 
0001
[ 6310.253665] GPR20: C8DE7800 0002 0001 
CF67FDC8
[ 6310.253688] GPR24: CF67FD40 0002 C001DBEC5C18 
0002
[ 6310.253708] GPR28: 0128 C003CB5B1D0C D006EB00 
C003CB5B1C80
[ 6310.253731] NIP [c000fa10] .dma_map_single+0x90/0xc0
[ 6310.253753] LR [c000f9b0] .dma_map_single+0x30/0xc0
[ 6310.253778] Call Trace:
[ 6310.253797] [cf3bb9f0] [c8de7800] 0xc8de7800 
(unreliable)
[ 6310.253838] [cf3bba90] [d005aee8] 
.ib_mad_post_receive_mads+0xb8/0x270 [ib_mad]
[ 6310.253880] [cf3bbb80] [d005c840] 
.ib_mad_init_device+0x350/0x660 [ib_mad]
[ 6310.253905] [cf3bbc70] [d004d0bc] 
.ib_register_client+0xdc/0x150 [ib_core]
[ 6310.253936] [cf3bbd00] [d0061e6c] 
.ib_mad_init_module+0x8c/0xf0 [ib_mad]
[ 6310.253999] [cf3bbd90] [c0070720] 
.sys_init_module+0x1e0/0x4d0
[ 6310.254030] [cf3bbe30] [c000d300] syscall_exit+0x0/0x18
[ 6310.254045] Instruction dump:
[ 6310.254053] 4e800421 e8410028 382100a0 e8010010 eb41ffd0 eb61ffd8 eb81ffe0 
eba1ffe8
[ 6310.254089] 7c0803a6 4e800020 6000 6000 <0fe0> 382100a0 
3860e8010010
[ 6310.254206]  Segmentation fault

I'm also attaching part of an opensm log file.

(the full copy is at http://scl.ameslab.gov/~troy/osm-ehca.log )

The IBM galaxy adapters are at:
Initial path: [0][1][16]
Initial path: [0][1][13]

00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00

00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00

Oct 13 10:42:05 978875 [42FFF970] -> umad_receiver: ERR 5409: send completed 
with error (method=1 attr=16) -- dropping.
Oct 13 10:42:05 978883 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 
hop count 2 DR SLID 0x0 DR DLID 0x0
Oct 13 10:42:05 978892 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: 
MAD completed in error (IB_TIMEOUT).
Oct 13 10:42:05 978925 [42FFF970] -> SMP dump:
base_ver0x1
mgmt_class..0x81
class_ver...0x1
method..0x1 (SubnGet)
D bit...0x0
status..0x0
hop_ptr.0x0
hop_count...0x2
trans_id0x1810
attr_id.0x16 (P_KeyTable)
resv0x0
attr_mod0x3E
m_key...0x
dr_slid.0x
dr_dlid.

Re: [openib-general] Re: IBM eHCA testing..

[Troy Benjegerdes]

Hal Rosenstock wrote:


On Thu, 2005-10-13 at 18:46, Troy Benjegerdes wrote:
 


I'm also attaching part of an opensm log file.

(the full copy is at http://scl.ameslab.gov/~troy/osm-ehca.log )

The IBM galaxy adapters are at:
Initial path: [0][1][16]
Initial path: [0][1][13]

   



The OpenSM is just saying that a SMP transaction it issued (in this
case, SM Get P_KeyTable) is timing out (no response made it back to
OpenSM).

BTW, what svn rev is OpenSM up to ?

-- Hal
 


So, how about a patch to opensm to report what svn rev it was built from ;)

I just discovered another problem.. We have been running pfvs2 over 
IPoIB on the same subnet, and in debugging this, I restarted opensm 
several times, and somewhere in the stack a PVFS2 write failed. I 
wouldn't think that a short downtime of the SM from restarting it would 
cause any IPoIB TCP sessions to fall over..

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Cray XD1 and OpenSM.. (ignoreing certain guids?)

[Troy Benjegerdes]

In the interest of plugging absolutely everything I have with infiniband
ports together and seeing what falls over, I connected a Cray XD1 to a
small (2 machine) infiniband network running OpenSM.

Ideally, I'd like to find out what sort of minimal emulation code needs
to be running on the XD1 nodes to answer standards compliant SM
requests. Failing that, Is there a way we can tell OpenSM to ignore parts 
of the IB network? I am seeing a lot of stuff in the osm.log like this:

(port 16 is on the internal mellanox chip in the XD1)

ct 14 18:07:22 646959 [43005960] -> umad_receiver: ERR 5409: send
completed with error (method=1 attr=15) -- dropping.
Oct 14 18:07:22 646972 [43005960] -> umad_receiver: ERR 5411: DR SMP hop
ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Oct 14 18:07:22 646983 [43005960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
3113: MAD completed in error (IB_TIMEOUT).
Oct 14 18:07:22 647020 [43005960] -> SMP dump:
base_ver0x1
mgmt_class..0x81
class_ver...0x1
method..0x1 (SubnGet)
D bit...0x0
status..0x0
hop_ptr.0x0
hop_count...0x3
trans_id0x1420
attr_id.0x15 (PortInfo)
resv0x0
attr_mod0x0
m_key...0x
dr_slid.0x
dr_dlid.0x

Initial path: [0][1][4][16]
Return path:  [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]

00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00

00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00

00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00

00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Cray XD1 and OpenSM.. (ignoreing certain guids?)

[Troy Benjegerdes]

I'm unaware of such an option. Not sure how you would specify which
nodes to ignore. Why would you want them on the net if they are to be
ignored ? 


Nodes are supposed to be IB compliant: SMA is a required component of
all nodes.
   



So I presume there is no SMA for the Cray XD1. If someone is going to
implement this, we can document what portion of the SMA needs to
implemented to work with OpenSM. That wouldn't necessarily guarantee it
should work with any SM as other SMs may rely on some slightly different
things or do things in a slightly different way since there is much more
flexibility allowed on the SM side.

-- Hal
 

I would be interested to know what minimal set of the SMA needs to be 
implemented. Does anyone have any SMA code in openib project yet? If so 
it might be a (theoretically) simple matter of running that code on the 
Cray XD1 nodes, and having the cray hardware pass the SMA packets up to 
the linux kernel.


Is the SMA implemented in firmware on mellanox cards? And if there's a 
'soft-sma'  available, could it be made to work on pathscale cards as well?

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Cray XD1 and OpenSM.. (ignoreing certain guids?)

[Troy Benjegerdes]

It's more than that: in addition to the IB hardware/driver difference,
it will need to be ported from Linux to whatever Cray OS is.

 

The Cray XD1 is actually running Linux.. I've even managed to build and 
boot my own kernel on one. They are actually using a derivative of the 
OpenIB SDP code that has been ported to what they call "RapidArray", 
which is infiniband at the wire-protocol level. They don't implement any 
of the higher level stuff (like, obviously,  SMA)


 

And if there's a 
'soft-sma'  available, could it be made to work on pathscale cards as well?
   



PathScale has an soft SMA for their HCAs already (for Linux at least).

-- Hal

 

I'm going to take a look at pathscale's sma when I get a chance. (which 
in reality probably won't be until after SC)

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] where is IB_WARN defined?

[Troy Benjegerdes]

I'm trying to rebuild opensm, and the libibumad configure is failing because 
IB_WARN is apparently not defined anyplace I can find it.

-- 
--
Troy Benjegerdes'da hozer'[EMAIL PROTECTED]  

Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] EHCA ipoib error..

[Troy Benjegerdes]

I get the following errors when trying to bring up ipoib:

10:~# modprobe ib_hcad_mod ehca_nr_ports=1
FATAL: Module ib_hcad_mod not found.
10:~# modprobe hcad_mod ehca_nr_ports=1
[ 1401.993165] eHCA Infiniband Device Driver (Rel.: EHCA2_0028)
[ 1401.994486] xics_enable_irq: irq=36868: ibm_int_on returned fffd
[ 1489.471617] PU 00060103:ehca_parse_ec  EHCA port 1 is available.
10:~# modprobe ib_ipoib
10:~# ifconfig ib0 10.40.4.52 netmask 255.255.0.0
[ 1697.767708] PU 000b0544:internal_modify_qp HCAD_ERROR   ehca_qp=c001cd7b2c80 qp_num=1 req_mask=21
opt_mask=88040 submitted_mask=3 qp_type=1
[ 1697.767734] ib_mad: mad_error_handler - ib_modify_qp to RTS : -22

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] where is IB_WARN defined?

[Troy Benjegerdes]

On Wed, Oct 19, 2005 at 04:30:06PM -0400, Hal Rosenstock wrote:
> On Wed, 2005-10-19 at 16:09, Troy Benjegerdes wrote:
> > I'm trying to rebuild opensm, and the libibumad configure is failing 
> > because 
> > IB_WARN is apparently not defined anyplace I can find it.
> 
> It should be IBWARN. Where is IB_WARN or do you mean IBWARN ?
> 
> It should be found in libibcommon/include/infiniband/common.h
> 
> What svn version are you using ? Have you updated all your management
> libraries ?

Hrrm..  it looks like for some reason the top level makefile didn't
rebuild libibcommon. 

Also, is there a clean way we could use a common autocong/automake setup
for the diags? The build spends more time running configure than
compiling ;)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [RFC] OpenSM Interactive Console

[Troy Benjegerdes]

On Tue, Oct 18, 2005 at 03:10:31PM -0400, Hal Rosenstock wrote:
> Currently, OpenSM does not support an interactive console. There has
> been a desire to introduce the ability to change certain parameters (as
> well as display things) once OpenSM has started. This patch introduces
> the first most basic commands: help and loglevel. I am investgating
> adding smpriority to this. The console is invoked by specifying -console
> as an option on the opensm command line.
> 
> If you have a request for a command you would like in the console, I
> would like to compile a list of these. 
> 
> Comments ?

As well as a console, I'd like an API for some way for external programs 
(say a cluster queue manager) to be able to query the SM (or the sm + some
helper library) for the following things:

* Topology
* guid/lid/IPoIB address/switch port mappings
* link state

Future neat things to do:

* An interface to dynamically partition the fabric
* Register for notifications for certain events (excessive traffic
queueing, or error counts)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] EHCA-0028 userspace build fails with openib svn 3774

[Troy Benjegerdes]

make[1]: Entering directory `/usr/src/openib-src/userspace/libehca'
if /bin/sh ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I.-O3 -g -Wall  -D_GNU_SOURCE -DP_SERIES -I../libibverbs/include -Isrc
-g -O2 -MTsrc_libehca_la-ehca_umain.lo -MD -MP -MF
".deps/src_libehca_la-ehca_umain.Tpo" -c -o src_libehca_la-ehca_umain.lo
`test -f 'src/ehca_umain.c' || echo './'`src/ehca_umain.c; \
then mv -f ".deps/src_libehca_la-ehca_umain.Tpo"
".deps/src_libehca_la-ehca_umain.Plo"; else rm -f
".deps/src_libehca_la-ehca_umain.Tpo"; exit 1; fi
mkdir .libs
 gcc -DHAVE_CONFIG_H -I. -I. -I. -O3 -g -Wall -D_GNU_SOURCE -DP_SERIES
 -I../libibverbs/include -Isrc -g -O2 -MT src_libehca_la-ehca_umain.lo
 -MD -MP -MF .deps/src_libehca_la-ehca_umain.Tpo -c src/ehca_umain.c
 -fPIC -DPIC -o .libs/src_libehca_la-ehca_umain.o
 src/ehca_umain.c: In function 'ehcau_query_device':
 src/ehca_umain.c:66: warning: passing argument 3 of
 'ibv_cmd_query_device' fromincompatible pointer type
 src/ehca_umain.c:66: warning: passing argument 4 of
 'ibv_cmd_query_device' makes pointer from integer without a cast
 src/ehca_umain.c:66: error: too few arguments to function
 'ibv_cmd_query_device'
 make[1]: *** [src_libehca_la-ehca_umain.lo] Error 1

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] where is IB_WARN defined?

[Troy Benjegerdes]

> > Hrrm..  it looks like for some reason the top level makefile didn't
> > rebuild libibcommon.
> 
> Not sure why that would be. 
> 
> In my top level generated Makefile,
> LIBS:=libibcommon libibumad libibmad
> 
> @for i in $(LIBS); do\
> if [ -x $$i/autogen.sh ]; then\
> if !(cd $$i; ./autogen.sh && ./configure && make
> && make install); then exit 1; fi\
> fi\
> done
> 
> What does yours look like ?

It looks like the same thing, so I'm going to assume for the moment this
was some error that scrolled by that didn't occur when I did it
manually.

> 
> > Also, is there a clean way we could use a common autocong/automake setup
> > for the diags? The build spends more time running configure than
> > compiling ;)
> 
> I've had it on my list for a while to combine all the diags
> subdirectories into 1 and just have one diags build to simplify this but
> I haven't gotten to this yet.
> 
> -- Hal
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [RFC] OpenSM Interactive Console

[Troy Benjegerdes]

> > * Topology
> 
> This can be done via SA queries currently.
> 
> > * guid/lid/IPoIB address/switch port mappings
> 
> The SM does not know (see) IPoIB addresses. The only thing it sees is
> the part of the subnet address.
> 
> The rest can be done via SA queries currently.
> 
> > * link state
> 
> This can be done via SA query currently.
> 
> This argues for a higher layer API to make these queries easy.
> 
> > Future neat things to do:
> > 
> > * An interface to dynamically partition the fabric
> 
> Is this referring to IB partitioning ?

I think so, but IB partitioning may not actually map to what I'm
interested in.  From the high-level (application) point of view, I want to
ensure that communication traffic for one cluster job minimally affects 
another job.
 
> > * Register for notifications for certain events (excessive traffic
> > queueing, or error counts)
> 
> Not sure what you mean by excessive traffic queuing.

I guess I'd like to know whenever utilization on a single link exceeds
90%, or the queuing delay exceeds XXX nanoseconds.

> It is the event set which is of interest to me. Are there others ?
> 
> There are a set of events which can be subscribed to currently. The ones
> along these lines are local link integrity threshold reached on a port,
> excessive buffer overrun threshold reached on a port, flow control and
> update watchdog timer expired on a switch port.
> 
> If you are referring to the PortCounters, these would need to be polled
> (at some periodicity) and then an event created as there is no event for
> this defined in IBA.
> 
> Higher layer APIs could help with this area too.

Some of this stuff may not necessarily belong in the OpenSM process either..
Stuff like getting IPoIB address from GUID's would be usefull in a
library, but isn't the SM's responsibility.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] EHCA-0028 userspace build fails with openib svn 3774

[Troy Benjegerdes]

On Thu, Oct 20, 2005 at 08:46:12AM +0200, Heiko J Schick wrote:
> Hello Troy,
> 
> this problem should be solved in EHCA2_0033. The EHCA2_0028 package was
> only tested with OpenIB trunk 3615. The problem is EHCA_0028 doesn't
> included the raw_fw_ver pointer for ibv_cmd_query_device.
> 
> Please use EHCA2_0033 available via:
> 1: https://openib.org/svn/gen2/trunk/src/userspace/libehca/
> 2: http://prdownloads.sourceforge.net/ibmehcad/ehca2_EHCA2_0033.tgz?download
> 
> We've included yesterday EHCA2_0033 into the OpenIB tree.

A couple of nits..

infiniband/hw/ehca/Makefile doesn't fit 80 columns very well.

I also see a bunch of warnings about:

drivers/infiniband/hw/ehca/./hcp_if.h:1823: warning: ISO C90 forbids
mixed declarations and code

I need the following patches to build. However, I appear to be unable to
unload the previous 0028 hcad_mod, so I think I'm going to have to
reboot to actually test it.

Index: infiniband/Kconfig
===
--- infiniband/Kconfig  (revision 3828)
+++ infiniband/Kconfig  (working copy)
@@ -32,6 +32,8 @@

 source "drivers/infiniband/hw/mthca/Kconfig"

+source "drivers/infiniband/hw/ehca/Kconfig"
+
 source "drivers/infiniband/ulp/ipoib/Kconfig"

 source "drivers/infiniband/ulp/sdp/Kconfig"
Index: infiniband/Makefile
===
--- infiniband/Makefile (revision 3828)
+++ infiniband/Makefile (working copy)
@@ -1,6 +1,7 @@
 obj-$(CONFIG_INFINIBAND)   += core/
 obj-$(CONFIG_IPATH_CORE)   += hw/ipath/
 obj-$(CONFIG_INFINIBAND_MTHCA) += hw/mthca/
+obj-$(CONFIG_INFINIBAND_EHCA)   += hw/ehca/
 obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/
 obj-$(CONFIG_INFINIBAND_SDP)   += ulp/sdp/
 obj-$(CONFIG_INFINIBAND_SRP)   += ulp/srp/
Index: infiniband/hw/ehca/Makefile
===
--- infiniband/hw/ehca/Makefile (revision 3828)
+++ infiniband/hw/ehca/Makefile (working copy)
@@ -37,15 +37,6 @@
 #  $Id: Makefile.kernel_prod,v 1.13 2005/10/13 15:01:16 schickhj Exp $
 #

-
-
-#make for kernel 2.6 build
-
-ifndef GEN2_PATH_KERNEL
-GEN2_PATH_KERNEL = /home/source/trunk_3745/src/linux-kernel
-endif
-# GEN2_PATH_KERNEL = drivers #for gen2 code in kernel
-
 obj-m += hcad_mod.o

 hcad_mod-objs = ehca_main.o ehca_hca.o ipz_pt_fn.o ehca_classes.o
ehca_av.o \
@@ -60,7 +51,6 @@

 EXTRA_CFLAGS +=-DP_SERIES -DEHCA_USE_HCALL -DEHCA_USE_HCALL_KERNEL \
-I$(src)/.  \
-   -I$(GEN2_PATH_KERNEL)/infiniband/include/rdma \
-   -I$(GEN2_PATH_KERNEL)/infiniband/core
+   -Idrivers/infiniband/include

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: ehca testing

[Troy Benjegerdes]

On Thu, Oct 20, 2005 at 04:47:07PM +0200, Christoph Raisch wrote:
> Can't promise the opensm part, but we'll try.
> We have some intel machines with mellanox cards. A second possibility 
> would be to just use the mellanox cards in our power5 boxes for opensm and 
> see what happens.

This is strange.. This machine has a mellanox card, but no ehca card.
It looks like when hcad_mod and ib_mthca are both loaded something
conflicts.

10:/usr/lib/infiniband# modprobe ib_ipoib
[ 1920.053089] mthca0: ib_query_pkey port 0 failed (ret = -22)
10:/usr/lib/infiniband# lsmod
Module  Size  Used by
ib_ipoib   57200  0
ib_sa  19704  1 ib_ipoib
hcad_mod  989040  0
ib_uverbs  50896  0
openafs   847024  3
ib_mthca  154656  0
ib_mad 54692  2 ib_sa,ib_mthca
ib_core61488  6
ib_ipoib,ib_sa,hcad_mod,ib_uverbs,ib_mthca,ib_mad

...

[  145.817358] ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23,
2005)
[  145.817373] ib_mthca: Initializing :d9:00.0
[  145.818257] PCI: Enabling device: (:d9:00.0), cmd 142
[  152.401571] openafs: module license
'http://www.openafs.org/dl/license10.html' taints kernel.
[  152.404580] Found system call table at 0xc0013e68 (scan:
close+ioctl)
[  152.420156] Starting AFS cache scan...Memory cache: Allocating 12500
dcache entries...found 0 non-empty cache files (0%).
[ 1580.877118] eHCA Infiniband Device Driver (Rel.: EHCA2_0033)
[ 1920.053089] mthca0: ib_query_pkey port 0 failed (ret = -22)

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] EHCA-0028 userspace build fails with openib svn 3774

[Troy Benjegerdes]

On Thu, Oct 20, 2005 at 08:46:12AM +0200, Heiko J Schick wrote:
> Hello Troy,
> 
> this problem should be solved in EHCA2_0033. The EHCA2_0028 package was
> only tested with OpenIB trunk 3615. The problem is EHCA_0028 doesn't
> included the raw_fw_ver pointer for ibv_cmd_query_device.

EHCA_0033 (the version in openib subversion) still has some serious problems.
When the module is unloaded, it doesn't clean up kernel threads, and
setting a different value of 'port_act_time' when loading the module the
second time does not have any effect.

The firmware is also broken and not responding to certain opensm
queries.

This is part of osm.log..

Oct 20 11:16:59 946091 [43005960] -> umad_receiver: ERR 5409: send
completed with error (method=0x1 attr=0x16 trans_id=0x9567) -- dropping.
Oct 20 11:16:59 946115 [43005960] -> umad_receiver: ERR 5411: DR SMP hop
ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Oct 20 11:16:59 946126 [43005960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
3113: MAD completed in error (IB_TIMEOUT).
Oct 20 11:16:59 946155 [43005960] -> SMP dump:
base_ver0x1
mgmt_class..0x81
class_ver...0x1
method..0x1 (SubnGet)
D bit...0x0
status..0x0
hop_ptr.0x0
hop_count...0x3
trans_id0x9567
attr_id.0x16 (P_KeyTable)
resv0x0
attr_mod0x1
m_key...0x
dr_slid.0x
dr_dlid.0x

Initial path: [0][1][13][2]
Return path:  [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]

00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00
00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00
00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00
00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 
00


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ehca testing

[Troy Benjegerdes]

On Thu, Oct 20, 2005 at 10:48:35AM -0700, Roland Dreier wrote:
> Troy> This is strange.. This machine has a mellanox card, but no
> Troy> ehca card.  It looks like when hcad_mod and ib_mthca are
> Troy> both loaded something conflicts.
> 
> Have you confirmed that it works without hcad_mod loaded?

I've since found I have the same problem without hcad_mod. I don't see
any errors in dmesg except for:

[ 7415.421699] mthca0: ib_query_pkey port 0 failed (ret = -22)

Any ideas? What debug options should I try next?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ehca testing

[Troy Benjegerdes]

On Thu, Oct 20, 2005 at 11:03:28AM -0700, Roland Dreier wrote:
> Troy> I've since found I have the same problem without hcad_mod. I
> Troy> don't see any errors in dmesg except for:
> 
> Troy> [ 7415.421699] mthca0: ib_query_pkey port 0 failed (ret = -22)
> 
> It's strange that IPoIB is querying port 0 of a CA.  Could you have
> mismatched versions of modules, so that some were compiled with a
> different version of ?
> 
> You could add printk calls to ipoib_add_one and mthca_register_device
> and make sure that they see the same value of node_type for the struct
> ib_device (make sure to add the printk to mthca_register_device after
> the place where it assigns the node_type field).
> 

There must be some poorly specified dependency information in the kernel
makefiles, since I just did a clean build and it loads fine now.

I've been updating the subversion ocassionally, and doing 'make
modules'. Apparently I had an old object file that didn't get rebuilt.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ehca testing

[Troy Benjegerdes]

On Thu, Oct 20, 2005 at 11:03:28AM -0700, Roland Dreier wrote:
> Troy> I've since found I have the same problem without hcad_mod. I
> Troy> don't see any errors in dmesg except for:
> 
> Troy> [ 7415.421699] mthca0: ib_query_pkey port 0 failed (ret = -22)
> 
> It's strange that IPoIB is querying port 0 of a CA.  Could you have
> mismatched versions of modules, so that some were compiled with a
> different version of ?

There is some sort of strange initializiation error going on here..

When ib_mthca is loaded by udev on startup, and then I modprobe
ib_ipoib, I get the ib_querry_pkey error.

But unloading all the ib modules and reloading them manually works just
fine. (I have noticed a 5-10 second delay before ipoib starts working
right)

This is kernel 2.6.13.3 svnversion 3829


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [RFC] OpenSM Interactive Console

[Troy Benjegerdes]

For me, the only purpose for an SNMP MIB would be to get the information
into a network management system. In my case, I'll be using something
that's open-source or has a plugin architecture like Nagios, and I'd
really rather just have the network management system communicate with
the subnet manager or SMA packets directly rather than introducting an
extra translation to SNMP.

SNMP is only usefull to me because it is (in theory) an interoperable
cross-vendor standard. In the infiniband case, we already have a
cross-vendor standard implementation (OpenIB), and adding SNMP is
another dependency and layer of complexity that can break and be
difficult to set up.

If I knew of an open-source tool that was actually able to use SNMP to
query a random ethernet vendor's switch and be able to tell me what port
a particular MAC address was plugged into, I might be more positive. But
as far as I know, each vendor's SNMP implementation is broken in subtly
different ways, so that this gets to be a nightmare to actually
implement.

I guess the point of all this is find a end-user use-case for the SM
MIB, and work back from there to decide if haveing a MIB actually helps
solve the problem.

On Thu, Oct 27, 2005 at 08:03:57AM +0200, Eitan Zahavi wrote:
> Yes this MIB needs some cleanup.
> I would love to hear from the community some feedback regarding SM MIB
> usefulness.
> 
> In the past we did not get any push for interactive SM or online
> configurable SM so I did not see any reason to work on it. 
> 
> I do not think it is a huge task to make SM MIB work with OpenSM. At least
> not the 90% of it that I glanced through.
> 
> 
> Eitan Zahavi
> Design Technology Director
> Mellanox Technologies LTD
> Tel:+972-4-9097208
> Fax:+972-4-9593245
> P.O. Box 586 Yokneam 20692 ISRAEL
> 
> 
> > -Original Message-
> > From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> > Sent: Wednesday, October 26, 2005 7:44 PM
> > To: Eitan Zahavi
> > Cc: Troy Benjegerdes; openib-general@openib.org
> > Subject: RE: [openib-general] [RFC] OpenSM Interactive Console
> > 
> > Hi Eitan,
> > 
> > I sit corrected. There are R/W parameters in the SM MIB as you indicate. I
> was
> > thinking of all the other IPoIB MIBs. It's been a while since I looked at
> the SM MIB.
> > 
> > Also, the SM MIB (draft-ietf-ipoib-subnet-manager-mib-00) expired a while
> ago. At a
> > minimum, it needs to be dusted off. That would include updating it for IBA
> 1.2.
> > 
> > -- Hal
> > 
> > 
> > 
> > From: Eitan Zahavi [mailto:[EMAIL PROTECTED]
> > Sent: Tue 10/25/2005 5:19 AM
> > To: Hal Rosenstock
> > Cc: Troy Benjegerdes; openib-general@openib.org
> > Subject: Re: [openib-general] [RFC] OpenSM Interactive Console
> > 
> > 
> > 
> > Hal Rosenstock wrote:
> > > On Mon, 2005-10-24 at 14:38, Eitan Zahavi wrote:
> > >
> > >>Hal Rosenstock wrote:
> > >>
> > >>>On Mon, 2005-10-24 at 03:08, Eitan Zahavi wrote:
> > >>>
> > >>>
> > >>>>I would suggest to use SNMP for the tasks below. IETF IPoIB group
> > >
> > > has
> > >
> > >>>>defined an SNMP MIB that can support the required functionality
> > >
> > > below.
> > >
> > >>>
> > >>>The IETF SNMP MIBs are one way of presenting the information to the
> > >>>outside world. There are other possible management interfaces. The
> > >
> > > SNMP
> > >
> > >>>MIB instrumentation would need to use lower layer APIs to get this
> > >>>information out of the SM.
> > >>
> > >>Yes but the IETF SM MIB is the only one that is close to a standard
> > >
> > > way.
> > >
> > >>It does not require low level interface if it will integrate into the
> > >
> > > OpenSM code.
> > >
> > >>One way to do it is buy extending OpenSM with an AgentX interface.
> > >>
> > >>IMO one clear advantage of using SNMP for SM integration is that the
> > >
> > > code will work with any SM that is IETF compliant.
> > >
> > >>Also if you want to write a "client server" type of application on top
> > >
> > > of an SM you
> > >
> > >>can either stick to sending MADs which translate into SA client based
> > >
> > > application or
> > >
> > >>you better stay with some known protocol for management (like SNMP)
> > >
> > > and not develop 

[openib-general] ib_mthca panic on PPC64

[Troy Benjegerdes]

I got this the other day (before I had a chance to add the debug code)

p5l0:~# [443954.161068] mthca0: ib_query_pkey port 0 failed (ret = -22)
[443988.334644] mthca0: ib_query_pkey port 0 failed (ret = -22)
[444037.579342] ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
[444037.579360] ib_mthca: Initializing :d9:00.0
[444101.503664] ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
[444101.503682] ib_mthca: Initializing :d9:00.0
[444107.815375] Oops: Kernel access of bad area, sig: 7 [#1]
[444107.815389] SMP NR_CPUS=8 NUMA PSERIES LPAR
[444107.815401] Modules linked in: ib_ipoib ib_sa ib_mthca ib_mad ib_core openaf
s
[444107.815425] NIP: D98BF638 XER: 2018 LR: C0057B2C CTR: D0
00098BF5D0
[444107.815440] REGS: c001ee79b490 TRAP: 0300   Tainted: P   (2.6.13.3-p
ower5)
[444107.815455] MSR: 80009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 2800
0084
[444107.815469] DAR: d10082189a04 DSISR: 4000
[444107.815481] TASK: c001ee7950e0[0] 'swapper' THREAD: c001ee798000 CPU
: 6
[444107.815494] GPR00: 0010 C001EE79B710 D98D6540 D1
0082189A04
[444107.815515] GPR04: 0008 0001009D0180  00
000800
[444107.815535] GPR08: C003DDA91910  C001EE79B840 
D10082189A04
[444107.815556] GPR12: 4882 C04BF400  
00C00060
[444107.815576] GPR16: 0006   

[444107.815595] GPR20:  C05F7ED8 C05F7F40 
C0606500
[444107.815617] GPR24: C001ECEFC498 C001EE79B840 C001EE798000 
C003DDA91000
[444107.815639] GPR28: 0100 C003DDA91000 D98D4EC0 

[444107.815661] NIP [d98bf638] .poll_catas+0x68/0x2f0 [ib_mthca]
[444107.815699] LR [c0057b2c] .run_timer_softirq+0x15c/0x260
[444107.815717] Call Trace:
[444107.815725] [c001ee79b710] [c001ee79b7c0] 0xc001ee79b7c0 
(unreliable)
[444107.815744] [c001ee79b7d0] [c0057b2c] 
.run_timer_softirq+0x15c/0x260
[444107.815764] [c001ee79b890] [c0051e68] .__do_softirq+0xe8/0x1c0
[444107.815783] [c001ee79b950] [c0051fc4] .do_softirq+0x84/0x90
[444107.815801] [c001ee79b9d0] [c00108f0] .timer_interrupt+0xd0/0x41
0
[444107.815821] [c001ee79bad0] [c000a2b4] 
decrementer_common+0xb4/0x100
[444107.815838] --- Exception: 901 at .pseries_dedicated_idle+0x104/0x280
[444107.815857] LR = .pseries_dedicated_idle+0x1e0/0x280
[444107.815868] [c001ee79be90] [c000f460] .cpu_idle+0x40/0x60
[444107.815886] [c001ee79bf00] [c0032fa0] 
.start_secondary+0x120/0x150
[444107.815905] [c001ee79bf90] [c000ba7c] .enable_64b_mode+0x0/0x28
[444107.815922] Instruction dump:
[444107.815930] 3be0 4820 2fab 381f0001 7c1f07b4 409e0058 801d0908 
7f9f0040
[444107.815955] 409c00c8 e97d08f8 7be91764 7c6b4a14 <7c001c2c> 0c00 
4c00012c 780b0020
[444107.815983]  <0>Kernel panic - not syncing: Fatal exception in interrupt
[444107.815998]

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: ehca testing

[Troy Benjegerdes]

On Thu, Oct 20, 2005 at 03:32:13PM -0700, Roland Dreier wrote:
> Troy> There is some sort of strange initializiation error going on here..
> 
> Yes, very strange.  Can you add
> 
>   printk(KERN_ERR "hca->node_type = %d\n", hca->node_type);
> 
> to the beginning of ipoib_add_port(), and
> 
>   printk(KERN_ERR "dev->ib_dev.node_type = %d\n", dev->ib_dev.node_type);
> 
> right before the call to ib_register_device() in
> mthca_register_device() and send the output that you get when hotplug
> loads ib_mthca vs. when you load ib_mthca by hand?

When loaded at boot:

[586811.915831] ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23,
2005)
[586811.915849] ib_mthca: Initializing :d9:00.0
[586811.916634] PCI: Enabling device: (:d9:00.0), cmd 142
[586818.501595] openafs: module license
'http://www.openafs.org/dl/license10.html' taints kernel.
[586818.504651] Found system call table at 0xc0013e68 (scan:
close+ioctl)
[586818.520240] Starting AFS cache scan...Memory cache: Allocating 12500
dcacheentries...found 0 non-empty cache files (0%).
[586875.848354] afs: Lost contact with volume location server
147.155.137.10 incell scl.ameslab.gov
[586875.848374] afs: Lost contact with volume location server
147.155.137.10 incell scl.ameslab.gov
[587154.758768] hca->node_type = 236
[587154.760578] hca->node_type = 236
[587154.761511] hca->node_type = 236
[587154.761572] mthca0: ib_query_pkey port 3 failed (ret = -22)
[587154.761584] hca->node_type = 236
[587154.761633] mthca0: ib_query_pkey port 4 failed (ret = -22)
[587154.761644] hca->node_type = 236
[587154.762506] hca->node_type = 236
[587154.763422] hca->node_type = 236
[587154.763480] mthca0: ib_query_pkey port 7 failed (ret = -22)
[587154.763491] hca->node_type = 236
[587154.763542] mthca0: ib_query_pkey port 8 failed (ret = -22)
[587154.763553] hca->node_type = 236
[587154.765698] hca->node_type = 236
[587154.767136] hca->node_type = 236
[587154.767312] mthca0: ib_query_pkey port 11 failed (ret = -22)
[587154.767324] hca->node_type = 236
[587154.767455] mthca0: ib_query_pkey port 12 failed (ret = -22)
[587154.767471] hca->node_type = 236
[587154.769140] hca->node_type = 236
[587154.772116] hca->node_type = 236
[587154.772180] mthca0: ib_query_pkey port 15 failed (ret = -22)
[587154.772192] hca->node_type = 236
[587154.772243] mthca0: ib_query_pkey port 16 failed (ret = -22)
[587154.772255] hca->node_type = 236
[587154.773401] hca->node_type = 236
[587154.776817] hca->node_type = 236
[587154.776974] mthca0: ib_query_pkey port 19 failed (ret = -22)
[587154.776986] hca->node_type = 236
[587154.778179] mthca0: ib_query_pkey port 20 failed (ret = -22)
[587154.778198] hca->node_type = 236
[587154.780159] hca->node_type = 236
[587154.785406] hca->node_type = 236
[587154.785512] mthca0: ib_query_pkey port 23 failed (ret = -22)
[587154.785523] hca->node_type = 236
[587154.785582] mthca0: ib_query_pkey port 24 failed (ret = -22)
[587154.785599] hca->node_type = 236
[587154.789427] hca->node_type = 236
[587154.794314] hca->node_type = 236
[587154.794458] mthca0: ib_query_pkey port 27 failed (ret = -22)
[587154.794474] hca->node_type = 236
[587154.794634] mthca0: ib_query_pkey port 28 failed (ret = -22)
[587154.794646] hca->node_type = 236
[587154.797133] hca->node_type = 236
[587154.803507] hca->node_type = 236
[587154.803597] mthca0: ib_query_pkey port 31 failed (ret = -22)
[587154.803608] hca->node_type = 236
[587154.803667] mthca0: ib_query_pkey port 32 failed (ret = -22)
[587154.803679] hca->node_type = 236
[587154.820947] hca->node_type = 236
[587154.829795] hca->node_type = 236
[587154.831921] mthca0: ib_query_pkey port 35 failed (ret = -22)
[587154.831934] hca->node_type = 236
[587154.834932] mthca0: ib_query_pkey port 36 failed (ret = -22)
[587154.834946] hca->node_type = 236
[587154.844314] hca->node_type = 236
[587154.853591] hca->node_type = 236
[587154.853680] mthca0: ib_query_pkey port 39 failed (ret = -22)
[587154.853692] hca->node_type = 236
[587154.853745] mthca0: ib_query_pkey port 40 failed (ret = -22)
[587154.853761] hca->node_type = 236
[587154.869483] hca->node_type = 236
[587154.874749] hca->node_type = 236
[587154.874952] mthca0: ib_query_pkey port 43 failed (ret = -22)
[587154.874969] hca->node_type = 236
[587154.875609] mthca0: ib_query_pkey port 44 failed (ret = -22)
[587154.875624] hca->node_type = 236
[587154.894612] hca->node_type = 236
[587154.908058] hca->node_type = 236
[587154.909244] mthca0: ib_query_pkey port 47 failed (ret = -22)
[587154.909261] hca->node_type = 236
[587154.909323] mthca0: ib_query_pkey port 48 failed (ret = -22)
[587154.909334] hca->node_type = 236
[587154.918749] hca->node_type = 236
[587154.939629] hca->node_type = 236
[587154.939729] mthca0: ib_query_pkey port 51 failed (ret = -22)
[587154.939745] hca->node_type = 236
[587154.939866] mthca0: ib_query_pkey port 52 failed (ret = -22)
[587154.939883] hca->node_type = 236
[587154.957219] hca->node_type = 236
[587154.971523] hca->node_type = 236

Re: [openib-general] Re: ehca testing

[Troy Benjegerdes]

On Thu, Oct 27, 2005 at 10:03:17AM -0700, Roland Dreier wrote:
> OK, looks like you have two problems.  First of all, you seem to have
> two versions of ib_mthca, one of which gets picked up by hotplug on
> boot and one of which gets picked up by modprobe.  Notice how you
> don't see the
> 
> dev->ib_dev.node_type = 1
> 
> line when mthca runs on boot?  The only explanation I can come up with
> for that would be that you have an old version of it in an initrd or
> something that's screwing thing up.

Whoops, that's exactly what's going on.. Now to figure out how to not
have IB stuff included in my initrd..
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] prototype version of ebus driver

[Troy Benjegerdes]

On Wed, Oct 26, 2005 at 04:56:08PM +0200, IBMEHCA DD wrote:
> on kernel 2.6.13 and 14 a "ebus" driver is needed to enable the ehca 
> driver on power5.
> I just uploaded a prototype patch to gen2/users/ehca svn 3879
> 

Please get some responses from the PPC64 maintainers, or possibly
linux-kernel.

I'd like to see ehca get reviewed as well, but it may be a little early
for that ;)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] opensm errors with ehca

[Troy Benjegerdes]

The firmware on the IBM eHCA causes opensm to spit out these kinds of
errors all the time..

Is there a way we can either not send P_KeyTable requests to any eHCA
guids, or figure out what (if anything) is broken in their firmware?

Is this a spec violation, or just ambiguities in implementation?

Oct 30 17:49:46 053820 [43005960] -> umad_receiver: ERR 5409: send
completed wit
h error (method=0x1 attr=0x16 trans_id=0x158c) -- dropping.
Oct 30 17:49:46 053830 [43005960] -> umad_receiver: ERR 5411: DR SMP hop
ptr 0 h
op count 2 DR SLID 0x0 DR DLID 0x0
Oct 30 17:49:46 053839 [43005960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
3113: MA
D completed in error (IB_TIMEOUT).
Oct 30 17:49:46 053861 [43005960] -> SMP dump:
base_ver0x1
mgmt_class..0x81
class_ver...0x1
method..0x1 (SubnGet)
D bit...0x0
status..0x0
hop_ptr.0x0
hop_count...0x2
trans_id0x158c
attr_id.0x16 (P_KeyTable)
resv0x0
attr_mod0x26
m_key...0x
dr_slid.0x
dr_dlid.0x

Initial path: [0][1][16]
Return path:  [0][0][0]
Reserved: [0][0][0][0][0][0][0]

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] libehca causes segfault when not physically present..

[Troy Benjegerdes]

On an Openpower720 system with a mellanox HCA (and no IBM ehca
installed), I get the following when trying to run ibv_rc_pingpong:

Starting program:
/usr/src/openib-src/userspace/libibverbs/examples/.libs/ibv_rc_pingpong
[Thread debugging using libthread_db enabled]
[New Thread 4398046660640 (LWP 6167)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 4398046660640 (LWP 6167)]
hipz_galpa_store (galpa={fw_handle = 0}, offset=48, value=0)
at src/hcp_phyp.c:72
72  *(u64 *) addr = value;
(gdb) bt
#0  hipz_galpa_store (galpa={fw_handle = 0}, offset=48, value=0)
at src/hcp_phyp.c:72
#1  0x10001b7c in pp_post_recv (ctx=0x100177d0, n=-3807848)
at verbs.h:844
#2  0x10002364 in main (argc=Variable "argc" is not available.
) at examples/rc_pingpong.c:566


I assume this means something somewhere is not actually checking sysfs
to see if the driver is actually there and active.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] opensm errors with ehca

[Troy Benjegerdes]

> Can you try the following opensm patch and see if this eliminates those
> timeout messages ?
> 
> This patch clears the high part of the attribute modifier when not a
> switch (when obtaining the PKeyTable).
> 
> -- Hal
> 
> Index: osm_port_info_rcv.c
> ===
> --- osm_port_info_rcv.c   (revision 3906)
> +++ osm_port_info_rcv.c   (working copy)
> @@ -430,6 +430,7 @@ void osm_pkey_get_tables(
>osm_dr_path_t path;
>uint8_t  port_num;
>uint16_t block_num, max_blocks;
> +  uint32_t attr_mod_ho;
>osm_switch_t* p_switch;
>  
>OSM_LOG_ENTER( p_log, osm_physp_has_pkey );
> @@ -455,7 +456,7 @@ void osm_pkey_get_tables(
>else
>{
>  /* This is a switch, and not a management port. The maximum blocks is 
> defined
> -   on the switch info partition enforcement cap. */
> +   in the switch info partition enforcement cap. */
>  p_switch = osm_get_switch_by_guid(p_subn, p_node->node_info.node_guid);
>  
>  if (! p_switch)
> @@ -472,10 +473,14 @@ void osm_pkey_get_tables(
>  
>for (block_num = 0 ; block_num < max_blocks  ; block_num++)
>{
> +if (osm_node_get_type( p_node ) != IB_NODE_TYPE_SWITCH)
> +  attr_mod_ho = block_num;
> +else
> +  attr_mod_ho = block_num | (port_num << 16);
>  status = osm_req_get( p_req,
>&path,
>IB_MAD_ATTR_P_KEY_TABLE,
> -  cl_hton32(block_num | (port_num << 16) ),
> +  cl_hton32(attr_mod_ho),
>CL_DISP_MSGID_NONE,
>&context );
>  

This seems to ignore the IBM logical HCA, but gives the same thing
on the IBM Logical switch. Is there a way to ignore this as well?

switchguids=0x255038580
Switch  63 "S-000255038580" # IBM Logical Switch 1 port 0
lid 21
[2] "H-000255038500"[1]
[1] "S-0002c90200402917"[22]


I still get:

Nov 01 22:34:08 660205 [43005960] -> umad_receiver: ERR 5409: send
completed wit
h error (method=0x1 attr=0x16 trans_id=0x13c9) -- dropping.
Nov 01 22:34:08 660213 [43005960] -> umad_receiver: ERR 5411: DR SMP hop
ptr 0 h
op count 2 DR SLID 0x0 DR DLID 0x0
Nov 01 22:34:08 660221 [43005960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
3113: MA
D completed in error (IB_TIMEOUT).
Nov 01 22:34:08 660243 [43005960] -> SMP dump:
base_ver0x1
mgmt_class..0x81
class_ver...0x1
method..0x1 (SubnGet)
D bit...0x0
status..0x0
hop_ptr.0x0
hop_count...0x2
trans_id0x13c9
attr_id.0x16
(P_KeyTable)
resv0x0
attr_mod0x1
m_key...0x
dr_slid.0x
dr_dlid.0x

Initial path: [0][1][16]
Return path:  [0][0][0]
Reserved: [0][0][0][0][0][0][0]



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] OpenSM errors question..

[Troy Benjegerdes]

What does the following mean? (the ERR 1B11, in particular)

Nov 02 16:18:33 656702 [41001960] -> osm_report_notice: Reporting
Generic Notice type:4 num:144 from LID:0x001B
GID:0xfe80,0x0002c90200402789
Nov 02 16:18:33 674607 [41802960] -> osm_ucast_mgr_process: Min Hop
Tables configured on all switches.
Nov 02 16:18:34 197522 [41802960] -> osm_ucast_mgr_process: Min Hop
Tables configured on all switches.
Nov 02 16:19:59 917207 [41802960] -> osm_report_notice: Reporting
Generic Notice type:3 num:66 from LID:0x001B
GID:0xfe80,0x0002c90200402789
Nov 02 16:19:59 917610 [41001960] -> osm_report_notice: Reporting
Generic Notice type:3 num:66 from LID:0x001B
GID:0xfe80,0x0002c90200402789
Nov 02 16:19:59 926829 [41001960] -> osm_mcmr_rcv_join_mgrp: ERR 1B11:
method =SubnAdmSet,scope_state = 0x1, component mask =
0x00010083, expected comp mask = 0x000130c7, MGID:
0xff12601b : 0x0016
Nov 02 16:20:01 670893 [41802960] -> osm_mcmr_rcv_join_mgrp: ERR 1B11:
method =SubnAdmSet,scope_state = 0x1, component mask =
0x00010083, expected comp mask = 0x000130c7, MGID:
0xff12601b : 0x0002


(I got this on an isolated subnet with 3 machines.. two opterons with
mellanox cards and an IBM with the eHCA card)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] libehca causes segfault when not physically present..

[Troy Benjegerdes]

On Thu, Nov 03, 2005 at 11:13:58AM -0800, Roland Dreier wrote:
> Heiko> this bug should be fixed in OpenIB trunk 3960.
> 
> It's good to see this fixed and all the other cleanups in this
> checkin.  I'll have to go back to my ehca code reviewing
> 
> However, when this code moves upstream, you'll have to make your
> changes in smaller digestible chunks.  The diff between r3959 and
> r3960 is rather gigantic:
> 
>  33 files changed, 945 insertions(+), 1163 deletions(-)
> 
> And this piece:
> 
> > -MODULE_VERSION("EHCA2_0035");
> > +MODULE_VERSION("EHCA2_0037");
> 
> indicates that there was a 0036 that you never let anyone see.

I'll second the comment about smaller digestible chunks. A second thing
I don't completely understand is the vast size difference between the
ehca and mthca drivers. Is the ehca really that much more complex?

I also want to comment that EHCA is the only thing that's versioned that
is easy to tell what version of the module is actually loaded at the
moment. I'd rather have versions I don't see float by than see every 
file in mthca get updated, but no version rev.

I tried adding some code to generate a version string from the outupt of
svnversion but it didn't work too well.

The same goes for OpenSM as well.. the only version string you get when
starting it is 'Opensm-1.1.0', which isn't very usefull.

And once that's figured out, maybe we can start thinking about how to
make sure kernel module versions match userspace versions. Personally,
I'd like to see the ehca functions exported as a VDSO.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] OpenSM and Wrong SM_Key

[Troy Benjegerdes]

On Wed, Nov 09, 2005 at 09:46:06AM +0200, Eitan Zahavi wrote:
> Hi Hal,
> 
> I would like to bring this to MgtWG before we change anything.
> IMO the situation when this happens is really not "legal" since if the
> SM's are not coordinated at least in their SM_Key it will cause the two
> masters on the subnet. 
> 
> >From our experience it is always better to cause a fatal flow and exit
> the SM rather then report the event in some log - normally it will not
> be seen ...
> 
> I know this is a controversial issue. 

Okay, so you're telling me you *WANT* behavior where a rogue node can
trivially cause the running subnet manager to exit and take over
management of the network?

Opensm needs to have a well documented config file, instead of 3 pages
of command line options, and different levels of logging. What to do in
the above situation is a site-local policy config decision, not something 
that should be hard-coded in the SM source code.

The logs might actually get looked at if there wasn't junk in the log
every time something timed out.

The linux kernel has 'WARN, NOTICE, and CRITICAL' level log messages.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] OpenSM and Wrong SM_Key

[Troy Benjegerdes]

On Sat, Nov 12, 2005 at 07:34:44PM +0200, Eitan Zahavi wrote:
> Hi Troy,
> 
> Good to get a straight forward message.
> 
> What I hear you saying is:
> 1. There needs to be a parameter to control the SM behavior if it finds
> another SM with non matching SM Key: 
> -> Either to ignore it or to die. We can do that. No problem!

Is it possible to have another option as well, to attempt to disable the
port the SM with the non-matching key is connected to?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [PATCH] Opensm - lid assignment issues

[Troy Benjegerdes]

Yael Kalka wrote:


Hi Hal,

During some windows tests we've discovered that there is still another
problem in the lid_mgr. The problem happend when 2  HCAs had the same
lid - opensm entered an infinite loop.
The following patch fixes this.

Thanks,
Yael

Signed-off-by:  Yael Kalka <[EMAIL PROTECTED]>

Index: opensm/osm_lid_mgr.c
===
--- opensm/osm_lid_mgr.c(revision 4032)
+++ opensm/osm_lid_mgr.c(working copy)
@@ -550,6 +550,9 @@ __osm_lid_mgr_init_sweep(
  {
  /* This port will use its local lid, and consume the entire 
required lid range.
 Thus we can skip that range. */
+  /* If the disc_max_lid is greater then lid - we can skip right to it, 
+ since we've done all neccessary checks on the lids in between. */

+  if (disc_max_lid > lid)
lid = disc_max_lid;
  }
}
@@ -593,7 +596,14 @@ __osm_lid_mgr_init_sweep(
  {
p_range =
  (osm_lid_mgr_range_t *)cl_malloc(sizeof(osm_lid_mgr_range_t));
-p_range->min_lid = 1;
+/* 
+   The p_range can be NULL in one of 2 cases:

+   1. If max_defined_lid == 0. In this case, we want the entire range.
+   2. If all lids discovered in the loop where mapped. In this case
+  no free range exists, and we want to define it after the last 
+  mapped lid.

+*/
+p_range->min_lid = lid;
  }
  p_range->max_lid = p_mgr->p_subn->max_unicast_lid_ho - 1;
  cl_qlist_insert_tail( &p_mgr->free_ranges, &p_range->item );


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
 


The opensm on the show floor is showing the following in oprofile:

with a unit mask of 0x01 (mandatory) count 10
samples  %app name symbol name
5970354  51.7020  libpthread-2.3.4.so  
pthread_cond_timedwait@@GLIBC_2.3.2

5037621  43.6247  libosmcomp.so.1.0.0  __cl_timer_prov_cb
66241 0.5736  libosmcomp.so.1.0.0  anonymous symbol from section 
.plt

55929 0.4843  oprofiled(no symbols)
49918 0.4323  opensm   __osm_ucast_mgr_process_neighbors
39585 0.3428  vmlinux  hpet_readl
25333 0.2194  oprofile (no symbols)
22734 0.1969  opreport (no symbols)
14724 0.1275  libcrypto.so.0.9.7a  (no symbols)
14296 0.1238  libc-2.3.4.so__tzfile_compute
13901 0.1204  vmlinux  __copy_to_user_ll


Is this the same loop?
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] another opensm crash

[Troy Benjegerdes]

(gdb) bt
#0  0x08071ff3 in osm_si_rcv_process (p_rcv=0x8090138, p_madw=0x80a1de0)
   at osm_sw_info_rcv.c:679
#1  0xb7fb0213 in __cl_disp_worker (context=0x8090da4) at 
cl_dispatcher.c:108

#2  0xb7fb8557 in __cl_thread_pool_routine (context=0x8090de4)
   at cl_threadpool.c:78
#3  0xb7fb834d in __cl_thread_wrapper (arg=0x8091408) at cl_thread.c:61
#4  0x46cde341 in start_thread () from /lib/tls/libpthread.so.0
#5  0x46b6e6fe in clone () from /lib/tls/libc.so.6

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general