Re: [openib-general] [PATCH v3 3/3] ofed_1_2 Provide generic allocator backport to2.6.20.

[Matt Leininger]

On Sun, 2007-01-14 at 12:46 -0600, Steve WIse wrote:
> It is blocked for me too today.

  The hard disk on the Sandia server filled up.  I did a bit of house
cleaning.  Michael Lee will have to do some more, or else add another
hard drive to the system.

  Thanks,

- Matt
> 
> 
> 
> On Sun, 2007-01-14 at 20:45 +0200, Michael S. Tsirkin wrote:
> > BTW, is openib-general working for you?
> > Seems to be blocked for me.
> > 
> 
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [openfabrics-ewg] Reminder: OFED 1.2

[Matt Leininger]

Sandia and LLNL would also find MVAPICH2 useful.

Thanks,

- Matt

On Fri, 2007-01-12 at 17:00 -0500, Stephen Poole wrote:
> I would find it very useful.
> 
> Steve...
> 
> Steve Poole
> 
> Chief Scientist / Director of Special Projects
> Computer Science and Mathematics Division
> 
> Chief Systems Architect
> Leadership Computing Facility
> 
> Oak Ridge National Laboratory
> 865.574.9008
> "Wisdom is not a product of schooling, but of the lifelong attempt to
> acquire it" Albert Einstein
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] creating releases for the libraries you own

[Matt Leininger]

On Sun, 2006-10-29 at 12:52 +0200, Tziporet Koren wrote:
> Roland Dreier wrote:
> >  > I want to suggest that you will create releases to the libraries you own 
> >
> > To make this simpler, is there any way we can give maintainers the
> > ability to put library releases somewhere on the new server so that
> > they show up on the downloads page automatically?  Right now it is
> > somewhat cumbersome to create library releases, since the poor
> > sysadmins have to manually add tarballs to the downloads page.
> >
> >  - R.
> >   
> Hi Matt,
> I think Roland's suggestion is very good.
> Since Hal and Sean also agreed to create a release of their libraries it 
> can serve all.
> 
  I agree with Roland.  Sandia will not be running the webpages or wiki
on the new server.  I think the market folks will run the webpages and
the developers can run the wiki.  Any preferences for what wiki to use?
Trac (http://trac.edgewall.org/) was one suggestion.

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] New server svn up

[Matt Leininger]

On Thu, 2006-10-26 at 04:16 +0200, Sasha Khapyorsky wrote:
> Hi Matt,
> 
> On 18:40 Wed 25 Oct     , Matt Leininger wrote:
> > 
> >   Subversion is up and running on the new server at
> > https://69.55.231.195/svn. 
> 
> That is great.
> 
> What will be syncronization policy in staging period (with old, yet
> current SVN repository)? Now I can see that commits >= r9957 are not
> in new repo yet.
> 
  That's up to the developers.  I suggest folks try out the new server
and move over to using git/svn on it as soon as possible.   We can
figure out how to clean up or remove the svn user space tree during the
summit as SC06.  

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] New server svn up

[Matt Leininger]

  Subversion is up and running on the new server at
https://69.55.231.195/svn.  Those who have write access to the current
svn should have write access on the new server.  Let us know if any
issues arise.

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Tools for development

[Matt Leininger]

On Tue, 2006-10-17 at 07:49 -0700, Roland Dreier wrote:
> Michael> The tool versions installed on openib are ancient.  Can
> Michael> site admins please install latest svn and git versions
> Michael> from source?
> 
> What distro is on the new openfabrics.org server?

  Ubuntu.

>  If it's something
> like Fedora or Ubuntu, then it would probably be better to install the
> distros versions of svn and git, so that keeping up with security
> updates is easiser.

  Developers had requested git 1.4, but Ubuntu had an older version.  We
went ahead and installed git from source.  I'd prefer to stick to Ubuntu
packages if possible.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2.6.18 kernel support in the main trunk.

[Matt Leininger]

On Wed, 2006-10-04 at 14:47 +0200, Michael S. Tsirkin wrote:
> Quoting r. Matt Leininger <[EMAIL PROTECTED]>:
> >   We just got approval to spend OFA money on a new hosted server.  The
> > arrangements are being made but we don't have a date for when we will
> > get access to this new machine or when it will be set up.  If I had to
> > guess I'd say we will start setting  up the server in the next couple
> > weeks. 
> > 
> >   Thanks,
> > 
> >   - Matt
> 
> Thanks.
> A couple of more requests as far as you are working on the infrastructure
> - updated svn server
>   enables fast mirroring better web access and other goodies

   Are you referring to svn 1.4?  our plan is to upgrade to 1.4.

> - add bugzilla email gateway
>   (as seen e.g. at kernel.org) that supports accepting Cc mail
>   where you put "[Bug ]" in the subject (where  is the bug number) 
> and cc
>   [EMAIL PROTECTED]

  I'll add that to the list.

  - Matt

> 
> Could these be addressed?
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2.6.18 kernel support in the main trunk.

[Matt Leininger]

On Tue, 2006-10-03 at 09:25 -0500, Steve Wise wrote:
> > Someday soon I hear, OFA will be able to host git repositories, so my 
> > preference 
> > is to delay any svn to git transition until then.  (I cannot host git from 
> > inside Intel's firewall, nor can I access a git repository which isn't 
> > hosted at 
> > kernel.org.)  How would you handle merging in changes from the main branch 
> > to 
> > side branches?
> > 
> 
> Can OFA give us a date on when this will happen?

  We just got approval to spend OFA money on a new hosted server.  The
arrangements are being made but we don't have a date for when we will
get access to this new machine or when it will be set up.  If I had to
guess I'd say we will start setting  up the server in the next couple
weeks. 

  Thanks,

  - Matt
   


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2.6.18 kernel support in the main trunk.

[Matt Leininger]

On Thu, 2006-09-28 at 13:55 -0700, Roland Dreier wrote:
> Matt>   RedHat and SuSE have stated several times that they want
> Matt> an OFED like process that takes the OF code and runs it
> Matt> through a rigorous suite of regression and performance
> Matt> tests.  The purpose of OFED is to get into the commercially
> Matt> supported distros (e.g RHEL and SLES).  That is what the
> Matt> majority of end customers want/need.  That said spinning out
> Matt> "pre-OFED" releases of each component would help to get the
> Matt> code into the other distros (FC, Debian, Ubuntu, Gentoo,
> Matt> etc.) which, of course, is a very good thing to do.
> 
> I think we've gotten mixed up about "release" vs. "distribution"
> again.  I would say that all the packaging crap, which OFED does as a
> short-term thing to make it possible for naive users to install, is
> actually a big negative for RH and Novell -- they would rather package
> and build software themselves.

  Fair point.  I don't like the way OFED is packaged.  It's messy and
just causes more problems than it is worth.  What I do like about OFED
is the rigorous testing that each company does.  It would be great if we
can include this rigorous testing into the OF release process. 

  
> 
> What is missing is the tested, coordinated tarball release of OF
> userspace stuff -- http://www.gnome.org/start/2.16/ might be a useful
> model, particularly the "Getting GNOME 2.16" section.
> 
  Yes, we need something like this.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2.6.18 kernel support in the main trunk.

[Matt Leininger]

On Thu, 2006-09-28 at 10:59 -0700, Roland Dreier wrote:
> Matt>   I'd add one more thing.  To make the OFED release process
> Matt> go more smoothly I'd like to see the maintainers for each
> Matt> stack component spin out releases from time to time.  Roland
> Matt> has been doing this with libmthca and libibverbs.  If we had
> Matt> the development releases for other kernel and all user space
> Matt> components then OFED could simple combine the latest
> Matt> development releases and start more through testing.
> 
> Yes, I strongly support that, although the OFED benefits are just a
> side effect to me.  The real reason to have these releases is to
> support distributions other than OFED -- for example having tarball
> releases of all the components makes it possible to get this stuff
> further upstream into real Linux distros.
> 
  RedHat and SuSE have stated several times that they want an OFED like
process that takes the OF code and runs it through a rigorous suite of
regression and performance tests.  The purpose of OFED is to get into
the commercially supported distros (e.g RHEL and SLES).   That is what
the majority of end customers want/need.  That said spinning out
"pre-OFED" releases of each component would help to get the code into
the other distros (FC, Debian, Ubuntu, Gentoo, etc.) which, of course,
is a very good thing to do.

   Thanks,

- Matt
 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2.6.18 kernel support in the main trunk.

[Matt Leininger]

On Thu, 2006-09-28 at 10:33 -0700, Roland Dreier wrote:
> Matt>   So what it your proposal (Roland and Bryan)?  Do you want
> Matt> to move all kernel development into Roland's git tree, and
> Matt> have the user space code stay in svn (at least for the time
> Matt> being)?
> 
> My proposal would be to leave userspace in svn, and make Linus's git
> tree the definitive source for Linux kernel code.  My git tree may be
> useful for people who want to try things that haven't been merged
> upstream yet, but other developers of Linux kernel code may want to
> host their work too (either as a git tree, a patch set, or however
> else they want).  This would match existing practice for other
> subsystems pretty closely.
> 
  That sounds reasonable to me.  

  I'd add one more thing.  To make the OFED release process go more
smoothly I'd like to see the maintainers for each stack component spin
out releases from time to time.  Roland has been doing this with
libmthca and libibverbs.  If we had the development releases for other
kernel and all user space components then OFED could simple combine the
latest development releases and start more through testing.

  Thoughts?

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2.6.18 kernel support in the main trunk.

[Matt Leininger]

  If we move forward with a git repository then we should move all
kernel code into git.  I don't want to get into a situation where kernel
components are spread out over various repositories and servers.  I'm
all for making your development lives easier.  The entire development
tree has gotten very confusing over the past few months.  The ipath
driver is never up to date (therefore it's always broken).  Iwarp is
upstream but not in the main line development tree.  If a simpler
process can fix this then I'm all for it.

  So what it your proposal (Roland and Bryan)?  Do you want to move all
kernel development into Roland's git tree, and have the user space code
stay in svn (at least for the time being)?  This would allow OFED
releases to be pulled direct from Roland's git tree (kernel) and the
openfabrics svn (user space).   BTW if it is useful we can set up a git
repository on openfabrics once we move the server to its new provider.

 Thanks,

  - Matt



On Thu, 2006-09-28 at 09:31 -0700, Bryan O'Sullivan wrote:
> On Thu, 2006-09-28 at 12:21 -0400, James Lentini wrote:
> 
> > As a user of the SVN repository, I'm confused about what this means 
> > going forward. 
> > 
> > Are you going to completely remove the mthca and ipath code from SVN 
> > or just stop updating the code that is there?
> 
> I will let Roland speak for the mthca driver, but we have stopped
> maintaining the ipath driver in the SVN tree, and I expect that we will
> remove it entirely in perhaps a month or so.
> 
> > Will the other components that are upstream (SRP, iSER, IPoIB, CM, 
> > RDMA CM, SA, MAD, CORE, ...) be removed? What rules are you using to 
> > determine if the SVN version will be kept up to date?
> 
> I have no stake in what happens to those components, but I would not
> personally mind if they moved into Roland's git tree.  I don't care for
> git, but I vastly prefer using it to waiting for SVN.
> 
> > In the future, how will users work 
> > with new features that are not yet upstream?
> 
> One possibility would be to pull the same components out of a branch of
> a git tree; same procedure, different source.
> 
>
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] OpenFabrics IBTA DevCon 2006 presentations

[Matt Leininger]

Most of the presentations from the OpenFabrics IBTA DevCon 2006 in San
Francisco yesterday have been posted online at

http://openfabrics.org/conference/sep2006devcon/

and 

http://www.infinibandta.org/events/DevCon2006_presentations


Thanks to everyone who helped set up this event and to those that
participated.

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] OpenFabrics server scheduled downtime Sat. Sept 23

[Matt Leininger]

 
  The OpenFabrics server will be offline Saturday September 23 from 6am
PST to 6pm PST due to a scheduled maintenance on a power substation at
Sandia.  These outages usually last less than the scheduled 12 hours.
We will bring the OpenFabrics server back online as soon as possible
after the scheduled outage. 

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [openfabrics-ewg] OFED 1.1 planning meeting - summary

[Matt Leininger]

On Tue, 2006-07-25 at 18:39 +0300, Tziporet Koren wrote:
> Matt Leininger wrote:
> >> 5. SRP:
> >>
> >> –   GA quality
> >>
> >> –   DM (Device Mapper) - for high availability
> >>
> >> –   Basic failover/failback testing with daemon+srp+XVM/MPP and
> >> Engenio target
> >>
> >> 
> > Tziporet,
> >
> >   Are there any plans to test with the DDN SRP target?  Several DoE
> > sites are testing/using the DDN IB based storage.
> >
> >
> >   
> Mellanox does not have DDN SRP target. We will be happy to test it of 
> DDN will loan us a system.
> 
> Another option is that DDN will take OFED 1.1 RCs and test it in their labs.
> Can you approach them and ask this. If yes then I can cc them on the RCs 
> mails so they can do it.
> 
> Is there any other vendor who has DDN SRP target, and going to test OFED 
> with it?

  I thought Cisco had a DDN SRP target.  

   - Matt



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] [openfabrics-ewg] OFED 1.1 planning meeting - summary

[Matt Leininger]

On Tue, 2006-07-25 at 00:45 +0300, Tziporet Koren wrote:
> Hi all,
> 
> This is the outcome of the meeting we had today regarding OFED 1.1
> schedule and features.

> 5. SRP:
> 
> –   GA quality
> 
> –   DM (Device Mapper) - for high availability
> 
> –   Basic failover/failback testing with daemon+srp+XVM/MPP and
> Engenio target
> 
Tziporet,

  Are there any plans to test with the DDN SRP target?  Several DoE
sites are testing/using the DDN IB based storage.

  Thanks,

- Matt



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] IBED-1.0-rc3 README.txt

[Matt Leininger]

Why does the README.txt refer to all things as "Mellanox"

Here is the title in the README

---

Mellanox IBED Distribution v1.0 for Linux
***




more here

---

IBED Home Page:  https://docs.mellanox.com/dm/ibgold/ReadMe.html

Please email bugs and error reports to your local Field Application
Engineer.

even more a few lines down

-

1. HW and SW Requirements:
==
1) Server platform with InfiniBand HCA (see Mellanox IBED Distribution
Release Notes for details)

2) Linux OS (see Mellanox IBED Distribution Release Notes for details)

--

This needs to be cleaned up and updated.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] RFC userspace / MPI multicast support

[Matt Leininger]

On Thu, 2006-04-20 at 13:09 -0700, Sean Hefty wrote:
> >  Not always.  For most of the older VAPI based stack we never turned on
> >IPoIB (or did and it didn't work).  I don't think we want to assume
> >IPoIB is always set up when MPI is running.
> 
> This means that you can't use the rdma_cm to establish connections.

  Requiring IPoIB to be running for MPI to work is a new dependency that
users are not accustomed to.  Maybe we need to do this through the cm
rather than rdma_cm.  We shouldn't require IPoIB to be running for MPI
to use IB.


  - Matt

  

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] RFC userspace / MPI multicast support

[Matt Leininger]

On Thu, 2006-04-20 at 13:00 -0400, Hal Rosenstock wrote:
> On Thu, 2006-04-20 at 12:58, Sean Hefty wrote:
> > >On Wed, 2006-04-19 at 15:05, Sean Hefty wrote:
> > >> I'd like to get some feedback regarding the following approach to 
> > >> supporting
> > >> multicast groups in userspace, and in particular for MPI.  Based on side
> > >> conversations, I need to know if this approach would meet the needs of 
> > >> MPI
> > >> developers.
> > >>
> > >> To join / leave a multicast group,
> > >
> > >MC groups also need to be created and deleted as well. Creating and
> > >deleting the group are assumed under the covers (first joiner, last
> > >leaver) so the additional MC parameters for creation need to be
> > >available on all adds.
> > 
> > Creation / deletion would be automatic.  The creation parameters for
> > RDMA_PROTO_IP would use the same settings as the ipoib broadcast group.
> > 
> > >> /* Bind to multicast group. */
> > >> mcast_ip = 224.0.0.74.71; /* some fine mcast addr */
> > >
> > >How are the MGIDs formed from this IP address ? Is the same algorithm as
> > >IPoIB used ?
> > >
> > >Are the MGIDs constrained to use 0x401B in the signature part (and
> > >0x601B if this is extended to IPv6) ?
> > 
> > The MGIDs would be formed using the same algorithm as ipoib.  I hadn't 
> > decided
> > on whether to use the same signature, or a different one.  My initial 
> > thought
> > was to use a different signature, but I'm not sure that it's necessary.
> 
> Guess it comes down to how much control is needed over the entire MGID
> by MPI as well as whether they can share the IPoIB broadcast group
> characteristics for all their multicast groups.
> 
> Also, is IPoIB always setup when running MPI ?

  Not always.  For most of the older VAPI based stack we never turned on
IPoIB (or did and it didn't work).  I don't think we want to assume
IPoIB is always set up when MPI is running.

 - Matt

> 
> -- Hal
> 
> > >BTW, this example has too many bytes...
> > 
> > Just a typo...
> > 
> > >> ip_mreq.imr_multiaddr = mcast_ip.in_addr;
> > >> rdma_set_option(id, RDMA_PROTO_IP, IP_ADD_MEMBERSHIP, &ip_mreq,
> > >>  sizeof(ip_mreq));
> > >
> > >The API only supports ADD/DROP. It lacks support for JoinStates.
> > >(I don't think the IP semantics are rich enough for IB; this was
> > >previously pointed out in the context of IP routers quite a while ago).
> > 
> > Additional join states are IB specific, so would be handled by using the
> > RDMA_PROTO_IB option.  As an alternative, we could replace IP_ADD_MEMBERSHIP
> > with RDMA_ADD_FULL_MEMBER, RDMA_ADD_SEND_MEMBER, etc.
> > 
> > >> The multicast group information is created / managed by the rdma_cm.  The
> > >> rdma_cm defines the mgid, q_key, p_key, sl, flowlabel, tclass, and 
> > >> joinstate.
> > >> Except for mgid, these would most likely match the values used by the 
> > >> ipoib
> > >> broadcast group.  The mgid mapping would be similar to that used by 
> > >> ipoib.
> > >
> > >Does that limit the MGIDs to use IP signatures ?
> > 
> > Yes - unless the RDMA_PROTO_IB option were used.
> > 
> > - Sean
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] IB + Dual-processor, dual-core Opteron + PCI-E

[Matt Leininger]

On Thu, 2006-04-20 at 10:09 -0400, Charles Taylor wrote:
>   
> We have  202 node cluster where each node is configured as follows...
> 
> dual-processor, dual-core Opteron 275
> Asus K8N-DRE Motherboard
> TopSpin/Cisco LionCub HCA in a 16x PCI-E slot
> 4 GB RAM (DDR 400)
> 
> IB fabric is two-tiered fat tree with 14 Cisco 7000 switches  on the  
> edge and Cisco 7008s (2) in the
> first tier.
> 
> We can scale HPL runs up to about 136 nodes/544 cpus reliably on any  
> set of nodes.   Above that
> number of nodes/processors, our HPL runs begin to fail residuals. 
> We can run across all 202 nodes
> successfully if we use only two procs/node but 4 procs/node will  
> *always* fail residuals.   It feels like
> a data corruption issue in the IB stack.
> 
> We have tried various combinations of the following software.
> 
> Kernel: 2.6.9-22, 2.6.9-34
> IB stack: topspin 3.2.0b82, OpenIB (IBGD 1.8.2)
> MPI: mvapich 092/095 (topspin), mvapich 096 (osu), OpenMPI 1.0.2
> Blas Libs: Goto 1.00, 1.02, ACM 3.0.0
> 
> The result is the same in every case.   We seem to be able run HPL  
> reliably up to about 544 - 548 processors.  It doesn't
> matter whether we run one mpi task per processor or 1 mpi task per  
> node with OMP_NUM_THREADS=4.   The result
> is always failed HPL residuals when we run across any subset of the  
> cluster above about 136 nodes using all four procs.
> 
> I'm wondering if anyone knows of any other large IB clusters using  
> dual-processor, dual-core Opterons + PCI-E with more
> than 136 nodes and if so, have they been able to successfully scale  
> MPI apps across their entire cluster?
> 
  Charles,

 If you see the problem after trying various combinations of the
software you listed above, then it's likely a hardware issue.

 I know of several ~256 node dual-proc dual-core Opteron IB clusters
that are running linpack.  I've heard there can be issues with "silent"
data corruption on the Opteron CPUs if they get too hot.  Are you
monitoring the node/cpu temps?  If CPU temp is an issue you should see a
problem whether you are running a single linpack across all 202 nodes,
or running simultaneous smaller linpacks (say 4 50 node runs).  I'll see
if I can find the bug report for this problem.

  - Matt

  

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: Compile problems with core code and pathscale for svn6462 and linux-2.6.17-rc1

[Matt Leininger]

On Fri, 2006-04-14 at 09:22 -0700, Bryan O'Sullivan wrote:
> On Fri, 2006-04-14 at 09:19 -0700, Sean Hefty wrote:
> > Matt Leininger wrote:
> > >   Ok.  So the current state is that the mainline devel branch will be
> > > broken for a while?
> > 
> > The trunk is always suppose to work, let alone compile.  This needs to be 
> > fixed 
> > quickly, or the offending code moved to a branch.
> 
> There is nothing that needs to be fixed.  Matt was just not using the
> right combination of bits when we was trying to compile the world.
> 
 I was using the right bits, I just had to just Rolands patch to the
Makefile to get things to compile.

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] 2.6.17-rc1 IPoIB netperf results

[Matt Leininger]

Here are the latest IPoIB results:

For mthca I saw a range of 380-424 MB/s.  The local CPU utilization on
the send side dropped for the 380 MB/s, from 98% to 70%

For ipath it was 310 MB/s.  The local CPU utilization on the send side
was always around 30%.  

  - Matt

Mellanox benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)
patch 1 - remove changeset 314324121f9b94b2ca657a494cf2b9cb0e4a28cc
patch 2 - remove changeset b8259d9ad1d0f8d0c5ea0e37bb15080b0bd395b5
msi_x=1 for all tests
PathScale benchmarks are with RHEL4 x86_64 with HTX HCA
dual-socket dual-core Opteron 2.4 GHz 



netperf -f -M -c -C -H IP_ADDRESS

KernelOpenIB netperf (MB/s)  
2.6.17-rc1   in-kernel424 (mthca ipoib)
2.6.17-rc1   in-kernel310 (ipath ipoib)
2.6.16   svn 6307 367 (mthca ipoib)
2.6.16   svn 6307 319 (ipath ipoib)
2.6.16   svn 6083 371 (mthca ipoib)
2.6.16   svn 6083 304 (ipath ipoib)
2.6.16   svn 5938 380 (mthca ipoib)
2.6.16   svn 5938 300 (ipath ipoib)
2.6.16   in-kernel364
2.6.16-rc5   in-kernel367  
2.6.15   in-kernel382
2.6.14-rc4 patch 12  in-kernel436 
2.6.14-rc4 patch 1   in-kernel434 
2.6.14-rc4   in-kernel385 
2.6.14-rc3   in-kernel374 
2.6.13.2 svn3627  386 
2.6.13.2 patch 1 svn3627  446 
2.6.13.2 in-kernel394 
2.6.13-rc3 patch 12  in-kernel442 
2.6.13-rc3 patch 1   in-kernel450 
2.6.13-rc3   in-kernel395
2.6.13-rc2 patch 1   in-kernel409
2.6.13-rc1 patch 1   in-kernel408
2.6.12.5-lustre  in-kernel399  
2.6.12.5 patch 1 in-kernel464
2.6.12.5 in-kernel402 
2.6.12   in-kernel406 
2.6.12-rc6 patch 1   in-kernel470 
2.6.12-rc6   in-kernel407
2.6.12-rc5   in-kernel405 
2.6.12-rc5 patch 1   in-kernel474
2.6.12-rc4   in-kernel470 
2.6.12-rc3   in-kernel466 
2.6.12-rc2   in-kernel469 
2.6.12-rc1   in-kernel466
2.6.11   in-kernel464 
2.6.11   svn3687  464 
2.6.9-11.ELsmp   svn3513  425  (Woody's results, 3.6Ghz EM64T) 


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: Compile problems with core code and pathscale for svn6462 and linux-2.6.17-rc1

[Matt Leininger]

On Thu, 2006-04-13 at 16:54 -0700, Bryan O'Sullivan wrote:
> On Thursday 13 April 2006 16:51, Matt Leininger wrote:
> 
> > > Are you building the ipath driver out of the kernel.org tree, or out of
> > > svn? If the latter, you have to patch the kernel and rebuild it first.
> >
> >   Out of svn.  I have the drivers/infiniband pointing to the svn tree.
> 
> Yes, that won't work, because the svn include directory has a bunch of stuff 
> that's no upstream.
> 
  Ok.  So the current state is that the mainline devel branch will be
broken for a while?

  BTW, the linux-2.6.17-rc1 in-kernel IB compiled fine.

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: Compile problems with core code and pathscale for svn6462 and linux-2.6.17-rc1

[Matt Leininger]

On Thu, 2006-04-13 at 16:40 -0700, Bryan O'Sullivan wrote:
> On Thursday 13 April 2006 16:32, Matt Leininger wrote:
> > I'm trying to compile the svn 6462 snapshot with linux-2.6.17-rc1 on a
> > RHEL4 based system.
> 
> Are you building the ipath driver out of the kernel.org tree, or out of svn?  
> If the latter, you have to patch the kernel and rebuild it first.

  Out of svn.  I have the drivers/infiniband pointing to the svn tree.  

  I'll try using the drivers in the kernel.org tree.

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Compile problems with core code and pathscale for svn6462 and linux-2.6.17-rc1

[Matt Leininger]

I'm trying to compile the svn 6462 snapshot with linux-2.6.17-rc1 on a
RHEL4 based system.

I get the following error for addr.c:

  CC [M]  drivers/infiniband/core/index.o
  CC [M]  drivers/infiniband/core/addr.o
In file included from drivers/infiniband/core/addr.c:38:
drivers/infiniband/include/rdma/ib_addr.h:43: error: field `dev_type'
has incomplete type
drivers/infiniband/core/addr.c: In function `copy_addr':
drivers/infiniband/core/addr.c:95: error: `RDMA_NODE_IB_CA' undeclared
(first use in this function)
drivers/infiniband/core/addr.c:95: error: (Each undeclared identifier is
reported only once
drivers/infiniband/core/addr.c:95: error: for each function it appears
in.)
drivers/infiniband/core/addr.c:98: error: `RDMA_NODE_RNIC' undeclared
(first use in this function)
make[3]: *** [drivers/infiniband/core/addr.o] Error 1
make[2]: *** [drivers/infiniband/core] Error 2
make[1]: *** [drivers/infiniband] Error 2


If I remove include/rdma (which I had to do in the past) then some of
the pathscale code fails to compile.  Here is the error:

  LD [M]  drivers/infiniband/core/rdma_ucm.o
  CC [M]  drivers/infiniband/hw/ipath/ipath_cq.o
In file included from drivers/infiniband/hw/ipath/ipath_cq.c:36:
drivers/infiniband/hw/ipath/ipath_verbs.h:40:26: rdma/ib_pack.h: No such
file or directory
In file included from drivers/infiniband/hw/ipath/ipath_cq.c:36:
drivers/infiniband/hw/ipath/ipath_verbs.h:128: error: field `grh' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:147: error: field `mgid' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:155: error: field `ibmr' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:161: error: field `ibfmr' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:168: error: field `ibpd' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:174: error: field `ibah' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:175: error: field `attr' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:223: error: field `ibcq' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:239: error: field `wr' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:269: error: field `ibsrq' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:284: error: field `ibqp' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:288: error: field
`remote_ah_attr' has incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:331: error: field `path_mtu'
has incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:412: error: field `ibdev' has
incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h:485: error: field `ibucontext'
has incomplete type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_imr':
drivers/infiniband/hw/ipath/ipath_verbs.h:490: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:490: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_ifmr':
drivers/infiniband/hw/ipath/ipath_verbs.h:495: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:495: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_ipd':
drivers/infiniband/hw/ipath/ipath_verbs.h:500: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:500: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_iah':
drivers/infiniband/hw/ipath/ipath_verbs.h:505: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:505: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_icq':
drivers/infiniband/hw/ipath/ipath_verbs.h:510: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:510: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_isrq':
drivers/infiniband/hw/ipath/ipath_verbs.h:515: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:515: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_iqp':
drivers/infiniband/hw/ipath/ipath_verbs.h:520: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:520: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: In function `to_idev':
drivers/infiniband/hw/ipath/ipath_verbs.h:525: warning: type defaults to
`int' in declaration of `__mptr'
drivers/infiniband/hw/ipath/ipath_verbs.h:525: warning: initialization
from incompatible pointer type
drivers/infiniband/hw/ipath/ipath_verbs.h: At top level:
drivers/infiniband/hw/ipath/ipath_verbs.h:533: warning: "

Re: [openib-general] Re: Re: TSO and IPoIB performance degradation

[Matt Leininger]

On Mon, 2006-03-20 at 12:31 -0800, Matt Leininger wrote:
> On Mon, 2006-03-20 at 11:02 +0200, Michael S. Tsirkin wrote:
> > 
> > BTW, Matt, it might be interesting to compare
> > 2.6.13-rc3 patch 1 against -rc1 and -rc2 with patch 1, to try and track
> > down the last bit of performance degradation in IPoIB.
> > 
> > Could you look into this?
> 
 Here are the latest IPoIB performance numbers.  I added the two kernels
you requested and some 2.6.16 results.


All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)
ipath results RHE4 x86_64 with HTX HCA dual socket dual core 2.4 GHz
 
patch 1 - remove changeset 314324121f9b94b2ca657a494cf2b9cb0e4a28cc
patch 2 - remove changeset b8259d9ad1d0f8d0c5ea0e37bb15080b0bd395b5
msi_x=1 for all PCIe tests

netperf -f -M -c -C -H IP_ADDRESS

KernelOpenIB netperf (MB/s)  
2.6.16   svn 6083 371 (mthca ipoib)
2.6.16   svn 6083 304 (ipath ipoib)
2.6.16   svn 5938 300 (ipath ipoib)
2.6.16   svn 5938 380 (mthca ipoib)
2.6.16   in-kernel364
2.6.16-rc5   in-kernel367  
2.6.15   in-kernel382
2.6.14-rc4 patch 12  in-kernel436 
2.6.14-rc4 patch 1   in-kernel434 
2.6.14-rc4   in-kernel385 
2.6.14-rc3   in-kernel374 
2.6.13.2 svn3627  386 
2.6.13.2 patch 1 svn3627  446 
2.6.13.2 in-kernel394 
2.6.13-rc3 patch 12  in-kernel442 
2.6.13-rc3 patch 1   in-kernel450 
2.6.13-rc3   in-kernel395
2.6.13-rc2 patch 1   in-kernel409
2.6.13-rc1 patch 1   in-kernel408
2.6.12.5-lustre  in-kernel399  
2.6.12.5 patch 1 in-kernel464
2.6.12.5 in-kernel402 
2.6.12   in-kernel406 
2.6.12-rc6 patch 1   in-kernel470 
2.6.12-rc6   in-kernel407
2.6.12-rc5   in-kernel405 
2.6.12-rc5 patch 1   in-kernel474
2.6.12-rc4   in-kernel470 
2.6.12-rc3   in-kernel466 
2.6.12-rc2   in-kernel469 
2.6.12-rc1   in-kernel466
2.6.11   in-kernel464 
2.6.11   svn3687  464 
2.6.9-11.ELsmp   svn3513  425  (Woody's results, 3.6Ghz EM64T) 


 - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Problem configuring ipath_ether

[Matt Leininger]

On Fri, 2006-03-24 at 22:38 -0800, Matt Leininger wrote:
> On Fri, 2006-03-24 at 16:42 -0800, Greg Lindahl wrote:
> > On Thu, Mar 23, 2006 at 11:50:18PM -0800, Matt Leininger wrote:
> > 
> > Matt,
> > 
> > ipath_ether uses InfiniBand protocols but is not the same as IPoIB. So
> > it's better to ask [EMAIL PROTECTED] about it. (The reason we ship
> > it at all is that it's faster than IPoIB -- this necessitates a
> > different wire format -- and it behaves more like an ethernet device
> > than IPoIB.)
> 
  Greg et. al.

I'm still having problems configuring the ipath_ether device.  The
modules build and load fine using an openfabrics svn 6083 snapshot.  I
have the following modules loaded with a 2.6.16 kernel, RHEL4 base
distro, iPath HTX, with an iWill HTX motherboard.

Module  Size  Used by
ipath_ether81264  0
ipath_core151572  1 ipath_ether
ib_ucm 15624  0
ib_cm  31504  1 ib_ucm
ib_uverbs  34864  1 ib_ucm
ib_mad 35108  1 ib_cm
ib_core45184  4 ib_ucm,ib_cm,ib_uverbs,ib_mad

The ipath_ether device seems to be mapped to eth2.  However, when I do
an 'ifconfig eth2 IP_ADDRESS' it hangs.  Here is the strace.

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b7aa5fc9000
read(6, "# Locale name alias data base.\n#"..., 4096) = 2528
read(6, "", 4096)   = 0
close(6)= 0
munmap(0x2b7aa5fc9000, 4096)= 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1
ENOENT (No such file or directory)ioctl(4, SIOCSIFADDR, 0x7f805800)
= 0
ioctl(4, SIOCGIFFLAGS, 0x7f805730)  = 0
ioctl(4, SIOCSIFFLAGS 

It's hanging (and then failing) and the ioctl call for SIOCSIFFLAGS.

Am I missing something?  

 Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Problem configuring ipath_ether

[Matt Leininger]

On Fri, 2006-03-24 at 16:42 -0800, Greg Lindahl wrote:
> On Thu, Mar 23, 2006 at 11:50:18PM -0800, Matt Leininger wrote:
> 
> > I have the ipath driver up, running, and working with IPoIB.  I'm using
> > 2.6.16 with svn 5938.  The ipath_ether comes up as eth2.  I can set the
> > netmask and broadcast, but when I try to set the ip address for this
> > device I get the following error:
> 
> Matt,
> 
> ipath_ether uses InfiniBand protocols but is not the same as IPoIB. So
> it's better to ask [EMAIL PROTECTED] about it. (The reason we ship
> it at all is that it's faster than IPoIB -- this necessitates a
> different wire format -- and it behaves more like an ethernet device
> than IPoIB.)

  That's why I want to try it.  :)

  I figured since the code was in OpenIB that this should go to this
mail list.  I'll cc [EMAIL PROTECTED] with any future issues.
> 
> My guess is that you're trying to use it with ib_mad (the in-kernel
> SMA).  Do you see anything in /var/log/messages like "ipath_ether_open
> timed out waiting for MLID"? The relevant entry in RELEASE-NOTES.txt
> under the KNOWN LIMITATIONS section is:

   That could be.  I'll check.  I thought ipath_ether was suppose to
work when IPoIB was enabled.
> 
>* IPoIB and ipath_ether do not work together, as ipath_ether requires
>  the InfiniPath SMA, which needs to be disabled to use OpenIB.  This
>  will be fixed in a future release.

>From a previous email I thought IPoIB and ipath_ether would work
together.

http://openib.org/pipermail/openib-general/2006-March/018206.html
> 
> Apologies for this not being more obvious.

  No problem.  Thanks for the help.

  - Matt
> 
> -- greg
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Problem configuring ipath_ether

[Matt Leininger]

I have the ipath driver up, running, and working with IPoIB.  I'm using
2.6.16 with svn 5938.  The ipath_ether comes up as eth2.  I can set the
netmask and broadcast, but when I try to set the ip address for this
device I get the following error:


[EMAIL PROTECTED] infiniband]# ifconfig eth2 10.128.20.103
SIOCSIFFLAGS: Operation not permitted


Here is the same command with an strace.


[EMAIL PROTECTED] infiniband]# strace ifconfig eth2 10.128.20.103
execve("/sbin/ifconfig", ["ifconfig", "eth2", "10.128.20.103"], [/* 31
vars */]) = 0
uname({sys="Linux", node="opt1", ...})  = 0
brk(0)  = 0x61
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b6edea52000
access("/etc/ld.so.preload", R_OK)  = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY)  = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=145885, ...}) = 0
mmap(NULL, 145885, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b6edea53000
close(3)= 0
open("/lib64/tls/libc.so.6", O_RDONLY)  = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\305A|<"...,
640) = 640
lseek(3, 624, SEEK_SET) = 624
read(3, "\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\4\0\0\0"..., 32)
= 32
fstat(3, {st_mode=S_IFREG|0755, st_size=1489988, ...}) = 0
mmap(0x3c7c40, 2301864, PROT_READ|PROT_EXEC, MAP_PRIVATE|
MAP_DENYWRITE, 3, 0) = 0x3c7c40
mprotect(0x3c7c529000, 1085352, PROT_NONE) = 0
mmap(0x3c7c628000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|
MAP_DENYWRITE, 3, 0x128000) = 0x3c7c628000
mmap(0x3c7c62e000, 16296, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|
MAP_ANONYMOUS, -1, 0) = 0x3c7c62e000
close(3)= 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b6edea77000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b6edea78000
mprotect(0x3c7c628000, 12288, PROT_READ) = 0
arch_prctl(0x1002, 0x2b6edea77b00)  = 0
munmap(0x2b6edea53000, 145885)  = 0
brk(0)  = 0x61
brk(0x631000)   = 0x631000
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=39550704, ...}) = 0
mmap(NULL, 39550704, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b6edea79000
close(3)= 0
uname({sys="Linux", node="opt1", ...})  = 0
access("/proc/net", R_OK)   = 0
access("/proc/net/unix", R_OK)  = 0
socket(PF_FILE, SOCK_DGRAM, 0)  = 3
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
access("/proc/net/if_inet6", R_OK)  = 0
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 5
access("/proc/net/ax25", R_OK)  = -1 ENOENT (No such file or
directory)
access("/proc/net/nr", R_OK)= -1 ENOENT (No such file or
directory)
access("/proc/net/rose", R_OK)  = -1 ENOENT (No such file or
directory)
access("/proc/net/ipx", R_OK)   = -1 ENOENT (No such file or
directory)
access("/proc/net/appletalk", R_OK) = -1 ENOENT (No such file or
directory)
access("/proc/sys/net/econet", R_OK)= -1 ENOENT (No such file or
directory)
access("/proc/sys/net/ash", R_OK)   = -1 ENOENT (No such file or
directory)
access("/proc/net/x25", R_OK)   = -1 ENOENT (No such file or
directory)
open("/usr/share/locale/locale.alias", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=2528, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b6ee1031000
read(6, "# Locale name alias data base.\n#"..., 4096) = 2528
read(6, "", 4096)   = 0
close(6)= 0
munmap(0x2b6ee1031000, 4096)= 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/net-tools.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/net-tools.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/net-tools.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
ioctl(4, SIOCSIFADDR, 0x7fbc8620)   = 0
ioctl(4, SIOCGIFFLAGS, 0x7fbc8550)  = 0
ioctl(4, SIOCSIFFLAGS, 0x7fbc8550)  = -1 EPERM (Operation not
permitted)
dup(2)  = 6
fcntl(6, F_GETFL)   = 0x8002 (flags O_RDWR|
O_LARGEFILE|0x8000)
fstat(6, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2b6ee1031000
lseek(6, 0, SEEK_CUR)   = -1 ESPIPE (Illegal seek)
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
o

[openib-general] Re: SRP compile error

[Matt Leininger]

On Wed, 2006-03-22 at 14:41 -0800, Roland Dreier wrote:
>  > drivers/infiniband/ulp/srp/ib_srp.c:1409:5: warning:
>  > "LINUX_VERSION_CODE" is not defined
>  > drivers/infiniband/ulp/srp/ib_srp.c:1409:27: warning: "KERNEL_VERSION"
>  > is not defined
>  > drivers/infiniband/ulp/srp/ib_srp.c:1409:41: missing binary operator
>  > before token "("
> 
> I don't see any reference to LINUX_VERSION_CODE in the latest svn srp code.
> So I think you must have a stale file somewhere.
> 
  Yeah, the date on ib_srp.c looks correct (today) but it's contents did
match that of the latest ib_srp.c in svn.

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] SRP compile error

[Matt Leininger]

I get the following error when trying to compile SRP (from the devel
branch) with 2.6.16.

  LD  drivers/infiniband/ulp/srp/built-in.o
  CC [M]  drivers/infiniband/ulp/srp/ib_srp.o
drivers/infiniband/ulp/srp/ib_srp.c:1409:5: warning:
"LINUX_VERSION_CODE" is not defined
drivers/infiniband/ulp/srp/ib_srp.c:1409:27: warning: "KERNEL_VERSION"
is not defined
drivers/infiniband/ulp/srp/ib_srp.c:1409:41: missing binary operator
before token "("
make[3]: *** [drivers/infiniband/ulp/srp/ib_srp.o] Error 1
make[2]: *** [drivers/infiniband/ulp/srp] Error 2
make[1]: *** [drivers/infiniband] Error 2
make: *** [drivers] Error 2

Should ib_srp.c have #include ?

Adding it fixes the compile error.

  - Matt



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: Re: TSO and IPoIB performance degradation

[Matt Leininger]

On Mon, 2006-03-20 at 11:02 +0200, Michael S. Tsirkin wrote:
> Quoting r. Matt Leininger <[EMAIL PROTECTED]>:
> > KernelOpenIBmsi_x  netperf (MB/s)  
> > 2.6.16-rc5   in-kernel1 367
> > 2.6.15   in-kernel1 382
> > 2.6.14-rc4 patch 1   in-kernel1 434 
> > 2.6.14-rc4   in-kernel1 385 
> > 2.6.14-rc3   in-kernel1 374 
> > 2.6.13.2 svn3627  1 386 
> > 2.6.13.2 patch 1 svn3627  1 446 
> > 2.6.13.2 in-kernel1 394 
> > 2.6.13-rc3 patch 12  in-kernel1 442 
> > 2.6.13-rc3 patch 1   in-kernel1 450 
> > 2.6.13-rc3   in-kernel1 395
> > 2.6.12.5-lustre  in-kernel1 399  
> > 2.6.12.5 patch 1 in-kernel1 464
> > 2.6.12.5 in-kernel1 402 
> > 2.6.12   in-kernel1 406 
> > 2.6.12-rc6 patch 1   in-kernel1 470 
> > 2.6.12-rc6   in-kernel1 407
> > 2.6.12-rc5   in-kernel1 405 
> > 2.6.12-rc5 patch 1   in-kernel1 474
> > 2.6.12-rc4   in-kernel1 470 
> > 2.6.12-rc3   in-kernel1 466 
> > 2.6.12-rc2   in-kernel1 469 
> > 2.6.12-rc1   in-kernel1 466
> > 2.6.11   in-kernel1 464 
> > 2.6.11   svn3687  1 464 
> > 2.6.9-11.ELsmp   svn3513  1 425  (Woody's results, 3.6Ghz
> > EM64T) 
> > 
> 
> BTW, Matt, it might be interesting to compare
> 2.6.13-rc3 patch 1 against -rc1 and -rc2 with patch 1, to try and track
> down the last bit of performance degradation in IPoIB.
> 
> Could you look into this?

  Sure, but I'm not sure it will tell us that much more.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: TSO and IPoIB performance degradation

[Matt Leininger]

On Tue, 2006-03-07 at 13:49 -0800, Stephen Hemminger wrote:
> On Tue, 07 Mar 2006 13:44:51 -0800
> Matt Leininger <[EMAIL PROTECTED]> wrote:
> 
> > On Mon, 2006-03-06 at 19:13 -0800, Shirley Ma wrote:
> > > 
> > > > More likely you are getting hit by the fact that TSO prevents the
> > > congestion
> > > window from increasing properly. This was fixed in 2.6.15 (around mid
> > > of Nov 2005). 
> > > 
> > > Yep, I noticed the same problem. After updating to the new kernel, the
> > > performance are much better, but it's still lower than before.
> > 
> >  Here is an updated version of OpenIB IPoIB performance for various
> > kernels with and without one of the TSO patches.  The netperf
> > performance for the latest kernels has not improved the TSO performance
> > drop.
> > 
> >   Any comments or suggestions would be appreciated.
> > 
> >   - Matt
> 
> Configuration information? like did you increase the tcp_rmem, tcp_wmem?
> Tcpdump traces of what is being sent and available window?
> Is IB using NAPI or just doing netif_rx()?

  I used the standard setting for tcp_rmem and tcp_wmem.   Here are a
few other runs that change those variables.  I was able to improve
performance by ~30MB/s to 403 MB/s, but this is still a ways from the
474 MB/s before the TSO patches.

 Thanks,

- Matt

All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)
patch 1 - remove changeset 314324121f9b94b2ca657a494cf2b9cb0e4a28cc
msi_x=1 for all tests

KernelOpenIB netperf (MB/s)  
2.6.16-rc5   in-kernel403  
tcp_wmem 4096 87380 16777216 tcp_rmem 4096 87380 16777216

2.6.16-rc5   in-kernel395  
tcp_wmem 4096 102400 16777216 tcp_rmem 4096 102400 16777216

2.6.16-rc5   in-kernel392  
tcp_wmem 4096 65536 16777216 tcp_rmem 4096 87380 16777216

2.6.16-rc5   in-kernel394  
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 102400 16777216

2.6.16-rc5   in-kernel377  
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 153600 16777216

2.6.16-rc5   in-kernel377  
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 131072 16777216

2.6.16-rc5   in-kernel353  
tcp_wmem 4096 262144 16777216 tcp_rmem 4096 262144 16777216

2.6.16-rc5   in-kernel305  
tcp_wmem 4096 262144 16777216 tcp_rmem 4096 524288 16777216

2.6.16-rc5   in-kernel303  
tcp_wmem 4096 131072 16777216 tcp_rmem 4096 524288 16777216

2.6.16-rc5   in-kernel290  
tcp_wmem 4096 524288 16777216 tcp_rmem 4096 524288 16777216

2.6.16-rc5   in-kernel367  default tcp values


All with standard tcp settings
KernelOpenIB netperf (MB/s)  
2.6.16-rc5   in-kernel367  
2.6.15   in-kernel382
2.6.14-rc4 patch 12  in-kernel436 
2.6.14-rc4 patch 1   in-kernel434 
2.6.14-rc4   in-kernel385 
2.6.14-rc3   in-kernel374 
2.6.13.2 svn3627  386 
2.6.13.2 patch 1 svn3627  446 
2.6.13.2 in-kernel394 
2.6.13-rc3 patch 12  in-kernel442 
2.6.13-rc3 patch 1   in-kernel450 
2.6.13-rc3   in-kernel395
2.6.12.5-lustre  in-kernel399  
2.6.12.5 patch 1 in-kernel464
2.6.12.5 in-kernel402 
2.6.12   in-kernel406 
2.6.12-rc6 patch 1   in-kernel470 
2.6.12-rc6   in-kernel407
2.6.12-rc5   in-kernel405 
2.6.12-rc5 patch 1   in-kernel474
2.6.12-rc4   in-kernel470 
2.6.12-rc3   in-kernel466 
2.6.12-rc2   in-kernel469 
2.6.12-rc1   in-kernel466
2.6.11   in-kernel464 
2.6.11   svn3687  464 
2.6.9-11.ELsmp   svn3513  425  (Woody's results, 3.6Ghz EM64T) 


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] RC1 status

[Matt Leininger]

On Tue, 2006-03-07 at 10:06 -0800, Bryan O'Sullivan wrote:
> I have x86_64 RPMs ready for FC4, and I need to verify that I can build
> RPMs from the source tarballs.  I can't build OpenSM on i386, which I've
> already told Hal about.
> 
> If the tarballs build, I'll put up FC4 and SUSE10 x86_64 RPMs some time
> today.  I don't have an i386 SUSE10 machine to build on, unfortunately.
> 
  RPMs for RHEL4 would also be useful.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: TSO and IPoIB performance degradation

[Matt Leininger]

On Mon, 2006-03-06 at 19:13 -0800, Shirley Ma wrote:
> 
> > More likely you are getting hit by the fact that TSO prevents the
> congestion
> window from increasing properly. This was fixed in 2.6.15 (around mid
> of Nov 2005). 
> 
> Yep, I noticed the same problem. After updating to the new kernel, the
> performance are much better, but it's still lower than before.

 Here is an updated version of OpenIB IPoIB performance for various
kernels with and without one of the TSO patches.  The netperf
performance for the latest kernels has not improved the TSO performance
drop.

  Any comments or suggestions would be appreciated.

  - Matt

> 
All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)
patch 1 - remove changeset 314324121f9b94b2ca657a494cf2b9cb0e4a28cc

KernelOpenIBmsi_x  netperf (MB/s)  
2.6.16-rc5   in-kernel1 367
2.6.15   in-kernel1 382
2.6.14-rc4 patch 1   in-kernel1 434 
2.6.14-rc4   in-kernel1 385 
2.6.14-rc3   in-kernel1 374 
2.6.13.2 svn3627  1 386 
2.6.13.2 patch 1 svn3627  1 446 
2.6.13.2 in-kernel1 394 
2.6.13-rc3 patch 12  in-kernel1 442 
2.6.13-rc3 patch 1   in-kernel1 450 
2.6.13-rc3   in-kernel1 395
2.6.12.5-lustre  in-kernel1 399  
2.6.12.5 patch 1 in-kernel1 464
2.6.12.5 in-kernel1 402 
2.6.12   in-kernel1 406 
2.6.12-rc6 patch 1   in-kernel1 470 
2.6.12-rc6   in-kernel1 407
2.6.12-rc5   in-kernel1 405 
2.6.12-rc5 patch 1   in-kernel1 474
2.6.12-rc4   in-kernel1 470 
2.6.12-rc3   in-kernel1 466 
2.6.12-rc2   in-kernel1 469 
2.6.12-rc1   in-kernel1 466
2.6.11   in-kernel1 464 
2.6.11   svn3687  1 464 
2.6.9-11.ELsmp   svn3513  1 425  (Woody's results, 3.6Ghz
EM64T) 


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] RE: Error with 2.6.16-rc5 and latest main branch

[Matt Leininger]

On Fri, 2006-03-03 at 16:46 -0800, Sean Hefty wrote:
> >I'm getting the following error when trying to compile svn 5606 with a
> >2.6.16-rc5 kernel.
> 
> Looks like you're picking up the standard 2.6.16-rc5 include files.
> 
 That was the problem.  

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Error with 2.6.16-rc5 and latest main branch

[Matt Leininger]

I'm getting the following error when trying to compile svn 5606 with a
2.6.16-rc5 kernel.  

 CC [M]  drivers/infiniband/core/addr.o
In file included from drivers/infiniband/core/addr.c:43:
drivers/infiniband/include/rdma/ib_addr.h:45: error: field `dev_type'
has incomplete type
drivers/infiniband/core/addr.c: In function `copy_addr':
drivers/infiniband/core/addr.c:101: error: `RDMA_NODE_IB_CA' undeclared
(first use in this function)
drivers/infiniband/core/addr.c:101: error: (Each undeclared identifier
is reported only once
drivers/infiniband/core/addr.c:101: error: for each function it appears
in.)
drivers/infiniband/core/addr.c:104: error: `RDMA_NODE_RNIC' undeclared
(first use in this function)
make[3]: *** [drivers/infiniband/core/addr.o] Error 1
make[2]: *** [drivers/infiniband/core] Error 2
make[1]: *** [drivers/infiniband] Error 2
make: *** [drivers] Error 2


 - Matt




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] RFC: SDP plans

[Matt Leininger]

On Mon, 2006-02-27 at 15:27 -0800, Sean Hefty wrote:
> Bryan O'Sullivan wrote:
> > So.  We can move the 1.0 release date to "whenever SDP gets into an
> > upstream kernel" without knowing when that might be, or we can do the
> > best we can with what we have now.
> > 
> > My preference is strongly for the latter.
> 
> SDP should not be in a release until it is release quality, and I would based 
> that on an upstream submission.
> 
> What's wrong with shipping release 1.0 without SDP, then shipping an updated 
> release (1.1) once SDP is ready?  SDP doesn't get there faster by delaying 
> the 
> 1.0 release.

  Sounds reasonable to me.   Put whatever components are ready into the
1.0 release.  Other components can come in later.  Don't try and solve
every problem now, just work towards a solid baseline for code releases.

  - Matt



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Suggested components to support in 1.0

[Matt Leininger]

On Fri, 2006-02-24 at 17:25 -0800, Bryan O'Sullivan wrote:
> On Fri, 2006-02-24 at 22:27 +0100, Christoph Hellwig wrote:

> > >   * SDP
> > 
> > There's various political problems involved here.
> 
> Pardon my ignorance, but what kinds?
> 
  MS claims that an SDP implementation *may* use some of their IP.  Of
course they don't tell you which patents.  The task of deciding what
risk is associated with SDP (licensing, etc.) is left as an exercise to
the reader.  Various individuals and companies on this list have looked
into the SDP licensing issue, but I haven't seen much (needed)
discussion lately.  

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] We have an OpenIB code release team

[Matt Leininger]

On Tue, 2006-02-14 at 07:00 -0800, Roland Dreier wrote:
> It's great that so many people want to help with the release, but the
> whole point of having a release team was to have a small number of
> people that can move the release rapidly.  I think that the three
> members of the team is the maximum number already.
> 
> Please give the release team time to identify places where they need
> help.  Once they have had a chance to begin working, I'm sure there
> will be ample opportunity to volunteer.

  Limiting the "release team" to three people is an attempt to keep this
process manageable.  However, for the OpenIB code release process to
work well we will need to take advantage of the Q&A resources at several
companies (as well as the wider community).  To first order I'd like to
see Mellanox, Cisco, Voltaire, and SilverStorm, working with the OpenIB
release team, to take advantage of each companies Q&A process.  It's
good that Moni (Voltaire) and Tziporet (Mellanox) would like to be the
points of contact for their companies working with the release team.  
  
   Would anyone from Cisco and SilverStorm be willing to interface
between your internal Q&A process and the OpenIB release team?  

  Thanks,

- Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] We have an OpenIB code release team

[Matt Leininger]

OpenIB Community,

   At the OpenIB workshop we discussed putting together an OpenIB code
release team.  The first task is to work on a 1.0 release of the current
development branch of the OpenIB code.  We have some volunteers so we
now have our 1.0 release team 
 
Bryan O'Sullivan has signed up to be the release manager for 1.0,
working with Robert Woodruff and Hal Rosenstock. 

The charter of the release team includes:

- determine release mechanics (eg how and when snapshot is taken,
bug handling, etc - to be discussed on mailing list)
- make sure release meets the needs for RedHat, Novell, and other
distros
- provide some oversight to the testing matrix
- leverage the Q&A resources of the OpenIB industry members
- publish what gets tested, at some high level
- publish some release criteria for 1.0 features
  (eg, what makes a feature 1.0 as opposed to Beta)
- post release candidates for testing by the wider community
- ship the release, hopefully in not too many weeks.

The release team will be following up to discuss how this
will all work.
 
Thanks to all who have volunteered their time for this effort. I'd like
to encourage the OpenIB community to download the code release
candidates (when they are ready) and test it in your particular
computing environments.  This is an important step towards getting
OpenIB hardened and ready for inclusion into the various Linux
distributions.  

  Thanks,

- Matt
 




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] OpenIB Developers Workshop Presentation Available

[Matt Leininger]

On Mon, 2006-02-13 at 12:10 -0800, Grant Grundler wrote:
> On Sun, Feb 12, 2006 at 08:07:06PM -0800, Matt Leininger wrote:
> >   Most of the presentations from the OpenIB Developers Workshop are
> > available for download at
> > http://www.openib.org/conference/sonoma2006/index.html
> 
> Matt - big thanks for posting those!
> 
> Of the ones still missing, I'd really like to review
>   "Oracle Need for RDS"
> 
> Richard, any ETA when you can email it to Matt?
> (or post it here?)
> 
  Thanks to Richard for sending them to me.  They are posted now.

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] OpenIB Developers Workshop Presentation Available

[Matt Leininger]

  Most of the presentations from the OpenIB Developers Workshop are
available for download at
http://www.openib.org/conference/sonoma2006/index.html

This link is available from the main openib webpage.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Timeline of IPoIB performance

[Matt Leininger]

On Wed, 2005-10-12 at 11:28 -0700, Matt Leininger wrote:
> On Wed, 2005-10-12 at 09:53 -0700, Roland Dreier wrote:
> > Herbert> Try reverting the changeset
> > 
> > Herbert> 314324121f9b94b2ca657a494cf2b9cb0e4a28cc
> > 
> > Herbert> which lies between these two points and may be relevant.
> > 
> > Matt, I pulled this out of git for you.  I guess Herbert is suggesting
> > to patch -R the below against 2.6.12-rc5:

> I applied your patch suggest by Herbert:
> 
> http://www.mail-archive.com/openib-general%40openib.org/msg11415.html
> 
  I backed out this patch out of a few other kernels and always see a
performance improvement.  This gets back ~50-60 MB/s of the 90-100 MB/s
drop off in IPoIB performance.   

  Is it still worth testing the TSO patches that Herbert suggested for
some of the 2.6.13-rc kernels?
 
  Thanks,

   - Matt



All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)

Kernel   OpenIBmsi_x  netperf (MB/s)  
2.6.14-rc4  in-kernel1 434  (backed out patch)
2.6.14-rc4  in-kernel1 385 

2.6.13.2svn3627  1 446  (backed out patch)
2.6.13.2svn3627  1 386 
2.6.13.2in-kernel1 394 

2.6.12.5in-kernel1 464  (backed out patch)
2.6.12.5in-kernel1 402 

2.6.12-rc6  in-kernel1 470  (backed out patch) 
2.6.12-rc6  in-kernel1 407

2.6.12-rc5  in-kernel1 474 (backed out patch)
2.6.12-rc5  in-kernel1 405 


2.6.9-11.ELsmp  svn3513  1 425  (Woody's results, 3.6Ghz EM64T) 


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Timeline of IPoIB performance

[Matt Leininger]

On Wed, 2005-10-12 at 09:53 -0700, Roland Dreier wrote:
> Herbert> Try reverting the changeset
> 
> Herbert> 314324121f9b94b2ca657a494cf2b9cb0e4a28cc
> 
> Herbert> which lies between these two points and may be relevant.
> 
> Matt, I pulled this out of git for you.  I guess Herbert is suggesting
> to patch -R the below against 2.6.12-rc5:

I applied your patch suggest by Herbert:

http://www.mail-archive.com/openib-general%40openib.org/msg11415.html

to my 2.6.12-rc5 tree and IPoIB performance improved back to the ~475
MB/s range for my EM64T system.  The data is below.

I'm building/testing 2.6.14-rc4 with and without this patch now.


All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)

Kernel   OpenIBmsi_x  netperf (MB/s)  
2.6.14-rc3  in-kernel1 374 
2.6.13.2svn3627  1 386 
2.6.13.2in-kernel1 394 
2.6.12.5-lustre in-kernel1 399  
2.6.12.5in-kernel1 402 
2.6.12  in-kernel1 406 
2.6.12-rc6  in-kernel1 407
2.6.12-rc5  in-kernel1 405
2.6.12-rc5
 - remove changeset 314324121f9b94b2ca657a494cf2b9cb0e4a28cc  
in-kernel1 474
2.6.12-rc4  in-kernel1 470 
2.6.12-rc3  in-kernel1 466 
2.6.12-rc2  in-kernel1 469 
2.6.12-rc1  in-kernel1 466
2.6.11  in-kernel1 464 
2.6.11  svn3687  1 464 
2.6.9-11.ELsmp  svn3513  1 425  (Woody's results, 3.6Ghz EM64T) 

  - Matt




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Timeline of IPoIB performance

[Matt Leininger]

On Mon, 2005-10-10 at 16:38 -0700, Roland Dreier wrote:
> Matt>   Pretty consistent.  Here are a few runs with 2.6.12-rc5
> Matt> with reboots in between each run.  I'm using netperf-2.3pl1.
> 
> That's interesting.  I'm guessing you're using mem-ful HCAs?

  Yes, I'm using mem-full HCAs.  I could try reflashing the firmware for
memfree if that's of interest.
> 
> Given that your results are more stable than mine, if you're up for
> it, you could install git, clone Linus's tree, and then do a git
> bisect between 2.6.12-rc4 and 2.6.12-rc5 to narrow down the regression
> to a single commit (if in fact that's possible).
  
 I was hoping someone else would do this.  :)
 
 I'll start working on it tomorrow if no one else gets to it.

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Lustre Network Driver - KDAPL or verbs?

[Matt Leininger]

On Sun, 2005-10-09 at 17:17 -0400, Peter J. Braam wrote:
> Cluster File Systems, Inc and its customers have been wondering if the
> Lustre Network Driver (LND) for OpenIb gen2, which we will begin to
> develop during the coming months, should be based on kdapl or verbs.
>  
> The driver we plan to develop should strive to address several goals: 
>  - high reliability and performance
>  - allow interoperability between user and kernel level
>  - allow interoperability, or better, portability among different
> operating systems (Linux, OS X, Windows, Solaris)
>  - be suitable for inclusion in the Linux kernel
>  
  These last two bullets are mutually exclusive.  Submitting code, for
inclusion into Linux, that contains an OS abstraction is a sure way to
get your code rejected.  It happened to the IBAL stack and it will
happen again unless you focus on a Linux specific "Lustre network
driver".  

  As a custom of IB products and Lustre, I'd recommend coding to the
OpenIB Verbs layer and use the new CM code as it develops (as Fab
described).  It's not difficult to port from VAPI to OpenIB Verbs  so
your current VAPI NAL would be a good starting point.

  It would be great to see fewer Lustre kernel patches and more of
Lustre in the Linux kernel.


  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Timeline of IPoIB performance

[Matt Leininger]

On Mon, 2005-10-10 at 11:23 -0700, Roland Dreier wrote:
>  > 2.6.12-rc5  in-kernel1 405   <
>  > 2.6.12-rc4  in-kernel1 470   <
> 
> I was optimistic when I saw this, because the changeover to git
> occurred with 2.6.12-rc2, so I thought I could use git bisect to track
> down exactly when the performance regression happened.
> 
> However, I haven't been able to get numbers that are stable enough to
> track this down.  I have two systems, both HP DL145s with dual Opteron
> 875s and two-port mem-free PCI Express HCAs.  I use MSI-X with the
> completion interrupt affinity set to CPU 0, and "taskset 2" to run
> netserver and netperf on CPU 1.
> 
> With default netperf parameters (just "-H otherguy") I get numbers
> between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5.
> The numbers are quite consistent between reboots, but if I reboot the
> system (even keeping the kernel identical), I see large performance
> changes.  Presumably something is happening like the cache coloring of
> some hot data structures changing semi-randomly depending on the
> timing of various initialations.
> 
> Matt, how stable are your numbers?


  Pretty consistent.  Here are a few runs with 2.6.12-rc5 with reboots
in between each run.  I'm using netperf-2.3pl1.

Run 1:
TCP STREAM TEST to 10.128.20.6
Recv   SendSend  Utilization   Service
Demand
Socket Socket  Message  Elapsed  Send Recv Send
Recv
Size   SizeSize Time Throughput  localremote   local
remote
bytes  bytes   bytessecs.KBytes  /s  % T  % T  us/KB
us/KB

 87380  16384  1638410.00  410302.39   99.8992.094.869
4.489

Run 2: (after another reboot)
TCP STREAM TEST to 10.128.20.6
Recv   SendSend  Utilization   Service
Demand
Socket Socket  Message  Elapsed  Send Recv Send
Recv
Size   SizeSize Time Throughput  localremote   local
remote
bytes  bytes   bytessecs.KBytes  /s  % T  % T  us/KB
us/KB

 87380  16384  1638410.00  409510.33   99.8991.594.879
4.473

Run 3: (after reboot)
TCP STREAM TEST to 10.128.20.6
Recv   SendSend  Utilization   Service
Demand
Socket Socket  Message  Elapsed  Send Recv Send
Recv
Size   SizeSize Time Throughput  localremote   local
remote
bytes  bytes   bytessecs.KBytes  /s  % T  % T  us/KB
us/KB

 87380  16384  1638410.00  404354.11   99.8991.394.941
4.520


I see the same variance in netperf results if I don't reboot between
runs.  

  - Matt



  


> 
  

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Timeline of IPoIB performance

[Matt Leininger]

I'm adding netdev to this thread to see if they can help.

I'm seeing an IPoIB (IP over InfiniBand) netperf performance drop off,
of up to 90 MB/s, when using kernels newer than 2.6.11.  This doesn't
appear to be an OpenIB IPoIB issue since the older in-kernel IB for
2.6.11 and a recent svn3687 snapshot both have the same performance (464
MB/s) with 2.6.11.  I used the same kernel config file as a starting
point for each of these kernel builds.  Have there been any changes in
Linux that would explain these results?

Here is the hardware setup and netperf results using 'netperf -f -M -c
-C -H IPoIB_ADDRESS

All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)

Kernel   OpenIBmsi_x  netperf (MB/s)  
2.6.14-rc3  in-kernel1 374 
2.6.13.2svn3627  1 386 
2.6.13.2in-kernel1 394 
2.6.12.5-lustre in-kernel1 399  
2.6.12.5in-kernel1 402 
2.6.12  in-kernel1 406 
2.6.12-rc6  in-kernel1 407
2.6.12-rc5  in-kernel1 405   <
2.6.12-rc4  in-kernel1 470   <
2.6.12-rc3  in-kernel1 466 
2.6.12-rc2  in-kernel1 469 
2.6.12-rc1  in-kernel1 466
2.6.11  in-kernel1 464 
2.6.11  svn3687  1 464 
2.6.9-11.ELsmp  svn3513  1 425  (Woody's results, 3.6Ghz EM64T) 

 Thanks,

- Matt



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Timeline of IPoIB performance

[Matt Leininger]

On Fri, 2005-10-07 at 18:16 -0700, Roland Dreier wrote:
> I wonder if this BIC bug has anything to do with it: 
> http://lkml.org/lkml/2005/10/7/230
> 
  I'm not sure this helps.  I'm seeing the performance drop of happen
between 2.6.12-rc4 (470 MB/s) and 2.6.12-rc5 (405 MB/s).

  I'll send out my new data and cc netdev.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Timeline of IPoIB performance

[Matt Leininger]

I'm seeing an IPoIB netperf performance drop off, up to 90 MB/s, when
using kernels newer than 2.6.11.  This doesn't appear to be an OpenIB
IPoIB issue since the in-kernel and a recent svn3687 snapshot both have
the same performance (464 MB/s) with 2.6.11.  I used the same kernel
config file as a starting point for each of these kernel builds.  Have
there been any changes in Linux that would explain these results?


All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0
dual EM64T 3.2 GHz PCIe IB HCA (memfull)

Kernel   OpenIBmsi_x  netperf (MB/s)  
2.6.14-rc3  in-kernel1 374 
2.6.13.2svn3627  1 386 
2.6.13.2in-kernel1 394 
2.6.12  in-kernel1 406 
2.6.11  in-kernel1 464 
2.6.11  svn3687  1 464 
2.6.9-11.ELsmp  svn3513  1 425  (Woody's results, 3.6Ghz EM64T) 

  Thanks,

- Matt



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] mvapich-gen2 question

[Matt Leininger]

Including [EMAIL PROTECTED] in this thread.

  - Matt


On Tue, 2005-09-13 at 18:31 -0700, Makia Minich wrote:
> I'm using a RHEL4 based system with the backport-2.6.9 svn drop (svn3279).
> Building the mvapich-gen2 from subversion against this, everything seems to
> be ok, and installing it goes well.  The problem is when I run a test I
> get the following error:
> 
> ::
> => mpicc -o osu-bw osu-bw.c 
> => mpirun_rsh -rsh -hostfile ~/machines -np 2 ./osu-bw
> /benchmarks/osu/src
> /benchmarks/osu/src
> [1] Abort: Error creating CQ
>  at line 121 in file viainit.c
> mpirun: executable version 1 does not match our version 2.
> 
> done.
> => 
> ::
> 
> I see in the code for mvapich (in ch-gen2) that there is a check against the
> version, but I'm not quite sure where this version is defined in my compiled
> code.  Perhaps there's something I'm just not seeing.
> 
> Thanks
> 
> (())
>  Makia Minich  Money is the Devil's toothpaste.
>  925.XXX.  --The Flea (Mucha Lucha)
> (())
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Datacenter Fabric Workshop talks

[Matt Leininger]

 Talks from last weeks OpenIB and Intel sponsored Datacenter Fabric
Workshop are available at http://openib.org/doc.html 

 If we are missing your talk please send it to me.

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] RE: OpenSM Work

[Matt Leininger]

On Thu, 2005-08-04 at 09:09 +0300, Eitan Zahavi wrote:

> > > 
> > > The mode of work we suggest is that she will work offline. 
> >  
> > Not sure by what you mean by offline here. 
> [EZ] Offline means she will do the entire merge and then commit. I
> propose she will commit the changes into a branch and then you can
> review it and do the merge to the main trunk yourself.

 Eitan,

  I'm glad to see the continued interest in OpenSM.  Thanks for the
help.

  Please submit OpenSM changes as patches to the mail list so that Hal
and others in the community can review them.  No sense in doing a bunch
of work until we know what things you are trying to add.  Start with
header files and work out from there.

  My understanding of your "offline" is that it is closed development
followed by open release.  That's not how OpenIB works.  Please submit
the patches to the list so everyone can follow and commit on the OpenSM
code changes.

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Re: [Infiniband-sock_direct] SDP Query

[Matt Leininger]

  OpenIB does have an SDP module that would likely meet your
expectations.  I've cc'd the openib-general mail list and Libor (the SDP
maintainer).  

   - Matt

On Wed, 2005-08-03 at 08:16 -0400, Michael Speth wrote:
> Matt,
>Does OpenIB have a module like the Offload Protocol Module (OPS)?
> 
> Thanks
> 
> On 8/3/05, Matt Leininger <[EMAIL PROTECTED]> wrote:
> > Rajib,
> > 
> >All active IB development has shifted to OpenIB (www.openib.org).
> > Try submitting your question to the OpenIB developers mail list
> > (openib-general@openib.org).
> > 
> >  Thanks,
> > 
> >   - Matt
> > 
> > 
> > On Tue, 2005-08-02 at 17:01 +0800, Majumder, Rajib wrote:
> > > Hello,
> > >
> > > I had a query and wanted to clarify it from this list.
> > >
> > > My firm is planning to migrate to IB. From ULP standpoint of view, our 
> > > plan is to use SDP to take advantage of IB fabric and also without making 
> > > any code changes.
> > >
> > > We have some processes that communicate on the LOCAL host using TCP 
> > > SOCK_STREAM. If we use SDP for these processes, do you expect a 
> > > performance gain?
> > >
> > > Does SDP behave the same way (offloaded stack, RDMA, kernel bypass, zcopy 
> > > etc) while the processes run on the SAME physical host?
> > > Do you have any latency/throughput data available for this test scenario?
> > >
> > > Any opinion would be highly appreciated.
> > >
> > > Thanks for your time!
> > >
> > > Rajib Majumder
> > > Credit Suisse First Boston
> > >
> > >
> > > ==
> > > Please access the attached hyperlink for an important electronic 
> > > communications disclaimer:
> > >
> > > http://www.csfb.com/legal_terms/disclaimer_external_email.shtml
> > >
> > > ==
> > >
> > >
> > >
> > > ---
> > > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> > > from IBM. Find simple to follow Roadmaps, straightforward articles,
> > > informative Webcasts and more! Get everything you need to get up to
> > > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> > > ___
> > > Infiniband-sock_direct mailing list
> > > [EMAIL PROTECTED]
> > > https://lists.sourceforge.net/lists/listinfo/infiniband-sock_direct
> > >
> > 
> > 
> > 
> > ---
> > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> > from IBM. Find simple to follow Roadmaps, straightforward articles,
> > informative Webcasts and more! Get everything you need to get up to
> > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> > ___
> > Infiniband-sock_direct mailing list
> > [EMAIL PROTECTED]
> > https://lists.sourceforge.net/lists/listinfo/infiniband-sock_direct
> > 
> 
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] kernel VM monitor for memory registration caching

[Matt Leininger]

FWIW here is a link to the quadrics vm patch that David Addison posted
on LKML a couple of months ago.  

http://lkml.org/lkml/2005/4/26/198

  - Matt




On Sun, 2005-07-31 at 13:31 +0300, Gleb Natapov wrote:
> Hello Pete,
> 
> On Fri, Jul 29, 2005 at 01:42:25PM -0400, Pete Wyckoff wrote:
> > I'll be happy to discuss the code with anyone who ends up wanting
> > to use it or improve upon it.
> 
> I glanced over the code and I have couple of questions/improvments.
> 
>  First of all, you have one user_delta per mm that user can poll from 
> userspace. Is it possible to make user_delta to be part of dreg_region
> instead of dreg_context and module will set it whenever
> registration becomes invalid. Field 'invalid' will be added to buf_info
> structure and pointer to it will be passed to kernel at registration
> time.
>  This way the userpace can look up cache and check if registration is
> still valid. No need to rescan cache from userspace, we already scanned
> it once from kernel after all. With your current approach userspace will
> need to search for mr_handle in the cache and invalidate the entry that
> holds it.
> 
> 
>  You change vma_ops in vma to catch open/close events. What about
> nopage() method in vma_ops? We have to forward it to original vma_ops?
> 
> Something like included patch (not even compiled).
> 
> 
> --- dreg.c.org2005-07-31 13:10:17.375403091 +0300
> +++ dreg.c2005-07-31 13:24:35.404872561 +0300
> @@ -162,7 +162,10 @@
>  
>  pr_debug("%s: reg %p vma %p addr %lx\n", __func__, reg, vma, reg->addr);
>  if (vma)
> +{
> + kfree (vma->vm_ops);
>   vma->vm_ops = reg->orig_ops;
> +}
>  if (reg->addr)
>   mem_deregister(dc, reg);
>  list_del(®->subordinate_list);
> @@ -305,6 +308,7 @@
>   * forget about it and do not build a new region for it.
>   */
>  if (list_empty(&temp_new_subordinate_list)) {
> + kfree (newvma->vm_ops);
>   newvma->vm_ops = orig_ops;
>  } else {
>   reg = kmem_cache_alloc(dreg_region_cache, GFP_KERNEL);
> @@ -510,7 +514,7 @@
>vma->vm_start, vma->vm_end, reg);
>  
>  reg->orig_ops = vma->vm_ops;
> -if (vma->vm_ops == &dreg_vm_ops) {
> +if (vma->vm_ops->close == dreg_vm_ops.close) {
>   /* chain off proper owner */
>   struct dreg_region *topreg;
>   pr_debug("%s: marked subordinate\n", __func__);
> @@ -523,10 +527,22 @@
>   }
>   list_add(®->subordinate_list, &topreg->subordinate_list);
>  } else {
> + struct vm_operations_struct *tmp_vm_ops;
>   /* non subordinate */
>   reg->vma = vma;
>   INIT_LIST_HEAD(®->subordinate_list);
> - vma->vm_ops = &dreg_vm_ops;  /* own this vma */
> +
> + tmp_vm_ops = kmalloc (sizeof (struct vm_operations_struct), GFP_KERNEL);
> + memcpy (tmp_vm_ops, &dreg_vm_ops, sizeof (struct vm_operations_struct));
> + if (vma->vm_ops)
> + {
> + tmp_vm_ops->nopage = vma->vm_ops->nopage;
> +#ifdef CONFIG_NUMA
> + tmp_vm_ops->set_policy = vma->vm_ops->set_policy;
> + tmp_vm_ops->get_policy = vma->vm_ops->get_policy;
> +#endif
> + }
> + vma->vm_ops = tmp_vm_ops;  /* own this vma */
>   reg->orig_vm_start = vma->vm_start;
>   reg->orig_vm_end = vma->vm_end;
>  }
> --
>   Gleb.
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] OpenIB server is back up

[Matt Leininger]

  The OpenIB webpages, mail list, and svn are back up.


   - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] OpenIB main server power outage this weekend

[Matt Leininger]

Hi folks,

   There is a scheduled power outage this Saturday (July 30th) for the
building that houses the main OpenIB server.  We expect the power to be
out from 7am-6pm PDT.  We'll be bringing down the server a few hours
before then and bring it back up as soon as the power is turned back on.
We've managed to schedule this outage on a Saturday to overlap with
everyone's weekend and minimize the impact to OpenIB work.

  Thanks and sorry for the inconvenience,

 - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] OpenIB Workshop Linux Track: Proposed Agenda

[Matt Leininger]

On Mon, 2005-07-18 at 10:33 -0700, Sujal Das wrote:

> 
> Track L7 
> 13.20-14.00 
> Open MPI  
> Speaker: [PLEASE ADVICE ON WHO THIS SHOULD BE]
> 
   I'd suggest Tim Woodall (LANL) and/or Jeff Squyres (UIndiana).

   - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] mem register failing

[Matt Leininger]

Deanan,

   The stack you are using is not part of OpenIB.  I suggest you contact
Mellanox or your IB software vendor to help you through these support
issues.  

  - Matt


On Sun, 2005-07-17 at 16:18 -0700, Deanan wrote:
> Hi All,
> 
> Has anyone seen vapi_register_mr fail with "Not enough memory" at above 
> ~1MB? I'm getting this error in
> the kernel:
> Jul 18 04:11:24 e09 kernel:  MOSAL(1): ddr[9548]: 
> usr/mellanox/src/vapi/kernel/mlxsys/mosal_iobuf.c:513: region not in 
> process's address space. va=0x5028B0 size=0x20D000
> Jul 18 04:11:24 e09 kernel:  VIPKL(1): 
> usr/mellanox/src/vapi/kernel/vip/mmu.c[242]: make_iobuf: 
> MOSAL_iobuf_register failed: va=0x5028B0size=0x20D000
> Jul 18 04:11:24 e09 kernel:  VIPKL(1): [create_mr] MM_bld_hh_mr failed 
> (-252:VAPI_ENOMEM)
> 
> Thanks,
> 
> Deanan
> 
> 
> 
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Error in opening IB cm device

[Matt Leininger]

On Fri, 2005-07-15 at 01:47 -0400, amith rajith mamidala wrote:
> Hi,
> 
> We have installed the latest openib stack (Rev:2861) on the X86_64
> platform. While running the pingpong tests I am encountering the
> following error:
> 
> libucm: Error <-1:2> couldn't open IB cm device 
> [1] Abort: Error getting HCA context
> 
> I am not sure why this happening ?
> 

What does 'ls -l /dev/infiniband' say?  You should have:

[EMAIL PROTECTED] ~]# ls -l /dev/infiniband/
total 0
crw-rw-rw-1 root root 231, 255 Jul 13 10:44 ucm
crw-rw-rw-1 root root 231, 192 Jul 13 10:45 uverbs0
crw-rw-rw-1 root root 231, 193 Jul 13 10:45 uverbs1

Try,

modprobe ib_uverbs
mknod -m 0666 /dev/infiniband/uverbs0 c 231 192
mknod -m 0666 /dev/infiniband/uverbs1 c 231 193


modprobe ib_ucm
mknod /dev/infiniband/ucm c 231 255

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] RE: IBDM and IBMgtSim Proposal Comments

[Matt Leininger]

On Fri, 2005-07-08 at 00:02 +0300, Eitan Zahavi wrote:
> > On Thu, 2005-07-07 at 16:33, Eitan Zahavi wrote: 
> > There is no reason the MAD and UMAD libraries couldn't be ported to 
> > Windows. 
> [EZ] It is not impossible - just does not make sense to me. If you
> port the applications to OSMV you get the porting to
> Gen1/OpenIB/WinIB/IBMgtSim for free.

  OpenIB is not supporting Gen1.  If Mellanox wants to support Gen1
that's your problem.

  - Matt



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] http://www.openib.org/doc.html update

[Matt Leininger]

On Tue, 2005-06-14 at 11:38 -0700, Tom Duffy wrote:
> Can the folks at Sandia please update the http://www.openib.org/doc.html
> page to point to the wiki site?
> 
> Also, we should move the FAQ's to WIKI.  I can do this if you want.
> 
  I'll add a link to doc.html pointing to the wiki.  Once the FAQ's are
moved to the wiki I can just have doc.html point directly to the wiki.

  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] email archives for March 2004

[Matt Leininger]

On Tue, 2005-06-07 at 11:43 -0400, Sayantan Sur wrote:
> Hi,
> 
> I am trying to access the archives for March 2004 ... more specifically
> this message link:
> 
> www.openib.org/pipermail/openib-general/2004-March/001513.html
> 
> However, the archives on this webpage:
> 
> http://openib.org/pipermail/openib-general/
> 
> date back only to July 2004. Is there any way I can access that message
> link?

  Yes, use one of the archives that have the full openib-general
history.  

 See http://www.openib.org/contact.html



- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Gen2 test suites

[Matt Leininger]

On Tue, 2005-05-17 at 15:26 -0700, Shirley Ma wrote:
> 
> > Community help or a testsuite? ;) 
> Of course testsuites :-).  
> 
> I am setting up two blaze servers for nightly build and regression
> test. There are some tests and utilities in Gen2 stack already. If you
> have any nice opensouce test suites, I am glad to integrate them.  
> 
> Please give me any advice on what kind of tests you are interested in.
> 
  Sandia, LANL, and LLNL are very interested in an automated test suite
for OpenIB.  Sandia and LANL have four IB clusters that could be used
for running the test suite.   These mid-sized cluster testbeds range
between 128-256 nodes [ 138 (x86), 128 (EM64T), 128 (x86), and 256
(Opteron) ].  I have a few PPC970 nodes we could use as well.  An
automated way to do nightly functionality and performance testing of
OpenIB is a great way to provide feedback to developers and other folks
in the OpenIB community.

  What are you using the run the tests?  Your own scripts? Something
else?

  Thanks,


  - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] Broken link in www.openib.org

[Matt Leininger]

I fixed the link that's why it works now.  

Thanks for finding the problem.

- Matt

On Mon, 2005-05-02 at 10:04 -0700, Ashit Shah wrote:
> The link works ok.  May be acrobat reader is not installed on your
> node.
> 
> Ashit Shah
> 
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Roy at
> SEVENtwentyfour
> Sent: Sunday, May 01, 2005 4:44 AM
> To: openib-general@openib.org
> Subject: [openib-general] Broken link in www.openib.org
> 
> There appears to be a problem on this page of your site.
> 
> On page http://www.openib.org/workshop.html
> when you click on "pdf",
> the link to
> http://www.openib.org/docs/oib_wkshp_022005/openmpi-sandia-sukalski.pdf
> gives the error: Not found.
> 
> As recommended by the Robot Guidelines, this email is to explain 
> our robot's visit to your site, and to let you know about one of 
> the problems we found. We don't store or publish the content of 
> your pages, but rather use the link information to update our map 
> of the World Wide Web.
> 
> Are these reports helpful? I'd love some feedback. If you prefer 
> not to receive these occasional error notices please let me know.
> 
> Roy Bryant
> 
>  
>  Roy Bryant, [EMAIL PROTECTED]
>  President
>  SEVENtwentyfour Inc.
>  http://www.seventwentyfour.com
>   
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] rendering openib.org on Firefox/Linux

[Matt Leininger]

On Wed, 2005-04-27 at 10:25 -0700, Roland Dreier wrote:
> Tom> The problem seems to stem from the fact that the horizontal
> Tom> blue bar does not move when the font is increased or
> Tom> decreased.  Here is a series of screenshots to demonstrate
> Tom> the issue:
> 
> Looks like there's some absolute positioning hard-coded in the html:
> 
>   href="index.html"> style="border: 0px solid ; width: 128px; height: 56px;">
> 
> even better is:
> 
> 
> 
  We'll look into it.

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] SM Bad Port Handling

[Matt Leininger]

On Thu, 2005-04-07 at 23:02 +0300, Eitan Zahavi wrote:
> >  
> > At this point, should it attempt to revive the port by bringing the 
> > physical link down and back up ? Should it try this several times
> before 
> > declaring the port as "bad" ? In any case, this is a refinement on
> the 
> > basic strategy for dealing with this scenario. 
> >  
> > Also, there could also be a periodic "ping" at a slower rate to
> check if 
> > the "bad" ports revive. 
> [EZ] This will be released in gen1 within 2 weeks or so. The
> enhancement to light sweep will include the irresponsive ports in the
> light sweep. Once they respond a new heavy sweep will be generated.
> 
 Are you submitting these changes to gen2?  If not, why not?  


   - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] moving gen1 branch to an archive directory

[Matt Leininger]

On Sun, 2005-03-27 at 09:28 +0200, Tziporet Koren wrote:
> Hi Matt,
> 
> From time to time we see people that try to work with gen1 although
> this tree is not really working or supported. 
> Since our focus is gen2 now I suggest to move this directory to some
> archive directory.
> 
  I agree.  How about creating an archive directory in the top level
directory and move gen1 into it?

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] http://openib.org/downloads/

[Matt Leininger]

On Tue, 2005-03-08 at 17:12 -0600, Timur Tabi wrote:
> Why is this directory empty?  I'm trying to download all the openib code 
> (or at least, all the driver code), but I can't find any tarballs.  Can 
> anyone tell me where I can download the OpenIB software?
> 

  You can grab the openib source code from the subversion repository.
See  http://www.openib.org/tools.html.   If you want everything run 'svn
co https://openib.org/svn'  

   Most of the work to date has been for kernel-space IB support (now in
the 2.6.11 kernel).  At some point, in the near future, the user-space
support will be stable/tested enough that we _may_ start posting tar
files, but until then subversion checkout is the best way to get the
source.

- Matt
  

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] 2005 OpenIB Developers Workshop presentations

[Matt Leininger]

On Wed, 2005-02-16 at 09:49 -0500, James Lentini wrote:
> 
> On Mon, 14 Feb 2005, Matt Leininger wrote:
> 
> >  Several developers have volunteered to work on uDAPL.  The DAT
> > collaborative is working to get a GPL/BSD version of uDAPL to OpenIB.
> > James and Arkady from NetApp gave a DAT talk.  They mentioned needing 1
> > month to get the GPL code, and another month or two to fork the code
> > base.  I'm not sure why 2-3 months is needed to do this.  Perhaps James
> > and/or Arkady can comment.
> 
> My 2-3 month estimate was based on 1 month for the DAT Collaborative 
> to discuss and vote on licensing the code under the GPL and 1-2 months 
> to port and test on OpenIB Gen 2.

   ok that sounds better.  We could follow Woody's suggestion of
starting uDAPL development on OpenIB with the BSD code and then dual
license it GPL/BSD after the DAT Collaborative gets the licensing issues
figure out.  
> 
> > I think we need to see the GPL/BSD uDAPL code for OpenIB in the next 
> > couple of weeks.  Better to put the code in the open so everyone can 
> > see what changes are required.  Same goes for kDAPL.  Was the 
> > u/kDAPL implementation for OpenIB going to live at openib.org or 
> > stay under cvs at the sourceforge dapl project?
> 
> We are proposing to maintain the code in both locations. The OpenIB 
> specific implementation would be on openib.org while a 
> platform/transport independent implementation would continue on 
> our SourceForge site.
> 
   That's reasonable.  

   Thanks,

- Matt
   

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop

[Matt Leininger]

On Tue, 2005-02-15 at 17:32 -0800, Ryan, Jim wrote:
> Tom Duffy wrote:
> > On Mon, 2005-02-14 at 16:47 -0800, Woodruff, Robert J wrote:
> >>  Hi Arkady,
> >> 
> >> As I mentioned in the BOF, I have a person (Arlin Davis) that can
> >> help with developing a uDAPL provider for the openib.org verbs.
> >> After discussing it more
> >> with folks here, is seems to us that perhaps for the uDAPL user-mode
> >> library,  it be provided to openib.org under a dual BSD + LGPL
> >> library rather than a BSD + GPL since people normally want to use
> >> LGPL for libraries.
> > 
> > I think using an LGPL instead of GPL would require a change to the
> > openib.org bylaws.
> > 
> > Jim, is that right?
> > 
> > Honestly, if they are licensed BSD too, people can do whatever they
> > want.  BSD/GPL will be fine on these libraries, IMO.
> > 
> > -tduffy
> 
> Tom, good to hear from you. 
> 
> Yes, it would require a change in our bylaws which are very clear wrt
> GPL and BSD.
> 
> Please let me know if you'd like to discuss further. 
> 
  GPL/BSD sounds good to me.

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Mailing

[Matt Leininger]

On Tue, 2005-02-15 at 09:02 -0800, Roland Dreier wrote:
> Eitan> 1. Use more mailing lists (e.g. openib-ib-manegement )
> 
> I've always resisted splitting openib-general.  I still think it's a
> bad idea for several reasons:
> 
>  - openib-general is still a low-traffic list.  linux-kernel works
>fine with 10 times the number of messages per day.
>  - we end up wasting time telling people they need to post to a
>different list, and users end up cross-posting to all the lists
>(and developers end up subscribing to all the lists)
>  - splitting the list dilutes the community.
> 
  I agree.  The classic example is lkml.  They seem to do fine with one
list and their traffic is much heavier than ours.  

  We could put some suggested mail list keywords on the webpage.  But
then we may end up with a mail archive where every SM bug has a subject
line "SM problem".  

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [openib-general] 2005 OpenIB Developers Workshop presentations

[Matt Leininger]

On Sun, 2005-02-13 at 23:45 -0800, Bill Thompson wrote:
> Hope everyone had a nice time in Sonoma.  Sorry I missed it... 
> 
> Got a question:
> 
> Can someone take a SWAG at when an OpenIB stack (OpenSM, OpenMPI,
> uDAPL, iSCSI, iSER, IPoIB, SDP, kDAPL) will be in RHE 4?
> 
> Now that could be CVS or Licensed from one of your companies...

  At the workshop a RedHat spokesperson mentioned that they want an
OpenIB stack the has multi-vendor support so your "licensed from one of
your companies" stack probably won't fly with RedHat.  

  This spokesperson also mentioned that RedHat would have pieces of
OpenIB in RHEL 4.  How much of the OpenIB stack get into RHEL 4 depends
on how far along we are.  I'd guess the initial RH release would have
support for IPoIB, with other ULPs being added as updates as they
stabilize on OpenIB.  It just depends on how long it takes to stabilize
the various components.  

  Open-MPI and uDAPL work can start once we have a user-space access
layer (verbs, cm, etc.).  Roland mentioned that this would be ready for
early testing in a couple of weeks.  Sandia and Los Alamos Labs will
then start adding OpenIB support to Open-MPI.  

  Several developers have volunteered to work on uDAPL.  The DAT
collaborative is working to get a GPL/BSD version of uDAPL to OpenIB.
James and Arkady from NetApp gave a DAT talk.  They mentioned needing 1
month to get the GPL code, and another month or two to fork the code
base.  I'm not sure why 2-3 months is needed to do this.  Perhaps James
and/or Arkady can comment.  I think we need to see the GPL/BSD uDAPL
code for OpenIB in the next couple of weeks.  Better to put the code in
the open so everyone can see what changes are required.  Same goes for
kDAPL.  Was the u/kDAPL implementation for OpenIB going to live at
openib.org or stay under cvs at the sourceforge dapl project?

I think iSCSI support will come from the Linux-iSCSI project on
SourceForge.  Libor has an SDP implementation for OpenIB that iSCSI can
use, but it still needs some tuning and features added.  There is also
the issue of Microsoft IP in SDP.  This was discussed at the workshop -
I thought is was moved to a BOF, but I didn't hear how that turned
out.  

iSER requires kDAPL.  The recent kDAPL discussions on this list and the
linux-iscsi-devel list point to several issues that need to be resolved
if kDAPL is ever going to have a chance of getting in the kernel.  

  Thanks,

- Matt




___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2005 OpenIB Developers Workshop presentations

[Matt Leininger]

On Sun, 2005-02-13 at 17:32 -0800, Tom Duffy wrote:
> On Sat, 2005-02-12 at 11:40 -0800, Matt Leininger wrote: 
> > fixed.
> 
> Thanks.
> 
> Although, the SRP talk link is b0rked.

   I removed the bad link.  I don't have this talk yet.  
> 
> And the IPoIB talk should link to 
> http://www.openib.org/docs/oib_wkshp_022005/ipoib-sdp-topspin-lmichalek.pdf
> 
  fixed

 Thanks,

- Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] 2005 OpenIB Developers Workshop presentations

[Matt Leininger]

On Sat, 2005-02-12 at 06:09 -0800, Tom Duffy wrote:
> On Fri, 2005-02-11 at 23:48 -0800, Matt Leininger wrote:
> >   The presentations given at the 2005 OpenIB Developers Workshop in
> > Sonoma this week are available at www.openib.org/workshop.html.  We
> > still have a few more presentations to track down, but most of them are
> > there.  
> 
> Matt,
> 
> I am getting 404 on most of the pdfs.
> 
fixed.

   - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] 2005 OpenIB Developers Workshop presentations

[Matt Leininger]

  The presentations given at the 2005 OpenIB Developers Workshop in
Sonoma this week are available at www.openib.org/workshop.html.  We
still have a few more presentations to track down, but most of them are
there.  

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop

[Matt Leininger]

On Fri, 2005-02-11 at 00:58 -0700, Eric W. Biederman wrote:
> Matt Leininger <[EMAIL PROTECTED]> writes:
> 
> > On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> > > On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> > > >   uDAPL - Oracle, MPI
> > > >   kDAPL - iSER, NFS over RDMA, Lustre?
> > > 
> > > Lustre will use Sandia Portals AFAIK.
> > > Anyone know what Portals will use?
> > > They might directly program to VAPI or something.
> > > 
> >   There will be a Portals over verbs.  At some point there may be a
> > Portals over kDAPL to support both RDMA ethernet and IB.  
> 
> 
> Cluster filesystems has already implemented native IB support in Lustre
> against gen1. I assume that is a portals.
> 
> You might want to ask them about it some time...
> 
  Already know.  Tri-labs are paying for it.  :)

   - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop

[Matt Leininger]

On Thu, 2005-02-10 at 21:11 +0100, Christoph Hellwig wrote:
> On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> >   kDAPL - iSER, NFS over RDMA, Lustre?
> 
> Okay, we have iSER code and maybe there will be NFS code.  Lustre
> doesn't matter at all for any possible design because it's not freely
> available.

   Good point.  Lustre is open source, but only after the code is old.
I wish they would open up the project and do real open development.  Oh
well.

   - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop

[Matt Leininger]

On Thu, 2005-02-10 at 12:27 -0800, Grant Grundler wrote:
> On Thu, Feb 10, 2005 at 12:05:58PM -0800, Matt Leininger wrote:
> >   uDAPL - Oracle, MPI
> >   kDAPL - iSER, NFS over RDMA, Lustre?
> 
> Lustre will use Sandia Portals AFAIK.
> Anyone know what Portals will use?
> They might directly program to VAPI or something.
> 
  There will be a Portals over verbs.  At some point there may be a
Portals over kDAPL to support both RDMA ethernet and IB.  

   - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: FW: [openib-general] Minutes from DAPL BOF at OpenIB Workshop

[Matt Leininger]

On Thu, 2005-02-10 at 10:51 -0800, Tom Duffy wrote:
> On Thu, 2005-02-10 at 18:46 +0100, Christoph Hellwig wrote:
> > Maybe you should lay down the requirement first.
> 
> I'll take a crack at it.  Let me know where I am off base.
> 
> >  - why do we need an intermediate API
> 
> To get things working today with the code that people have already
> written.  To create a proof of concept.
> 
> In any event, I am not proposing this API get kernel inclusion.  Only a
> wider audience than in the sourceforge DAPL project cvs tree.
> 
> >  - what differences does it
> 
> ??
> 
> >  - what devices does it abstract
> 
> IB for now, any other RDMA capable transports later.
> 
> >  - what are the users
> 
> NFS over RDMA, maybe -- so RPC.  Anybody else know others?
> 
  uDAPL - Oracle, MPI
  kDAPL - iSER, NFS over RDMA, Lustre?

  - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] RFC on SDP checkin

[Matt Leininger]

On Fri, 2005-02-04 at 11:42 -0800, Libor Michalek wrote:
> On Fri, Feb 04, 2005 at 11:33:18AM -0800, Tom Duffy wrote:
> > On Fri, 2005-02-04 at 11:21 -0800, Libor Michalek wrote:
> > >   3) Since the rest of the tree is not destabilized or effected in
> > >  any other way by the code, just check it into tree itself.
> > >  (e.g. gen2/trunk/src/linux-kernel/infiniband/ulp/sdp)
> > > 
> > >   Anyone have thoughts on this? Personally I'm leaning towards #3,
> > > but that's because it's the least amount of work for me. :)
> > 
> > I would like #3 as well.  Libor, would it be possible to get a preview
> > of what is going to be checked in?  Maybe with a small one-pager about
> > how to use it, etc?
> 
>   You mean a quick description of the code, like the primary contents of
> each file and how to get it to do something ?
> 
  How about a description of the code, a todo list, and an sdp_faq we
can put on the website and in svn.  

 Thanks,

   - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] pls correct openib_faq.txt (was Re: patches from subversion)

[Matt Leininger]

On Wed, 2005-01-26 at 17:02 +0200, Michael S. Tsirkin wrote:
> Quoting r. Tom Duffy <[EMAIL PROTECTED]>:
> > > |   (or modify ~/.subversion/config: diff-cmd = ~/bin/mydiff)
> > 
> > This didn't work for me.  I needed to spell out the whole path to
> > "mydiff".  For some reason, I guess "~" was not be expanded properly.
> > 
> > Also, you have to put this line under the [helpers] section, which you
> > may need to create if it is not already in your .subversion/config.
> > 
> 
> 
> It just bit me on a new machine I was configuring.
> Please put the following in FAQ:
> 
> 
> It is preferred that patches are generated with the diff -up option.  
> Here is a sample command line: 
> svn diff --diff-cmd "/usr/bin/diff" -x -up FILENAME
> 
> or using a "mydiff" wrapper:
> 
> svn diff --diff-cmd $HOME/bin/mydiff
> or modify ~/.subversion/config by adding the following lines:
> --
> [helpers]
> diff-cmd = //mydiff
> 
> --
> where  should be replaced by the full path
> to $HOME/bin/mydiff, and where "$HOME/bin/mydiff" contains:
> #!/usr/bin/perl
> exec ("diff", "-up",@ARGV);
> 
> 
> 
> 
> (Please note an empty line is required at the end of the helpers section).
> Thanks,
> MST

  The OpenIB faq has been updated.  Please double-check it.

   Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Re: [KJ] [RFC] TODO file cleanups

[Matt Leininger]

On Thu, 2005-01-20 at 10:21 -0800, Greg KH wrote:
> On Thu, Jan 20, 2005 at 10:02:24AM -0800, Sean Hefty wrote:
> > Greg KH wrote:
> > 
> > >Personally, I think it's a stupid thing to try to license this code in a
> > >dual way, as any port someone is going to have to do to get this code to
> > >work in another os will be almost a complete rewrite in the end
> > >anyway...
> > 
> > I think that companies want to be able to make derivative works without 
> > needing to make the derivative open source, versus porting it to 
> > another OS.
> 
> And then run that derivitave work on a Linux GPL kernel?  Hah, good luck
> with your lawyers if you try to do that.  And good luck trying to work
> around symbols that the openib code is using that are marked
> EXPORT_SYMBOL_GPL().
> 
> Why do people try to do such stupid things, haven't the IB members
> learned from the past...
> 
> Remind me to _never_ send in an openib kernel patch if this is the
> reason why the license is what it is.
> 
  The idea was for folks to be able to take the OpenIB code, under BSD,
and port it to OSes other than Linux.  I agree that having a BSD only
stack running on Linux would be silly and should be avoided.  In the end
I only care about Linux.  I think performance should drive what Linux
features OpenIB uses.  If it improves performance then it's worth
using.  

   - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Power outage - OpenIB server down

[Matt Leininger]

  We had a power outage for our entire building around 5:15pm PST today.
Things are back up and the OpenIB server is up again.

  Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] patches from subversion

[Matt Leininger]

On Wed, 2005-01-19 at 22:32 -0800, Grant Grundler wrote:
> On Wed, Jan 19, 2005 at 07:48:33PM -0800, Matt Leininger wrote:
> >Added to the OpenIB FAQ. Check  www.openib.org/docs/openib_faq.txt to
> > make sure I've captured all the suggestions.
> 
> Thanks!
> 
> | It is preferred that patches are generated with the diff -up option.
> | How are a few ways to do this.
> 
> s/How/Here
> 
> | 
> | 2) svn diff --diff-cmd "/usr/bin/diff" -x -up FILENAME
> | 
> | 3) svn diff --diff-cmd mydiff
> 
> Can you replace it with the following?
> 
> | It is preferred that patches are generated with the diff -up option.
> | Here is a sample command line:
> | svn diff --diff-cmd "/usr/bin/diff" -x -up FILENAME
> |
> | or using a "mydiff" wrapper:
> | svn diff --diff-cmd mydiff
> |   (or modify ~/.subversion/config: diff-cmd = ~/bin/mydiff)
> |
> | where "~/bin/mydiff contains:
> | #!/usr/bin/perl
> | exec ("diff", "-up",@ARGV);
> 
  Done.

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] patches from subversion

[Matt Leininger]

On Wed, 2005-01-19 at 10:15 -0800, Grant Grundler wrote:
> Please add MST's suggestions and mine to the FAQ as part of "how to use SVN"
> and not a requirement to submit patches.
> 
   Added to the OpenIB FAQ. Check  www.openib.org/docs/openib_faq.txt to
make sure I've captured all the suggestions.

   Thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] OpenSM License

[Matt Leininger]

On Wed, 2005-01-19 at 14:53 -0800, Sean Hefty wrote:
> Matt Leininger wrote:
> 
> >   It appears that the OpenSM license is either the GPL license or Intel
> > BSD + patent licence.  I thought we had agreed to change the Intel BSD +
> > patent license to the standard BSD license.  Why hasn't this been
> > changed yet?   
> 
> Can you change the license like that on existing code?
> 
  Intel said they would.  :)

   - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] OpenSM License

[Matt Leininger]

  It appears that the OpenSM license is either the GPL license or Intel
BSD + patent licence.  I thought we had agreed to change the Intel BSD +
patent license to the standard BSD license.  Why hasn't this been
changed yet?   

 Thanks,

   - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] got ipoib up once but not twice :-)

[Matt Leininger]

On Sat, 2005-01-15 at 07:27 -0500, Hal Rosenstock wrote:
> On Fri, 2005-01-14 at 23:10, Ronald G. Minnich wrote:
> > Hmm, it's back. I guess I was not patient enough. Not sure when it all got
> > back. I will have to time it next time, I assume it won't take 6 hours 
> > each time :-)
> > 
> > I'm working on making this 256-node cluster work over infiniband only, 
> > same as our myrinet clusters which are myrinet-only.
> 
> How many 96 port switches ? I'd be curious how long it does take to
> initialize this (as I do not have access to a large cluster). Also,
> right now I'm pretty sure things are being done without pipelining on so
> it is likely slower. More on this later.
> 
 
  Ron has 9 96 port switches, 3 in the spine and 6 leaf switches all
based on the InfiniScale II switch ASIC, to make a 288 port fabric.
It's not quite a true fat-tree network since there are no spine bypass
cards for the older 96 port switch (needed on the leaf switches).  This
is an interesting test case because Ron's network has a total of 612
switch chips.  A 1152 port fat-tree fabric based on InfiniScale III
would have 240 switch chips.  

   - Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] Gentoo has OpenIB code

[Matt Leininger]

Gentoo seems to be the latest distribution that contains the OpenIB
code.  Gentoo is using the 2.6.11-rc1 as the "development-sources"
kernel.  It's great to see the OpenIB code trickling down to the Linux
distros.   Nice work everyone.

   - Matt

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] another shutdown reminder

[Matt Leininger]

The openib server will be down from 3:30pm to 4:30pm PST.  Sorry for the
trouble, but we have to me the machine into our new building.

  Thanks,

 
  - Matt


On Wed, 2005-01-05 at 17:08 -0500, Hal Rosenstock wrote:
> On Wed, 2005-01-05 at 17:06, Michael Paichi Lee wrote:
> > This is another reminder that the openib.org site will be down from
> > 3:30pm to 4:30pm for a system move.  All mail, http, and subversion
> > subversions will be unavailable during this window.
> 
> What TZ ? PST ?
> 
> Thanks.
> 
> -- Hal
> 
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [PATCH] Re: [openib-general] Re: IPoIB Failure CQ overrun

[Matt Leininger]

On Tue, 2005-01-04 at 10:24 -0800, Tom Duffy wrote:
> On Thu, 2004-12-23 at 11:07 +0200, Tziporet Koren wrote:
> > Hi, 
> > We found an issue with the FW that causing this overrun. 
> > The bug happened when incrementing CQ consumer index in more then 1
> > and a CQE is written at the same time on this CQ. 
> > This bug is the same in 3.3.1 and 4.6.1 versions.
> > 
> > A FW with a fix will be provided next week.
> 
> Can these firmware revs be checked in somewhere?  Or put on the OpenIB
> website?
> 
> I don't even have tavor 3.3.1 or arbel 4.6.1.
> 
> Thanks,
> 
> -tduffy
> 
  Can someone from Mellanox comment on this idea?  

   thanks,

- Matt


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Latest IPoIB FAQ

[Matt Leininger]

I'll post the updated IPoIB FAQ on our webpages.

We started looking at Wiki's and then got sidetracked.  Is there a
preferred wiki?

  Thanks,

- Matt



On Mon, 2004-12-06 at 09:13 -0800, Roland Dreier wrote:
> This looks good.
> 
> Matt, it might be worth putting this on the web site, and longer term
> I think this is yet another reason to set up some sort of Wiki.
> 
>  - R.
> ___
> openib-general mailing list
> [EMAIL PROTECTED]
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 

___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] troubles with IPoIB

[Matt Leininger]

On Mon, 2004-11-22 at 23:26 -0500, Hal Rosenstock wrote:
> Hi Josh,
> 
> On Mon, 2004-11-22 at 20:26, Josh England wrote:
> > I've got an 85-node x86_64 PCIe cluster I'd like to run (and test)
> > openIB on.  I've built a kernel using the latest patches from SVN,
> > loaded all the modules, and I see ACTIVE on the ports, but IPoIB does
> > not seem to want to work.
> 
> What is the firmware version of the PCIe adapters ? I have seen problems
> like this when not all the adapters were at 4.5.3.
> 
> You can get this via:
> 
> cat /sys/class/infiniband/mthca0/fw_ver
> 

  We are using fw_ver 4.5.0.  Looks like we need to upgrade.  Time to
try the user space firmware burning tools. 

- Matt


___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] *****SPAM***** [PATCH][RFC/v1][1/12] Add core InfiniBand support

[Matt Leininger]

On Thu, 2004-11-18 at 11:01 -0800, Roland Dreier wrote:
> Hmm... looks like our spamassassin is a little trigger happy :)
> 

  Well since Roland is sending us all spam I can either boot him off the
list or increase the spamassassin threshold.  :)  I decide to increase
the threshold to 7.5 (all the IB patches got a spam score of 6.6) so
future kernel patches shouldn't be listed as spam.  

  - Matt


___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] Signed-off-by: lines

[Matt Leininger]

On Mon, 2004-11-15 at 10:16 -0800, Roland Dreier wrote:
> By the way, for our initial submission upstream, I am planning on
> submitting all the patches with my own
> 
> Signed-off-by: Roland Dreier <[EMAIL PROTECTED]>
> 
> line, of course preserving any other Signed-off-by: lines that already
> exist.  However, for the future, it would be a good idea to make sure
> that all patches come with a properly formatted Signed-off-by: line(s)
> and preserve all such lines in the svn commit messages.
> 
> (Read Documentation/SubmittingPatches in the kernel tree for full details)
> 
   I added the "signed-off by" requirement to the OpenIB FAQ.   We
probably need to have an 'SVN acceptable use policy' that covers the
licensing and "signed-off by" requirements.  I'll put something together
and put it up on openib.org for review.

- Matt


___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] New OpenIB webpages

[Matt Leininger]

On Thu, 2004-11-11 at 10:47 -0800, Grant Grundler wrote:
> On Thu, Nov 11, 2004 at 01:14:37PM -0500, Hal Rosenstock wrote:
> > Not indicating the current version (2.6.9) makes for less frequent web
> > page updates. Is just saying latest 2.6 kernel sufficient ?
> 
> Probably not since SLES9-ia64 is based on 2.6.5 and it won't work as-is.
> Making ithe FAQ a wiki (tduffy) is a good idea.
> 
  FAQ wiki does sound good.  I'll look into it.  

- Matt


___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] New OpenIB webpages

[Matt Leininger]

On Thu, 2004-11-11 at 13:14 -0500, Hal Rosenstock wrote:
> On Thu, 2004-11-11 at 12:52, Roland Dreier wrote:
> > in "What version of the Linux kernel do you support?"
> > 
> > I suggest changing the answer to something like OpenIB
> > supports the latest 2.6 kernel (currently 2.6.9).
> 
> Not indicating the current version (2.6.9) makes for less frequent web
> page updates. Is just saying latest 2.6 kernel sufficient ?
> 
  I don't mind keeping it updated.

- Matt

___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] New OpenIB webpages

[Matt Leininger]

On Thu, 2004-11-11 at 09:52 -0800, Roland Dreier wrote:
> Matt> The FAQ and a few other items are still a work in progress.
> 
> A couple of suggestions for the FAQ:
> 
> in "How do I submit source code patches?"
> 
> I suggest adding something like "Please make sure that patches are
> licensed under the same terms as the original code (dual GPL/BSD
> for most of the OpenIB stack)."
> 
> in "What version of the Linux kernel do you support?"
> 
> I suggest changing the answer to something like OpenIB
> supports the latest 2.6 kernel (currently 2.6.9).
> 
> in "What are all these upper layer protocols like IPoIB, DAPL, MPI, SDP,
> SRP, and others?"
> 
> add a link to the IETF ipoib WG at 
> 
> 
  Done.  Thanks.

- Matt


___
openib-general mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general