Progress still continues on this issue... (Nathan and I are actually sitting together in a room this week and are continuing to work on this)
We just put up a new FAQ item about this issue: http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem On Jul 13, 2012, at 7:02 PM, Jeff Squyres wrote: > On Jul 12, 2012, at 12:04 PM, Paul Kapinos wrote: > >> a long time ago, I reported about an error in Open MPI: >> http://www.open-mpi.org/community/lists/users/2012/02/18565.php >> >> Well, in the 1.6 the behaviour has changed: the test case don't hang forever >> and block an InfiniBand interface, but seem to run through, and now this >> error message is printed: >> -------------------------------------------------------------------------- >> The OpenFabrics (openib) BTL failed to register memory in the driver. >> Please check /var/log/messages or dmesg for driver specific failure >> reason. > > We updated our mechanism, but accidentally left this warning message in (it > has since been removed). > > Here's what's happening: Mellanox changed the default amount of registered > memory that is available -- they dramatically reduced it. We haven't gotten > a good answer yet as to *why* this change was made. > > You can change some kernel-level parameters to increase it again, and then > OMPI should work fine. Here's an IBM article about it: > > http://www.ibm.com/developerworks/wikis/display/hpccentral/Using+RDMA+with+pagepool+larger+than+8GB > > And here's some comments that Mellanox made on a ticket about this issue > (including some corrections/clarifications to that IBM article): > > https://svn.open-mpi.org/trac/ompi/ticket/3134#comment:12 > > ----- > > Basically, what's happening is that OMPI is behaving badly when it runs out > of registered memory. We have tried two things to make this better (i.e., > still perform *correctly*, albeit at a lower performance level), and we're > not sure yet whether they work properly. > > 1. When OMPI tries to register more memory for an RDMA message transaction > and fails, it falls back to send-receive (where we already have > pre-registered memory available to use). However, this can still end up > hanging because of OMPI's "lazy connection" scheme -- where OMPI doesn't open > IB connections between MPI processes until the first time each pair of > processes communicate. So if OMPI runs out of registered memory and then > tries to open a new IB connection to a new peer -- kaboom. > > 2. When OMPI starts it, it guesstimates how much memory can be registered and > equally divides it between all the OMPI processes *in that job* on the same > node. We had mixed reports of this working or not. I made a 1.6.x tarball > with this fix in it, if you could give it a whirl (with the default low > registered memory kernel parameters, to ensure that you can invoke the "out > of registered memory" issue): > > http://www.open-mpi.org/~jsquyres/unofficial/ > Use the openmpi-1.6.1ticket3131r26612M.tar.bz2 tarball > > #2 is the latest attempt to fix it, but we haven't had good testing of it. > Could you give it a whirl and let us know what happens? > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/