Unfortunately verbose error reports are not so friendly...anyway , I may think about 2 issues:
1. You trying to open open too much QPs. By default ib devices support fairly large amount of QPs and it is quite hard to push it to this corner. But If your job is really huge it may be the case. Or for example, if you share the compute nodes with some other processes that create a lot of qps. The maximum amount of supported qps you may see in ibv_devinfo. 2. The memory limit for registered memory is too low, as result driver fails allocate and register memory for QP. This scenario is most common. Just happened to me recently, system folks pushed some crap into limits.conf. Regards, Pavel (Pasha) Shamis --- Application Performance Tools Group Computer Science and Math Division Oak Ridge National Laboratory On Jan 27, 2011, at 5:56 PM, Barrett, Brian W wrote: > All - > > On one of our clusters, we're seeing the following on one of our > applications, I believe using Open MPI 1.4.3: > > [xxx:27545] *** An error occurred in MPI_Scatterv > [xxx:27545] *** on communicator MPI COMMUNICATOR 5 DUP FROM 4 > [xxx:27545] *** MPI_ERR_OTHER: known error not in list > [xxx:27545] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) > [xxx][[31806,1],0][connect/btl_openib_connect_oob.c:857:qp_create_one] error > creating qp errno says Resource temporarily unavailable > -------------------------------------------------------------------------- > mpirun has exited due to process rank 0 with PID 27545 on > node rs1891 exiting without calling "finalize". This may > have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > -------------------------------------------------------------------------- > > > The problem goes away if we modify the eager protocol msg sizes so that there > are only two QPs necessary instead of the default 4. Is there a way to bump > up the number of QPs that can be created on a node, assuming the issue is > just running out of available QPs? If not, any other thoughts on working > around the problem? > > Thanks, > > Brian > > -- > Brian W. Barrett > Dept. 1423: Scalable System Software > Sandia National Laboratories > > > > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
