Hi Dorian, thank you for your message.
doriankrause wrote:
The trouble is with an MPI code that runs fine with an openmpi 1.1.2 library compiled without infiniband support (I have tested the scalability of the code up to 64 cores, the nodes are 4 or 8 cores, the results are exactly what I expect), but if I try to use a version compiled for infiniband, then only a subset of comunications (the ones connecting cores in the same node) are enabled, and because of this the program fails (gets stuck in a perennial waiting phase, in particular). This happens with any combination of compilers/library releases (1.1.2, 1.2.7, 1.2.8) I have tried. On other codes, and in particular on benchmarks downloaded from the net, openmpi over infiniband seems to work (I compared the latency with the tcp btl, so I am pretty sure that infiniband works). The two variables I kept fixed are SGE and the OFED module stack. I would like not to touch them, if possible, because the cluster seems to run fine for other purposes.
Does the problem only show up with openmpi? Did you tried to use mvapich (http://mvapich.cse.ohio-state.edu/) to test whether it is a hardware or software problem? (I don't know any other open-source MPI implementation which supports infiniband)
I have had bad experiences with mpich, on which mvapich is based. The short answer to your question is yes, and it did not work for other reasons (not even over ethernet). The interesting development today is that Intel MPI (which should be more or less mvapich2 if I am not wrong) seems to work (I will verify this also with mvapich2). This seems to point towards a problem with the OpenMPI libraries, but I have reservations: they seem to work for even complicated benchmarking tests (like the Intel Benchmark) AND I have troubles also with mpich, which I did not sort out. A possibility is that the problem is generated by the interaction MPI-SGE-my code. I would love if someone more experienced than me would give a look at the code (which unfortunately is fortran). I will try to trim down the over 4000 lines to a manageable proof of concept, if anyone is interested in following this up, but it is unlikely to happen before new year :-)
Thanks again, Biagio
