I am using Open MPI 2.1.0 on RHEL 7. My application has one unavoidable pinch point where a large amount of data needs to be transferred (about 8 GB of data needs to be both sent to and received all other ranks), and I'm seeing worse performance than I would expect; this step has a major impact on my overall runtime. In the real application, I am using MPI_Alltoall() for this step, but for the purpose of a simple benchmark, I simplified to simply do a single MPI_Send() / MPI_Recv() between two ranks of a 2 GB buffer.
I'm running this in AWS with instances that have 10 Gbps connectivity in the same availability zone (according to tracepath, there are no hops between them) and MTU set to 8801 bytes. Doing a non-MPI benchmark of sending data directly over TCP between these two instances, I reliably get around 4 Gbps. Between these same two instances with MPI_Send() / MPI_Recv(), I reliably get around 2.4 Gbps. This seems like a major performance degradation for a single MPI operation. I compiled Open MPI 2.1.0 with gcc 4.9.1 and default settings. I'm connecting between instances via ssh and using I assume TCP for the actual network transfer (I'm not setting any special command-line or programmatic settings). The actual command I'm running is: mpirun -N 1 --bind-to none --hostfile hosts.txt my_app Any advice on other things to test or compilation and/or runtime flags to set would be much appreciated! -Adam
_______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
