Are there are tuning parameters than I can use to reduce the amount of memory used by OpenMPI? I would very much like to use OpenMPI instead of MVAPICH, but I'm on a cluster where memory usage is the most important consideration. Here are three results which capture the problem:
With the "leave_pinned" behavior turned on, I get good performance (19.528, lower is better) mpirun --prefix /usr/mpi/intel/openmpi-1.2.8 --machinefile /var/spool/torque/aux/7972.fwnaeglingio -np 28 --mca btl ^tcp --mca mpi_leave_pinned 1 --mca mpool_base_use_mem_hooks 1 -x LD_LIBRARY_PATH -x MPI_ENVIRONMENT=1 /tmp/7972.fwnaeglingio/falconv4_ibm_openmpi -cycles 100 -ri restart.0 -ro /tmp/7972.fwnaeglingio/restart.0 Compute rate (processor-microseconds/cell/cycle): 19.528 Total memory usage: 38155.3477 MB (38.1553 GB) Turning off the leave_pinned behavior, I get considerably slower performance (28.788), but the memory usage is unchanged (still 38 GB) mpirun --prefix /usr/mpi/intel/openmpi-1.2.8 --machinefile /var/spool/torque/aux/7972.fwnaeglingio -np 28 -x LD_LIBRARY_PATH -x MPI_ENVIRONMENT=1 /tmp/7972.fwnaeglingio/falconv4_ibm_openmpi -cycles 100 -ri restart.0 -ro /tmp/7972.fwnaeglingio/restart.0 Compute rate (processor-microseconds/cell/cycle): 28.788 Total memory usage: 38335.7656 MB (38.3358 GB) Using MVAPICH, the performance is in the middle (23.6), but the memory usage is reduced by 5 to 6 GB out of 38 GB, a significant decrease to me. /usr/mpi/intel/mvapich-1.1.0/bin/mpirun_rsh -ssh -np 28 -hostfile /var/spool/torque/aux/7972.fwnaeglingio LD_LIBRARY_PATH="/usr/mpi/intel/mvapich-1.1.0/lib/shared:/usr/mpi/intel/openmpi-1.2.8/lib64:/appserv/intel/fce/10.1.008/lib:/appserv/intel/cce/10.1.008/lib" MPI_ENVIRONMENT=1 /tmp/7972.fwnaeglingio/falconv4_ibm_mvapich -cycles 100 -ri restart.0 -ro /tmp/7972.fwnaeglingio/restart.0 Compute rate (processor-microseconds/cell/cycle): 23.608 Total memory usage: 32753.0586 MB (32.7531 GB) I didn't see anything in the FAQ that discusses memory usage other than the impact of the "leave_pinned" option, which apparently does not affect the memory usage in my case. But I figure there must be a justification why OpenMPI would use 6 GB more than MVAPICH on the same case. Thanks for any insights. Also attached is the output of ompi_info -a.
ompi_info.output
Description: ompi_info.output
