Hi, I think that I found the explanation/solution. But I would be happy if somebody with experience with OpenMPI and Slurm could comment on this!
On Tuesday, December 4, 2018 at 12:30:15 PM UTC+1, Daniel Garcia-Sanchez wrote: > > But it is striking that If I run 16 MPI jobs of one process per MPI job in > the same machine, then I get an important decrease in performance. Is there > an MPI overhead per MPI job? I thought that a single MPI process takes only > one single core. Am I missing something? > > mpiexec -n 1 ./membrane membrane_step_0.h5 & mpiexec -n 1 ./membrane > membrane_step_1.h5 & mpiexec -n 1 ./membrane membrane_step_2.h5 & mpiexec > -n 1 ./membrane membrane_step_3.h5 & mpiexec -n 1 ./membrane > membrane_step_4.h5 & mpiexec -n 1 ./membrane membrane_step_5.h5 & mpiexec > -n 1 ./membrane membrane_step_6.h5 & mpiexec -n 1 ./membrane > membrane_step_7.h5 & mpiexec -n 1 ./membrane membrane_step_8.h5 & mpiexec > -n 1 ./membrane membrane_step_9.h5 & mpiexec -n 1 ./membrane > membrane_step_10.h5 & mpiexec -n 1 ./membrane membrane_step_11.h5 & mpiexec > -n 1 ./membrane membrane_step_12.h5 & mpiexec -n 1 ./membrane > membrane_step_13.h5 & mpiexec -n 1 ./membrane membrane_step_14.h5 & mpiexec > -n 1 ./membrane membrane_step_15.h5 & > > +---------------------------------------------+------------+------------+ > > | Total wallclock time elapsed since start | 25.8s | | > | | | | > | Section | no. calls | wall time | % of total | > > +---------------------------------+-----------+------------+------------+ > | assembly | 1 | 24.9s | 96% | > | output | 1 | 3.14e-06s | 0% | > > | setup | 1 | 0.343s | 1.3% | > | solve | 1 | 0.539s | 2.1% | > +---------------------------------+-----------+------------+------------+ > > I have a 16 core machine with 32 hyper-threads. When I run 16 MPI jobs with one process per job using mpiexec of OpenMPI, the machine does not use all the cores. I think that that this is related to "MPI process affinity". Now, if I use slurm instead of mpiexec, the process load is well distributed on the machine and I get the expected "total wallclock time". Interestingly slurm seems to balance even better the load than running several non-MPI processes in parallel using bash. srun -n 1 ./membrane membrane_step_0.h5 & srun -n 1 ./membrane membrane_step_1.h5 & srun -n 1 ./membrane membrane_step_2.h5 & srun -n 1 ./membrane membrane_step_3.h5 & srun -n 1 ./membrane membrane_step_4.h5 & srun -n 1 ./membrane membrane_step_5.h5 & srun -n 1 ./membrane membrane_step_6.h5 & srun -n 1 ./membrane membrane_step_7.h5 & srun -n 1 ./membrane membrane_step_8.h5 & srun -n 1 ./membrane membrane_step_9.h5 & srun -n 1 ./membrane membrane_step_10.h5 & srun -n 1 ./membrane membrane_step_11.h5 & srun -n 1 ./membrane membrane_step_12.h5 & srun -n 1 ./membrane membrane_step_13.h5 & srun -n 1 ./membrane membrane_step_14.h5 & srun -n 1 ./membrane membrane_step_15.h5 & +---------------------------------------------+------------+------------+ | Total wallclock time elapsed since start | 12.9s | | | | | | | Section | no. calls | wall time | % of total | +---------------------------------+-----------+------------+------------+ | assembly | 1 | 12.3s | 95% | | output | 1 | 2.17e-06s | 0% | | setup | 1 | 0.256s | 2% | | solve | 1 | 0.314s | 2.4% | +---------------------------------+-----------+------------+------------+ I would be happy to hear your comments about this. Thanks! Daniel -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
