On Thu, 23 Jul 2020, Jeff Hammond wrote: > Open-MPI refuses to let users over subscribe without an extra flag to > mpirun.
Yes - and when using this flag - it lets the run through - but there is still performance degradation in oversubscribe mode. > I think Intel MPI has an option for blocking poll that supports > oversubscription “nicely”. What option is this? Is it compile time option or something for mpiexec? Satish > MPICH might have a “no local” option that > disables shared memory, in which case nemesis over libfabric with the > sockets or TCP provider _might_ do the right thing. But you should ask > MPICH people for details. > > Jeff > > On Thu, Jul 23, 2020 at 12:40 PM Jed Brown <j...@jedbrown.org> wrote: > > > I think we should default to ch3:nemesis when --download-mpich, and only > > do ch3:sock when requested (which we would do in CI). > > > > Satish Balay via petsc-dev <petsc-dev@mcs.anl.gov> writes: > > > > > Primarily because ch3:sock performance does not degrade in oversubscribe > > mode - which is developer friendly - i.e on your laptop. > > > > > > And folks doing optimized runs should use a properly tuned MPI for their > > setup anyway. > > > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate > > if using --download-mpich [and not using a separate/optimized MPI] > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > Satish > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis > > had a > > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > >> core speedup. > > >> > > >> Why do we default to sock? > > >> > > >> Thanks, > > >> > > >> Matt > > >> > > >> > > >