Re: [petsc-dev] Question about MPICH device we use
I have this change at: https://gitlab.com/petsc/petsc/-/merge_requests/2990 Satish On Thu, 23 Jul 2020, Satish Balay via petsc-dev wrote: > The primary reason is for users - developing on laptops/desktop and doing > development runs in oversubscribed mode. > > The choice was few percent loss in performance for sock vs exponential cost > for oversubscribed usage of nemesis [so we defaulted to sock]. > > I think we should preserve this behavior for at-least the debug builds. [i.e > switch only optimized builds to nemesis] > > In CI we do pay this extra cost for some of the builds [that explicitly test > with wemesis, openmpi etc..] > > Satish > > On Thu, 23 Jul 2020, Jed Brown wrote: > > > I think we should default to ch3:nemesis when --download-mpich, and only do > > ch3:sock when requested (which we would do in CI). > > > > Satish Balay via petsc-dev writes: > > > > > Primarily because ch3:sock performance does not degrade in oversubscribe > > > mode - which is developer friendly - i.e on your laptop. > > > > > > And folks doing optimized runs should use a properly tuned MPI for their > > > setup anyway. > > > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if > > > using --download-mpich [and not using a separate/optimized MPI] > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > Satish > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis > > >> had a > > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > >> core speedup. > > >> > > >> Why do we default to sock? > > >> > > >> Thanks, > > >> > > >> Matt > > >> > > >> > > >
Re: [petsc-dev] Question about MPICH device we use
Should also note: the test suite is also run by users - not just CI. Only yesterday I suggested Oana to try nemesis for a different issues [on WSL] - and the response was 'test suite is slow' so reverted back to sock [and tried a different workaround for that issue] Satish On Thu, 23 Jul 2020, Satish Balay via petsc-dev wrote: > The primary reason is for users - developing on laptops/desktop and doing > development runs in oversubscribed mode. > > The choice was few percent loss in performance for sock vs exponential cost > for oversubscribed usage of nemesis [so we defaulted to sock]. > > I think we should preserve this behavior for at-least the debug builds. [i.e > switch only optimized builds to nemesis] > > In CI we do pay this extra cost for some of the builds [that explicitly test > with wemesis, openmpi etc..] > > Satish > > On Thu, 23 Jul 2020, Jed Brown wrote: > > > I think we should default to ch3:nemesis when --download-mpich, and only do > > ch3:sock when requested (which we would do in CI). > > > > Satish Balay via petsc-dev writes: > > > > > Primarily because ch3:sock performance does not degrade in oversubscribe > > > mode - which is developer friendly - i.e on your laptop. > > > > > > And folks doing optimized runs should use a properly tuned MPI for their > > > setup anyway. > > > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if > > > using --download-mpich [and not using a separate/optimized MPI] > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > Satish > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis > > >> had a > > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > >> core speedup. > > >> > > >> Why do we default to sock? > > >> > > >> Thanks, > > >> > > >> Matt > > >> > > >> > > >
Re: [petsc-dev] Question about MPICH device we use
On Thu, Jul 23, 2020 at 11:35 PM Satish Balay via petsc-dev < petsc-dev@mcs.anl.gov> wrote: > On Thu, 23 Jul 2020, Jeff Hammond wrote: > > > Open-MPI refuses to let users over subscribe without an extra flag to > > mpirun. > > Yes - and when using this flag - it lets the run through - but there is > still performance degradation in oversubscribe mode. > > > I think Intel MPI has an option for blocking poll that supports > > oversubscription “nicely”. > > What option is this? Is it compile time option or something for mpiexec? > I only found configure time options, --enable-nemesis-dbg-nolocal, alias for --enable-dbg-nolocal --enable-dbg-nolocalenables debugging mode where shared-memory communication is disabled Satish > > > MPICH might have a “no local” option that > > disables shared memory, in which case nemesis over libfabric with the > > sockets or TCP provider _might_ do the right thing. But you should ask > > MPICH people for details. > > > > Jeff > > > > On Thu, Jul 23, 2020 at 12:40 PM Jed Brown wrote: > > > > > I think we should default to ch3:nemesis when --download-mpich, and > only > > > do ch3:sock when requested (which we would do in CI). > > > > > > Satish Balay via petsc-dev writes: > > > > > > > Primarily because ch3:sock performance does not degrade in > oversubscribe > > > mode - which is developer friendly - i.e on your laptop. > > > > > > > > And folks doing optimized runs should use a properly tuned MPI for > their > > > setup anyway. > > > > > > > > In this case --download-mpich-device=ch3:nemesis is likely > appropriate > > > if using --download-mpich [and not using a separate/optimized MPI] > > > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > > > Satish > > > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on > the > > > >> Firedrake list where it ended up that reconfiguring using > ch3:nemesis > > > had a > > > >> 2x performance boost on his 16-core proc, and noticeable effect on > the 4 > > > >> core speedup. > > > >> > > > >> Why do we default to sock? > > > >> > > > >> Thanks, > > > >> > > > >> Matt > > > >> > > > >> > > > > > >
Re: [petsc-dev] Question about MPICH device we use
On Thu, 23 Jul 2020, Jeff Hammond wrote: > Open-MPI refuses to let users over subscribe without an extra flag to > mpirun. Yes - and when using this flag - it lets the run through - but there is still performance degradation in oversubscribe mode. > I think Intel MPI has an option for blocking poll that supports > oversubscription “nicely”. What option is this? Is it compile time option or something for mpiexec? Satish > MPICH might have a “no local” option that > disables shared memory, in which case nemesis over libfabric with the > sockets or TCP provider _might_ do the right thing. But you should ask > MPICH people for details. > > Jeff > > On Thu, Jul 23, 2020 at 12:40 PM Jed Brown wrote: > > > I think we should default to ch3:nemesis when --download-mpich, and only > > do ch3:sock when requested (which we would do in CI). > > > > Satish Balay via petsc-dev writes: > > > > > Primarily because ch3:sock performance does not degrade in oversubscribe > > mode - which is developer friendly - i.e on your laptop. > > > > > > And folks doing optimized runs should use a properly tuned MPI for their > > setup anyway. > > > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate > > if using --download-mpich [and not using a separate/optimized MPI] > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > Satish > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis > > had a > > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > >> core speedup. > > >> > > >> Why do we default to sock? > > >> > > >> Thanks, > > >> > > >> Matt > > >> > > >> > > >
Re: [petsc-dev] Question about MPICH device we use
The primary reason is for users - developing on laptops/desktop and doing development runs in oversubscribed mode. The choice was few percent loss in performance for sock vs exponential cost for oversubscribed usage of nemesis [so we defaulted to sock]. I think we should preserve this behavior for at-least the debug builds. [i.e switch only optimized builds to nemesis] In CI we do pay this extra cost for some of the builds [that explicitly test with wemesis, openmpi etc..] Satish On Thu, 23 Jul 2020, Jed Brown wrote: > I think we should default to ch3:nemesis when --download-mpich, and only do > ch3:sock when requested (which we would do in CI). > > Satish Balay via petsc-dev writes: > > > Primarily because ch3:sock performance does not degrade in oversubscribe > > mode - which is developer friendly - i.e on your laptop. > > > > And folks doing optimized runs should use a properly tuned MPI for their > > setup anyway. > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if > > using --download-mpich [and not using a separate/optimized MPI] > > > > Having defaults that satisfy all use cases is not practical. > > > > Satish > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis had a > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > >> core speedup. > >> > >> Why do we default to sock? > >> > >> Thanks, > >> > >> Matt > >> > >> >
Re: [petsc-dev] Question about MPICH device we use
Open-MPI refuses to let users over subscribe without an extra flag to mpirun. I think Intel MPI has an option for blocking poll that supports oversubscription “nicely”. MPICH might have a “no local” option that disables shared memory, in which case nemesis over libfabric with the sockets or TCP provider _might_ do the right thing. But you should ask MPICH people for details. Jeff On Thu, Jul 23, 2020 at 12:40 PM Jed Brown wrote: > I think we should default to ch3:nemesis when --download-mpich, and only > do ch3:sock when requested (which we would do in CI). > > Satish Balay via petsc-dev writes: > > > Primarily because ch3:sock performance does not degrade in oversubscribe > mode - which is developer friendly - i.e on your laptop. > > > > And folks doing optimized runs should use a properly tuned MPI for their > setup anyway. > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate > if using --download-mpich [and not using a separate/optimized MPI] > > > > Having defaults that satisfy all use cases is not practical. > > > > Satish > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis > had a > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > >> core speedup. > >> > >> Why do we default to sock? > >> > >> Thanks, > >> > >> Matt > >> > >> > -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/
Re: [petsc-dev] Question about MPICH device we use
I think we should default to ch3:nemesis when --download-mpich, and only do ch3:sock when requested (which we would do in CI). Satish Balay via petsc-dev writes: > Primarily because ch3:sock performance does not degrade in oversubscribe mode > - which is developer friendly - i.e on your laptop. > > And folks doing optimized runs should use a properly tuned MPI for their > setup anyway. > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if > using --download-mpich [and not using a separate/optimized MPI] > > Having defaults that satisfy all use cases is not practical. > > Satish > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the >> Firedrake list where it ended up that reconfiguring using ch3:nemesis had a >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 >> core speedup. >> >> Why do we default to sock? >> >> Thanks, >> >> Matt >> >>
