If I want to know whether only rank 0 is slow (since it may has more io) or actually a portion of cores are slow, what tools can I start with?
Thanks. Xiangdong On Fri, Feb 5, 2016 at 5:27 PM, Jed Brown <j...@jedbrown.org> wrote: > Matthew Knepley <knep...@gmail.com> writes: > >> I attached the full summary. At the end, it has > >> > >> Average time to get PetscTime(): 0 > >> Average time for MPI_Barrier(): 8.3971e-05 > >> Average time for zero size MPI_Send(): 7.16746e-06 > >> > >> Is it an indication of slow network? > >> > > > > I think so. It takes nearly 100 microseconds to synchronize processes. > > Edison with 65536 processes: > Average time for MPI_Barrier(): 4.23908e-05 > Average time for zero size MPI_Send(): 2.46466e-06 > > Mira with 16384 processes: > Average time for MPI_Barrier(): 5.7075e-06 > Average time for zero size MPI_Send(): 1.33179e-05 > > Titan with 131072 processes: > Average time for MPI_Barrier(): 0.000368595 > Average time for zero size MPI_Send(): 1.71567e-05 >
