Matthew Knepley <knep...@gmail.com> writes: > On Thu, Jul 26, 2018 at 2:43 PM Jed Brown <j...@jedbrown.org> wrote: > >> Matthew Knepley <knep...@gmail.com> writes: >> >> > On Thu, Jul 26, 2018 at 12:56 PM Fande Kong <fdkong...@gmail.com> wrote: >> > >> >> >> >> >> >> On Thu, Jul 26, 2018 at 10:35 AM, Junchao Zhang <jczh...@mcs.anl.gov> >> >> wrote: >> >> >> >>> On Thu, Jul 26, 2018 at 11:15 AM, Fande Kong <fdkong...@gmail.com> >> wrote: >> >>> >> >>>> >> >>>> >> >>>> On Thu, Jul 26, 2018 at 9:51 AM, Junchao Zhang <jczh...@mcs.anl.gov> >> >>>> wrote: >> >>>> >> >>>>> Hi, Pierre, >> >>>>> From your log_view files, I see you did strong scaling. You used 4X >> >>>>> more cores, but the execution time only dropped from 3.9143e+04 >> >>>>> to 1.6910e+04. >> >>>>> From my previous analysis of a GAMG weak scaling test, it looks >> >>>>> communication is one of the reasons that caused poor scaling. In >> your >> >>>>> case, VecScatterEnd time was doubled from 1.5575e+03 to 3.2413e+03. >> Its >> >>>>> time percent jumped from 1% to 17%. This time can contribute to the >> big >> >>>>> time ratio in MatMultAdd ant MatMultTranspose, misleading you guys >> thinking >> >>>>> there was load-imbalance computation-wise. >> >>>>> The reason is that I found in the interpolation and restriction >> >>>>> phases of gamg, the communication pattern is very bad. Few processes >> >>>>> communicate with hundreds of neighbors with message sizes of a few >> bytes. >> >>>>> >> >>>> >> >>>> We may need to truncate interpolation/restriction operators. Also do >> >>>> some aggressive coarsening. Unfortunately, GAMG currently does not >> support. >> >>>> >> >>> >> >>> Are these gamg options the truncation you thought? >> >>> >> >> >> >>> -pc_gamg_threshold[] <thresh,default=0> - Before aggregating the graph >> >>> GAMG will remove small values from the graph on each level >> >>> -pc_gamg_threshold_scale <scale,default=1> - Scaling of threshold on >> each >> >>> coarser grid if not specified >> >>> >> >> >> >> Nope. Totally different things. >> >> >> > >> > Well, you could use _threshold to do more aggressive coarsening, but not >> > for thinning out >> > the interpolation. >> >> Increasing the threshold results in slower coarsening. >> > > Hmm, I think we have to change the webpage then: > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html > > I read it the opposite way.
More coarse points is "better" (stronger), but higher complexity.
