Before you make any more timing runs you should register three stages with
PETSC_EXTERN PetscErrorCode PetscLogStageRegister(const char[],PetscLogStage*);
and then put
PETSC_EXTERN PetscErrorCode PetscLogStagePush(PetscLogStage);
PETSC_EXTERN PetscErrorCode PetscLogStagePop(void);
around the advection solve. Put the second set around the call to KSPSetUp() on
the projection step and the
third set around the projection solves. This will cause the -log_view to
display separately the information about each type of solve and the setup so
it is a bit easier to understand.
To make test runs with GAMG as Mark suggestions you must use the master
branch of the PETSc repository. We have been aggressively optimizing parts of
GAMG recently. The set up time of AMG is largely determined by the performance
of the sparse matrix triple product P^T A P, PETSc currently has four versions
of this product which make tradeoffs in memory and time through different
algorithmic approaches. They can be accessed with
-matptap_via scalable or nonscalable or allatonce or allatonce_merged or hypre
the final one uses the matrix triple product of hypre (but not the rest of
hypre BoomerAMG.) Since your project problem remains the same for all
time-steps (I assume) you can add the option
-mat_freeintermediatedatastructures to save memory usage.
We'd be very interested in hearing about your performance results
Barry
> On May 6, 2019, at 6:41 PM, Raphael Egan via petsc-users
> <petsc-users@mcs.anl.gov> wrote:
>
> Dear Petsc developer(s),
>
> I am assessing the strong scalability of our incompressible Navier-Stokes
> solver on distributed Octree grids developed at the University of California,
> Santa Barbara.
>
> I intend to show satisfactory strong scaling behavior, even on very large
> numbers of cores for computational problems that involve a mesh that is big
> enough. My scaling test case involves 10 time steps of our solver, applied to
> the three-dimensional flow past a sphere on an extremely fine (but
> non-uniform) mesh made of about 270,000,000 computational cells. We use the
> p4est library for grid management and Petsc solvers for all linear algebra
> problems. I am working on Stampede 2 supercomputer's KNL nodes, using 64
> cores per node.
>
> The most elementary usage of the solver can be summarized as follows:
> - "viscosity step": velocity components are decoupled from each other and
> solved for successively, leading to three successive Poisson problems with an
> added positive diagonal term;
> - "projection step": a projection step is solved to enforce the velocity
> field to be divergence-free.
> The solution of the "projection step" updates the appropriate boundary
> conditions for the "viscosity step" and the process may thus be repeated
> until a desired convergence threshold is reached for the current time step.
>
> Given the discretization that we use for the "viscosity step", the three
> corresponding linear systems are slightly nonsymmetric but have a dominant
> diagonal. We use BiCGStab with PCSOR as a preconditioner invariably for those
> problems (only a few iterations are required for those problems in this
> case). On the contrary, the linear system to be solved for the "projection
> step" is symmetric and positive definite, so we use a conjugate gradient
> solver. PCSOR or PCHYPRE can be chosen by the user as a preconditioner for
> that task. If PCHYPRE is chosen, "-pc_hypre_boomeramg_strong_threshold" is
> set to "0.5", "-pc_hypre_boomeramg_coarsen_type" is set to "Falgout" and
> "-pc_hypre_boomeramg_truncfactor" is set to "0.1". Tests on my local machine
> (on a much smaller problem) revealed that PCHYPRE outperforms PCSOR (both in
> terms of execution time and number of iterations) and I expected the
> conclusion to hold for (much) bigger problems on Stampede 2, as well.
>
> However that was wrong: PCSOR actually outperforms PCHYPRE quite
> significantly when using large numbers of cores for my scaling test problem.
> Worse than that, strong scaling of the projection step is actually very poor
> when using PCHYPRE as preconditioner: the time spent on that part of the
> problem grows with the number of cores. See the results here below:
>
> NB: the scaling test that is considered here below consists of 2 time steps
> of the solver starting from an initial state read from disk, of around
> 270,000,000 computational cells. Each time step involves two sub-iterations,
> i.e., every time step solves a first viscosity step (3 calls to KSPSolve of
> type bcgs with zero initial guesses), then a first projection step (1 call to
> KSPSolve of type cg with zero initial guess) then a second viscosity step (3
> more calls to KSPSolve of type bcgs with nonzero initial guess), and a second
> projection step (1 call to KSPSolve of type cg with nonzero initial guess).
> This makes a total of 16 calls to KSPSolves, 12 of them are for the viscosity
> step (not under investigation here) and 4 others are for the projection steps
>
> Mean execution time for the projection step (only):
> Number of cores: 4096 --> mean execution time for projection step using
> PCSOR: 125.7 s |||||| mean exectution time for projection step using
> PCHYPRE: 192.7 s
> Number of cores: 8192 --> mean execution time for projection step using
> PCSOR: 81.58 s |||||| mean exectution time for projection step using
> PCHYPRE: 281.8 s
> Number of cores: 16384 --> mean execution time for projection step using
> PCSOR: 48.2 s |||||| mean exectution time for projection step using
> PCHYPRE: 311.5 s
>
> The tests were all run with "-ksp_monitor", "-ksp_view" and "-log_view" for
> gathering more information about the processes (see files attached). While it
> is clear that the number of iterations is dramatically reduced when using
> PCHYPRE, it looks like a very long time is spent in PCSetUp and PCApply when
> using PCHYPRE for the projection step. When using PCHYPRE, the time spent in
> PCSetUp, PCApply and KSPSolve grows with the number of cores that are used.
> In particular, the time spent in PCSetUp alone when using PCHYPRE is larger
> than the total KSPSolve time when using PCSOR (although thousands of
> iterations are required in that case)...
>
> I am very surprised by these results and I don't quite understand them, I
> used PCHYPRE with similar options and option values in other elliptic solver
> contexts previously with very satisfactory results, even on large problems. I
> can't figure out why we loose strong scaling in this case?
>
> Would you have some advice about a better preconditioner to use and or a
> better set of options and/or option values to set in order to recover strong
> scaling when using PCHYPRE?
>
> --
> Raphael M. Egan
>
> Department of Mechanical Engineering
> Engineering II Building, Office 2317
> University of California, Santa Barbara
> Santa Barbara, CA 93106
> Email: raphaele...@ucsb.edu
> <petsc_test_flow_past_sphere_Re_500_scaling_test_16384_cg_hypre><petsc_test_flow_past_sphere_Re_500_scaling_test_8192_cg_hypre><petsc_test_flow_past_sphere_Re_500_scaling_test_4096_cg_hypre><petsc_test_flow_past_sphere_Re_500_scaling_test_4096_cg_sor><petsc_test_flow_past_sphere_Re_500_scaling_test_8192_cg_sor><petsc_test_flow_past_sphere_Re_500_scaling_test_16384_cg_sor>
