Oh I see the coarse grid solver in your full solver output now. You still want an accurate coarse grid solve. Usually (the default in GAMG) you use a direct solver on one process, and cousin until the coarse grid is small enough to make that cheap.
On Mon, Sep 29, 2025 at 8:07 AM Moral Sanchez, Elena < [email protected]> wrote: > Hi, I doubled the system size and changed the tolerances just to show a > better example of the problem. This is the output of the callbacks in the > first iteration: > CG Iter 0/1 | res = 2.25e+00/1.00e-09 | 0.1 s > MG lvl 0 (s=884): CG Iter 0/15 | res = 2.25e+00/1.00e-01 | 0.3 s > MG lvl 0 (s=884): CG Iter 1/15 | res = 1.43e+00/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 2/15 | res = 1.17e+00/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 3/15 | res = 1.32e+00/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 4/15 | res = 5.01e-01/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 5/15 | res = 3.57e-01/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 6/15 | res = 2.49e-01/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 7/15 | res = 3.04e-01/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 8/15 | res = 2.78e-01/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 9/15 | res = 1.68e-01/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 10/15 | res = 1.21e-01/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 11/15 | res = 9.45e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 12/15 | res = 8.31e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 13/15 | res = 5.47e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 14/15 | res = 4.36e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 15/15 | res = 5.08e-02/1.00e-01 | 0.1 s > ConvergedReason MG lvl 0: 4 > MG lvl -1 (s=524): CG Iter 0/15 | res = 8.15e-02/1.00e-01 | 3.0 s > ConvergedReason MG lvl -1: 3 > MG lvl 0 (s=884): CG Iter 0/15 | res = 5.08e-02/1.00e-01 | 0.3 s > MG lvl 0 (s=884): CG Iter 1/15 | res = 2.93e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 2/15 | res = 3.26e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 3/15 | res = 4.14e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 4/15 | res = 4.82e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 5/15 | res = 3.20e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 6/15 | res = 3.46e-02/1.00e-01 | 0.3 s > MG lvl 0 (s=884): CG Iter 7/15 | res = 3.41e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 8/15 | res = 4.69e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 9/15 | res = 3.37e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 10/15 | res = 4.07e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 11/15 | res = 2.66e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 12/15 | res = 2.83e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 13/15 | res = 2.98e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 14/15 | res = 3.53e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 15/15 | res = 2.33e-02/1.00e-01 | 0.2 s > ConvergedReason MG lvl 0: 4 > CG Iter 1/1 | res = 2.42e-02/1.00e-09 | 5.6 s > MG lvl 0 (s=884): CG Iter 0/15 | res = 2.42e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 1/15 | res = 1.76e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 2/15 | res = 1.40e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 3/15 | res = 1.42e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 4/15 | res = 1.62e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 5/15 | res = 1.35e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 6/15 | res = 1.39e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 7/15 | res = 1.51e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 8/15 | res = 1.28e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 9/15 | res = 1.24e-02/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 10/15 | res = 9.59e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 11/15 | res = 9.02e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 12/15 | res = 1.19e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 13/15 | res = 1.08e-02/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 14/15 | res = 8.19e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 15/15 | res = 7.61e-03/1.00e-01 | 0.1 s > ConvergedReason MG lvl 0: 4 > MG lvl -1 (s=524): CG Iter 0/15 | res = 1.38e-02/1.00e-01 | 5.2 s > ConvergedReason MG lvl -1: 3 > MG lvl 0 (s=884): CG Iter 0/15 | res = 7.61e-03/1.00e-01 | 0.2 s > MG lvl 0 (s=884): CG Iter 1/15 | res = 5.62e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 2/15 | res = 6.64e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 3/15 | res = 8.71e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 4/15 | res = 6.40e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 5/15 | res = 7.23e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 6/15 | res = 6.20e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 7/15 | res = 7.04e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 8/15 | res = 7.19e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 9/15 | res = 6.35e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 10/15 | res = 7.31e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 11/15 | res = 6.64e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 12/15 | res = 7.24e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 13/15 | res = 4.97e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 14/15 | res = 5.36e-03/1.00e-01 | 0.1 s > MG lvl 0 (s=884): CG Iter 15/15 | res = 5.84e-03/1.00e-01 | 0.1 s > ConvergedReason MG lvl 0: 4 > CG ConvergedReason: -3 > > For completeness, I add here the -ksp_view of the whole solver: > KSP Object: 1 MPI process > type: cg > variant HERMITIAN > maximum iterations=1, nonzero initial guess > tolerances: relative=1e-08, absolute=1e-09, divergence=10000. > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI process > type: mg > type is MULTIPLICATIVE, levels=2 cycles=v > Cycles per PCApply=1 > Not using Galerkin computed coarse grid matrices > Coarse grid solver -- level 0 ------------------------------- > KSP Object: (mg_coarse_) 1 MPI process > type: cg > variant HERMITIAN > maximum iterations=15, nonzero initial guess > tolerances: relative=0.1, absolute=0.1, divergence=1e+30 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: (mg_coarse_) 1 MPI process > type: none > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: python > rows=524, cols=524 > Python: Solver_petsc.LeastSquaresOperator > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (mg_levels_1_) 1 MPI process > type: cg > variant HERMITIAN > maximum iterations=15, nonzero initial guess > tolerances: relative=0.1, absolute=0.1, divergence=1e+30 > left preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: (mg_levels_1_) 1 MPI process > type: none > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: python > rows=884, cols=884 > Python: Solver_petsc.LeastSquaresOperator > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Mat Object: 1 MPI process > type: python > rows=884, cols=884 > Python: Solver_petsc.LeastSquaresOperator > > Regarding Mark's Email: What do you mean with "the whole solver doesn't > have a coarse grid"? I am using my own Restriction and Interpolation > operators. > Thanks for the help, > Elena > > ------------------------------ > *From:* Mark Adams <[email protected]> > *Sent:* 28 September 2025 20:13:54 > *To:* Barry Smith > *Cc:* Moral Sanchez, Elena; petsc-users > *Subject:* Re: [petsc-users] setting correct tolerances for MG smoother > CG at the finest level > > Not sure why your "whole"solver does not have a coarse grid but this is > wrong: > > KSP Object: (mg_coarse_) 1 MPI process > type: cg > variant HERMITIAN > maximum iterations=100, initial guess is zero > tolerances: relative=0.1, absolute=0.1, divergence=1e+30 > > The coarse grid has to be accurate. The defaults are a good place to > start: max_it=10.000, rtol=1e-5, atol=1e-30 (ish) > > > On Fri, Sep 26, 2025 at 3:21 PM Barry Smith <[email protected]> wrote: > >> Looks reasonable. Send the output running with >> >> -ksp_monitor -mg_levels_ksp_monitor -ksp_converged_reason >> -mg_levels_ksp_converged_reason >> >> On Sep 26, 2025, at 1:19 PM, Moral Sanchez, Elena < >> [email protected]> wrote: >> >> Dear Barry, >> >> This is -ksp_view for the smoother at the finest level: >> >> KSP Object: (mg_levels_1_) 1 MPI process >> type: cg >> variant HERMITIAN >> maximum iterations=10, nonzero initial guess >> tolerances: relative=0.1, absolute=0.1, divergence=1e+30 >> left preconditioning >> using UNPRECONDITIONED norm type for convergence test >> PC Object: (mg_levels_1_) 1 MPI process >> type: none >> linear system matrix = precond matrix: >> Mat Object: 1 MPI process >> type: python >> rows=524, cols=524 >> Python: Solver_petsc.LeastSquaresOperator >> >> And at the coarsest level: >> >> KSP Object: (mg_coarse_) 1 MPI process >> type: cg >> variant HERMITIAN >> maximum iterations=100, initial guess is zero >> tolerances: relative=0.1, absolute=0.1, divergence=1e+30 >> left preconditioning >> using UNPRECONDITIONED norm type for convergence test >> PC Object: (mg_coarse_) 1 MPI process >> type: none >> linear system matrix = precond matrix: >> Mat Object: 1 MPI process >> type: python >> rows=344, cols=344 >> Python: Solver_petsc.LeastSquaresOperator >> >> And for the whole solver: >> >> KSP Object: 1 MPI process >> type: cg >> variant HERMITIAN >> maximum iterations=100, nonzero initial guess >> tolerances: relative=1e-08, absolute=1e-09, divergence=10000. >> left preconditioning >> using UNPRECONDITIONED norm type for convergence test >> PC Object: 1 MPI process >> type: mg >> type is MULTIPLICATIVE, levels=2 cycles=v >> Cycles per PCApply=1 >> Not using Galerkin computed coarse grid matrices >> Coarse grid solver -- level 0 ------------------------------- >> KSP Object: (mg_coarse_) 1 MPI process >> type: cg >> variant HERMITIAN >> maximum iterations=100, initial guess is zero >> tolerances: relative=0.1, absolute=0.1, divergence=1e+30 >> left preconditioning >> using UNPRECONDITIONED norm type for convergence test >> PC Object: (mg_coarse_) 1 MPI process >> type: none >> linear system matrix = precond matrix: >> Mat Object: 1 MPI process >> type: python >> rows=344, cols=344 >> Python: Solver_petsc.LeastSquaresOperator >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object: (mg_levels_1_) 1 MPI process >> type: cg >> variant HERMITIAN >> maximum iterations=10, nonzero initial guess >> tolerances: relative=0.1, absolute=0.1, divergence=1e+30 >> left preconditioning >> using UNPRECONDITIONED norm type for convergence test >> PC Object: (mg_levels_1_) 1 MPI process >> type: none >> linear system matrix = precond matrix: >> Mat Object: 1 MPI process >> type: python >> rows=524, cols=524 >> Python: Solver_petsc.LeastSquaresOperator >> Up solver (post-smoother) same as down solver (pre-smoother) >> linear system matrix = precond matrix: >> Mat Object: 1 MPI process >> type: python >> rows=524, cols=524 >> Python: Solver_petsc.LeastSquaresOperator >> >> Best, >> Elena >> >> ------------------------------ >> *From:* Barry Smith <[email protected]> >> *Sent:* 26 September 2025 19:05:02 >> *To:* Moral Sanchez, Elena >> *Cc:* [email protected] >> *Subject:* Re: [petsc-users] setting correct tolerances for MG smoother >> CG at the finest level >> >> >> Send the output using -ksp_view >> >> Normally one uses a fixed number of iterations of smoothing on level >> with multigrid rather than a tolerance, but yes PETSc should respect such a >> tolerance. >> >> Barry >> >> >> On Sep 26, 2025, at 12:49 PM, Moral Sanchez, Elena < >> [email protected]> wrote: >> >> Hi, >> I am using multigrid (multiplicative) as a preconditioner with a V-cycle >> of two levels. At each level, I am setting CG as the smoother with certain >> tolerance. >> >> What I observe is that in the finest level the CG continues iterating >> after the residual norm reaches the tolerance (atol) and it only stops when >> reaching the maximum number of iterations at that level. At the coarsest >> level this does not occur and the CG stops when the tolerance is reached. >> >> I double-checked that the smoother at the finest level has the right >> tolerance. And I am using a Monitor function to track the residual. >> >> Do you know how to make the smoother at the finest level stop when >> reaching the tolerance? >> >> Cheers, >> Elena. >> >> >>
