On Mon, May 8, 2023 at 7:27 AM Mark Adams <mfad...@lbl.gov> wrote: > > > On Mon, May 8, 2023 at 2:32 AM Sebastian Blauth < > sebastian.bla...@itwm.fraunhofer.de> wrote: > >> Hello everyone, >> >> I wanted to briefly follow up on my question (see my last reply). >> Does anyone know / have an idea why the LSC preconditioner in PETSc does >> not seem to scale well with the problem size (the outer fgmres solver I >> am using nearly scale nearly linearly with the problem size in my >> example). >> I have also already tried using -ksp_diagonal_scale but the results are >> identical. >> Any help is really appreciated. >> > > I would start by finding results in the literature that you like, in that > they are on a problem similar to yours and you like the performance, and > reproduce them in your code. > If you can do that then you have a research problem to see how to get your > problem to work well. > If your solver does not reproduce published results then we might be > able to provide some advice. >
I have not seen LSC scale well. The only thing I have seen to actually work is patch smoothing (I gave a paper reference). It could be that we have a bug in LSC, but I thought we verified it with the Shuttleworth paper. THanks, Matt > Mark > > >> >> Thanks a lot, >> Sebastian >> >> On 03.05.2023 09:07, Sebastian Blauth wrote: >> > First of all, yes you are correct that I am trying to solve the >> > stationary incompressible Navier Stokes equations. >> > >> > On 02.05.2023 21:33, Matthew Knepley wrote: >> >> On Tue, May 2, 2023 at 2:29 PM Jed Brown <j...@jedbrown.org >> >> <mailto:j...@jedbrown.org>> wrote: >> >> >> >> Sebastian Blauth <sebastian.bla...@itwm.fraunhofer.de >> >> <mailto:sebastian.bla...@itwm.fraunhofer.de>> writes: >> >> >> >> > I agree with your comment for the Stokes equations - for these, >> I >> >> have >> >> > already tried and used the pressure mass matrix as part of a >> >> (additive) >> >> > block preconditioner and it gave mesh independent results. >> >> > >> >> > However, for the Navier Stokes equations, is the Schur >> complement >> >> really >> >> > spectrally equivalent to the pressure mass matrix? >> >> >> >> No, it's not. You'd want something like PCD (better, but not >> >> algebraic) or LSC. >> >> >> > >> > I would like to take a look at the LSC preconditioner. For this, I did >> > also not achieve mesh-independent results. I am using the following >> > options (I know that the tolerances are too high at the moment, but it >> > should just illustrate the behavior w.r.t. mesh refinement). Again, I >> am >> > using a simple 2D channel problem for testing purposes. >> > >> > I am using the following options >> > >> > -ksp_type fgmres >> > -ksp_gmres_restart 100 >> > -ksp_gmres_cgs_refinement_type refine_ifneeded >> > -ksp_max_it 1000 >> > -ksp_rtol 1e-10 >> > -ksp_atol 1e-30 >> > -pc_type fieldsplit >> > -pc_fieldsplit_type schur >> > -pc_fieldsplit_schur_fact_type full >> > -pc_fieldsplit_schur_precondition self >> > -fieldsplit_0_ksp_type preonly >> > -fieldsplit_0_pc_type lu >> > -fieldsplit_1_ksp_type gmres >> > -fieldsplit_1_ksp_pc_side right >> > -fieldsplit_1_ksp_gmres_restart 100 >> > -fieldsplit_1_ksp_gmres_cgs_refinement_type refine_ifneeded >> > -fieldsplit_1_ksp_max_it 1000 >> > -fieldsplit_1_ksp_rtol 1e-10 >> > -fieldsplit_1_ksp_atol 1e-30 >> > -fieldsplit_1_pc_type lsc >> > -fieldsplit_1_lsc_ksp_type preonly >> > -fieldsplit_1_lsc_pc_type lu >> > -fieldsplit_1_ksp_converged_reason >> > >> > Again, the direct solvers are used so that only the influence of the >> LSC >> > preconditioner is seen. I have suitable preconditioners for all of >> these >> > available (using boomeramg). >> > >> > At the bottom, I attach the output for different discretizations. As >> you >> > can see there, the number of iterations increases nearly linearly with >> > the problem size. >> > >> > I think that the problem could occur due to a wrong scaling. In your >> > docs https://petsc.org/release/manualpages/PC/PCLSC/ , you write that >> > the LSC preconditioner is implemented as >> > >> > inv(S) \approx inv(A10 A01) A10 A00 A01 inv(A10 A01) >> > >> > However, in the book of Elman, Sylvester and Wathen (Finite Elements >> and >> > Fast Iterative Solvers), the LSC preconditioner is defined as >> > >> > inv(S) \approx inv(A10 inv(T) A01) A10 inv(T) A00 inv(T) A01 >> > inv(A10 inv(T) A01) >> > >> > where T = diag(Q) and Q is the velocity mass matrix. >> > >> > There is an options -pc_lsc_scale_diag, which states that it uses the >> > diagonal of A for scaling. I suppose, that this means, that the >> diagonal >> > of the A00 block is used for scaling - however, this is not the right >> > scaling, is it? Even in the source code for the LSC preconditioner, in >> > /src/ksp/pc/impls/lsc/lsc.c it is mentioned, that a mass matrix should >> > be used... >> > Is there any way to implement this in PETSc? Maybe by supplying the >> mass >> > matrix as Pmat? >> > >> > Thanks a lot in advance, >> > Sebastian >> > >> >> >> >> I think you can do a better job than that using something like >> >> >> >> https://arxiv.org/abs/1810.03315 <https://arxiv.org/abs/1810.03315> >> >> >> >> Basically, you use an augmented Lagrangian thing to make the Schur >> >> complement well-conditioned, >> >> and then use a special smoother to handle that perturbation. >> >> >> >> > And even if it is, the convergence is only good for small >> >> Reynolds numbers, for moderately high ones the convergence really >> >> deteriorates. This is why I am trying to make >> >> fieldsplit_schur_precondition selfp work better (this is, if I >> >> understand it correctly, a SIMPLE type preconditioner). >> >> >> >> SIMPLE is for short time steps (not too far from resolving CFL) and >> >> bad for steady. This taxonomy is useful, though the problems are >> >> super academic and they don't use high aspect ratio. >> >> >> > >> > Okay, I get that I cannot expect the SIMPLE preconditioning >> > (schur_precondition selfp) to work efficiently. I guess the reason it >> > works for small time steps (for the instationary equations) is that the >> > velocity block becomes diagonally dominant in this case, so that >> diag(A) >> > is a good approximation of A. >> > >> > >> >> https://doi.org/10.1016/j.jcp.2007.09.026 >> >> <https://doi.org/10.1016/j.jcp.2007.09.026> >> >> >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> -- >> >> What most experimenters take for granted before they begin their >> >> experiments is infinitely more interesting than any results to which >> >> their experiments lead. >> >> -- Norbert Wiener >> >> >> >> https://www.cse.buffalo.edu/~knepley/ >> >> <http://www.cse.buffalo.edu/~knepley/> >> > >> > >> > And here is the output of my scaling tests >> > >> > 8x8 discretization >> > >> > Newton solver: iter, abs. residual (abs. tol), rel. residual (rel. >> tol) >> > >> > Newton solver: 0, 1.023e+03 (1.00e-30), 1.000e+00 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations 38 >> > Newton solver: 1, 1.313e+03 (1.00e-30), 1.283e+00 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations 76 >> > Newton solver: 2, 1.198e+02 (1.00e-30), 1.171e-01 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations 74 >> > Newton solver: 3, 7.249e-01 (1.00e-30), 7.084e-04 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations 74 >> > Newton solver: 4, 3.883e-05 (1.00e-30), 3.795e-08 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations 74 >> > Newton solver: 5, 2.778e-12 (1.00e-30), 2.714e-15 >> (1.00e-10) >> > >> > >> > >> > 16x16 discretization >> > >> > Newton solver: iter, abs. residual (abs. tol), rel. residual (rel. >> tol) >> > >> > Newton solver: 0, 1.113e+03 (1.00e-30), 1.000e+00 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations 62 >> > Newton solver: 1, 8.316e+02 (1.00e-30), 7.475e-01 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 141 >> > Newton solver: 2, 5.806e+01 (1.00e-30), 5.218e-02 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 119 >> > Newton solver: 3, 3.309e-01 (1.00e-30), 2.974e-04 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 118 >> > Newton solver: 4, 9.085e-06 (1.00e-30), 8.166e-09 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 120 >> > Newton solver: 5, 3.475e-12 (1.00e-30), 3.124e-15 >> (1.00e-10) >> > >> > >> > >> > 32x32 discretization >> > >> > Newton solver: iter, abs. residual (abs. tol), rel. residual (rel. >> tol) >> > >> > Newton solver: 0, 1.330e+03 (1.00e-30), 1.000e+00 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations 98 >> > Newton solver: 1, 5.913e+02 (1.00e-30), 4.445e-01 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 183 >> > Newton solver: 2, 3.214e+01 (1.00e-30), 2.416e-02 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 152 >> > Newton solver: 3, 2.059e-01 (1.00e-30), 1.547e-04 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 151 >> > Newton solver: 4, 6.949e-06 (1.00e-30), 5.223e-09 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 149 >> > Newton solver: 5, 5.300e-12 (1.00e-30), 3.983e-15 >> (1.00e-10) >> > >> > >> > >> > 64x64 discretization >> > >> > Newton solver: iter, abs. residual (abs. tol), rel. residual (rel. >> tol) >> > >> > Newton solver: 0, 1.707e+03 (1.00e-30), 1.000e+00 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 198 >> > Newton solver: 1, 4.259e+02 (1.00e-30), 2.494e-01 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 357 >> > Newton solver: 2, 1.706e+01 (1.00e-30), 9.993e-03 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 266 >> > Newton solver: 3, 1.134e-01 (1.00e-30), 6.639e-05 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 261 >> > Newton solver: 4, 4.285e-06 (1.00e-30), 2.510e-09 >> (1.00e-10) >> > Linear fieldsplit_1_ solve converged due to CONVERGED_RTOL >> iterations >> > 263 >> > Newton solver: 5, 9.650e-12 (1.00e-30), 5.652e-15 >> (1.00e-10) >> > >> >> -- >> Dr. Sebastian Blauth >> Fraunhofer-Institut für >> Techno- und Wirtschaftsmathematik ITWM >> Abteilung Transportvorgänge >> Fraunhofer-Platz 1, 67663 Kaiserslautern >> Telefon: +49 631 31600-4968 >> sebastian.bla...@itwm.fraunhofer.de >> www.itwm.fraunhofer.de >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>