You are referring to
8 KSP Residual norm 8.464122393957e-07 Residual norms for fieldsplit_p_lsc_ solve. 0 KSP Residual norm 1.000000000000e+00 1 KSP Residual norm 1.627170127450e-05 2 KSP Residual norm 1.448537328052e-05 3 KSP Residual norm 1.309526985856e-05 4 KSP Residual norm 1.308667010157e-05 5 KSP Residual norm 1.254021842629e-05 6 KSP Residual norm 9.490928295823e-06 where because the default rtol is 1.e-5 it takes 6 iterations, mostly to get the last little bit? The thing I notice is that "Residual norms for fieldsplit_p_ solve.:" are converging pretty slowly, around 1 digit per iteration. That tells me that the inner inner solve is "oversolving", that is giving more accuracy than is likely needed. Of course, when it takes 1 iteration there is no way for you to back it off to take 1/2 an iteration :-). Some things to consider: 1) do not depend on the rtol at all but just set that inner inner solve to do a single iteration; clearly in this particular case this would work fine. (you could even turn off norm calculations in this inner solve, just use Richardson in this inner inner solve, to save a tiny bit of time), of course, only experimentation would tell if that always works for your family of problems. 2) or just use an rtol of say 1.e-3 in that inner inner solve. Likely that is ok given the relative slow convergence of the fieldsplit_p_ solve. Now if the "Residual norms for fieldsplit_p_ solve.:" was dropping say 4 or more digits each iteration, then I would not generally back off the inner inner solve since likely in that case it would not be oversolving. I don't think the scaling factor as part of the FGMRES algorithm is relevant here. Barry > On Jul 6, 2023, at 3:09 PM, Alexander Lindsay <alexlindsay...@gmail.com> > wrote: > > Hi all, I've found that having fgmres as an outer solve and the initial > residual scaling that comes with it can cause difficulties for inner solves > as I'm approaching convergence, presumably because I'm running out of > precision. This is the kind of thing I would normally set an absolute > tolerance for, but it feels a little silly to set `-ksp_atol 1e-4` when at > the beginning of solves there is no problem reaching that tolerance. Here is > an example solve history. You can see that the fieldsplit_p_lsc_solve > converges in a single iteration until the very end where it's clearly > struggling to hit the rtol of 1e-5. Is the scaling of the fgmres residual > important to the algorithm? git blame showed that the comment regarding > "scale VEC_VV (the initial residual)" was last hit in 1999 (and it existed > before that time), see line 125 of fgmres.c > > 0 KSP Residual norm 2.586864431320e-06 > Residual norms for fieldsplit_p_ solve. > 0 KSP Residual norm 4.692593309369e-02 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 4.048060575312e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 4.951804839158e+01 > 1 KSP Residual norm 9.672178594775e-05 > 1 KSP Residual norm 2.076409133334e-02 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.346618231753e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 8.119436471232e+01 > 1 KSP Residual norm 1.233237431511e-04 > 2 KSP Residual norm 8.753379232559e-03 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.023335686682e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.095058035930e+02 > 1 KSP Residual norm 8.452076098586e-05 > 3 KSP Residual norm 1.589938296474e-03 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 7.680981295089e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 7.768759831332e+01 > 1 KSP Residual norm 5.120395415891e-04 > 4 KSP Residual norm 4.191115394200e-04 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 8.815992679344e-07 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 5.245482238296e+01 > 1 KSP Residual norm 8.672309412797e-05 > 5 KSP Residual norm 1.053200310197e-04 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.450184043299e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 9.058043325461e+01 > 1 KSP Residual norm 1.643116692318e-04 > 6 KSP Residual norm 2.315708746597e-05 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.406236994541e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 5.158957906917e+01 > 1 KSP Residual norm 8.910977751230e-05 > 7 KSP Residual norm 2.378083073391e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 7.981173164359e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 7.271546763656e+01 > 1 KSP Residual norm 2.211194524633e-04 > 8 KSP Residual norm 8.464122393957e-07 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 1.000000000000e+00 > 1 KSP Residual norm 1.627170127450e-05 > 2 KSP Residual norm 1.448537328052e-05 > 3 KSP Residual norm 1.309526985856e-05 > 4 KSP Residual norm 1.308667010157e-05 > 5 KSP Residual norm 1.254021842629e-05 > 6 KSP Residual norm 9.490928295823e-06 > Residual norms for fieldsplit_p_lsc_ solve. > 0 KSP Residual norm 7.084510697783e+01 > 1 KSP Residual norm 1.444782053473e-04 > 9 KSP Residual norm 3.334507715334e-07 > Linear fieldsplit_p_ solve converged due to CONVERGED_RTOL iterations 9