Colin, 1) What equations are you solving?
2) In your second case, you set the outer ksp to preonly, thus we are unable to see the ksp_monitor for the (firedrake_0_) solver. Set it to gmres and see if you have a similar output to your first case: 0 KSP preconditioned resid norm 4.985448866758e+00 true resid norm 1.086016610848e-03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.245615753306e-13 true resid norm 2.082000915439e-14 ||r(i)||/||b|| 1.917098591903e-11 Because according to the first ksp_view output, after one lu sweep for the (firedrake_0_fieldsplit_1_) solver. That is, going from: 0 KSP preconditioned resid norm 8.819238435108e-02 true resid norm 1.797571993221e-01 ||r(i)||/||b|| 1.000000000000e+00 to 1 KSP preconditioned resid norm 1.025167319984e-02 true resid norm 3.340583874349e-02 ||r(i)||/||b|| 1.858386694356e-01 appeared to give you an exact schur complement. Justin On Mon, Mar 18, 2019 at 2:18 PM Cotter, Colin J via petsc-users < petsc-users@mcs.anl.gov> wrote: > Sorry, just to clarify, in the second case I see several *inner* > iterations, even though I'm using LU on a supposedly exact Schur complement > as the preconditioner for the Schur system. > ------------------------------ > *From:* petsc-users <petsc-users-boun...@mcs.anl.gov> on behalf of > Cotter, Colin J via petsc-users <petsc-users@mcs.anl.gov> > *Sent:* 18 March 2019 20:14:48 > *To:* petsc-users@mcs.anl.gov > *Subject:* [petsc-users] Confusing Schur preconditioner behaviour > > > Dear petsc-users, > > I'm solving a 2x2 block system, for which I can construct the Schur > complement analytically (through compatible FEM stuff), > > which I can pass as the preconditioning matrix. > > > When using gmres on the outer iteration, and preonly+lu on the inner > iterations with a Schur complement preconditioner, > > I see convergence in 1 iteration as expected. However, when I set gmres+lu > on the inner iteration for S, I see several iterations. > > > This seems strange to me, as the first result seems to confirm that I have > an exact Schur complement, but the second result > > implies not. > > What could be going on here? > > > I've appended output to the bottom of this message, first the preonly+lu > and then for gmres+lu. > > > all the best > > --Colin > > > Linear firedrake_0_fieldsplit_1_ solve converged due to CONVERGED_ITS > iterations 1 > Residual norms for firedrake_0_ solve. > 0 KSP preconditioned resid norm 4.985448866758e+00 true resid norm > 1.086016610848e-03 ||r(i)||/||b|| 1.000000000000e+00 > Linear firedrake_0_fieldsplit_1_ solve converged due to CONVERGED_ITS > iterations 1 > 1 KSP preconditioned resid norm 1.245615753306e-13 true resid norm > 2.082000915439e-14 ||r(i)||/||b|| 1.917098591903e-11 > KSP Object: (firedrake_0_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-07, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (firedrake_0_) 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from A11 > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 1. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=6144, cols=6144 > package used to perform factorization: petsc > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: seqaij > rows=6144, cols=6144 > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is 5 > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 5.09173 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=15360, cols=15360 > package used to perform factorization: petsc > total: nonzeros=1360836, allocated nonzeros=1360836 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > linear system matrix followed by preconditioner matrix: > Mat Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=15360, cols=15360 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: seqaij > rows=15360, cols=15360 > total: nonzeros=267264, allocated nonzeros=267264 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=15360, cols=6144 > total: nonzeros=73728, allocated nonzeros=73728 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > KSP of A00 > KSP Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 1. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=6144, cols=6144 > package used to perform factorization: petsc > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 2048 nodes, limit > used is 5 > linear system matrix = precond matrix: > Mat Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: seqaij > rows=6144, cols=6144 > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is > 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=6144, cols=15360 > total: nonzeros=73728, allocated nonzeros=73728 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is 5 > Mat Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: seqaij > rows=15360, cols=15360 > total: nonzeros=267264, allocated nonzeros=267264 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > linear system matrix followed by preconditioner matrix: > Mat Object: (firedrake_0_) 1 MPI processes > type: nest > rows=21504, cols=21504 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : type=seqaij, rows=15360, cols=15360 > (0,1) : type=seqaij, rows=15360, cols=6144 > (1,0) : type=seqaij, rows=6144, cols=15360 > (1,1) : type=seqaij, rows=6144, cols=6144 > Mat Object: (firedrake_0_) 1 MPI processes > type: nest > rows=21504, cols=21504 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : prefix="firedrake_0_fieldsplit_1_", type=seqaij, > rows=15360, cols=15360 > (0,1) : type=seqaij, rows=15360, cols=6144 > (1,0) : type=seqaij, rows=6144, cols=15360 > (1,1) : prefix="firedrake_0_fieldsplit_0_", type=seqaij, > rows=6144, cols=6144 > > ===== > > > Residual norms for firedrake_0_fieldsplit_1_ solve. > 0 KSP preconditioned resid norm 8.819238435108e-02 true resid norm > 1.797571993221e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.025167319984e-02 true resid norm > 3.340583874349e-02 ||r(i)||/||b|| 1.858386694356e-01 > 2 KSP preconditioned resid norm 1.235104644359e-03 true resid norm > 8.148396804822e-03 ||r(i)||/||b|| 4.533001646417e-02 > 3 KSP preconditioned resid norm 1.624748553125e-04 true resid norm > 1.612221957927e-03 ||r(i)||/||b|| 8.968886720573e-03 > 4 KSP preconditioned resid norm 2.233373761266e-05 true resid norm > 3.292437172839e-04 ||r(i)||/||b|| 1.831602397710e-03 > 5 KSP preconditioned resid norm 1.895393184017e-06 true resid norm > 4.091207337005e-05 ||r(i)||/||b|| 2.275962994770e-04 > 6 KSP preconditioned resid norm 1.699212495729e-07 true resid norm > 3.851173419652e-06 ||r(i)||/||b|| 2.142430697728e-05 > Linear firedrake_0_fieldsplit_1_ solve converged due to CONVERGED_RTOL > iterations 6 > KSP Object: (firedrake_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (firedrake_0_) 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, factorization FULL > Preconditioner for the Schur complement formed from A11 > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 1. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=6144, cols=6144 > package used to perform factorization: petsc > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: seqaij > rows=6144, cols=6144 > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is 5 > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 5.09173 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=15360, cols=15360 > package used to perform factorization: petsc > total: nonzeros=1360836, allocated nonzeros=1360836 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > linear system matrix followed by preconditioner matrix: > Mat Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=15360, cols=15360 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: seqaij > rows=15360, cols=15360 > total: nonzeros=267264, allocated nonzeros=267264 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > A10 > Mat Object: 1 MPI processes > type: seqaij > rows=15360, cols=6144 > total: nonzeros=73728, allocated nonzeros=73728 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > KSP of A00 > KSP Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 1. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=6144, cols=6144 > package used to perform factorization: petsc > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues > calls =0 > using I-node routines: found 2048 nodes, limit > used is 5 > linear system matrix = precond matrix: > Mat Object: (firedrake_0_fieldsplit_0_) 1 MPI processes > type: seqaij > rows=6144, cols=6144 > total: nonzeros=18432, allocated nonzeros=18432 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is > 5 > A01 > Mat Object: 1 MPI processes > type: seqaij > rows=6144, cols=15360 > total: nonzeros=73728, allocated nonzeros=73728 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 2048 nodes, limit used is 5 > Mat Object: (firedrake_0_fieldsplit_1_) 1 MPI processes > type: seqaij > rows=15360, cols=15360 > total: nonzeros=267264, allocated nonzeros=267264 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 5120 nodes, limit used is 5 > linear system matrix followed by preconditioner matrix: > Mat Object: (firedrake_0_) 1 MPI processes > type: nest > rows=21504, cols=21504 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : type=seqaij, rows=15360, cols=15360 > (0,1) : type=seqaij, rows=15360, cols=6144 > (1,0) : type=seqaij, rows=6144, cols=15360 > (1,1) : type=seqaij, rows=6144, cols=6144 > Mat Object: (firedrake_0_) 1 MPI processes > type: nest > rows=21504, cols=21504 > Matrix object: > type=nest, rows=2, cols=2 > MatNest structure: > (0,0) : prefix="firedrake_0_fieldsplit_1_", type=seqaij, > rows=15360, cols=15360 > (0,1) : type=seqaij, rows=15360, cols=6144 > (1,0) : type=seqaij, rows=6144, cols=15360 > (1,1) : prefix="firedrake_0_fieldsplit_0_", type=seqaij, > rows=6144, cols=6144 > > > >