On Wed, Oct 26, 2016 at 11:15 AM, Olivier Mesnard < olivier.mesna...@gmail.com> wrote:
> On 26 October 2016 at 09:38, Mark Adams <mfad...@lbl.gov> wrote: > >> Please run with -info and grep on GAMG and send that. (-info is very >> noisy). >> >> I cat the grep at the end of the log file (see attachment > petsc-3.7.4-n2.log). > Also, increasing the local number of iterations in SOR, as suggested by > Barry, removed the indefinite preconditioner (file > petsc-3.7.4-n2-lits2.log). > The eigen estimate on the second level looks like it could be low. You have a small stencil. Is this a 2D problem? You might try adding '-pc_gamg_square_graph 0', just to see what happens. (AMG can be a real pain sometimes.) > > >> I'm not sure what is going on here. Divergence with parallelism. Here >> are some suggestions. >> >> Note, you do not need to set the null space for a scalar (Poisson) >> problem unless you have some special null space. And not getting it set >> (with the 6 rigid body modes) for the velocity (elasticity) equation will >> only degrade convergence rates. >> >> There was a bug for a while (early 3.7 versions) where the coarse grid >> was not squeezed onto one processor, which could result in very bad >> convergence, but not divergence, on multiple processors (the -info output >> will report the number of 'active pes'). Perhaps this bug is causing >> divergence for you. We had another subtle bug where the eigen estimates >> used a bad seed vector, which gives a bad eigen estimate. This would cause >> divergence but it should not be a parallelism issue (these two bugs were >> both regressions in around 3.7) >> >> Divergence usually comes from a bad eigen estimate in a Chebyshev >> smoother, but this is not highly correlated with parallelism. The -info >> data will report the eigen estimates but that is not terribly useful but >> you can see if it changes (gets larger) with better parameters. Add these >> parameters, with the correct prefix, and use -options_left to make sure >> that "there are no unused options": >> >> -mg_levels_ksp_type chebyshev >> -mg_levels_esteig_ksp_type cg >> -mg_levels_esteig_ksp_max_it 10 >> >> -mg_levels_ksp_chebyshev_esteig 0,.1,0,1.05 >> >> petsc-3.7.4-n2-chebyshev.log contains the output when using the default > KSP Chebyshev. > When estimating the eigenvalues using cg with the translations [0, 0.1; 0, > 1.05] (previously using default gmres with translations [0, 0.1; 0, 1.1]), > the max eigenvalue decreases from 1.0931 to 1.04366 and the indefinite > preconditioner appears ealier after 2 iterations (3 previously). > I attached the log (see petsc-3.7.4-chebyshev.log). > > >> chebyshev is the default, as Barry suggested, replace this with gmres or >> richardson (see below) and verify that this fixed the divergence problem. >> >> > Using gmres (-poisson_mg_levels_ksp_type gmres) fixes the divergence > problem > (file petsc-3.7.4-n2-gmres.log) > . > > Same observation with richardson (file petsc-3.7.4-n2-richardson.log). > > >> If your matrix is symmetric positive definite then use >> '-mg_levels_esteig_ksp_type cg', if not then use the default gmres. >> > > I checked and I still get an indefinite preconditioner when using gmres to > estimate the eigenvalues. > > If you matrix is symmetric (and positive) then CG will give you a better estimate (provable?) > >> Increase/decrease '-mg_levels_esteig_ksp_max_it 10', you should see the >> estimates increase and converge with higher max_it. Setting this to a huge >> number, like 100, should fix the bad seed vector problem mentioned above. >> >> I played with the maximum number of iterations. Here are the min/max > eigenvalue estimates for the two levels: > - max_it 5: (min=0.0975079, max=1.02383) on level 1, (0.0975647, 1.02443) > on level 2 > - max_it 10: (0.0991546, 1.04112), (0.0993962, 1.04366) > - max_it 20: (0.0995918, 1.04571), (0.115723, 1.21509) > - max_it 50: (0.0995651, 1.04543), (0.133744, 1.40431) > - max_it 100: (0.0995651, 1.04543), (0.133744, 1.40431) > > Note that all those runs ended up with an indefinite preconditioner, > except when increasing the maximum number of iterations to 50 (and 100, > which did not improve the eigenvalue estimates). > Are you saying that the indefinite PC error goes away when '-mg_levels_esteig_ksp_max_it 50' ? or is this MG smoother iterations? > > >> If eigen estimates are a pain, like with non SPD systems, then >> richardson is an option (instead of chebyshev): >> >> -mg_levels_ksp_type richardson >> -mg_levels_ksp_richardson_scale 0.6 >> >> You then need to play with the scaling (that is what chebyshev does for >> you essentially). >> >> >> On Tue, Oct 25, 2016 at 10:22 PM, Matthew Knepley <knep...@gmail.com> >> wrote: >> >>> On Tue, Oct 25, 2016 at 9:20 PM, Barry Smith <bsm...@mcs.anl.gov> wrote: >>> >>>> >>>> Olivier, >>>> >>>> Ok, so I've run the code in the debugger, but I don't not think the >>>> problem is with the null space. The code is correctly removing the null >>>> space on all the levels of multigrid. >>>> >>>> I think the error comes from changes in the behavior of GAMG. GAMG >>>> is relatively rapidly moving with different defaults and even different >>>> code with each release. >>>> >>>> To check this I added the option -poisson_mg_levels_pc_sor_lits 2 >>>> and it stopped complaining about KSP_DIVERGED_INDEFINITE_PC. I've seen this >>>> before where the smoother is "too weak" and so the net result is that >>>> action of the preconditioner is indefinite. Mark Adams probably has better >>>> suggestions on how to make the preconditioner behave. Note you could also >>>> use a KSP of richardson or gmres instead of cg since they don't care about >>>> this indefinite business. >>> >>> >>> I think old GAMG squared the graph by default. You can see in the 3.7 >>> output that it does not. >>> >>> Matt >>> >>> >>>> >>>> Barry >>>> >>>> >>>> >>>> > On Oct 25, 2016, at 5:39 PM, Olivier Mesnard < >>>> olivier.mesna...@gmail.com> wrote: >>>> > >>>> > On 25 October 2016 at 17:51, Barry Smith <bsm...@mcs.anl.gov> wrote: >>>> > >>>> > Olivier, >>>> > >>>> > In theory you do not need to change anything else. Are you using >>>> a different matrix object for the velocity_ksp object than the poisson_ksp >>>> object? >>>> > >>>> > The matrix is different for the velocity_ksp and the poisson_ksp. >>>> > >>>> > The code change in PETSc is very little but we have a report from >>>> another CFD user who also had problems with the change so there may be some >>>> subtle bug that we can't figure out causing things to not behave properly. >>>> > >>>> > First run the 3.7.4 code with -poisson_ksp_view and verify that >>>> when it prints the matrix information it prints something like has attached >>>> null space if it does not print that it means that somehow the matrix is >>>> not properly getting the matrix attached. >>>> > >>>> > When running with 3.7.4 and -poisson_ksp_view, the output shows that >>>> the nullspace is not attached to the KSP (as it was with 3.5.4); however >>>> the print statement is now under the Mat info (which is expected when >>>> moving from KSPSetNullSpace to MatSetNullSpace?). >>>> > >>>> > Though older versions had MatSetNullSpace() they didn't >>>> necessarily associate it with the KSP so it was not expected to work as a >>>> replacement for KSPSetNullSpace() with older versions. >>>> > >>>> > Because our other user had great difficulty trying to debug the >>>> issue feel free to send us at petsc-ma...@mcs.anl.gov your code with >>>> instructions on building and running and we can try to track down the >>>> problem. Better than hours and hours spent with fruitless email. We will, >>>> of course, not distribute the code and will delete in when we are finished >>>> with it. >>>> > >>>> > The code is open-source and hosted on GitHub ( >>>> https://github.com/barbagroup/PetIBM). >>>> > I just pushed the branches `feature-compatible-petsc-3.7` and >>>> `revert-compatible-petsc-3.5` that I used to observe this problem. >>>> > >>>> > PETSc (both 3.5.4 and 3.7.4) was configured as follow: >>>> > export PETSC_ARCH="linux-gnu-dbg" >>>> > ./configure --PETSC_ARCH=$PETSC_ARCH \ >>>> > --with-cc=gcc \ >>>> > --with-cxx=g++ \ >>>> > --with-fc=gfortran \ >>>> > --COPTFLAGS="-O0" \ >>>> > --CXXOPTFLAGS="-O0" \ >>>> > --FOPTFLAGS="-O0" \ >>>> > --with-debugging=1 \ >>>> > --download-fblaslapack \ >>>> > --download-mpich \ >>>> > --download-hypre \ >>>> > --download-yaml \ >>>> > --with-x=1 >>>> > >>>> > Our code was built using the following commands: >>>> > mkdir petibm-build >>>> > cd petibm-build >>>> > export PETSC_DIR=<directory of PETSc> >>>> > export PETSC_ARCH="linux-gnu-dbg" >>>> > export PETIBM_DIR=<directory of PetIBM git repo> >>>> > $PETIBM_DIR/configure --prefix=$PWD \ >>>> > CXX=$PETSC_DIR/$PETSC_ARCH/bin/mpicxx \ >>>> > CXXFLAGS="-g -O0 -std=c++11" >>>> > make all >>>> > make install >>>> > >>>> > Then >>>> > cd examples >>>> > make examples >>>> > >>>> > The example of the lid-driven cavity I was talking about can be >>>> found in the folder `examples/2d/convergence/lidDrivenCavity20/20/` >>>> > >>>> > To run it: >>>> > mpiexec -n N <path-to-petibm-build>/bin/petibm2d -directory >>>> <path-to-example> >>>> > >>>> > Let me know if you need more info. Thank you. >>>> > >>>> > Barry >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > > On Oct 25, 2016, at 4:38 PM, Olivier Mesnard < >>>> olivier.mesna...@gmail.com> wrote: >>>> > > >>>> > > Hi all, >>>> > > >>>> > > We develop a CFD code using the PETSc library that solves the >>>> Navier-Stokes equations using the fractional-step method from Perot (1993). >>>> > > At each time-step, we solve two systems: one for the velocity >>>> field, the other, a Poisson system, for the pressure field. >>>> > > One of our test-cases is a 2D lid-driven cavity flow (Re=100) on a >>>> 20x20 grid using 1 or 2 procs. >>>> > > For the Poisson system, we usually use CG preconditioned with GAMG. >>>> > > >>>> > > So far, we have been using PETSc-3.5.4, and we would like to update >>>> the code with the latest release: 3.7.4. >>>> > > >>>> > > As suggested in the changelog of 3.6, we replaced the routine >>>> `KSPSetNullSpace()` with `MatSetNullSpace()`. >>>> > > >>>> > > Here is the list of options we use to configure the two solvers: >>>> > > * Velocity solver: prefix `-velocity_` >>>> > > -velocity_ksp_type bcgs >>>> > > -velocity_ksp_rtol 1.0E-08 >>>> > > -velocity_ksp_atol 0.0 >>>> > > -velocity_ksp_max_it 10000 >>>> > > -velocity_pc_type jacobi >>>> > > -velocity_ksp_view >>>> > > -velocity_ksp_monitor_true_residual >>>> > > -velocity_ksp_converged_reason >>>> > > * Poisson solver: prefix `-poisson_` >>>> > > -poisson_ksp_type cg >>>> > > -poisson_ksp_rtol 1.0E-08 >>>> > > -poisson_ksp_atol 0.0 >>>> > > -poisson_ksp_max_it 20000 >>>> > > -poisson_pc_type gamg >>>> > > -poisson_pc_gamg_type agg >>>> > > -poisson_pc_gamg_agg_nsmooths 1 >>>> > > -poissonksp_view >>>> > > -poisson_ksp_monitor_true_residual >>>> > > -poisson_ksp_converged_reason >>>> > > >>>> > > With 3.5.4, the case runs normally on 1 or 2 procs. >>>> > > With 3.7.4, the case runs normally on 1 proc but not on 2. >>>> > > Why? The Poisson solver diverges because of an indefinite >>>> preconditioner (only with 2 procs). >>>> > > >>>> > > We also saw that the routine `MatSetNullSpace()` was already >>>> available in 3.5.4. >>>> > > With 3.5.4, replacing `KSPSetNullSpace()` with `MatSetNullSpace()` >>>> led to the Poisson solver diverging because of an indefinite matrix (on 1 >>>> and 2 procs). >>>> > > >>>> > > Thus, we were wondering if we needed to update something else for >>>> the KSP, and not just modifying the name of the routine? >>>> > > >>>> > > I have attached the output files from the different cases: >>>> > > * `run-petsc-3.5.4-n1.log` (3.5.4, `KSPSetNullSpace()`, n=1) >>>> > > * `run-petsc-3.5.4-n2.log` >>>> > > * `run-petsc-3.5.4-nsp-n1.log` (3.5.4, `MatSetNullSpace()`, n=1) >>>> > > * `run-petsc-3.5.4-nsp-n2.log` >>>> > > * `run-petsc-3.7.4-n1.log` (3.7.4, `MatSetNullSpace()`, n=1) >>>> > > * `run-petsc-3.7.4-n2.log` >>>> > > >>>> > > Thank you for your help, >>>> > > Olivier >>>> > > <run-petsc-3.5.4-n1.log><run-petsc-3.5.4-n2.log><run-petsc-3 >>>> .5.4-nsp-n1.log><run-petsc-3.5.4-nsp-n2.log><run-petsc-3.7.4 >>>> -n1.log><run-petsc-3.7.4-n2.log> >>>> > >>>> > >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >