> On Jun 22, 2018, at 3:33 PM, Mark Adams <mfad...@lbl.gov> wrote:
>
> We are using KNL (Cori) and hypre is not working when configured with
> '--with-openmp=1', even when not using threads (as far as I can tell, I never
> use threads).
It does seem to run correctly without the --with-openmp option?
PETSc code does not know about --with-openmp at all so my guess is some bug
in the compiler when building hypre with OpenMP.
Barry
>
> Hypre is not converging, for instance with an optimized build:
>
> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type
> cg -pc_hypre_type boomeramg
> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.
> 0 KSP Residual norm 7.366251922394e+22
> 1 KSP Residual norm 3.676434682799e+22
> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2
>
> Interestingly in debug mode it almost looks good but it is dying:
>
> 05:09 nid02516 maint *= ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$
> make
> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp
> PETSC_ARCH="" run
> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type
> cg -pc_hypre_type boomeramg
> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.
> 0 KSP Residual norm 7.882081712007e+02
> 1 KSP Residual norm 2.500214073037e+02
> 2 KSP Residual norm 3.371746347713e+01
> 3 KSP Residual norm 2.918759396143e+00
> 4 KSP Residual norm 9.006505495017e-01
> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5
>
> This test runs fine on Xeon nodes. I assume that Hypre has been tested on
> KNL. GAMG runs fine, of coarse and the initial residual is similar to this
> debug run.
>
> Could PETSc be messing up the matrix conversion to hypre '--with-openmp=1' ?
>
> Any ideas?
>
> Thanks,
> Mark
>
