Re: [petsc-users] Cross-compilation cluster
Hmmn, ok. I will try it out. On Thu, Mar 14, 2019 at 9:34 AM Matthew Knepley wrote: > On Thu, Mar 14, 2019 at 10:05 AM Amneet Bhalla > wrote: > >> I need the serial built of Silo and HDF5 libraries. I am sure that the >> MPI wrappers are coming from the serial compilers. >> >> i.e, If I do >> >> gcc —version and mpicc —version, I see the same GCC version. >> > > That is in a perfect world. Unfortunately, flags for compilers can make > incompatible libraries. Its no problem > to give the MPI compilers to Silo and HDF5, even if you do not want to > link the libraries. > >Matt > > >> On Thu, Mar 14, 2019 at 6:44 AM Matthew Knepley >> wrote: >> >>> It is very dangerous to use different compilers. I would make sure that >>> all the compilers are the MPI compilers. >>> >>> Thanks, >>> >>> Matt >>> >>> On Thu, Mar 14, 2019 at 8:46 AM Amneet Bhalla >>> wrote: >>> Ah, Ok. Do serial compilers look OK to you? Can lib-32 and lib-64 (say -lm) operate simulataneously during runtime, or this is my imagination? On Thu, Mar 14, 2019 at 5:36 AM Matthew Knepley wrote: > On Thu, Mar 14, 2019 at 8:28 AM Amneet Bhalla > wrote: > >> Matt -- SAMRAI, PETSc, and libMesh configure logs are attached with >> this email. Also including some other log files in case they are useful. >> > > Okay, PETSc is not sticking in a /usr/lib (or /usr/lib64). However, I > can see that you mpif90 (and perhaps other of the wrappers) are > reporting both /usr/lib64 AND /usr/lib flags, and I am guessing that > is where the other configures are picking it up. > > Thanks, > > Matt > > >> Thanks, >> --Amneet >> >> On Thu, Mar 14, 2019 at 4:21 AM Matthew Knepley >> wrote: >> >>> In order to see why each flag was included, we need to see >>> configure.log. >>> >>> Thanks, >>> >>> Matt >>> >>> On Thu, Mar 14, 2019 at 2:40 AM Amneet Bhalla via petsc-users < >>> petsc-users@mcs.anl.gov> wrote: >>> Hi Folks, I am on a cluster that has -L/lib dir with 32-bit libraries and -L/lib64 with 64-bit libraries. During compilation of some of libraries required for my code (such as SAMRAI and libMesh) both paths get picked -L/lib and -L/lib64. I am seeing some sporadic behavior in runtime when at some timesteps PETSc does not converge. The same code with the same number of processors run just fine on my workstation that has just 64-bit version of libraries. Even during the final linking stage of the executable, the linker gives warnings like ld: skipping incompatible //lib/libm.so when searching for -lm ld: skipping incompatible /lib/libm.so when searching for -lm ld: skipping incompatible /lib/libm.so when searching for -lm ld: skipping incompatible //lib/libpthread.so when searching for -lpthread ld: skipping incompatible /lib/libpthread.so when searching for -lpthread ld: skipping incompatible /lib/libpthread.so when searching for -lpthread ld: skipping incompatible //lib/libdl.so when searching for -ldl ld: skipping incompatible //lib/libc.so when searching for -lc ld: skipping incompatible /lib/libc.so when searching for -lc ld: skipping incompatible /lib/libc.so when searching for -lc but the executable runs. This is during config of SAMRAI when it picks both -L/lib and -L/lib64: checking whether we are using the GNU Fortran 77 compiler... no checking whether ifort accepts -g... yes checking how to get verbose linking output from ifort... -v checking for Fortran 77 libraries of ifort... -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/
Re: [petsc-users] GAMG parallel convergence sensitivity
Mark Lohry writes: > It seems to me with these semi-implicit methods the CFL limit is still so > close to the explicit limit (that paper stops at 30), I don't really see > the purpose unless you're running purely incompressible? That's just my > ignorance speaking though. I'm currently running fully implicit for > everything, with CFLs around 1e3 - 1e5 or so. It depends what you're trying to resolve. Sounds like maybe you're stepping toward steady state. The paper is wishing to resolve vortex and baroclinic dynamics while stepping over acoustics and barotropic waves.
Re: [petsc-users] Cross-compilation cluster
It is very dangerous to use different compilers. I would make sure that all the compilers are the MPI compilers. Thanks, Matt On Thu, Mar 14, 2019 at 8:46 AM Amneet Bhalla wrote: > Ah, Ok. Do serial compilers look OK to you? > > Can lib-32 and lib-64 (say -lm) operate simulataneously during runtime, or > this is my imagination? > > > > On Thu, Mar 14, 2019 at 5:36 AM Matthew Knepley wrote: > >> On Thu, Mar 14, 2019 at 8:28 AM Amneet Bhalla >> wrote: >> >>> Matt -- SAMRAI, PETSc, and libMesh configure logs are attached with this >>> email. Also including some other log files in case they are useful. >>> >> >> Okay, PETSc is not sticking in a /usr/lib (or /usr/lib64). However, I can >> see that you mpif90 (and perhaps other of the wrappers) are >> reporting both /usr/lib64 AND /usr/lib flags, and I am guessing that is >> where the other configures are picking it up. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> --Amneet >>> >>> On Thu, Mar 14, 2019 at 4:21 AM Matthew Knepley >>> wrote: >>> In order to see why each flag was included, we need to see configure.log. Thanks, Matt On Thu, Mar 14, 2019 at 2:40 AM Amneet Bhalla via petsc-users < petsc-users@mcs.anl.gov> wrote: > Hi Folks, > > I am on a cluster that has -L/lib dir with 32-bit libraries and > -L/lib64 with 64-bit libraries. During compilation of some of > libraries required for my code (such as SAMRAI and libMesh) both paths > get picked -L/lib and -L/lib64. > > I am seeing some sporadic behavior in runtime when at some timesteps > PETSc does not converge. The same code with the same number of processors > run just fine on my workstation that has just 64-bit version of libraries. > > Even during the final linking stage of the executable, the linker > gives warnings like > > ld: skipping incompatible //lib/libm.so when searching for -lm > > ld: skipping incompatible /lib/libm.so when searching for -lm > > ld: skipping incompatible /lib/libm.so when searching for -lm > > ld: skipping incompatible //lib/libpthread.so when searching for > -lpthread > > ld: skipping incompatible /lib/libpthread.so when searching for > -lpthread > > ld: skipping incompatible /lib/libpthread.so when searching for > -lpthread > > ld: skipping incompatible //lib/libdl.so when searching for -ldl > > ld: skipping incompatible //lib/libc.so when searching for -lc > > ld: skipping incompatible /lib/libc.so when searching for -lc > > ld: skipping incompatible /lib/libc.so when searching for -lc > but the executable runs. > > > This is during config of SAMRAI when it picks both -L/lib and -L/lib64: > > checking whether we are using the GNU Fortran 77 compiler... no > > checking whether ifort accepts -g... yes > > checking how to get verbose linking output from ifort... -v > > checking for Fortran 77 libraries of ifort... > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ > -L/lib/../lib64 > -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/ > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/ > -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc > -lpthread -lgcc -lgcc_s -lirc_s -ldl > > libMesh is also picking that path > > libmesh_optional_LIBS : -lhdf5 -lhdf5_cpp -lz > -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib > -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib > -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64 > -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib > -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib > -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 > -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 >
Re: [petsc-users] Cross-compilation cluster
As these warnings indicate - 64bit compiler ignores 32bit libraries [so they don't get used]. i.e mixing 32bit and 64bit libraries ins not the cause of your problems on this machine.. Satish On Wed, 13 Mar 2019, Amneet Bhalla via petsc-users wrote: > Hi Folks, > > I am on a cluster that has -L/lib dir with 32-bit libraries and -L/lib64 > with 64-bit libraries. During compilation of some of libraries required for > my code (such as SAMRAI and libMesh) both paths > get picked -L/lib and -L/lib64. > > I am seeing some sporadic behavior in runtime when at some timesteps PETSc > does not converge. The same code with the same number of processors run > just fine on my workstation that has just 64-bit version of libraries. > > Even during the final linking stage of the executable, the linker gives > warnings like > > ld: skipping incompatible //lib/libm.so when searching for -lm > > ld: skipping incompatible /lib/libm.so when searching for -lm > > ld: skipping incompatible /lib/libm.so when searching for -lm > > ld: skipping incompatible //lib/libpthread.so when searching for -lpthread > > ld: skipping incompatible /lib/libpthread.so when searching for -lpthread > > ld: skipping incompatible /lib/libpthread.so when searching for -lpthread > > ld: skipping incompatible //lib/libdl.so when searching for -ldl > > ld: skipping incompatible //lib/libc.so when searching for -lc > > ld: skipping incompatible /lib/libc.so when searching for -lc > > ld: skipping incompatible /lib/libc.so when searching for -lc > but the executable runs. > > > This is during config of SAMRAI when it picks both -L/lib and -L/lib64: > > checking whether we are using the GNU Fortran 77 compiler... no > > checking whether ifort accepts -g... yes > > checking how to get verbose linking output from ifort... -v > > checking for Fortran 77 libraries of ifort... > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64 > -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/ > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/ > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/ > -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc > -lpthread -lgcc -lgcc_s -lirc_s -ldl > > libMesh is also picking that path > > libmesh_optional_LIBS : -lhdf5 -lhdf5_cpp -lz > -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib > -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib > -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64 > -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib > -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib > -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 > -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 > -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 > -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 > -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin > -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin > -lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core > -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport > -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s > -lstdc++ -ldl -L/lib -Wl,-rpath,/lib > -Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64 > -L/usr/local/mpi/intel/openmpi-4.0.0/lib64 > -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 > -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 > > Perhaps PETSc also picks up both versions (and there is a way to query it > from PETSc?), but I can't confirm this. Is there a way to instruct make to > select only -L/lib64? I want to rule out that 32-bit dynamic library is not > a culprit for the random non-convergence of PETSc solvers and the eventual > crash of the
