Dear dealii Community,
I am working on solving the a nonlinear coupled problem involving a vector
displacement field and a scalar phase-field variable. The code is MPI
parallelized using p:d:t and TrilinosWrappers for linear algebra.
Usually I use CG+AMG for solving the SLEs when solving for each of the
variables within a staggered scheme. But for certain scenarios, the
iterative linear solver fails and we switch to Amesos_Superludist solver.
The code is run on 2 nodes (144 MPI processes in total) and as shown by the
code performance monitor, the flop count of one of the nodes drops to
(almost) zero and only one one node seems to be doing the computations once
the solver switch from iterative to direct solver occurs. Please see
attached flops and memory bandwidth plots. The blue and red lines here
represent the two nodes. Similar observations were also made for a larger
problem involving 8 nodes.
These plots seem to hint that Superlu-dist solver does not scale across
multiple nodes. One possible reason I could think of is that I probably
missed some option while installing dealii with trilinos and superlu-dist
using spack. I also attach the spack spec which I installed on the
cluster. The gcc compiler and corresponding openmpi@4.1.2 are available
form the cluster.
Any ideas on solving this issue would be of great help.
Kind regards,
Paras Kumar
--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see
https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dealii/44fad345-7ca4-47b7-a20b-29ee006efc11n%40googlegroups.com.
Input spec
--------------------------------
dealii@9.0.1%gcc@11.2.0
^trilinos@12.10.1~explicit_template_instantiation+superlu-dist
Concretized
--------------------------------
dealii@9.0.1%gcc@11.2.0+adol-c~arborx+arpack+assimp~cuda~doc+examples~ginkgo+gmsh+gsl+hdf5~int64~ipo+metis+mpi+muparser~nanoflann~netcdf+oce+optflags+p4est+petsc~python+scalapack~simplex+slepc+sundials~symengine+threads+trilinos
build_type=DebugRelease cuda_arch=none cxxstd=default
patches=4282b32e96f2f5d376eb34f3fddcc4615fcd99b40004cca784eb874288d1b31c,61f217744b70f352965be265d2f06e8c1276685e2944ca0a88b7297dd55755da,6f876dc8eadafe2c4ec2a6673864fb451c6627ca80511b6e16f3c401946fdf33
arch=linux-almalinux8-icelake
^adol-c@2.7.2%gcc@11.2.0~advanced_branching+atrig_erf~boost+doc+examples~openmp~sparse
arch=linux-almalinux8-icelake
^arpack-ng@3.8.0%gcc@11.2.0+mpi+shared arch=linux-almalinux8-icelake
^cmake@3.21.4%gcc@11.2.0~doc+ncurses~openssl+ownlibs~qt
build_type=Release arch=linux-almalinux8-icelake
^ncurses@6.2%gcc@11.2.0~symlinks+termlib abi=none
arch=linux-almalinux8-icelake
^pkgconf@1.8.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^openblas@0.3.18%gcc@11.2.0~bignuma~consistent_fpcsr~ilp64+locking+pic+shared
threads=none arch=linux-almalinux8-icelake
^perl@5.34.0%gcc@11.2.0+cpanm+shared+threads
arch=linux-almalinux8-icelake
^berkeley-db@18.1.40%gcc@11.2.0+cxx~docs+stl
patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522
arch=linux-almalinux8-icelake
^bzip2@1.0.8%gcc@11.2.0~debug~pic+shared
arch=linux-almalinux8-icelake
^diffutils@3.8%gcc@11.2.0 arch=linux-almalinux8-icelake
^libiconv@1.16%gcc@11.2.0 libs=shared,static
arch=linux-almalinux8-icelake
^gdbm@1.19%gcc@11.2.0 arch=linux-almalinux8-icelake
^readline@8.1%gcc@11.2.0 arch=linux-almalinux8-icelake
^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared
arch=linux-almalinux8-icelake
^openmpi@4.1.2%gcc@11.2.0~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker+pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath
fabrics=ucx schedulers=slurm arch=linux-almalinux8-icelake
^assimp@5.0.1%gcc@11.2.0~ipo+shared build_type=RelWithDebInfo
arch=linux-almalinux8-icelake
^boost@1.76.0%gcc@11.2.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave
cxxstd=98 visibility=hidden arch=linux-almalinux8-icelake
^gmsh@4.8.4%gcc@11.2.0+alglib~cairo+cgns+compression~eigen~external+fltk+gmp~hdf5~ipo+med+metis+mmg~mpi+netgen+oce~opencascade~openmp~petsc~privateapi+shared~slepc+tetgen+voropp
build_type=RelWithDebInfo arch=linux-almalinux8-icelake
^cgns@4.2.0%gcc@11.2.0~base_scope~fortran+hdf5~int64~ipo~legacy~mem_debug+mpi+scoping+shared~static~testing
build_type=RelWithDebInfo arch=linux-almalinux8-icelake
^hdf5@1.10.7%gcc@11.2.0~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools
api=default build_type=RelWithDebInfo arch=linux-almalinux8-icelake
^numactl@2.0.14%gcc@11.2.0
patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296
arch=linux-almalinux8-icelake
^autoconf@2.69%gcc@11.2.0
patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee
arch=linux-almalinux8-icelake
^m4@1.4.19%gcc@11.2.0+sigsegv
patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89
arch=linux-almalinux8-icelake
^libsigsegv@2.13%gcc@11.2.0
arch=linux-almalinux8-icelake
^automake@1.16.3%gcc@11.2.0 arch=linux-almalinux8-icelake
^libtool@2.4.6%gcc@11.2.0 arch=linux-almalinux8-icelake
^fltk@1.3.7%gcc@11.2.0+gl+shared~xft arch=linux-almalinux8-icelake
^libx11@1.7.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^inputproto@2.3.2%gcc@11.2.0 arch=linux-almalinux8-icelake
^util-macros@1.19.3%gcc@11.2.0 arch=linux-almalinux8-icelake
^kbproto@1.0.7%gcc@11.2.0 arch=linux-almalinux8-icelake
^libxcb@1.14%gcc@11.2.0 arch=linux-almalinux8-icelake
^libpthread-stubs@0.4%gcc@11.2.0
arch=linux-almalinux8-icelake
^libxau@1.0.8%gcc@11.2.0 arch=linux-almalinux8-icelake
^xproto@7.0.31%gcc@11.2.0 arch=linux-almalinux8-icelake
^libxdmcp@1.1.2%gcc@11.2.0 arch=linux-almalinux8-icelake
^libbsd@0.11.3%gcc@11.2.0 arch=linux-almalinux8-icelake
^libmd@1.0.3%gcc@11.2.0
arch=linux-almalinux8-icelake
^xcb-proto@1.14.1%gcc@11.2.0 arch=linux-almalinux8-icelake
^xextproto@7.3.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^xtrans@1.3.5%gcc@11.2.0 arch=linux-almalinux8-icelake
^mesa@21.2.3%gcc@11.2.0+glx+llvm+opengl~opengles+osmesa~strip
buildtype=debugoptimized default_library=shared swr=auto
arch=linux-almalinux8-icelake
^bison@3.8.2%gcc@11.2.0 arch=linux-almalinux8-icelake
^expat@2.4.1%gcc@11.2.0+libbsd arch=linux-almalinux8-icelake
^flex@2.6.4%gcc@11.2.0+lex~nls
patches=09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3
arch=linux-almalinux8-icelake
^findutils@4.8.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^gettext@0.21%gcc@11.2.0+bzip2+curses+git~libunistring+libxml2+tar+xz
arch=linux-almalinux8-icelake
^libxml2@2.9.12%gcc@11.2.0~python
arch=linux-almalinux8-icelake
^xz@5.2.5%gcc@11.2.0~pic libs=shared,static
arch=linux-almalinux8-icelake
^tar@1.34%gcc@11.2.0 arch=linux-almalinux8-icelake
^help2man@1.47.16%gcc@11.2.0 arch=linux-almalinux8-icelake
^glproto@1.4.17%gcc@11.2.0 arch=linux-almalinux8-icelake
^libxext@1.3.3%gcc@11.2.0 arch=linux-almalinux8-icelake
^libxt@1.1.5%gcc@11.2.0 arch=linux-almalinux8-icelake
^libice@1.0.9%gcc@11.2.0 arch=linux-almalinux8-icelake
^libsm@1.2.3%gcc@11.2.0 arch=linux-almalinux8-icelake
^util-linux-uuid@2.36.2%gcc@11.2.0
arch=linux-almalinux8-icelake
^llvm@12.0.1%gcc@11.2.0~all_targets+clang~code_signing+compiler-rt~cuda~flang+gold+internal_unwind~ipo+libcxx+lld+lldb~llvm_dylib~mlir+omp_as_runtime~omp_debug~omp_tsan+polly~python~shared_libs~split_dwarf
build_type=Release cuda_arch=none arch=linux-almalinux8-icelake
^binutils@2.37%gcc@11.2.0~gas+gold~headers~interwork+ld~libiberty~lto+nls+plugins
libs=shared,static arch=linux-almalinux8-icelake
^hwloc@2.6.0%gcc@11.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared
arch=linux-almalinux8-icelake
^libpciaccess@0.16%gcc@11.2.0
arch=linux-almalinux8-icelake
^libedit@3.1-20210216%gcc@11.2.0
arch=linux-almalinux8-icelake
^perl-data-dumper@2.173%gcc@11.2.0
arch=linux-almalinux8-icelake
^python@3.8.12%gcc@11.2.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3~ssl~tix~tkinter~ucs4+uuid+zlib
patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4
arch=linux-almalinux8-icelake
^libffi@3.3%gcc@11.2.0
patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0
arch=linux-almalinux8-icelake
^sqlite@3.36.0%gcc@11.2.0+column_metadata+fts~functions~rtree
arch=linux-almalinux8-icelake
^swig@4.0.2%gcc@11.2.0 arch=linux-almalinux8-icelake
^pcre@8.44%gcc@11.2.0~jit+multibyte+utf
arch=linux-almalinux8-icelake
^z3@4.8.9%gcc@11.2.0~gmp~ipo~python
build_type=RelWithDebInfo arch=linux-almalinux8-icelake
^meson@0.60.0%gcc@11.2.0
patches=aa6c50d5a2aeb1a487d16f6712be4357fefb923aae37ab830699b07338388287
arch=linux-almalinux8-icelake
^ninja@1.10.2%gcc@11.2.0 arch=linux-almalinux8-icelake
^py-setuptools@58.2.0%gcc@11.2.0
arch=linux-almalinux8-icelake
^py-mako@1.1.4%gcc@11.2.0 arch=linux-almalinux8-icelake
^py-markupsafe@2.0.1%gcc@11.2.0
arch=linux-almalinux8-icelake
^xrandr@1.5.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^libxrandr@1.5.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^libxrender@0.9.10%gcc@11.2.0
arch=linux-almalinux8-icelake
^renderproto@0.11.1%gcc@11.2.0
arch=linux-almalinux8-icelake
^randrproto@1.5.0%gcc@11.2.0
arch=linux-almalinux8-icelake
^freetype@2.11.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^libpng@1.6.37%gcc@11.2.0 arch=linux-almalinux8-icelake
^gmp@6.2.1%gcc@11.2.0 arch=linux-almalinux8-icelake
^med@4.0.0%gcc@11.2.0+api23~fortran~ipo+mpi~shared
build_type=RelWithDebInfo
patches=ba351973779de38d658c62db0f97180449b40540c7e5be28dccf6098966cbf2a
arch=linux-almalinux8-icelake
^mesa-glu@9.0.1%gcc@11.2.0 arch=linux-almalinux8-icelake
^mmg@5.5.2%gcc@11.2.0~doc~ipo+scotch+shared build_type=RelWithDebInfo
arch=linux-almalinux8-icelake
^scotch@6.1.1%gcc@11.2.0+compression~esmumps~int64~metis+mpi+shared
arch=linux-almalinux8-icelake
^oce@0.18.3%gcc@11.2.0~X11+tbb arch=linux-almalinux8-icelake
^intel-tbb@2020.3%gcc@11.2.0~ipo+shared+tm
build_type=RelWithDebInfo cxxstd=default
patches=62ba015ebd1819c45bef47411540b789b493e31ca668c4ff4cb2afcbc306b476,ce1fb16fb932ce86a82ca87cf0431d1a8c83652af9f552b264213b2ff2945d73,d62cb666de4010998c339cde6f41c7623a07e9fc69e498f2e149821c0c2c6dd0
arch=linux-almalinux8-icelake
^gsl@2.7%gcc@11.2.0~external-cblas arch=linux-almalinux8-icelake
^metis@5.1.0%gcc@11.2.0~gdb~int64~real64+shared build_type=Release
patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1,b1225da886605ea558db7ac08dd8054742ea5afe5ed61ad4d0fe7a495b1270d2
arch=linux-almalinux8-icelake
^muparser@2.2.6.1%gcc@11.2.0 arch=linux-almalinux8-icelake
^netlib-scalapack@2.1.0%gcc@11.2.0~ipo~pic+shared build_type=Release
patches=1c9ce5fee1451a08c2de3cc87f446aeda0b818ebbce4ad0d980ddf2f2a0b2dc4,f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2
arch=linux-almalinux8-icelake
^p4est@2.8%gcc@11.2.0+mpi~openmp arch=linux-almalinux8-icelake
^petsc@3.16.1%gcc@11.2.0~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse~superlu-dist~tetgen~trilinos~valgrind
amdgpu_target=none clanguage=C cuda_arch=none arch=linux-almalinux8-icelake
^hypre@2.23.0%gcc@11.2.0~complex~cuda~debug+fortran~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist~unified-memory
cuda_arch=none arch=linux-almalinux8-icelake
^parmetis@4.0.3%gcc@11.2.0~gdb~int64~ipo+shared
build_type=RelWithDebInfo
patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d
arch=linux-almalinux8-icelake
^slepc@3.16.0%gcc@11.2.0+arpack~blopex~cuda~rocm amdgpu_target=none
cuda_arch=none arch=linux-almalinux8-icelake
^suite-sparse@5.10.1%gcc@11.2.0~cuda~openmp+pic~tbb
arch=linux-almalinux8-icelake
^mpfr@4.1.0%gcc@11.2.0 arch=linux-almalinux8-icelake
^autoconf-archive@2019.01.06%gcc@11.2.0
arch=linux-almalinux8-icelake
^texinfo@6.5%gcc@11.2.0
patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1
arch=linux-almalinux8-icelake
^sundials@3.2.1%gcc@11.2.0+ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~monitoring+mpi~openmp~petsc~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos
amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none precision=double
arch=linux-almalinux8-icelake
^trilinos@12.10.1%gcc@11.2.0~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus~explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~phalanx~piro~python~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu+superlu-dist~teko~tempus+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2
build_type=RelWithDebInfo cuda_arch=none cxxstd=14 gotype=long_long
arch=linux-almalinux8-icelake
^superlu-dist@5.3.0%gcc@11.2.0~cuda~int64~ipo~openmp+shared
build_type=RelWithDebInfo cuda_arch=none arch=linux-almalinux8-icelake
