Hi PETSc community,
I've been looking at using Spack to build PETSc, in particular I need to
disable the default metis/parmetis dependencies and use PTScotch instead,
for our software.
I've had quite a bit of trouble with this - it seems like something in the
resulting build of our simulator ends up badly optimised and an mpi
bottleneck, when I build against
PETSc built with Spack.
I've been trying to track this down, and noticed this in the PETSc Spack
build recipe:
# PTScotch: Currently disable Parmetis wrapper, this means
# nested disection won't be available thought PTScotch
depends_on("scotch+esmumps~metis+mpi", when="+ptscotch")
depends_on("scotch+int64", when="+ptscotch+int64")
Sure enough - when I compare the build with Spack and a traditional build
with ./configure etc, I see that, in the traditional build, Scotch is
always built with the parmetis wrapper,
but not in the Scotch build. In fact, I'm not sure how to turn off the
parmetis wrapper option for scotch, in the case of a traditional build
(i.e. there doesn't seem to be a flag in the
configure script for it) - which would be a very useful test for me (I can
of course do similar experiments by doing a classical build of petsc
against ptscotch built separately without the
wrappers, etc - will try that).
Does anyone know why the parmetis wrapper is always disabled in the spack
build options? Is there something about Spack that would prevent it from
working? It's notable - but I might
be missing it - that there's no warning that there's a difference in the
way ptscotch is built between the spack and classical builds:
- classical: ptscotch will always be built with parmetis wrappers, can't
seem to turn off
- spack: ptscotch will always be built without parmetis wrappers, can't
turn on
Any insight at all would be great, I'm new to Spack and am not super
familiar with the logic that goes into setting up builds for the system.
Here is the kind of command I give to Spack for PETSc builds, which may
well be less than ideal:
spack install petsc@3.19.1 ~metis +ptscotch ^hdf5 +fortran +hl
Separate tiny note: when building with hdf5, I have to ensure that the
fortran flag is set for it, as above. There's a fortran flag for the petsc
module, default true, and a fortran flag for the hdf5
module, default false. A naive user (i.e. me), will see the fortran flag
for the petsc module, and assume that all dependencies will correspondingly
be built with fortran capability - then see that
hdf5.mod is missing when trying to build their software against petsc. It's
the old "did you forget --with-hdf5-fortran-bindings?" issue, resurrected
for a new build system.
Thanks,
Daniel
