This is the complexity with maintaining dependencies (and dependencies
of dependencies), and different build systems
- Its not easy to keep the "defaults" in both builds exactly the same.
- And its not easy to expose all "variants" or keep the same variants in both
builds.
- And each pkg has its own issues that prevents some combinations to
work or not [or tested combinations vs untested].
This e-mail query has multiple things:
- understand "why" the current impl of [spack, petsc] build tools are the way
they are.
- if they can be improved
- and build use cases that you need working
- [and subsequently your code working]
Addressing them all is not easy - so lets stick with what you need to make
progress.
For one - we recommend using latest petsc version [i.e 3.21 - not 3.19] - any
fixes we have will address the current release.
> - spack: ptscotch will always be built without parmetis wrappers, can't turn
> on
diff --git a/var/spack/repos/builtin/packages/petsc/package.py
b/var/spack/repos/builtin/packages/petsc/package.py
index b7b1d86b15..ae27ba4c4e 100644
--- a/var/spack/repos/builtin/packages/petsc/package.py
+++ b/var/spack/repos/builtin/packages/petsc/package.py
@@ -268,9 +268,7 @@ def check_fortran_compiler(self):
depends_on("metis@5:~int64", when="@3.8:+metis~int64")
depends_on("metis@5:+int64", when="@3.8:+metis+int64")
- # PTScotch: Currently disable Parmetis wrapper, this means
- # nested disection won't be available thought PTScotch
- depends_on("scotch+esmumps~metis+mpi", when="+ptscotch")
+ depends_on("scotch+esmumps+mpi", when="+ptscotch")
depends_on("scotch+int64", when="+ptscotch+int64")
depends_on("hdf5@:1.10+mpi", when="@:3.12+hdf5+mpi")
Now you can try:
spack install petsc~metis+ptscotch ^scotch+metis
vs
spack install petsc~metis+ptscotch ^scotch~metis [~metis is the default for
scotch]
Note the following comment in
spack/var/spack/repos/builtin/packages/scotch/package.py
>>>>
# Vendored dependency of METIS/ParMETIS conflicts with standard
# installations
conflicts("metis", when="+metis")
conflicts("parmetis", when="+metis")
<<<<<
> - classical: ptscotch will always be built with parmetis wrappers, can't seem
> to turn off
Looks like spack uses cmake build of ptscotch. PETSc uses Makefile interface.
It likely doesn't support turning off metis wrappers [without hacks].
So you might either need to hack scotch build via petsc - or just install it
separately - and use it with petsc.
I see an effort to migrate scotch build in petsc to cmake
https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7242/__;!!G_uCfscf7eWS!dL00pokNVI6oaNk_chaSyfI1zWFeTgYA9jbRW6n9YT73s51VwLBuXYc-MAJWEKXr8uBgEFrmhFQ_VJOSlvzW6OA$
https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7495/__;!!G_uCfscf7eWS!dL00pokNVI6oaNk_chaSyfI1zWFeTgYA9jbRW6n9YT73s51VwLBuXYc-MAJWEKXr8uBgEFrmhFQ_VJOS6OuSrPs$
Satish
On Wed, 24 Apr 2024, Daniel Stone wrote:
> Hi PETSc community,
>
> I've been looking at using Spack to build PETSc, in particular I need to
> disable the default metis/parmetis dependencies and use PTScotch instead,
> for our software.
> I've had quite a bit of trouble with this - it seems like something in the
> resulting build of our simulator ends up badly optimised and an mpi
> bottleneck, when I build against
> PETSc built with Spack.
>
> I've been trying to track this down, and noticed this in the PETSc Spack
> build recipe:
>
> # PTScotch: Currently disable Parmetis wrapper, this means
> # nested disection won't be available thought PTScotch
> depends_on("scotch+esmumps~metis+mpi", when="+ptscotch")
> depends_on("scotch+int64", when="+ptscotch+int64")
>
>
> Sure enough - when I compare the build with Spack and a traditional build
> with ./configure etc, I see that, in the traditional build, Scotch is
> always built with the parmetis wrapper,
> but not in the Scotch build. In fact, I'm not sure how to turn off the
> parmetis wrapper option for scotch, in the case of a traditional build
> (i.e. there doesn't seem to be a flag in the
> configure script for it) - which would be a very useful test for me (I can
> of course do similar experiments by doing a classical build of petsc
> against ptscotch built separately without the
> wrappers, etc - will try that).
>
> Does anyone know why the parmetis wrapper is always disabled in the spack
> build options? Is there something about Spack that would prevent it from
> working? It's notable - but I might
> be missing it - that there's no warning that there's a difference in the
> way ptscotch is built between the spack and classical builds:
> - classical: ptscotch will always be built with parmetis wrappers, can't
> seem to turn off
> - spack: ptscotch will always be built without parmetis wrappers, can't
> turn on
>
> Any insight at all would be great, I'm new to Spack and am not super
> familiar with the logic that goes into setting up builds for the system.
>
> Here is the kind of command I give to Spack for PETSc builds, which may
> well be less than ideal:
>
> spack install petsc@3.19.1 ~metis +ptscotch ^hdf5 +fortran +hl
>
> Separate tiny note: when building with hdf5, I have to ensure that the
> fortran flag is set for it, as above. There's a fortran flag for the petsc
> module, default true, and a fortran flag for the hdf5
> module, default false. A naive user (i.e. me), will see the fortran flag
> for the petsc module, and assume that all dependencies will correspondingly
> be built with fortran capability - then see that
> hdf5.mod is missing when trying to build their software against petsc. It's
> the old "did you forget --with-hdf5-fortran-bindings?" issue, resurrected
> for a new build system.
>
> Thanks,
>
> Daniel
>
