Hi Jose, Pierre, Yes, I am not using ParMetis or Metis. This is how I am configuring PETSc: $ ./configure --with-debugging=no --with-mpi-dir=<openmpi> --with-scalar-type=complex --download-mumps --download-scalapack --with-blaslapack-dir=<intel mkl>
Would you happen to know if there is any option available to force MUMPS to be deterministic/repeatable in parallel? I couldn't find anything in the MUMPS user guide that suggests non-deterministic behavior. For the example matrix that I shared, the final computed eigenvalues are within machine precision on multiple runs. However, for some other bigger cases that I am testing, I get different eigenvalues on multiple runs and I am trying to figure out the source of these inconsistencies. Thanks, Varun On Sun, Jan 16, 2022 at 6:09 AM Pierre Jolivet <pie...@joliv.et> wrote: > Default renumbering is sequential. Since --download-parmetis requires > --download-metis, I doubt Varun is using ParMETIS since the appropriate > ICNTL flag is not explicitly set. > If we go back to the original “issue”, I believe this is because you are > doing an LU factorization with a large number of off diagonal pivots, check > INFOG(12), which are handled dynamically, and thus may yield different > factors. > > Thanks, > Pierre > > > On 16 Jan 2022, at 12:28 PM, Jose E. Roman <jro...@dsic.upv.es> wrote: > > > > Probably someone else can give a better answer, but if I remember > correctly ParMetis relies on certain random number generator that makes it > produce different partitions for different runs. I think this was fixed in > the ParMetis that --download-parmetis installs, but if I am not wrong you > are not using that version. This would explain what you get. > > > > Jose > > > > > >> El 16 ene 2022, a las 11:37, Varun Hiremath <varunhirem...@gmail.com> > escribió: > >> > >> Hi All, > >> > >> I am using SLEPc to compute eigenvalues and MUMPS for factorization. > >> > >> Please find attached: > >> 1) A simple test program slepc_eps_mumps_test.cpp that reads a given > PETSc matrix and computes the smallest eigenvalues using MUMPS for > factorization > >> 2) An example PETSc matrix MatA of size 581343 rows (sending in .gz > format via Google drive link, please extract it "gunzip MatA.gz" before > using). You should be able to reproduce this issue with any other matrix of > a similar or bigger size. > >> > >> I notice that when I run the attached test program in parallel with the > attached test matrix the MUMPS statistics printed (using the -eps_view > option in the command line) change with every run. > >> > >> This is how I run the test: > >> $ mpiexec -n 24 ./slepc_eps_mumps_test.o -nev 5 -f MatA -eps_view > >> and for example, the output of this includes the following MUMPS stats > >> ... > >> PC Object: (st_) 24 MPI processes > >> type: lu > >> out-of-place factorization > >> tolerance for zero pivot 2.22045e-14 > >> matrix ordering: external > >> factor fill ratio given 0., needed 0. > >> Factored matrix follows: > >> Mat Object: 24 MPI processes > >> type: mumps > >> rows=581343, cols=581343 > >> package used to perform factorization: mumps > >> total: nonzeros=348236349, allocated nonzeros=348236349 > >> MUMPS run parameters: > >> SYM (matrix type): 0 > >> ... > >> > >> I ran this test 10 times as follows and got a different number of > nonzeros (line highlighted above ) reported in each run. (If you save the > full output and compare, you will notice many other differences, but I > wouldn't have expected the nonzeros to change with every run.) > >> > >> $ for i in `seq 1 10`; do echo "run $i :-----"; mpiexec -n 24 > ./slepc_eps_mumps_test.o -nev 5 -f MatA -eps_view | grep -A 1 > "factorization: mumps"; done > >> > >> run 1 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=354789915, allocated nonzeros=354789915 > >> run 2 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=359811101, allocated nonzeros=359811101 > >> run 3 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=354834871, allocated nonzeros=354834871 > >> run 4 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=354830397, allocated nonzeros=354830397 > >> run 5 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=353942929, allocated nonzeros=353942929 > >> run 6 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=354147241, allocated nonzeros=354147241 > >> run 7 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=354980083, allocated nonzeros=354980083 > >> run 8 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=354980083, allocated nonzeros=354980083 > >> run 9 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=354214219, allocated nonzeros=354214219 > >> run 10 :----- > >> package used to perform factorization: mumps > >> total: nonzeros=355894047, allocated nonzeros=355894047 > >> > >> Can somebody please explain what causes these differences in MUMPS > stats? > >> > >> Thanks, > >> Varun > >> MatA.gz > >> > >> <slepc_eps_mumps_test.cpp> > > > >