Hi Hong, Thanks, I tested your branch and I think it is working fine. I don't see any increase in runtime, however with -log_view I see that the MatLUFactorSymbolic function is still being called twice, so is this expected? Is the second call a no-op?
$ ./ex52.o -use_mumps_lu -print_mumps_memory -log_view | grep MatLUFactorSym MatLUFactorSym 2 1.0 4.4411e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 Thanks, Varun On Mon, Jan 17, 2022 at 7:49 PM Zhang, Hong <hzh...@mcs.anl.gov> wrote: > Varun, > I created a branch hzhang/feature-mumps-mem-estimate, > see https://gitlab.com/petsc/petsc/-/merge_requests/4727 > > You may give it a try and let me know if this is what you want. > src/ksp/ksp/tutorials/ex52.c is an example. > > Hong > ------------------------------ > *From:* Varun Hiremath <varunhirem...@gmail.com> > *Sent:* Monday, January 17, 2022 12:41 PM > *To:* Zhang, Hong <hzh...@mcs.anl.gov> > *Cc:* Jose E. Roman <jro...@dsic.upv.es>; Peder Jørgensgaard Olesen via > petsc-users <petsc-users@mcs.anl.gov> > *Subject:* Re: [petsc-users] PETSc MUMPS interface > > Hi Hong, > > Thanks for looking into this. Here is the workflow that I might use: > > MatLUFactorSymbolic(F,A,perm,iperm,&info); > > // get memory estimates from MUMPS e.g. INFO(3), INFOG(16), INFOG(17) > // find available memory on the system e.g. RAM size > if (estimated_memory > available_memory) > { > // inform and stop; or > // switch MUMPS to out-of-core factorization > ICNTL(22) = 1; > } > else > { > // set appropriate settings for in-core factorization > } > > // Now we call the solve and inside if MatLUFactorSymbolic is already > called then it should be skipped > EPSSolve(eps); > > Thanks, > Varun > > On Mon, Jan 17, 2022 at 9:18 AM Zhang, Hong <hzh...@mcs.anl.gov> wrote: > > Varun, > I am trying to find a way to enable you to switch options after > MatLUFactorSymbolic(). > A hack is modifying the flag 'mumps->matstruc' > inside MatLUFactorSymbolic_AIJMUMPS() and MatFactorNumeric_MUMPS(). > > My understanding of what you want is: > // collect mumps memory info > ... > MatLUFactorSymbolic(F,A,perm,iperm,&info); > printMumpsMemoryInfo(F); > //--------- > if (memory is available) { > EPSSolve(eps); --> skip calling of MatLUFactorSymbolic() > } else { > //out-of-core (OOC) option in MUMPS > } > > Am I correct? I'll let you know once I work out a solution. > Hong > > ------------------------------ > *From:* Varun Hiremath <varunhirem...@gmail.com> > *Sent:* Sunday, January 16, 2022 10:10 PM > *To:* Zhang, Hong <hzh...@mcs.anl.gov> > *Cc:* Jose E. Roman <jro...@dsic.upv.es>; Peder Jørgensgaard Olesen via > petsc-users <petsc-users@mcs.anl.gov> > *Subject:* Re: [petsc-users] PETSc MUMPS interface > > Hi Jose, Hong, > > Thanks for the explanation. I have verified using -log_view that > MatLUFactorSymbolic > is indeed getting called twice. > > Hong, we use MUMPS solver for other things, and we typically run the > symbolic analysis first and get memory estimates to ensure that we have > enough memory available to run the case. If the available memory is not > enough, we can stop or switch to the out-of-core (OOC) option in MUMPS. We > wanted to do the same when using MUMPS via SLEPc/PETSc. Please let me know > if there are other ways of getting these memory stats and switching options > during runtime with PETSc. > Appreciate your help! > > Thanks, > Varun > > On Sun, Jan 16, 2022 at 4:01 PM Zhang, Hong <hzh...@mcs.anl.gov> wrote: > > Varun, > I believe Jose is correct. You may verify it by running your code with > option '-log_view', then check the number of calls to MatLUFactorSym. > > I guess I can add a flag in PCSetUp() to check if user has already called > MatLUFactorSymbolic() and wants to skip it. Normally, users simply allocate > sufficient memory in the symbolic factorization. Why do you want to check > it? > Hong > > ------------------------------ > *From:* Jose E. Roman <jro...@dsic.upv.es> > *Sent:* Sunday, January 16, 2022 5:11 AM > *To:* Varun Hiremath <varunhirem...@gmail.com> > *Cc:* Zhang, Hong <hzh...@mcs.anl.gov>; Peder Jørgensgaard Olesen via > petsc-users <petsc-users@mcs.anl.gov> > *Subject:* Re: [petsc-users] PETSc MUMPS interface > > Hong may give a better answer, but if you look at PCSetUp_LU() > https://petsc.org/main/src/ksp/pc/impls/factor/lu/lu.c.html#PCSetUp_LU > you will see that MatLUFactorSymbolic() is called unconditionally during > the first PCSetUp(). Currently there is no way to check if the user has > already called MatLUFactorSymbolic(). > > Jose > > > > El 16 ene 2022, a las 10:40, Varun Hiremath <varunhirem...@gmail.com> > escribió: > > > > Hi Hong, > > > > Thank you, this is very helpful! > > > > Using this method I am able to get the memory estimates before the > actual solve, however, I think my code may be causing the symbolic > factorization to be run twice. Attached is my code where I am using SLEPc > to compute eigenvalues, and I use MUMPS for factorization. I have commented > above the code that computes the memory estimates, could you please check > and tell me if this would cause the symbolic factor to be computed twice (a > second time inside EPSSolve?), as I am seeing a slight increase in the > overall computation time? > > > > Regards, > > Varun > > > > On Wed, Jan 12, 2022 at 7:58 AM Zhang, Hong <hzh...@mcs.anl.gov> wrote: > > PCFactorSetMatSolverType(pc,MATSOLVERMUMPS); > > PCFactorSetUpMatSolverType(pc); > > PCFactorGetMatrix(pc,&F); > > > > MatLUFactorSymbolic(F,A,...) > > You must provide row and column permutations etc, > petsc/src/mat/tests/ex125.c may give you a clue on how to get these inputs. > > > > Hong > > > > > > From: petsc-users <petsc-users-boun...@mcs.anl.gov> on behalf of > Junchao Zhang <junchao.zh...@gmail.com> > > Sent: Wednesday, January 12, 2022 9:03 AM > > To: Varun Hiremath <varunhirem...@gmail.com> > > Cc: Peder Jørgensgaard Olesen via petsc-users <petsc-users@mcs.anl.gov> > > Subject: Re: [petsc-users] PETSc MUMPS interface > > > > Calling PCSetUp() before KSPSetUp()? > > > > --Junchao Zhang > > > > > > On Wed, Jan 12, 2022 at 3:00 AM Varun Hiremath <varunhirem...@gmail.com> > wrote: > > Hi All, > > > > I want to collect MUMPS memory estimates based on the initial symbolic > factorization analysis before the actual numerical factorization starts to > check if the estimated memory requirements fit the available memory. > > > > I am following the steps from > https://petsc.org/main/src/ksp/ksp/tutorials/ex52.c.html > > > > PCFactorSetMatSolverType(pc,MATSOLVERMUMPS); > > PCFactorSetUpMatSolverType(pc); > > PCFactorGetMatrix(pc,&F); > > > > KSPSetUp(ksp); > > MatMumpsGetInfog(F,...) > > > > But it appears KSPSetUp calls both symbolic and numerical factorization. > So is there some other way to get these statistics before the actual > factorization starts? > > > > Thanks, > > Varun > > <slepc_eps_mumps_test.cpp> > >