In 3D problems it is recommended to use preconditioned iterative solvers. Unfortunately the spectrum slicing technique requires the full factorization (because it uses matrix inertia).
> El 25 nov 2019, a las 18:44, Perceval Desforges > <perceval.desfor...@polytechnique.edu> escribió: > > I am basically trying to solve a finite element problem, which is why in 3D I > have 7 non-zero diagonals that are quite farm apart from one another. In 2D I > only have 5 non-zero diagonals that are less far apart. So is it normal that > the set up time is around 400 times greater in the 3D case? Is there nothing > to be done? > > I will try setting up only one partition. > > Thanks, > > Perceval, > >> Probably it is not a preallocation issue, as it shows "total number of >> mallocs used during MatSetValues calls =0". >> >> Adding new diagonals may increase fill-in a lot, if the new diagonals are >> displaced with respect to the other ones. >> >> The partitions option is intended for running several nodes. If you are >> using just one node probably it is better to set one partition only. >> >> Jose >> >> >>> El 25 nov 2019, a las 18:25, Matthew Knepley <knep...@gmail.com> escribió: >>> >>> On Mon, Nov 25, 2019 at 11:20 AM Perceval Desforges >>> <perceval.desfor...@polytechnique.edu> wrote: >>> Hi, >>> >>> So I'm loading two matrices from files, both 1000000 by 10000000. I ran the >>> program with -mat_view::ascii_info and I got: >>> >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=1000000, cols=1000000 >>> total: nonzeros=7000000, allocated nonzeros=7000000 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> >>> 20 times, and then >>> >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=1000000, cols=1000000 >>> total: nonzeros=1000000, allocated nonzeros=1000000 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> >>> 20 times as well, and then >>> >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=1000000, cols=1000000 >>> total: nonzeros=7000000, allocated nonzeros=7000000 >>> total number of mallocs used during MatSetValues calls =0 >>> not using I-node routines >>> >>> 20 times as well before crashing. >>> >>> I realized it might be because I am setting up 20 krylov schur partitions >>> which may be too much. I tried running the code again with only 2 >>> partitions and now the code runs but I have speed issues. >>> >>> I have one version of the code where my first matrix has 5 non-zero >>> diagonals (so 5000000 non-zero entries), and the set up time is quite fast >>> (8 seconds) and solving is also quite fast. The second version is the same >>> but I have two extra non-zero diagonals (7000000 non-zero entries) and the >>> set up time is a lot slower (2900 seconds ~ 50 minutes) and solving is also >>> a lot slower. Is it normal that adding two extra diagonals increases solve >>> and set up time so much? >>> >>> >>> I can't see the rest of your code, but I am guessing your preallocation >>> statement has "5", so it does no mallocs when you create >>> your first matrix, but mallocs for every row when you create your second >>> matrix. When you load them from disk, we do all the >>> preallocation correctly. >>> >>> Thanks, >>> >>> Matt >>> Thanks again, >>> >>> Best regards, >>> >>> Perceval, >>> >>> >>> >>> >>> >>>> Then I guess it is the factorization that is failing. How many nonzero >>>> entries do you have? Run with >>>> -mat_view ::ascii_info >>>> >>>> Jose >>>> >>>> >>>>> El 22 nov 2019, a las 19:56, Perceval Desforges >>>>> <perceval.desfor...@polytechnique.edu> escribió: >>>>> >>>>> Hi, >>>>> >>>>> Thanks for your answer. I tried looking at the inertias before solving, >>>>> but the problem is that the program crashes when I call EPSSetUp with >>>>> this error: >>>>> >>>>> slurmstepd: error: Step 2140.0 exceeded virtual memory limit (313526508 > >>>>> 107317760), being killed >>>>> >>>>> I get this error even when there are no eigenvalues in the interval. >>>>> >>>>> I've started using BVMAT instead of BVVECS by the way. >>>>> >>>>> Thanks, >>>>> >>>>> Perceval, >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Don't use -mat_mumps_icntl_14 to reduce the memory used by MUMPS. >>>>>> >>>>>> Most likely the problem is that the interval you gave is too large and >>>>>> contains too many eigenvalues (SLEPc needs to allocate at least one >>>>>> vector per each eigenvalue). You can count the eigenvalues in the >>>>>> interval with the inertias, which are available at EPSSetUp (no need to >>>>>> call EPSSolve). See this example: >>>>>> http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex25.c.html >>>>>> You can comment out the call to EPSSolve() and run with the option >>>>>> -show_inertias >>>>>> For example, the output >>>>>> Shift 0.1 Inertia 3 >>>>>> Shift 0.35 Inertia 11 >>>>>> means that the interval [0.1,0.35] contains 8 eigenvalues (=11-3). >>>>>> >>>>>> By the way, I would suggest using BVMAT instead of BVVECS (the latter is >>>>>> slower). >>>>>> >>>>>> Jose >>>>>> >>>>>> >>>>>>> El 21 nov 2019, a las 18:13, Perceval Desforges via petsc-users >>>>>>> <petsc-users@mcs.anl.gov> escribió: >>>>>>> >>>>>>> Hello all, >>>>>>> >>>>>>> I am trying to obtain all the eigenvalues in a certain interval for a >>>>>>> fairly large matrix (1000000 * 1000000). I therefore use the spectrum >>>>>>> slicing method detailed in section 3.4.5 of the manual. The >>>>>>> calculations are run on a processor with 20 cores and 96 Go of RAM. >>>>>>> >>>>>>> The options I use are : >>>>>>> >>>>>>> -bv_type vecs -eps_krylovschur_detect_zeros 1 -mat_mumps_icntl_13 1 >>>>>>> -mat_mumps_icntl_24 1 -mat_mumps_cntl_3 1e-12 >>>>>>> >>>>>>> >>>>>>> >>>>>>> However the program quickly crashes with this error: >>>>>>> >>>>>>> slurmstepd: error: Step 2115.0 exceeded virtual memory limit (312121084 >>>>>>> > 107317760), being killed >>>>>>> >>>>>>> I've tried reducing the amount of memory used by slepc with the >>>>>>> -mat_mumps_icntl_14 option by setting it at -70 for example but then I >>>>>>> get this error: >>>>>>> >>>>>>> [1]PETSC ERROR: Error in external library >>>>>>> [1]PETSC ERROR: Error reported by MUMPS in numerical factorization >>>>>>> phase: INFOG(1)=-9, INFO(2)=82733614 >>>>>>> >>>>>>> which is an error due to setting the mumps icntl option so low from >>>>>>> what I've gathered. >>>>>>> >>>>>>> Is there any other way I can reduce memory usage? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Perceval, >>>>>>> >>>>>>> >>>>>>> >>>>>>> P.S. I sent the same email a few minutes ago but I think I made a >>>>>>> mistake in the address, I'm sorry if I've sent it twice. >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ > >
