On Wed, Oct 24, 2018 at 9:38 AM Jan Grießer <griesser....@googlemail.com> wrote:
> Hey, > i tried to run my program as you said with -bv_type vecs and/or -bv_type > mat, but instead of the PETScInt overflow i now get an MPI Error 9 > Send the actual error. Thanks, Matt > message, which i assume (after googling a little bit around) should be a > memory problem. I tried to run it also on slightly bigger compute nodes on > our cluster with 20 cores with each 12 GB and 24 GB but the problem still > remains. The actual limit appears to be a 1.5 Million x 1.5 Million where > i searched for NEV = 1500 and MPD = 500/ 200 eigenvalues. > Do you have maybe an idea what the error could be? I appended also the > python method i used to solve the problem. I also tried to solve the > problem with spectrum solving but the error message remains the same. > > def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension): > # Create the solver > # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem > solver) > E = SLEPc.EPS().create() > > # Set the preconditioner > pc = PETSc.PC().create() > pc.setType(pc.Type.BJACOBI) > > # Set the linear solver > # Create the KSP object > ksp = PETSc.KSP().create() > # Create the solver, in this case GMRES > ksp.setType(ksp.Type.GMRES) > # Set the tolerances of the GMRES solver > # Link it to the PC > ksp.setPC(pc) > > # Set up the spectral transformations > st = SLEPc.ST().create() > st.setType("shift") > st.setKSP(ksp) > # MPD stands for "maximum projected dimension". It has to due with > computational cost, please read Chap. 2.6.5 of SLEPc docu for > # an explanation. At the moment mpd is only a guess > E.setDimensions(nev=NEV, mpd = MPD) > # Eigenvalues should be real, therefore we start to order them from the > smallest real value |l.real| > E.setWhichEigenpairs(E.Which.SMALLEST_REAL) > # Since the dynamical matrix is symmetric and real it is hermitian > E.setProblemType(SLEPc.EPS.ProblemType.HEP) > # Use the Krylov Schur method to solve the eigenvalue problem > E.setType(E.Type.KRYLOVSCHUR) > # Set the convergence criterion to relative to the eigenvalue and the > maximal number of iterations > E.setConvergenceTest(E.Conv.REL) > E.setTolerances(tol = 1e-8, max_it = 5000) > # Set the matrix in order to solve > E.setOperators(DynMatrix_nn, None) > # Sets EPS options from the options database. This routine must be called > before `setUp()` if the user is to be allowed to set dthe solver type. > E.setFromOptions() > # Sets up all the internal data structures necessary for the execution of > the eigensolver. > E.setUp() > > # Solve eigenvalue problem > E.solve() > > Print = PETSc.Sys.Print > > Print() > Print("****************************") > Print("***SLEPc Solution Results***") > Print("****************************") > > its = E.getIterationNumber() > Print("Number of iterations of the method: ", its) > eps_type = E.getType() > Print("Solution method: ", eps_type) > nev, ncv, mpd = E.getDimensions() > Print("Number of requested eigenvalues: ", nev) > Print("Number of computeded eigenvectors: ", ncv) > tol, maxit = E.getTolerances() > Print("Stopping condition: (tol, maxit)", (tol, maxit)) > # Get the type of convergence > conv_test = E.getConvergenceTest() > Print("Selected convergence test: ", conv_test) > # Get the used spectral transformation > get_st = E.getST() > Print("Selected spectral transformation: ", get_st) > # Get the applied direct solver > get_ksp = E.getDS() > Print("Selected direct solver: ", get_ksp) > nconv = E.getConverged() > Print("Number of converged eigenpairs: ", nconv) > ..... > > > > Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. < > bsm...@mcs.anl.gov>: > >> >> >> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczh...@mcs.anl.gov> >> wrote: >> > >> > >> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grießer < >> griesser....@googlemail.com> wrote: >> > With more than 1 MPI process you mean i should use spectrum slicing in >> divide the full problem in smaller subproblems? >> > The --with-64-bit-indices is not a possibility for me since i >> configured petsc with mumps, which does not allow to use the 64-bit version >> (At least this was the error message when i tried to configure PETSc ) >> > >> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit >> integer feature" and "full 64-bit integer version" as well. >> >> They use to achieve this by compiling with special Fortran flags to >> promote integers to 64 bit; this is too fragile for our taste so we never >> hooked PETSc up wit it. If they have a dependable way of using 64 bit >> integers we should add that to our mumps.py and test it. >> >> Barry >> >> > >> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman < >> jro...@dsic.upv.es>: >> > To use BVVECS just add the command-line option -bv_type vecs >> > This causes to use a separate Vec for each column, instead of a single >> long Vec of size n*m. But it is considerably slower than the default. >> > >> > Anyway, for such large problems you should consider using more than 1 >> MPI process. In that case the error may disappear because the local size is >> smaller than 768000. >> > >> > Jose >> > >> > >> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knep...@gmail.com> >> escribió: >> > > >> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grießer < >> griesser....@googlemail.com> wrote: >> > > Hi all, >> > > i am using slepc4py and petsc4py to solve for the smallest real >> eigenvalues and eigenvectors. For my test cases with a matrix A of the size >> 30k x 30k solving for the smallest soutions works quite well, but when i >> increase the dimension of my system to around A = 768000 x 768000 or 3 >> million x 3 million and ask for the smallest real 3000 (the number is >> increasing with increasing system size) eigenvalues and eigenvectors i get >> the output (for the 768000): >> > > The product 4001 times 768000 overflows the size of PetscInt; >> consider reducing the number of columns, or use BVVECS instead >> > > i understand that the requested number of eigenvectors and >> eigenvalues is causing an overflow but i do not understand the solution of >> the problem which is stated in the error message. Can someone tell me what >> exactly BVVECS is and how i can use it? Or is there any other solution to >> my problem ? >> > > >> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices >> > > >> > > Thanks, >> > > >> > > Matt >> > > >> > > Thank you very much in advance, >> > > Jan >> > > >> > > >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > > >> > > https://www.cse.buffalo.edu/~knepley/ >> > >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>