On Wed, Feb 20, 2019 at 4:43 AM Sal Am <tempoho...@gmail.com> wrote: > Hi Matthew you were right, > > The matrix I have is very ill conditioned and my supervisor gave it for > testing purposes. Having said that, I was able to solve it previously > however, for some reason it said convergence reached at e-3 even though the > default convergence tol is e-5 which is why I put rel_tol to e-7 and it > crashed. > > Just to get something solving so you can check the discretization, I would >> >> a) Use SuperLU or MUMPS, -pc_type lu >> >> b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu >> >> Then finally >> >> c) For very big problems, I suspect that PCPATCHcould be made to work >> with the right choices, but this is a research problem >> > > That is very interesting, I have tried a) before and it does not work > (lack of memory presumably). I will try to do b), but I am very interested > in c). the final BOSS fight so to say is the 30million finite element model > of the JET A2 Antenna so it would be great if I can utilise PETSc solving > it. (Perhaps I should open another ticket for that later on) >
Yep. The reason I thought it might work is that there is theory for it working on positive semi-definite matrices (matrices which are almost nice but can have a large null space). The idea is to capture the local nullspace in each patch. There is an excellent writeup of the idea, with all appropriate references, here: https://arxiv.org/abs/1810.03315 But to stay on topic I have only used VecView to actually view the solution > *after* it was done solving. Is there an example on how people actually > utilise VecView or Jed's suggestion of KSPMonitor to have checkpoints > *during* iterations so we can pick up wherever it crashed? > Sure. Here we set a custom monitor https://bitbucket.org/petsc/petsc/src/6f3214441ec7b7dca9916d2bd112d01eef3e7185/src/ksp/ksp/examples/tutorials/ex9.c#lines-111 Just stick your VecView in there. In fact, I would use VecViewFromOptions() so you can decide what viewer to use on the command line. Thanks, Matt > Thanks > > On Tue, Feb 19, 2019 at 1:49 PM Matthew Knepley <knep...@gmail.com> wrote: > >> On Tue, Feb 19, 2019 at 8:21 AM Sal Am <tempoho...@gmail.com> wrote: >> >>> Matthew maybe indeed I was not clear and wrong in the wording. What I >>> meant is the matrix is undefined, but since it is a finite element method >>> (using second order elements). The pattern is like this: >>> >> >> Okay, the paper says that the formulation of Maxwell in frequency space >> is "well conditioned", and in the paper they solve using >> BiCG/Jacobi: >> >> -ksp_type bicg -pc_type jacobi >> >> Since you are taking thousands of iterates: >> >> a) They are mistaken and the formulation is not well-conditioned >> >> b) You have a bug in the discretization >> >> c) You are outside the parameter range/boundary conditions/forcing they >> were talking about for conditioning >> >> Just to get something solving so you can check the discretization, I would >> >> a) Use SuperLU or MUMPS, -pc_type lu >> >> b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu >> >> Then finally >> >> c) For very big problems, I suspect that PCPATCHcould be made to work >> with the right choices, but this is a research problem >> >> Thanks, >> >> Matt >> >> >>> [image: image.png] >>> We are indeed solving Maxwell's equations. I have added the paper. >>> >>> On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <knep...@gmail.com> >>> wrote: >>> >>>> On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users < >>>> petsc-users@mcs.anl.gov> wrote: >>>> >>>>> It is a finite-element problem of an RF antenna dielectric interaction >>>>> where all the non-zero elements are on the diagonal of the sparse matrix >>>>> (if that is relevant). >>>>> >>>> >>>> I am unsure whether this is what you mean. If you matrix is diagonal, >>>> you can invert it exactly in a single step so you >>>> do not need a Krylov solver. What equations are you solving? Maxwell? >>>> Electrostatic? What finite element are you using? >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> More about the matrix: 25Mx25M, 2B non-zeros. I checked the >>>>> KSPMonitor, it seems that I have to write my own routine? >>>>> >>>>>> Running for a Krylov method for tens of thousands of iterations is >>>>>> very rarely recommended. >>>>>> >>>>> GAMG and BCGS is the only ones that have actually worked for me so >>>>> far, I increased the max_it because it was not enough with the default >>>>> one. >>>>> With the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but >>>>> I need more accuracy so I increased the tol to 10^-7 and that is when it >>>>> crashed after ~51000 iterations. >>>>> >>>>> @Barry >>>>> >>>>>> I'm guessing you mange your own time stepping (and nonlinear solver >>>>>> if there is one)? >>>>>> >>>>> >>>>> It is the default from the solver (attached the short code). >>>>> >>>>>> As Jed says it doesn't make sense to save "partial" solutions within >>>>>> the linear solver. Just save it every several time-steps. >>>>>> >>>>> >>>>> Yes maybe I worded it wrong. I did mean to save checkpoints basically >>>>> such that the next time it is running it can pick up from where it >>>>> crashed. >>>>> >>>>> Also the code should not be "crashing" at seemingly long times >>>>>> (after hours) with a Segmentation fault. Send us the full error message >>>>>> and >>>>>> we'll see if there is some way we can help you fix this problem. >>>>>> >>>>> I have added this before, but seemingly they conclusion was that it is >>>>> memory error...although I am not sure how, but I have added the entire >>>>> error. >>>>> >>>>> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsm...@mcs.anl.gov> >>>>> wrote: >>>>> >>>>>> >>>>>> I'm guessing you mange your own time stepping (and nonlinear >>>>>> solver if there is one)? >>>>>> >>>>>> You can save the solution with a call to VecView() and then reload >>>>>> the solution with a VecLoad() but you need to manage any other restart >>>>>> data >>>>>> that you may need, like the value of the current time etc. >>>>>> >>>>>> As Jed says it doesn't make sense to save "partial" solutions >>>>>> within the linear solver. Just save it every several time-steps. >>>>>> >>>>>> Barry >>>>>> >>>>>> Also the code should not be "crashing" at seemingly long times >>>>>> (after hours) with a Segmentation fault. Send us the full error message >>>>>> and >>>>>> we'll see if there is some way we can help you fix this problem. >>>>>> >>>>>> >>>>>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users < >>>>>> petsc-users@mcs.anl.gov> wrote: >>>>>> > >>>>>> > Is there a function/command line option to save the solution as it >>>>>> is solving (and read in the file from where it crashed and keep iterating >>>>>> from there perhaps)? >>>>>> > Had a seg fault and all the results until that point seems to have >>>>>> been lost. >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> >>>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>