Re: [petsc-users] saving results

[Matthew Knepley via petsc-users]

On Wed, Feb 20, 2019 at 4:43 AM Sal Am  wrote:

> Hi Matthew you were right,
>
> The matrix I have is very ill conditioned and my supervisor gave it for
> testing purposes. Having said that, I was able to solve it previously
> however, for some reason it said convergence reached at e-3 even though the
> default convergence tol is e-5 which is why I put rel_tol to e-7 and it
> crashed.
>
> Just to get something solving so you can check the discretization, I would
>>
>>   a) Use SuperLU or MUMPS, -pc_type lu
>>
>>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>>
>> Then finally
>>
>>   c) For very big problems, I suspect that PCPATCHcould be made to work
>> with the right choices, but this is a research problem
>>
>
> That is very interesting, I have tried a) before and it does not work
> (lack of memory presumably). I will try to do b), but I am very interested
> in c). the final BOSS fight so to say is the 30million finite element model
> of the JET A2 Antenna so it would be great if I can utilise PETSc solving
> it. (Perhaps I should open another ticket for that later on)
>

Yep. The reason I thought it might work is that there is theory for it
working on positive semi-definite matrices (matrices which are
almost nice but can have a large null space). The idea is to capture the
local nullspace in each patch. There is an excellent writeup
of the idea, with all appropriate references, here:
https://arxiv.org/abs/1810.03315

But to stay on topic I have only used VecView to actually view the solution
> *after* it was done solving. Is there an example on how people actually
> utilise VecView or Jed's suggestion of KSPMonitor to have checkpoints
> *during* iterations so we can pick up wherever it crashed?
>

Sure. Here we set a custom monitor


https://bitbucket.org/petsc/petsc/src/6f3214441ec7b7dca9916d2bd112d01eef3e7185/src/ksp/ksp/examples/tutorials/ex9.c#lines-111

Just stick your VecView in there. In fact, I would use VecViewFromOptions()
so you can decide what viewer to use on the command line.

  Thanks,

 Matt


> Thanks
>
> On Tue, Feb 19, 2019 at 1:49 PM Matthew Knepley  wrote:
>
>> On Tue, Feb 19, 2019 at 8:21 AM Sal Am  wrote:
>>
>>> Matthew maybe indeed I was not clear and wrong in the wording. What I
>>> meant is the matrix is undefined, but since it is a finite element method
>>> (using second order elements). The pattern is like this:
>>>
>>
>> Okay, the paper says that the formulation of Maxwell in frequency space
>> is "well conditioned", and in the paper they solve using
>> BiCG/Jacobi:
>>
>>   -ksp_type bicg -pc_type jacobi
>>
>> Since you are taking thousands of iterates:
>>
>>   a) They are mistaken and the formulation is not well-conditioned
>>
>>   b) You have a bug in the discretization
>>
>>   c) You are outside the parameter range/boundary conditions/forcing they
>> were talking about for conditioning
>>
>> Just to get something solving so you can check the discretization, I would
>>
>>   a) Use SuperLU or MUMPS, -pc_type lu
>>
>>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>>
>> Then finally
>>
>>   c) For very big problems, I suspect that PCPATCHcould be made to work
>> with the right choices, but this is a research problem
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> [image: image.png]
>>> We are indeed solving Maxwell's equations. I have added the paper.
>>>
>>> On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley 
>>> wrote:
>>>
 On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
 petsc-users@mcs.anl.gov> wrote:

> It is a finite-element problem of an RF antenna dielectric interaction
> where all the non-zero elements are on the diagonal of the sparse matrix
> (if that is relevant).
>

 I am unsure whether this is what you mean. If you matrix is diagonal,
 you can invert it exactly in a single step so you
 do not need a Krylov solver. What equations are you solving? Maxwell?
 Electrostatic? What finite element are you using?
   Thanks,

 Matt


> More about the matrix: 25Mx25M, 2B non-zeros. I checked the
> KSPMonitor, it seems that I have to write my own routine?
>
>> Running for a Krylov method for tens of thousands of iterations is
>> very rarely recommended.
>>
> GAMG and BCGS is the only ones that have actually worked for me so
> far, I increased the max_it because it was not enough with the default 
> one.
> With the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but
> I need more accuracy so I increased the tol to 10^-7 and that is when it
> crashed after ~51000 iterations.
>
> @Barry
>
>> I'm guessing you mange your own time stepping (and nonlinear solver
>> if there is one)?
>>
>
> It is the default from the solver (attached the short code).
>
>> As Jed says it doesn't make sense to save "partial" solutions within
>> the linear solver.  Just save it 

Re: [petsc-users] saving results

[Smith, Barry F. via petsc-users]

   I'm guessing you mange your own time stepping (and nonlinear solver if there 
is one)? 

   You can save the solution with a call to VecView() and then reload the 
solution with a VecLoad() but you need to manage any other restart data that 
you may need, like the value of the current time etc. 

   As Jed says it doesn't make sense to save "partial" solutions within the 
linear solver.  Just save it every several time-steps.

   Barry

   Also the code should not be "crashing" at seemingly long times (after hours) 
with a Segmentation fault. Send us the full error message and we'll see if 
there is some way we can help you fix this problem.


> On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users  
> wrote:
> 
> Is there a function/command line option to save the solution as it is solving 
> (and read in the file from where it crashed and keep iterating from there 
> perhaps)?
> Had a seg fault and all the results until that point seems to have been lost. 
> 
> 
> 
> 

Re: [petsc-users] saving results

[Jed Brown via petsc-users]

What kind of problems are you solving?  Running for a Krylov method for
tens of thousands of iterations is very rarely recommended.

Regarding storage, it's significantly more expensive to store the Krylov
basis (even when it's a recurrence) than the current approximation.
Some methods require some work to compute the current approximation.
Anyway, you can use a KSPMonitor to write checkpoints at whatever
interval you like.  In case of a crash, use VecLoad() to read in the
checkpoint file and use KSPSetInitialGuessNonzero() to make it be used
as an initial guess.

Sal Am  writes:

> This is how I run it:
>
> -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable
> -build_twosided allreduce -ksp_monitor_true_residual
> -ksp_monitor_if_not_converged -log_view -ksp_max_it 10 -ksp_rtol 1.0e-7
>
> so BCGS with GAMG as preconditioner. I am guessing writing at every
> timestep would be expensive, maybe every hour? I am not sure what would be
> a good number here if the simulations lasts more than a day.
>
>
>
> On Mon, Feb 18, 2019 at 4:10 PM Jed Brown  wrote:
>
>> What kind of solver are you using and how often do you want to write?
>>
>> Sal Am via petsc-users  writes:
>>
>> > Is there a function/command line option to save the solution as it is
>> > solving (and read in the file from where it crashed and keep iterating
>> from
>> > there perhaps)?
>> > Had a seg fault and all the results until that point seems to have been
>> > lost.
>>

Re: [petsc-users] saving results

[Jed Brown via petsc-users]

What kind of solver are you using and how often do you want to write?

Sal Am via petsc-users  writes:

> Is there a function/command line option to save the solution as it is
> solving (and read in the file from where it crashed and keep iterating from
> there perhaps)?
> Had a seg fault and all the results until that point seems to have been
> lost.