Re: [petsc-users] Question about memory usage in Multigrid preconditioner

2016-09-14 Thread Dave May
On Thursday, 15 September 2016, Hengjie Wang wrote: > Hi Dave, > > Sorry, I should have put more comment to explain the code. > No problem. I was looking at the code after only 3 hrs of sleep > > The number of process in each dimension is the same: Px = Py=Pz=P. So is > the domain size. >

Re: [petsc-users] Question about memory usage in Multigrid preconditioner

2016-09-14 Thread Hengjie Wang
Hi Dave, Sorry, I should have put more comment to explain the code. The number of process in each dimension is the same: Px = Py=Pz=P. So is the domain size. So if the you want to run the code for a 512^3 grid points on 16^3 cores, you need to set "-N 512 -P 16" in the command line. I add more

Re: [petsc-users] Question about memory usage in Multigrid preconditioner

2016-09-14 Thread Dave May
On Thursday, 15 September 2016, Dave May wrote: > > > On Thursday, 15 September 2016, frank > wrote: > >> Hi, >> >> I write a simple code to re-produce the error. I hope this can help to >> diagnose the problem. >> The code just solves a 3d poisson equation. >> > > Why is the stencil width a run

Re: [petsc-users] Question about memory usage in Multigrid preconditioner

2016-09-14 Thread Dave May
On Thursday, 15 September 2016, frank wrote: > Hi, > > I write a simple code to re-produce the error. I hope this can help to > diagnose the problem. > The code just solves a 3d poisson equation. > Why is the stencil width a runtime parameter?? And why is the default value 2? For 7-pnt FD Laplac

Re: [petsc-users] Question about memory usage in Multigrid preconditioner

2016-09-14 Thread Dave May
Hi Frank, On Thursday, 15 September 2016, frank wrote: > Hi, > > I write a simple code to re-produce the error. I hope this can help to > diagnose the problem. > The code just solves a 3d poisson equation. > I run the code on a 1024^3 mesh. The process partition is 32 * 32 * 32. > That's when I

Re: [petsc-users] Question about memory usage in Multigrid preconditioner

2016-09-14 Thread frank
Hi, I write a simple code to re-produce the error. I hope this can help to diagnose the problem. The code just solves a 3d poisson equation. I run the code on a 1024^3 mesh. The process partition is 32 * 32 * 32. That's when I re-produce the OOM error. Each core has about 2G memory. I also run

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Harshad Sahasrabudhe
Actually, I was using TR with MUMPS before I tried GAMG and it was working pretty well. The reason I want to switch to GAMG is that I have to increase my system size, and the simulation just takes too long with MUMPS and doesn't scale well. On Wed, Sep 14, 2016 at 3:09 PM, Harshad Sahasrabudhe wr

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Harshad Sahasrabudhe
Hi Barry, Sorry, I had no idea that Newton TR would have anything to do with the linear solver. I was using TR because it is trustworthy and converges every time. I tried LS just now with GAMG+GMRES and it converges, so I don't have the CONVERGED_STEP_LENGTH problem anymore. Thanks for you help!

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Barry Smith
Oh, you are using SNESSolve_NEWTONTR ! Now it all makes sense! The trust region methods impose other conditions on the linear solution so it needs to have their own special convergence test. In particular it requires that the solution from the linear solve be inside the trust region (i.e

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Kong (Non-US), Fande
Hi Hsahara, I am not sure whether or not the dbg will also trace your libmesh code. We have a similar issue in the MOOSE with PETSc-3.7.x. No issues with the old PETSc. We finally get the problem fixed. In KSP, we could plugin any user converged test function, and there is a plugin-in function in

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Harshad Sahasrabudhe
Thanks. I now put a watchpoint on watch *( (PetscErrorCode (**)(KSP, PetscInt, PetscReal, KSPConvergedReason *, void *)) &(ksp->converged) ) The function pointer changes in the first iteration of SNES. It changed at the following place: Old value = (PetscErrorCode (*)(KSP, PetscInt, PetscRea

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Barry Smith
Super strange, it should never have switched to the function SNES_TR_KSPConverged_Private Fortunately you can use the same technique to track down where the function pointer changes. Just watch ksp->converged to see when the function pointer gets changed and send back the new stack trace.

Re: [petsc-users] link error and symbol error

2016-09-14 Thread Satish Balay
Do you have MacPorts installed? With Blas? Perhaps thats the source of conflict.. Satish On Wed, 14 Sep 2016, Barry Smith wrote: > > Send configure.log to petsc-ma...@mcs.anl.gov Looks like a troubled > BLAS/LAPACK install. > >Barry > > > On Sep 14, 2016, at 12:52 PM, Gideon Simpson

Re: [petsc-users] link error and symbol error

2016-09-14 Thread Barry Smith
Send configure.log to petsc-ma...@mcs.anl.gov Looks like a troubled BLAS/LAPACK install. Barry > On Sep 14, 2016, at 12:52 PM, Gideon Simpson wrote: > > I was trying to compile a new version of 3.7.3, and I noticed the following. > First, when building the library, I see, at the end:

[petsc-users] link error and symbol error

2016-09-14 Thread Gideon Simpson
I was trying to compile a new version of 3.7.3, and I noticed the following. First, when building the library, I see, at the end: CLINKER /opt/petsc-3.7.3/arch-darwin-c-debug/lib/libpetsc.3.7.3.dylib ld: warning: could not create compact unwind for _dhseqr_: stack subq instruction is too differ

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Harshad Sahasrabudhe
Hi Barry, I put a watchpoint on *((KSP_CONVERGED_REASON*) &( ((_p_KSP*)ksp)->reason )) in gdb. The ksp->reason switched between: Old value = KSP_CONVERGED_ITERATING New value = KSP_CONVERGED_RTOL 0x2b143054bef2 in KSPConvergedDefault (ksp=0x23c3090, n=12, rnorm=5.3617149831259514e-08, reason=

Re: [petsc-users] Time Stepping Terminology

2016-09-14 Thread Hong Zhang
> On Sep 13, 2016, at 9:47 PM, Gideon Simpson wrote: > > I was looking around to see if there was any built in routine for implicit > midpoint time stepping in petsc. By this I mean equation (1i) of > https://en.wikipedia.org/wiki/Midpoint_method. > > I noticed the TSTHETA solver, and I se

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Harshad Sahasrabudhe
I think I found the problem. I configured PETSc with COPTFLAGS=-O3. I'll remove that option and try again. Thanks! Harshad On Wed, Sep 14, 2016 at 10:06 AM, Harshad Sahasrabudhe wrote: > Hi Barry, > > Thanks for your inputs. I tried to set a watchpoint on > ((_p_KSP*)ksp)->reason, but gdb says

Re: [petsc-users] KSP_CONVERGED_STEP_LENGTH

2016-09-14 Thread Harshad Sahasrabudhe
Hi Barry, Thanks for your inputs. I tried to set a watchpoint on ((_p_KSP*)ksp)->reason, but gdb says no symbol _p_KSP in context. Basically, GDB isn't able to find the PETSc source code. I built PETSc with --with-debugging=1 statically and -fPIC, but it seems the libpetsc.a I get doesn't contain

Re: [petsc-users] block matrix without MatCreateNest

2016-09-14 Thread Klaij, Christiaan
Jed, Just a reminder, you haven't responded to this thread, notably Matt's question below whether you can fix the L2G mapping. Chris > > From: Matthew Knepley > > Sent: Tuesday, August 02, 2016 12:28 AM > > To: Klaij, Christiaan > > Cc: petsc-users@mcs.anl.gov; Jed Brown > > Subject: Re: [petsc