[petsc-users] something wrong with digest?

Today I got more than 20 petsc-user Digest emails in my inbox (Vol 141, Issue 78 to 101), many containing only one message and being sent within a few minutes of each other (10:39, 10:41 10:44, ...). Is this really how Digest is supposed to work? Chris dr. ir. Christiaan Klaij | Senior Rese

Re: [petsc-users] Finding which cell an arbitrary point belongs to in DMPlex

On Thu, Sep 24, 2020 at 6:09 PM Jeremy Theler wrote: > On Wed, 2020-09-16 at 14:29 +, Hapla Vaclav wrote: > > There is also DMPlexFindVertices() which finds the nearest vertex to > > the given coords in the given radius. > > At first I had understood that this function performed a nearest- >

Re: [petsc-users] Matrix Free Method questions

> On Sep 24, 2020, at 3:08 PM, Matthew Knepley wrote: > > On Thu, Sep 24, 2020 at 4:03 PM Blondel, Sophie via petsc-users > mailto:petsc-users@mcs.anl.gov>> wrote: > Hi Barry, > > I probably should have sent this output before (with -log_view_memory to get > an idea of where the vectors are

Re: [petsc-users] Finding which cell an arbitrary point belongs to in DMPlex

On Wed, 2020-09-16 at 14:29 +, Hapla Vaclav wrote: > There is also DMPlexFindVertices() which finds the nearest vertex to > the given coords in the given radius. At first I had understood that this function performed a nearest- neighbor search but after a closer look it sweeps the local DM an

Re: [petsc-users] Matrix Free Method questions

> On Sep 24, 2020, at 3:08 PM, Matthew Knepley wrote: > > On Thu, Sep 24, 2020 at 4:03 PM Blondel, Sophie via petsc-users > mailto:petsc-users@mcs.anl.gov>> wrote: > Hi Barry, > > I probably should have sent this output before (with -log_view_memory to get > an idea of where the vectors are

Re: [petsc-users] Tough to reproduce petsctablefind error

You might add code here like: if (ierr) { for (; iB->rmap->n; i++) { for ( jilen[i]; j++) { PetscInt data,gid1 = aj[B->i[i] + j] + 1; // don't use ierr print rank, gid1; } CHKERRQ(ierr); I am guessing that somehow you have a table that is bad and too small. It failed on

Re: [petsc-users] Tough to reproduce petsctablefind error

Oh, sorry, my mistake. > On Sep 24, 2020, at 3:31 PM, Junchao Zhang wrote: > > That error stack was from Fande Kong. We should wait for Chris's. > > --Junchao Zhang > > > On Thu, Sep 24, 2020 at 2:42 PM Barry Smith > wrote: > > The stack is listed below. It cr

Re: [petsc-users] Tough to reproduce petsctablefind error

That error stack was from Fande Kong. We should wait for Chris's. --Junchao Zhang On Thu, Sep 24, 2020 at 2:42 PM Barry Smith wrote: > > The stack is listed below. It crashes inside MatPtAP(). > > It is possible there is some subtle bug in the rather complex PETSc code > for MatPtAP() but

Re: [petsc-users] Tough to reproduce petsctablefind error

> On Sep 24, 2020, at 2:47 PM, Matthew Knepley wrote: > > On Thu, Sep 24, 2020 at 3:42 PM Barry Smith > wrote: > > The stack is listed below. It crashes inside MatPtAP(). > > What about just checking that the column indices that PtAP receives are > valid? Are we

Re: [petsc-users] Matrix Free Method questions

On Thu, Sep 24, 2020 at 4:03 PM Blondel, Sophie via petsc-users < petsc-users@mcs.anl.gov> wrote: > Hi Barry, > > I probably should have sent this output before (with -log_view_memory to > get an idea of where the vectors are created). I looked at it but it > doesn't help me much... > Just quicki

Re: [petsc-users] Proper GPU usage in PETSc

On Thu, Sep 24, 2020 at 3:48 PM Zhang, Chonglin wrote: > Hi Matt, > > Thanks for the comments and the nice example code. Right now our objective > is to use XGC unstructured flux-surface-following mesh (fixed in size), I > will keep your comment on mesh refinement in mind. > Okay. If you load th

Re: [petsc-users] Proper GPU usage in PETSc

Hi Matt, Thanks for the comments and the nice example code. Right now our objective is to use XGC unstructured flux-surface-following mesh (fixed in size), I will keep your comment on mesh refinement in mind. Thanks! Chonglin On Sep 24, 2020, at 3:26 PM, Matthew Knepley mailto:knep...@gmail.c

Re: [petsc-users] Tough to reproduce petsctablefind error

On Thu, Sep 24, 2020 at 3:42 PM Barry Smith wrote: > > The stack is listed below. It crashes inside MatPtAP(). > What about just checking that the column indices that PtAP receives are valid? Are we not doing that? Matt > It is possible there is some subtle bug in the rather complex PE

Re: [petsc-users] Tough to reproduce petsctablefind error

The stack is listed below. It crashes inside MatPtAP(). It is possible there is some subtle bug in the rather complex PETSc code for MatPtAP() but I am included to blame MPI again. I think we should add some simple low-overhead always on communication error detecting code to PetscSF wh

Re: [petsc-users] Proper GPU usage in PETSc

On Thu, Sep 24, 2020 at 3:08 PM Zhang, Chonglin wrote: > Hi Matt, > > I will quickly summarize what I found with “CreateMesh" for running ex12 > here: > https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex12.c. > If this is not a proper threads to discuss this, I can open a new one.

Re: [petsc-users] Proper GPU usage in PETSc

Hi Matt, I will quickly summarize what I found with “CreateMesh" for running ex12 here: https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex12.c. If this is not a proper threads to discuss this, I can open a new one. Commands used (relevant to mesh creation) to run ex12 (quad core

Re: [petsc-users] Proper GPU usage in PETSc

> > > > 12:46 master= ~/petsc/src/snes/tutorials$ make > PETSC_DIR=/ccs/home/adams/petsc PETSC_ARCH=arch-summit-opt-gnu-cuda-omp -f > mymake run NP=4 EXTRA='-dm_refine 2 -dm_view' > jsrun -n 1 -a 4 -c 4 -g 1 -r 1 --smpiargs "-gpu" ./ex13 -dm_plex_box_dim 3 > -dm_plex_box_simplex 0 -potential_petscs

Re: [petsc-users] Proper GPU usage in PETSc

On Thu, Sep 24, 2020 at 2:04 PM Mark Adams wrote: > On Thu, Sep 24, 2020 at 1:38 PM Matthew Knepley wrote: > >> On Thu, Sep 24, 2020 at 12:48 PM Zhang, Chonglin >> wrote: >> >>> Thanks Mark and Barry! >>> >>> A quick try of using “-pc_type jacobi” did reduce the number of count >>> for “CpuToGp

Re: [petsc-users] Proper GPU usage in PETSc

On Thu, Sep 24, 2020 at 1:38 PM Matthew Knepley wrote: > On Thu, Sep 24, 2020 at 12:48 PM Zhang, Chonglin wrote: > >> Thanks Mark and Barry! >> >> A quick try of using “-pc_type jacobi” did reduce the number of count for >> “CpuToGpu” and “GpuToCpu”, although using “-pc_type gamg” (the counts di

Re: [petsc-users] Proper GPU usage in PETSc

On Thu, Sep 24, 2020 at 12:48 PM Zhang, Chonglin wrote: > Thanks Mark and Barry! > > A quick try of using “-pc_type jacobi” did reduce the number of count for > “CpuToGpu” and “GpuToCpu”, although using “-pc_type gamg” (the counts did > not decrease in this case) solves the problem faster (may no

Re: [petsc-users] Tough to reproduce petsctablefind error

On Thu, Sep 24, 2020 at 1:22 PM Chris Hewson wrote: > Hi Guys, > > Thanks for all of the prompt responses, very helpful and appreciated. > > By "when debugging", did you mean when configure petsc --with-debugging=1 > COPTFLAGS=-O0 -g etc or when you attached a debugger? > - Both, I have run with

Re: [petsc-users] Proper GPU usage in PETSc

On Sep 24, 2020, at 1:11 PM, Barry Smith mailto:bsm...@petsc.dev>> wrote: On Sep 24, 2020, at 11:48 AM, Zhang, Chonglin mailto:zhang...@rpi.edu>> wrote: Thanks Mark and Barry! A quick try of using “-pc_type jacobi” did reduce the number of count for “CpuToGpu” and “GpuToCpu”, although usi

Re: [petsc-users] Tough to reproduce petsctablefind error

Hi Guys, Thanks for all of the prompt responses, very helpful and appreciated. By "when debugging", did you mean when configure petsc --with-debugging=1 COPTFLAGS=-O0 -g etc or when you attached a debugger? - Both, I have run with a debugger attached and detached, all compiled with the following

Re: [petsc-users] Proper GPU usage in PETSc

> On Sep 24, 2020, at 11:48 AM, Zhang, Chonglin wrote: > > Thanks Mark and Barry! > > A quick try of using “-pc_type jacobi” did reduce the number of count for > “CpuToGpu” and “GpuToCpu”, although using “-pc_type gamg” (the counts did not > decrease in this case) solves the problem faster (

Re: [petsc-users] Tough to reproduce petsctablefind error

Chris, We realize how frustrating this type of problem is to deal with. Here is the code: ierr = PetscTableCreate(aij->B->rmap->n,mat->cmap->N+1,&gid1_lid1);CHKERRQ(ierr); for (i=0; iB->rmap->n; i++) { for (j=0; jilen[i]; j++) { PetscInt data,gid1 = aj[B->i[i] +

Re: [petsc-users] Proper GPU usage in PETSc

Thanks Mark and Barry! A quick try of using “-pc_type jacobi” did reduce the number of count for “CpuToGpu” and “GpuToCpu”, although using “-pc_type gamg” (the counts did not decrease in this case) solves the problem faster (may not be of any meaning since the problem size is too small; the fun

Re: [petsc-users] Tough to reproduce petsctablefind error

On Thu, Sep 24, 2020 at 9:47 AM Chris Hewson wrote: > After about a month of not having this issue pop up, it has come up again > > We have been struggling with a similar PETSc Error for awhile now, the > error message is as follows: > > [7]PETSC ERROR: PetscTableFind() line 132 in > /home/chewso

Re: [petsc-users] Proper GPU usage in PETSc

MatSOR() runs on the CPU, this causes copy to CPU for each application of MatSOR() and then a copy to GPU for the next step. You can try, for example -pc_type jacobi better yet use PCGAMG if it amenable for your problem. Also the problem is way to small for a GPU. There will be co

Re: [petsc-users] Proper GPU usage in PETSc

This communication is all in PCApply. What -pc_type are you using? It looks like -pc_type ssor (or is it sor). That is not implemented on the GPU. You can use 'jacobi' On Thu, Sep 24, 2020 at 11:08 AM Zhang, Chonglin wrote: > Dear PETSc Users, > > I have some questions regarding the proper GPU

Re: [petsc-users] Tough to reproduce petsctablefind error

On Thu, Sep 24, 2020 at 8:47 AM Chris Hewson wrote: > After about a month of not having this issue pop up, it has come up again > > We have been struggling with a similar PETSc Error for awhile now, the > error message is as follows: > > [7]PETSC ERROR: PetscTableFind() line 132 in > /home/chewso

[petsc-users] Proper GPU usage in PETSc

Dear PETSc Users, I have some questions regarding the proper GPU usage. I would like to know the proper way to: (1) solve linear equation in SNES, using GPU in PETSc; what syntax/arguments should I be using; (2) how to avoid/reduce the “CpuToGpu count” and “GpuToCpu count” data transfer showed

Re: [petsc-users] Compute C*Ct using MatCreateTranspose for Ct

Olivier and Matt, MatPtAP with A=I gives Pt*P, not P*Pt. We have sequential MatRARt and MatMatTransposeMult(), but no support for mpiaij matrices. The problem is that we do not have a way to implement C*Ct without explicitly transpose C in parallel. We support MatTransposeMatMult (A*Bt) for m

Re: [petsc-users] Compute C*Ct using MatCreateTranspose for Ct

Indeed, we do not have MatCreateTranspose for mpaij matrix. I can adding such support. How soon do you need it? Hong From: petsc-users on behalf of Matthew Knepley Sent: Thursday, September 24, 2020 6:12 AM To: Mark Adams Cc: PETSc Subject: Re: [petsc-users] C

Re: [petsc-users] Tough to reproduce petsctablefind error

After about a month of not having this issue pop up, it has come up again We have been struggling with a similar PETSc Error for awhile now, the error message is as follows: [7]PETSC ERROR: PetscTableFind() line 132 in /home/chewson/petsc-3.13.3/include/petscctable.h key 24443 is greater than lar

Re: [petsc-users] Compute C*Ct using MatCreateTranspose for Ct

On Thu, Sep 24, 2020 at 6:48 AM Mark Adams wrote: > >> Is there a way to avoid the explicit transposition of the matrix? >> > > It does not look like we have A*B^T for mpiaij as the error message says. > I am not finding it in the code. > > Note, MatMatMult with a transpose shell matrix, I suspec

Re: [petsc-users] Compute C*Ct using MatCreateTranspose for Ct

Ahah, maybe I am using a too old version. I use the 3.12.3. I will try with a more recent one. On 24/09/2020 12:47, Mark Adams wrote: Is there a way to avoid the explicit transposition of the matrix? It does not look like we have A*B^T for mpiaij as the error message says. I am not fin

Re: [petsc-users] Compute C*Ct using MatCreateTranspose for Ct

> > > Is there a way to avoid the explicit transposition of the matrix? > It does not look like we have A*B^T for mpiaij as the error message says. I am not finding it in the code. Note, MatMatMult with a transpose shell matrix, I suspect that it does an explicit transpose internally, or it could

Re: [petsc-users] Compute C*Ct using MatCreateTranspose for Ct

But it seems that MatMatTransposeMult does not work for MPIAIJ matrices: [0]PETSC ERROR: MatMatTransposeMult not supported for A of type mpiaij On 23/09/2020 20:23, Zhang, Hong wrote: You can use MatMatTransposeMult(). Hong -